Friday, August 01, 2014

What happened in the year 774/775?

Something odd happened to the Earth around the year 774/775. For the background, head over to Gred Laden's blog, where this is just part of a larger question about climate reconstruction.

A few exerts follow to give you  a taste of what caught me eye.

A long time ago, probably in our galaxy but kind of far away, a cosmic event happened that caused the Earth to be bathed in Gamma rays in AD 774 or 775.
...
There is chemical and physical evidence, though, of the Gamma ray burst. The best evidence is the large scale conversion of stable Nitrogen isotopes into unstable Carbon–14 isotopes in the upper atmosphere. As you know, radioactive (meaning, unstable) Carbon–14 is created continuously but at a somewhat variable rate in the upper atmosphere. Some of that Carbon is incorporated, along with regular stable Carbon, into living tissues. After the living tissue is created and further biological activity that might retrofit some of the Carbon atoms ends (i.e., the thing dies) the ratio of radioactive Carbon to stable Carbon slowly changes as the radioactive Carbon changes back into Nitrogen. By measuring the ratio now, we can estimate how many years ago, plus or minus, the originally living thing lived and died. But it does vary. Solar activity, nuclear testing, other things, can change the amount of Carbon–14 that gets produced. And, a cosmic event that happened in 774/775 caused the production of enough Carbon–14 to throw off the chronology by hundreds of years.

Now why haven't I heard of this event before? How certain are we that it was a Gamma Ray Burst, and if so, where was the source?

Thursday, July 24, 2014

Fifteen years in orbit and awesome: The Chandra X-ray Observatory

Yesterday, July 25rd, 2014, marked the 15th anniversary of the launch of one of NASA's Great Observatories: the Chandra X-ray Observatory. I was one year into my first post-doc at Johns Hopkins at the time, and watched the launch in one of the conference rooms on the 1st floor. Later I spent many years working on data from Chandra. I can't believe its been so long.


Over-views of the 15 years of Chandra:

For information on the accompanying press release image go here.

Monday, July 21, 2014

An introduction to quantitative astrophotography on Linux: Part 2 - RAW to FITS and Dark Frames

In part 1 of this series I introduced my reasons for wanting to look into what I called quantitative astrophotography, by which I mean getting god quantitative estimates of the performance of a very simple, entry level, set of DSLR images of the night sky. I wanted to control and understand the quantitative changes made to the images, and hence chose to avoid the Windows-based GUI software common in the amateur astronomy world. Instead I'll use standard astronomical data reduction and analysis techniques and stick to open source tools running under Linux (actually in this case, OSX with macports).

By the end of part 1 I was in possession of ~30 RAW images taken with a Canon Digital Rebel XTi, including a series of dark frames and bias frames. This article covers the next step in the process: converting the RAW images into FITS format, along with a quick look at the dark frame properties. The process of finding the bad pixels and removing them from the images will be covered in Part 3.

Dark frame and bad pixel data reduction flow chart

The flow chart summarizes the actions that need to be performed on the RAW images to end up with FITS images that have had the bad pixels, the dark current and the bias removed.

The basic idea behind astronomical imaging is to use a series of calibration images to improve the final image of the target. At these early stages of reduction we're talking about using dark frames, bias frames, and bad pixel masks to remove the additional signal in the image that is not from the sky, but from the detector electronics itself. Originally I wrote a lengthy description of each of these calibration image types, but I know think it is better just to point interested readers at more extensive discussions at:
  1. Starizona's Guide to CCD Imaging.
  2. Mischa's Data Reduction web page at the University of Bonn.

The other reason I decided to cut down on the terminology discussion is that is very CCD-centric, as they are the classic detector type used in professional astronomy. However digital cameras almost always use CMOS detectors and not classic CCDs. The on-Camera data processors inside digital camera perform operations that alter the data values in ways that professional CCD imagers do not.

dcraw to FITS.

To create a FITS file from a RAW file we use the -c flag on the dcraw command line and pipe the output into a program that can convert PNM format to FITS. The dcraw web page recommends pnmtofits, e.g.
dcraw -c (other_flags) input_image.cr2 | pnmtofits > output.fits

However there is one problem with this. dcraw and many image processing utilties assume the first pixel is the top left of an image, whereas the FITS conventsion is the first pixel is the bottom left. So FITS files created with pnmtofits are upside-down. ImageMagick is smart enough to know about fits and corrects for this, so we can make correctly-oriented FITS files like this:
dcraw -c (other_flags) input_image.cr2 | convert pnm:- output.fits

Dark and/or Bias Frame Creation

The dark and bias frames are not exposed to light, so the Bayer RGB mask on top of the detector is not important and the signal doesn't depend on whether each pixel is R, G or B. This means we convert the RAW file to a single channel (grayscale) image for analysis.

It is important that both the raw count output from the camera's ADU and image orientation are preserved, i.e. no gamma correction or histogram manipulation should be performed on the raw file. We want the exact 12-bit (or 14-bit if you have a newer camera) values [except that we cannot control what the Camera's image processor does to the RAW images].

The corresponding dcraw command line options are -D -4 -j -t 0. Hence for each raw format dark and/or bias frame we initially run the following command to convert to FITS:
dcraw -c -D -4 -j -t 0 input_dark.cr2 | convert pnm:- output_dark.fits

Dark and Bias Frame Statistics

The following table summarizes some simple statistical properties of all the dark and bias frames I took. The file name also contains the ISO value (400, 800 or 1600) and whether bad pixel had been removed in the dcraw to FITS conversion, where "nobadpix" means "no bad pixel removal". We'll deal with the bad pixel identification and removal in the next post.


FileRowsColsMean +/- StdDevMedianMinimumMaximum
Prior to bad pixel removal
dark_0694_t30.0_800_Sat_Jan_18_22:04:46_2014_nobadpix.fits26023906256.11 +/- 6.39256.50170.54056.5
dark_0695_t30.0_800_Sat_Jan_18_22:05:41_2014_nobadpix.fits26023906256.11 +/- 6.34256.50177.54056.5
dark_0696_t30.0_1600_Sat_Jan_18_22:06:22_2014_nobadpix.fits26023906256.42 +/- 10.89257.5095.54057.5
dark_0697_t30.0_1600_Sat_Jan_18_22:06:57_2014_nobadpix.fits26023906256.38 +/- 10.84257.50105.54057.5
dark_0698_t30.0_400_Sat_Jan_18_22:07:40_2014_nobadpix.fits26023906255.99 +/- 4.03255.50219.54055.5
dark_0699_t30.0_400_Sat_Jan_18_22:08:14_2014_nobadpix.fits26023906256.00 +/- 4.02255.50218.54055.5
bias_0700_t0.00025_400_Sat_Jan_18_22:09:03_2014_nobadpix.fits26023906255.99 +/- 2.95255.50210.5333.5
bias_0701_t0.00025_400_Sat_Jan_18_22:09:09_2014_nobadpix.fits26023906256.00 +/- 2.94255.50216.5328.5
bias_0702_t0.00025_800_Sat_Jan_18_22:09:21_2014_nobadpix.fits26023906256.09 +/- 4.98256.50175.5397.5
bias_0703_t0.00025_800_Sat_Jan_18_22:09:23_2014_nobadpix.fits26023906256.09 +/- 4.98256.50185.5394.5
bias_0704_t0.00025_1600_Sat_Jan_18_22:09:31_2014_nobadpix.fits26023906256.39 +/- 9.27257.50101.5531.5
bias_0705_t0.00025_1600_Sat_Jan_18_22:09:32_2014_nobadpix.fits26023906256.39 +/- 9.27257.5099.5534.5
Bad pixels removed
dark_0694_t30.0_800_Sat_Jan_18_22:04:46_2014.fits26023906256.10 +/- 5.04256.50193.5320.5
dark_0695_t30.0_800_Sat_Jan_18_22:05:41_2014.fits26023906256.10 +/- 5.04256.50177.5693.5
dark_0696_t30.0_1600_Sat_Jan_18_22:06:22_2014.fits26023906256.40 +/- 9.41257.5095.51047.5
dark_0697_t30.0_1600_Sat_Jan_18_22:06:57_2014.fits26023906256.36 +/- 9.39257.50105.5670.5
dark_0698_t30.0_400_Sat_Jan_18_22:07:40_2014.fits26023906255.99 +/- 2.96255.50219.5451.5
dark_0699_t30.0_400_Sat_Jan_18_22:08:14_2014.fits26023906255.99 +/- 2.95255.50218.5316.5
bias_0700_t0.00025_400_Sat_Jan_18_22:09:03_2014.fits26023906255.99 +/- 2.95255.50210.5307.5
bias_0701_t0.00025_400_Sat_Jan_18_22:09:09_2014.fits26023906256.00 +/- 2.94255.50216.5299.5
bias_0702_t0.00025_800_Sat_Jan_18_22:09:21_2014.fits26023906256.09 +/- 4.98256.50175.5341.5
bias_0703_t0.00025_800_Sat_Jan_18_22:09:23_2014.fits26023906256.09 +/- 4.98256.50185.5340.5
bias_0704_t0.00025_1600_Sat_Jan_18_22:09:31_2014.fits26023906256.39 +/- 9.27257.50101.5418.5
bias_0705_t0.00025_1600_Sat_Jan_18_22:09:32_2014.fits26023906256.39 +/- 9.27257.50102.5434.5


We can immediately see that the RAW values have been altered by the on-camera image processing chip.
  1.  The mean and median values of all the dark frame and bias frames are essentially 256+/-1, irrespective of whether it is a 30 second dark frame or a 1/4000'th of a second bias exposure, and irrespective of the ISO value or whether bad pixels have been removed. That value of 256 is 2^8, which would be highly suggestive of a camera-processor induced value even if we didn't already suspect some on-board processing of the RAW frames.
  2. The standard deviations of the mean don't relate the mean values. 
In the next post in this series we will look at bad pixel detection and removal using dcraw.

Friday, July 11, 2014

How much does a square root cost?

An interesting diversion: how much (time) does calling the square root function sqrt() cost, and is it even worth trying alternative less-accurate implementations?

Some Assembly Required provides some answers: 

SQUARE ROOT
Method Total time Time per float Avg Error
Compiler sqrt(x) /
x87 FPU FSQRT
404.029ms 24ns 0.0000%
SSE intrinsic ssqrts 200.395ms 11.9ns 0.0000%
Carmack’s Magic Number rsqrt * x 72.682ms 4.33ns 0.0990%
SSE rsqrtss * x 20.495ms 1.22ns 0.0094%
SSE rsqrtss * x
with one NR step
53.401ms 3.18ns 0.0000%
SSE rsqrtss * x
with one NR step, unrolled by four
48.701ms 2.90ns 0.0000%
RECIPROCAL SQRT
Method Total time Time per float Avg Error
Carmack’s Magic Number rsqrt 59.378ms 3.54ns 0.0990%
SSE rsqrtss 14.202ms 0.85ns 0.0094%
SSE rsqrtss
with one NR step
45.952ms 2.74ns 0.0000%