An introduction to quantitative astrophotography on Linux: Part 2 - RAW to FITS and Dark Frames

In part 1 of this series I introduced my reasons for wanting to look into what I called quantitative astrophotography, by which I mean getting god quantitative estimates of the performance of a very simple, entry level, set of DSLR images of the night sky. I wanted to control and understand the quantitative changes made to the images, and hence chose to avoid the Windows-based GUI software common in the amateur astronomy world. Instead I'll use standard astronomical data reduction and analysis techniques and stick to open source tools running under Linux (actually in this case, OSX with macports).

By the end of part 1 I was in possession of ~30 RAW images taken with a Canon Digital Rebel XTi, including a series of dark frames and bias frames. This article covers the next step in the process: converting the RAW images into FITS format, along with a quick look at the dark frame properties. The process of finding the bad pixels and removing them from the images will be covered in Part 3.

The flow chart summarizes the actions that need to be performed on the RAW images to end up with FITS images that have had the bad pixels, the dark current and the bias removed.

The basic idea behind astronomical imaging is to use a series of calibration images to improve the final image of the target. At these early stages of reduction we're talking about using dark frames, bias frames, and bad pixel masks to remove the additional signal in the image that is not from the sky, but from the detector electronics itself. Originally I wrote a lengthy description of each of these calibration image types, but I know think it is better just to point interested readers at more extensive discussions at:

1. Starizona's Guide to CCD Imaging.
2. Mischa's Data Reduction web page at the University of Bonn.

The other reason I decided to cut down on the terminology discussion is that is very CCD-centric, as they are the classic detector type used in professional astronomy. However digital cameras almost always use CMOS detectors and not classic CCDs. The on-Camera data processors inside digital camera perform operations that alter the data values in ways that professional CCD imagers do not.

dcraw to FITS.

To create a FITS file from a RAW file we use the -c flag on the dcraw command line and pipe the output into a program that can convert PNM format to FITS. The dcraw web page recommends pnmtofits, e.g.

dcraw -c (other_flags) input_image.cr2 | pnmtofits > output.fits

However there is one problem with this. dcraw and many image processing utilties assume the first pixel is the top left of an image, whereas the FITS conventsion is the first pixel is the bottom left. So FITS files created with pnmtofits are upside-down. ImageMagick is smart enough to know about fits and corrects for this, so we can make correctly-oriented FITS files like this:

dcraw -c (other_flags) input_image.cr2 | convert pnm:- output.fits

Dark and/or Bias Frame Creation

The dark and bias frames are not exposed to light, so the Bayer RGB mask on top of the detector is not important and the signal doesn't depend on whether each pixel is R, G or B. This means we convert the RAW file to a single channel (grayscale) image for analysis.

It is important that both the raw count output from the camera's ADU and image orientation are preserved, i.e. no gamma correction or histogram manipulation should be performed on the raw file. We want the exact 12-bit (or 14-bit if you have a newer camera) values [except that we cannot control what the Camera's image processor does to the RAW images].

The corresponding dcraw command line options are -D -4 -j -t 0. Hence for each raw format dark and/or bias frame we initially run the following command to convert to FITS:

dcraw -c -D -4 -j -t 0 input_dark.cr2 | convert pnm:- output_dark.fits

Dark and Bias Frame Statistics

The following table summarizes some simple statistical properties of all the dark and bias frames I took. The file name also contains the ISO value (400, 800 or 1600) and whether bad pixel had been removed in the dcraw to FITS conversion, where "nobadpix" means "no bad pixel removal". We'll deal with the bad pixel identification and removal in the next post.

File	Rows	Cols	Mean +/- StdDev	Median	Minimum	Maximum
Prior to bad pixel removal
dark_0694_t30.0_800_Sat_Jan_18_22:04:46_2014_nobadpix.fits	2602	3906	256.11 +/- 6.39	256.50	170.5	4056.5
dark_0695_t30.0_800_Sat_Jan_18_22:05:41_2014_nobadpix.fits	2602	3906	256.11 +/- 6.34	256.50	177.5	4056.5
dark_0696_t30.0_1600_Sat_Jan_18_22:06:22_2014_nobadpix.fits	2602	3906	256.42 +/- 10.89	257.50	95.5	4057.5
dark_0697_t30.0_1600_Sat_Jan_18_22:06:57_2014_nobadpix.fits	2602	3906	256.38 +/- 10.84	257.50	105.5	4057.5
dark_0698_t30.0_400_Sat_Jan_18_22:07:40_2014_nobadpix.fits	2602	3906	255.99 +/- 4.03	255.50	219.5	4055.5
dark_0699_t30.0_400_Sat_Jan_18_22:08:14_2014_nobadpix.fits	2602	3906	256.00 +/- 4.02	255.50	218.5	4055.5
bias_0700_t0.00025_400_Sat_Jan_18_22:09:03_2014_nobadpix.fits	2602	3906	255.99 +/- 2.95	255.50	210.5	333.5
bias_0701_t0.00025_400_Sat_Jan_18_22:09:09_2014_nobadpix.fits	2602	3906	256.00 +/- 2.94	255.50	216.5	328.5
bias_0702_t0.00025_800_Sat_Jan_18_22:09:21_2014_nobadpix.fits	2602	3906	256.09 +/- 4.98	256.50	175.5	397.5
bias_0703_t0.00025_800_Sat_Jan_18_22:09:23_2014_nobadpix.fits	2602	3906	256.09 +/- 4.98	256.50	185.5	394.5
bias_0704_t0.00025_1600_Sat_Jan_18_22:09:31_2014_nobadpix.fits	2602	3906	256.39 +/- 9.27	257.50	101.5	531.5
bias_0705_t0.00025_1600_Sat_Jan_18_22:09:32_2014_nobadpix.fits	2602	3906	256.39 +/- 9.27	257.50	99.5	534.5
Bad pixels removed
dark_0694_t30.0_800_Sat_Jan_18_22:04:46_2014.fits	2602	3906	256.10 +/- 5.04	256.50	193.5	320.5
dark_0695_t30.0_800_Sat_Jan_18_22:05:41_2014.fits	2602	3906	256.10 +/- 5.04	256.50	177.5	693.5
dark_0696_t30.0_1600_Sat_Jan_18_22:06:22_2014.fits	2602	3906	256.40 +/- 9.41	257.50	95.5	1047.5
dark_0697_t30.0_1600_Sat_Jan_18_22:06:57_2014.fits	2602	3906	256.36 +/- 9.39	257.50	105.5	670.5
dark_0698_t30.0_400_Sat_Jan_18_22:07:40_2014.fits	2602	3906	255.99 +/- 2.96	255.50	219.5	451.5
dark_0699_t30.0_400_Sat_Jan_18_22:08:14_2014.fits	2602	3906	255.99 +/- 2.95	255.50	218.5	316.5
bias_0700_t0.00025_400_Sat_Jan_18_22:09:03_2014.fits	2602	3906	255.99 +/- 2.95	255.50	210.5	307.5
bias_0701_t0.00025_400_Sat_Jan_18_22:09:09_2014.fits	2602	3906	256.00 +/- 2.94	255.50	216.5	299.5
bias_0702_t0.00025_800_Sat_Jan_18_22:09:21_2014.fits	2602	3906	256.09 +/- 4.98	256.50	175.5	341.5
bias_0703_t0.00025_800_Sat_Jan_18_22:09:23_2014.fits	2602	3906	256.09 +/- 4.98	256.50	185.5	340.5
bias_0704_t0.00025_1600_Sat_Jan_18_22:09:31_2014.fits	2602	3906	256.39 +/- 9.27	257.50	101.5	418.5
bias_0705_t0.00025_1600_Sat_Jan_18_22:09:32_2014.fits	2602	3906	256.39 +/- 9.27	257.50	102.5	434.5

We can immediately see that the RAW values have been altered by the on-camera image processing chip.

1.  The mean and median values of all the dark frame and bias frames are essentially 256+/-1, irrespective of whether it is a 30 second dark frame or a 1/4000'th of a second bias exposure, and irrespective of the ISO value or whether bad pixels have been removed. That value of 256 is 2^8, which would be highly suggestive of a camera-processor induced value even if we didn't already suspect some on-board processing of the RAW frames.
2. The standard deviations of the mean don't relate the mean values. 

In the next post in this series we will look at bad pixel detection and removal using dcraw.