Mimir - Data Reduction notes


	Mimir Data Reduction Notes

Basic Mimir Image Processing -- 2006/10/02 - DPC

These constitute the basic steps necessary to creating images for which bad pixels have been flagged (and sometimes removed), all data values have been corrected for known InSb deviations from linearity, dark removal and flat fielding has been applied.

Links on this page go to the major routines, but not to their support routines. Please consult the SOFTWARE page to find all these routines and their support routines.

Further details related to photometric extractions (PSF fitting, etc), spectroscopy, and polarimetry analyses will be added later.

The steps in basic Mimir image processing are:

Create Electronic Logs of Each Night of Observations ("LOGS")

Verify all necessary data are present; test quality; delete bad ("TRIAGE")

Test whether column shifting needed ("KOKEPELI")

Make the images "sky true" if shifting needed ("SKY TRUE")

Detect, Correct RAW1 Instability & CDS Gain Changes ("JUMP") [note - no longer optional]

Build the Bad Pixel Image Masks ("BPM")

(Optional) Calculate Linearity Correction Tables

Assemble the images to make linearity correction tables ("LC LIST")

Compute the linearity correction look-up tables ("LC TABLES")

Compute the Ave RAW1 Bias Images ("AVE RAW1")

(Optional) Determine Detector Read Noise and Conversion Gain ("GAIN")

Linearity Correct All Images ("LC")

Compute average darks ("DARKS")

Compute average flat-field images ("FLATS")

Process science images ("SCIENCE")

In addition, many of these steps can be accelerated through pseudo-batch processing:

Batch processing Mimir IDL procedures - the routine batch.pro will accept a text file containing lines that list Mimir or intrinsic IDL commands, followed by the command line arguments needed by the routine. All entries are separated by semicolons, with no spaces before or after filenames. Quotes are not needed for string type arguments. A example input file (test_batch_input.dat) shows how many runs of one Mimir routine can be used to process many nights' data through the same step. Alternatively, consecutive processing steps could be listed in an input file to create a processing pipeline. batch.pro may also call itself, so a batch operation calling other batch operations could be constructed.

Create Electronic Logs

Detailed logs, with one line summaries for each image, are crucial for guiding the data processing. Build electronic logs and keep them with the data as they are processed to make your life easier.

Create a directory structure for the raw data from your observing run. I suggest a root directory with the month name and data type (e.g., "200505_raw").
Under this root, make subdirectories for each night (or day) of observing (e.g., "20050517").
Under each of these nightly directories, make two subdirectories, "CDS_RAW" and "RAW1_RAW". Use these names exactly, as some of the IDL routines expect to see them.
Create a directory structure to hold the results of your data processing steps. In parallel with the directory in step 1. above, I suggest creating a "200505_analysis" (hereafter, "Analysis" in all steps below) master directory.
In the "Analysis" directory, create a subdirectory to hold the nightly logs. I suggest "Step_0_logs"
Use the make_log.pro program on each night's CDS images, creating one log file for each night. I suggest using file names that match to the night of observation, such as "20050517_log.dat".
(optional) I like to make shorter summaries in Excel, grouped by similar object, filter, exposure combinations. Columns for each processing step make it easier to track data processing progress.

Verify that all necessary data are present; test quality; delete bad images
Currently, Mimir images are being stored as CDS FITS files and non-FITS RAW1 and RAW2 type files. RAW1 files are the raw array reads for the initial read after array reset. RAW2 are the reads that take place one integration time later. The CDS frames are the difference between RAW1 and RAW2, with suitable FITS headers added. The linearity correction needs knowledge of the RAW1 image data (RAW2 can be easily reconstituted from RAW1 and CDS images).

Users are cautioned to be sure to save the RAW1 images along with the CDS images when they bring data home.

Sometimes, either the RAW1 or RAW2 image for a particular integration fail to be stored and the corresponding CDS image is useless. Until we can fix these problems in hardware, these failures need to be identified and removed.
1. Copy your raw data from your telescope-written Mimir DVDs into the proper directory (from Step_0:1 above), separating the CDS and RAW1 images into their respective subdirectories.
2. Use the triage.pro routine to evaluate your data into good and bad images automatically, or for small numbers of images you can use fastlook_for_CDS.pro to identify bad CDS images.
  1. If using fastlook, remove the bad images from the disk directories
  2. If using triage:
    1. Under the "Analysis" directory, create a useful directory to hold the lists of good and bad image files for each night. I suggest "Step_1_triage" for the directory name and names matching the night designators for the lists, as in "20050517_good.dat" and "20050622_bad.dat".
    2. Plan to use the good image file lists (copied and edited as necessary) for driving the later analysis programs.
    3. You should also delete the bad image files using the delete_bad.pro program.
Note: fastlook and triage, as well as other routines, are set up to normally select groups of files using the file-chooser window of the window manager. An alternate approach is to pass to these routines the name of a file containing a list of image pathnames. Such a file can be generated with the operating system (e.g., "ls" in unix). To invoke one of the Mimir IDL routines to use only lists of files, that is, the file-chooser is disabled, add the /NOGUI keyword to the procedure call (e.g., "triage, /NOGUI").

Test whether the images need columns shifted
At present, some data transfer bugs have resulted in images with columns of data placed into the wrong order.

The best way to see this problem is to examine some CDS images carefully with DS9, paying particular attention to the linear nature of the legs of the central crack (aka "Kokepeli"). The crack will show gaps at column 511-513 if there has been a misordering of the columns. Look for missing columns from the center appearing instead in the first and last column positions.

An analysis package, consisting of several IDL programs, is available that will evaluate each of your images and make recommendations about what needs to be done to make the images "sky_true" by reordering columns of data in the different quadrants. Howerver, the package is not perfect, and some intelligent use and understanding is needed to properly guide it.
1. Start by making a new subdirectory below the "Analysis" level (e.g., "Step_2_kokepeli_analysis")
2. Run kokepeli_analysis.pro on each night's CDS image data. Input files should be the "good" files identified by triage.pro. Output files should be labeled by date (e.g., "20050621_kokepeli_analysis.dat") and put into the subdirectory
3. Examine and/or edit these files to be sure they contain sensible recommendations:
  1. Each line in a file lists the CDS image name, four numbers representing the column shifts being recommended for each quadrant (LL, LR, UL, UR), and letter "grades" of the program's perception of its certainty of recommendation.
  2. For most Mimir runs, bias frames contain very little contrast between the Kokepeli crack and the surrounding image. For these images, the kokepeli_analysis program has a hard time finding the crack and determining whether it is offset.
  3. If the recommendations appear spurious, edit the file to change the quadrant offset numbers (generally back to "0 0 0 0")

Make the images to be "sky true" if necessary
If the Kokepeli analysis step results in valid recommendations for column shifts of some quadrants of some (or all) images, use the make_sky_true.pro program.
1. Create a subdirectory in "Analysis" to hold the "Sky True" images after processing. I suggest "Step_3_sky_true"
2. Create subdirectories under that directory, one for each night's data.
3. Create subdirectories under each of these, one labelled "CDS_RAW" and one "RAW1_RAW" to match the structure of the original data directories.
4. Use make_sky_true.pro to apply the recommended changes from the kokepeli_analysis to fix a batch of CDS images and their corresponding RAW1 images and put the results in the new directory. These new data will be used in all subsequent steps. If disk space is tight, the original raw data directories may be deleted now.

Detect and Correct RAW1 Jumps (& CDS Gain Changes)
For Mimir data obtained prior to 20051001, instabilities in the timing of the global reset signal introduce jumps in the reset levels that are both quantized and affect the gain of the imaging array. Analysis has shown that it is possible to automatically detect which RAW1 images are affected and to establish and apply gain corrections to the CDS images. For data collected after Oct 1 2005 the RAW1 levels still jump, but are no longer correlated with CDS gain changes.

All Mimir data should be corrected for these RAW1 jumps, or later processing steps (notably linearity correction) will fail.

For data obtained since 20051001, perform the following:
1. Create a subdirectory in "Analysis" with a name like "Step_4_RAW1_jumps"
2. Create subdirectories within Step_4 called "Examination_results" and "Detection_results"
3. For each night of data:

In the directory of raw data for the night, rename the "/RAW1_RAW" directory to "/OLD_RAW1_RAW"
Run the program RAW1_examiner.pro on the contents of the "/OLD_RAW1_RAW", directing the output to the Examination_results directory with a name like "20060624_RAW1_examination.dat". This program looks at each RAW1 image to build a list of the mean count values for each quadrant and for each of the odd and even rows.
Run the program RAW1_jump_finder.pro, using the previous file as the input and saving the output results in the Detection_results directory with a name like "20060624_RAW1_jumps.dat". This interactive program is used to isolate and detect legitimate RAW1 value jumps in a night's data. Detailed instructions can be found at this link.
Run the program RAW1_jump_corrector.pro, using the file from step 3 as the input and pointing the output directory to the original raw data directory for the night (e.g., "20060624"). This program takes the original RAW1 images and the detected jumps file to produce corrected RAW1 images.
If desired, the three programs can be run again to look at the resulting corrected set of RAW1 images to verify all jumps were found and corrected.

For data obtained prior to 20051001, the steps necessary involve measuring the mean RAW1 levels in each image, calibrating the CDS gain change for a given RAW1 jump, automatically detecting RAW1 jumps, and applying corrections to the affected RAW1 and CDS images. These are presented in more detail on the RAW1_Instability_Correction page.

Build the bad pixel mask
Use the program make_bad_pix_mask.pro to create the bad pixel mask (or masks). The images to be input to this routine are any moderate number (~32) of bias or other short exposure frames. These can be listed in a file or selected using the file chooser. The program first asks about image size and masking, then reads all the bias frames and builds a histogram of pixel values from all the frames. The program will ask for a "low cut" value. Move the cursor to place it to the left of the gaussian, about 10-12 sigma to the left of the peak of the gaussian (generally around data value -50 to -70). Click the left mouse button to select this value. The program then asks about a "high cut" value. It is probably best to mirror the low cut value to the high cut one (ie, +50 to +70). This can be viewed with DS9. (2005/06/13 - I have added "ski jump" fitting to the program. This significantly slows the execution, but produces much better results -- the tops and bottoms are preserved and the total fraction of bad pixels is about 0.23%, that is 99.77% good pixels!)
1. Create a subdirectory in "Analysis" to hold the bad pixel image files. I suggest calling it "Step_5_bad_pixel_masks"
2. Run make_bad_pix_mask.pro, putting the results into the subdirectory, with file names reflecting the night identifier and a running number of masks for the night (e.g., "20050517_BPM_1.fits" or "20050622_BPM_3.fits"). Note that any number of BPMs can be calculated. I tend to compute one for each group of 32 bias images I collect.
3. Examine each BPM image file with DS9 to be sure it looks reasonable. Remember this is a bad pixel mask, so bad pixel locations have the value "1" and good pixel locations have the value "0"

(Optional) Calculate Linearity Correction Tables
This is optional in the sense that it needs to be done only once an observing run, so long as the detector parameters are stable. It does *not* need to be done nightly.

The steps involve collecting the necessary images, running code to create useful lists of the images, coadding the images with the same exposure times, and finally running the code to compute the 9 linearity tables (images).

The Aladdin III InSb array is intrinsically non-linear in its response to light. By computing the actual response of each pixel to a run of different exposure times (looking at a flat-field screen), the deviation from linearity can be modeled for each pixel and used to correct all other images. The correction involves using both the CDS and the RAW1 images.

The current correction performs a fourth-order polynomial fit of the counts ("ADUs") for each pixel versus exposure time, then does the reverse (another fourth order fit) to determine the "effective" exposure time seen by a pixel in a target (science) observation before applying the correction. In addition, the saturation level of each pixel is noted to be sure correction beyond saturation is not attempted. Thus there are a total of nine images that make up the linearity correction tables. These are the saturation level image, the five constants making up the first fourth order fit, and the three constants making up the quadratic, cubic, and quartic terms in the linearity correction. Corrected this way, the data are linearized to within their photometric uncertainty.

Assemble the files needed for the linearity correction computation
These will consist of a set of bias images and the 25-50 sets of linearity images obtained automatically using the lindata.tcl script against the flat field screen. Generally, several images are obtained at each different integration time. To perform the linearity correction computation, these multiple images must first be combined (using a median filter) into high-quality average images.
1. Create a subdirectory in "Analysis" with a name like "Step_6_linearity_tables".
2. Copy all the linearity image files and their associated (non-averaged) bias files for each linearity correction data collection event (usually a night) into one new subdirectory (do this for both the CDS FITS files and the non-FITS RAW1 files) with a name like "input_images"
3. Make a list of all of the CDS images, using ls.pro, naming the list something like "linearity_input_image_list.dat" and putting it at the "Step_6" directory level
4. Run make_linearity_list.pro using the list as an input file. This program reads the FITS headers of the CDS images to generate a sorted list of bias and linearity images, needed for guiding the coadder. Name the output file something like "organized_coadding_list.dat" and put it at the "Step_6" directory level.
5. Run coadd_linearity_images.pro to coadd the bias files into one bias image and the different groups of linearity images into single images for each exposure time. This routine writes the list of exposure times and pathnames needed by the next program (make_linearity_tables.pro) into a summary file. Name this output file something like "coadd_summary.dat" and put it at the "Step_6" directory level. Note: because of extensive use of the median filtering routine, this program takes a significant amount of time (hours) to run. Go get some coffee and maybe play a racket ball game...

Compute Linearity Correction Tables (Images)

Run make_linearity_tables.pro to create the nine FITS images making up the linearity correction. These should be saved in the "Step_6" directory with root file names such as "linearity_" (or "lin_0525_" to identify month, date).

Compute the Average RAW1 Bias (Reset Pedestal) Images
These are the mean RAW1 images used to correct the individual RAW1 images back to the pedestal reset time. Each set of zero exposure, dark images ("biases") have their RAW1 images averaged and used in all later steps.
1. Create a subdirectory of "Analysis" called something like "Step_7_ave_RAW1_biases"
2. Use coadd_median_with_bad_for_RAW.pro to create averaged RAW1 images. Put them in the "Step_7" subdirectory with names that reflect the observing night and the bias group number (e.g., "20050517_ave_RAW1_bias_1"). Note that these are *not* FITS images, so should not have ".fits" appended to their names.
3. (Optional) To look at the newly constructed average RAW1-type images, use raw1_to_FITS.pro to convert a RAW1 file to a FITS type file for examination using DS9.

(Optional) Determine Detector Read Noise and Conversion Gain
At least once per observing run, it is probably worth determining the read noise and conversion gain of the Mimir detector system. This starts with collecting the necessary set of flat-field images with the lights on and lights off for a range of exposure times. The procedure details are in the "How To" manual at this link.

Processing these data to determine read noise and conversion gain starts with computing mean RAW1 bias images, then performing the linearity correction on the CDS images. The corrected images are matched up by exposure time (lights on - lights off) to determine mean counts and count standard deviations in each quadrant. Plotting these quantities allows determination of read noise (related to the Y axis intercept) and the conversion gain (related to the linear slope).

Create a subdirectory in "Analysis" to hold the gain calibration analysis files. I suggest "Step_8_gain_calibration"
Identify the RAW1 average bias image located closest in time to the flat-field images to be used in the gain calibration
1. If none have been computed, use coadd_median_with_bad_for_RAW.pro to compute mean RAW1 images and put them in the "Step_8" directory
Create a subdirectory in "Step_8" with a name like "linearity_corrected"
Run linearity_correct.pro on the lights-on and lights-off images, storing the resulting images in the subdirectory
Create a list of the lights-on, linearity-corrected CDS images, using ls.pro, putting the list into the "Step_8" directory with a name like "lights_on_lc_list.dat"
Create a similar list of the lights-off, linearity-corrected CDS images, also into "Step_8" with a name like "lights_off_lc_list.dat"
Run make_linearity_list.pro on the lights-on list and then on the lights-off list, producing two output lists. Put these in the "Step_8" directory with names like "lights_on_sorted_list.dat" and "lights_off_sorted_list.dat"
Pare these lists to have matched numbers of entries at each exposure time for each list
Create a subdirectory in"Step_8" to store the differenced images for the next step. I suggest it be named "difference_images"
1. Run diff_on_off_from_lists.pro to produce the differenced images (on - off) and put these into the new subdirectory
2. Use DS9 to examine a few of the differenced images to look for anomalies or problems with the images
Run ls.pro on the step 9 subdirectory to make a new list, store it in the "Step_8" directory with a name like "difference_list.dat"
Edit this file to keep only pairs of images with the same exposure time, as consecutive entries in the file. (The output of ls.pro does not capture image exposure times, so it may be useful to again run make_linearity_list.pro, which does read the image FITS headers to capture exposure time, to sort the list by exposure time. When editing for pairs, however, edit the list produced by ls.pro, not the one produced by make_linearity_list.pro, as the input expected by the next program expects a simple file list.)
Run detector_characterization.pro with this list, producing a summary file that should be stored in the "Step_8" directory with a name like "quadrant_summaries.dat"
Run compute_gain.pro using the summary file from step 12. This will plot the gain data and compute the noise and conversion gain for each quadrant.

Linearity Correct All Image Data
At this stage, the linearity correction tables are in place, the ave RAW1 bias images are in place, so the linearity correct can be applied. It will generate new CDS-like images to replace the old ones. It also will incorporate all of the RAW1 information, so after this step, the raw CDS and RAW1 images may be deleted.
1. Create a subdirectory in "Analysis" to hold the linearity corrected files. I suggest "Step_9_linearity_corrected"
2. (Optional) To evaluate which bias averages to assign to groups of images:
  1. Run RAW1_read_examination.pro on the RAW1-jump-corrected RAW1 images for each night, saving the summary file to a directory
  2. Run RAW1_summary_plot.pro to produce a plot of the RAW1 means vs frame number
  3. Based on the plot, identify which groups of images should be groups with the different bias averages to perform the linearity correction
3. Run linearity_correct.pro, directing the output to the new directory.
  1. If correcting a large (> hundreds) number of images, set up a batch job, as described at the top of this page
4. (Optional) To check on the quality of the correction and to test whether bad bias averages or other problems have crept into the analysis, the output images can be quickly checked using count_bad_pixels.pro, which will generate a summary file containing the total number of bad pixels found in each image of a list or directory. If the numbers of bad pixels rise far beyond the mean of about 4400 (that is beyond 6000 per image), it is worth examining the pre-corrected and post-corrected images with DS9. If the differences are confined to the bottom and top rows, fine. If the central portion of the image shows large numbers of deleted pixels, stop and fix the problem.
5. (Optional) The output of the bad pixel counting can be quickly plotted using bad_pixel_summary_plot.pro. I found a couple of badly deinterlaced images with this routine that I thought would have been trapped by triage.pro but were not. The plotter is useful for finding these bad images by the large number of bad pixels.

Compute average darks

Darks are normally taken as a group of images with identical exposure times with one of the dark filters in the beam. At the present time, Mimir shows a time-dependent dark current, with a fairly large "prompt" dark contribution and a much slower "classical" dark current. Observers are strongly urged to obtain dark images with exposure times *identical* to those of their science and calibration images and NOT to scale darks by exposure time.

In this step, multiple frame darks are averaged using the median filtering averager.

Create a subdirectory in "Analysis" with a name such as "Step_10_Ave_Darks"
Identify the darks that will be averaged into one average dark image for each day and exposure time
1. The lists of images can be put into filelists using the ls.pro progra
Use coadd_median_with_bad.pro to create the averaged dark image, giving it a name like "20050517_ave_dark_3s.fits".
1. Multiple average dark images may be calculated using the batch.pro program, as described at the top of this page

Compute flat field images

Mimir flat field images are normally obtained as a large number of dome images with the illuminating lights on and then an equal number of images of the same exposure time with the lights off. Differencing these sets of images removes dark and bias contributions and other nasties. The steps necessary for producing averaged, background-removed, normalized flats for non-polarimetric flats are as follows:

Create a subdirectory in "Analysis" with a name such as "Step_11_Flats"
If only a small number of different types of flat field images were obtained, the user can run coadd_for_flats.pro directly. Usually, however, there are a large number of different flats to create, making batch-type processing preferred. The batch steps are:
1. Create a subdirectory in "Step_11" with a name like "batch_files"
2. Run ls.pro to select the list of linearity-cleaned FITS images making up the "lights on" set of flats, storing the resulting list of images in the "batch_files" directory with a name like "K_on_input_list.dat"
3. Repeat ls.pro for the "lights off" set, writing a file with a name like "K_off_input_list.dat"
4. Repeat 2 & 3 for all sets of lights-on and lights-off FITS images, building the list files in the "batch_files" directory
5. Create a batch input file in the "batch_files" directory with a name like "batch_input.dat". In it, write lines that will use coadd_for_flats.pro to process each group of files into one averaged file, with a name like "K_on_ave.fits"
6. Run the batch file, processing all the lights-on and lights-off sets of files into averaged files
Create a subdirectory in "Step_11" with a name like "differenced_images"
Difference the lights-on and lights-off images with diff_for_flats.pro, storing the resulting images in the "differenced_images" directory with names like "K_on-off.fits"
To achieve normalized flat-field images:
1. For imaging flats, use imhist.pro to determine the median value of each "on-off" image
2. For spectroscopic flats, use DS9 to find a representative "brightest" pixel.
3. Use normal.pro to normalize the flat image by the median (if imaging) or brightest pixel (if spectroscopy), saving the result in the "Step_11" directory with names like "K_flat.fits"
For polarimetric flats, because of the large number of different position angles sampled, more automation is needed. This is done by running prepare_polar_flats_for_coadd.pro, which will write the necessary batch input files and the batch file to be run to create the average lights-on and lights-off images.

There are many possible problems that may arise in this step. The user is strongly cautioned to use IDL's "journal" mode to capture all error messages, to review them and resolve outstanding errors before proceeding.
Correcting errors in FITS headers can be done using hedit.pro which allows changing the value of a FITS header for one or more images or adding additional COMMENT lines, if critical comments were left off at observing time (e.g., the status of the dome lights, as "Lights On" in a COMMENT field.)

After producing the averaged Lights-On and Lights-Off images, use ls.pro to make a list of the files, then edit it by hand into the form of a batch input file that calls diff_for_flats.pro, generating files with names like "0517_polar_On-Off_133.fits", where the "133" signifies the HWP motor step number.
Run batch.pro with the input file from the previous step, to produce the 32 difference images.
Use ls.pro to generate a file listing all of the "On-Off" images, with a name like "0517_difference_images.dat"
Compile and run imhist_for_polar_flats.pro, using as the input file the one produced in the previous step, and creating an output file with a name like "0517_histo_values.dat". This program will read each image and display a plot of the cumulative pixel count vs pixel value and mark the median pixel value and print that value on the plot. Position the mouse inside the plot and click any button to proceed. The next image will be analyzed and its plot generated, etc. The output file will be a list of the input files and their median values.
Edit this file into the form of a batch file (including having a "cd;path...." first line to point into the right working directory), with lines calling normal.pro, and output names such as "0517_polar_flat_0.fits", where "0" is the HWP motor step.
Run the batch.pro program, with the input file from the step above. This will generate the set of 32 autonormalized polarization flats in whatever directory you elect to save them into.
For data obtained prior to 2006/07/27, one of the FF images has likely been taken at the wrong PA value due to an error in the FF script. You can create a new FF image for the missing PA, via interpolation from the other PA images, using the fix_missing_pol_flat.pro tool.

Science Image Corrections and Processing
At this point, all science images will have already been linearity corrected. All that remains is to subtract darks and divide by flats. This is done while still respecting the bad pixel masks. The resulting files will have names like "20050521.1001_LDFC.fits" to recognize that they have been Linearity, Dark, and Flat field Corrected.
1. Create a directory within the "Analysis" directory with a name like "Step_12_Science_Images"
2. Create subdirectories in "Step_12" with the dates of the nights containing science data (e.g., "20050521")
3. Compile the program science_dark_and_flat_correct.pro
4. In the IDL window, or development editor, change directories to the "Analysis" directory (e.g., "cd,'/home/user/analysis'). This will make all following steps much quicker.
5. With the observing logs (both hand written and the electronics ones from Step_0) in hand, invoke science_dark_and_flat_correct.pro, select the linearity-corrected FITS-type images to be process, the corresponding bad pixel mask, the appropriate duration dark image, and the appropiate filter flat field image. Save the results (by specifying an output directory) in the subdirectory of "Step_12" with the proper night designation.
1. The above process works quite well for normal quantities of imaging and spectroscopy data.
2. For the vast quantities of data generated by polarimetry, or by occultation or other image-intensive programs, it will be best to use batch.pro to run science_dark_and_flat_correct.pro with suitably generated input list files.
3. The particular problem of polarimetric flat correction for specific HWP position angles is covered below.
- For processing of polarimetric data, characterized by lots of images taken through a number of different half-wave-plate position angles, more automation is needed to make the processing tractable.
1. For each night's polarimetry, create a subdirectory in the "Step_12" directory, with a name reflecting the night's designation (e.g., "20050522")
2. Also, create a subdirectory of "Step_12" named "batch_files" to hold the batch and journal files
3. Within that new directory, create two subdirectories. One should be named "corrected" (and will be where the dark and flat corrected images will go), and the other should be named "input_list_files", which is where the file lists for batch processing will go.
4. For a given night, all polarimetric data obtained with the same exposure times will be processed as a group. If multiple exposure times were used during the night, there will be multiple groups that will need to be processed.
5. If the number of frames in a group is more than about 1200, an extra few steps are needed:
  1. Use the ls.pro program to select about 600-800 frames from the linearity-corrected data repository ("Step_9"), and put the resulting list file into the "input_list_files" directory for that night. If the number of polarimetric frames with the same exposure time is, say, 2400, then you can plan on making three such lists, each with 800 files. As you save the list files, give them names like "first_third.dat", "second_third.dat", etc.
  2. Use an editor to combine all the lists into one single file (e.g., "whole_list.dat")
  3. Run the ls_select_pol_files.pro program, selecting the list file ("whole_list.dat") instead of a group of files.
6. If the number of frames in a group is less than about 1200, run the ls_select_pol_files.pro program, selecting as inputs the linearity-corrected frames you wish to process
7. ls_select_pol_files will read the headers of all the polarimetry files, then group by HWP position angles into lists of files to be processed together. It is most useful to make the root file names of the list files to be written reflect the exposure times used, hence "0522_pol_1s" for a root name indicates that the lists will be for the 1s exposure times that night. Designate the "input_list_files" directory as the place the list files are to be stored.
8. Depending on how many HWP position angles were found in the polarimetry images, that number of input file lists will be written, each with a combination of the root file name and a suffix to that name to indicate the HWP position angle (in stepper motor steps), e.g., "0522_pol_1s_HWP_67.dat"
9. Use ls.pro on the input file list directory, selecting all the input files, and putting the output into the "batch_files" directory, with a name like "all_list.dat"
10. Use ls_to_batch.pro to create a skeleton batch file, named something like "big_batch.dat", with one line for each of the input file lines. Edit this batch file into full batch file form, prepared for use by science_dark_and_flat_correct by including each input list file, the corresponding dark file, the bad pixel mask, the flat file for that HWP angle, and designating the "corrected" subdirectory (perhaps including a night designator subdirectory) for the output directory. An example of such a batch file is given here.
11. In IDL, compile the batch.pro program
12. Start up journal logging by designating a journal file to hold the output from all IDL and program messages (e.g., "journal, jfile" where jfile was chosen to be a useful file name like "journal_big_batch.dat" and selected to reside in the "batch_files" subdirectory). Journal logging is necessary because of the enormous number of files that will be processed and the need to locate errors on particular files. Note that science_dark_and_flat_correct.pro is fairly robust to errors and will report problems with a file then go on to the next file.
13. Run batch.pro with input from the "big_batch.dat" type file located in the "batch_files" directory
14. After completion of the batch run, stop journal logging by issuing another journal command, but with no file name (e.g., "journal")
15. Using your editor, search the journal file for the word "ERROR". If the only one found is in the last couple of lines before or just after the words "Batch is Done!" then the batch job ran without error. If errors are embedded in the journal, identify the problem (usually a mismatch of exposure times, or missing/misdesignated filenames) and rerun the batch job (or a subset thereof).

Mimir Data Reduction Notes

Basic Mimir Image Processing -- 2006/10/02 - DPC