Single-visit processing¶

In this page, we describe single-visit processing. Structure of output FITS image is also explained at the end of the page.

Single-visit processing, detrending and calibration¶

After preparing data for detrending, next step is the single-visit processing.

The single-visit processing starts with detrending; overscan subtraction, Bias and Dark subtraction, flatfielding, and Fringe subtraction. The mask values are given to indicate bad pixels, crosstalk, and saturated pixels. After Bias sabtraction, the linearity and brighter-fatter effect are corrected for each CCD. Then the calibration part stars. The sky background is subtracted using 128-pixel grids, and bright sources for calibration are detected. The measured PSF of those sources are used to estimate photometric zero-points by matching with reference catalog. The object detection and measurement are performed. During the detection process, objects are deblended. Coordinates, shape, and flux are measured for the detected objects, and mask information or events during measurement are flagged in each pixel.

The hscPipe command singleFrameDriver.py can cover the all process mentioned above.

# Assume a directory of reduction is ~/HSC and rerun name test. The object or region name can be the rerun name.
# In case of using local system,
singleFrameDriver.py ~/HSC --calib ~/HSC/CALIB --rerun test --id visit=18214..18220:2^18224..18230:2 --batch-type=smp --cores=8

# If you output the data after detrending,
# For investigation of helpdesk, these data are sometimes required.
singleFrameDriver.py ~/HSC --calib ~/HSC/CALIB --rerun test --id visit=18214..18220:2^18224..18230:2 --config processCcd.isr.doWrite=True --batch-type=smp --cores=8

# singleFrameDriver.py [data reduction directory] --calib [directory of data for detrending] --rerun [rerun name] --id [visit, field, filter..]
# Options
#       --batch-type: The singleFrameDriver.py command can be submitted to batch system.
#       --config processCcd.isr.doBias=False processCcd.isr.doDark=False: If you skip creating Bias and Dark data, add these options.

In case of the region where objects are not crowded, nearly 2 GB memory would be consumed in 1 process. Please confirm the memory capacity before running the command. It takes about 10 hours to complete processing of 10 shots with 8 cores on HSC data analysis machine for open use (hanaco).

After finishing all processes, you can find corrected CCD images and object catalogs. The images are created as the following files;

Corrected CCD image (sky background is subtracted) : ~/HSC/rerun/[rerun]/[pointing]/[filter]/corr/CORR-[visit]-[ccd].fits
Sky background image : ~/HSC/rerun/[rerun]/[pointing]/[filter]/corr/BKGD-[visit]-[ccd].fits
Images after overscan subtraction ： ~/HSC/rerun/[rerun]/[pointing]/[filter]/thumbs/oss-[visit]-[ccd].png
Images after flatfielding : ~/HSC/rerun/[rerun]/[pointing]/[filter]/thumbs/flattened-[visit]-[ccd].png
Images after detrending (postISR) : ~/HSC/CALIB/postISRCCD/v[visit]-f[filter]/c[ccd].fits

Figure 1：(from left to right) raw data, oss image, flattened image, sky background image, CORR image. Sky background image is adjusted its size because it has a low number of pixels. CORR and sky background image are lotated to align raw data.

The catalogs can be found in /rerun/[rerun]/[pointing]/[filter]/output/;

Catalog of bright sources for calibration : ICSRC-[visit]-[ccd].fits
Catalog of detected sources, output from the final stage of single-visit processing: SRC-[visit]-[ccd].fits
Catalog of detected sources cross-matched with the external reference : SRCMATCH-[visit]-[ccd].fits
Catalog of detected and matched reference sources : SRCMATCHFULL-[visit]-[ccd].fits

These catalogs are FITS BINTABLE format, so you cannot check the contents with imaging software (e.g., ds9). Fv (Fv FITS Viewer) or STSDAS/TABLES (TABLES) is useful. You can also use PyFITS or IDL.

For example, the case of using Fv is shown in figure 2.

Figure 2：Results of opening a SRC file with Fv. The bottom-left panel shows FITS Header, the right one catalog.

There are schema files of catalog in rerun/[rerun]/schema. Note that the column name has been significantly different with hscPipe4.

In hscPipe, “aperture corrections” for each photometry algorithm. It is a photometric calibration by using r = 12 pix (4”) diameter circular aperture. For PSF, CModel, or Kron flux, it is corrected as measured by r = 12 pix aperture.

Structure of output FITS image¶

The created CORR image has total 13 HDUs (Header Data Unit);

Primary HDU (HDU0): FITS header
2nd HDU (HDU1): Science image (32bit Float)
3rd HDU (HDU2): Mask image (16bit UInt)
4th HDU (HDU3): Variance
after 5th HDU : Other information (e.g., PSF or aparture correction)

Figure 3 shows science and mask image. You have to specify which image you open by ds9;

# Open science image with ds9
ds9 CORR-[visit]-[ccd].fits
# Or
ds9 CORR-[visit]-[ccd].fits[1]

# Open mask image with ds9
ds9 CORR-[visit]-[ccd].fits[2]

# Open variance image with ds9
ds9 CORR-[visit]-[ccd].fits[3]

Figure 3：(left) science image of a CORR file. (right) mask image of the same fits.

In order to check the definition of mask bit, open header of mask image (figure 4). The mask bit and its event are shown in it (enclosed by red line). For example, the pixel where the value is 1024 = 2^10 is flagged as crosstalk because of “HIERARCH MP_CROSSTALK = 10” in the header. For the another pixel, the value 2080 (= 2048 + 32 = 2^11 + 2^5) indicates “Not Deblended” and “Detected”. The relation between bit and event could be changed in each reduction.

Figure 4: Header of mask image.

Loading catalogs by PyFITS¶

As the example of PyFITS, plot the object in the SRC catalog on CORR image. In the following example, the interactive mode of python is used. The result is shown in figure 5.

# Setup hscPipe
setup-hscpipe

# Launch python
python

# Import modules
>>> import pyfits
>>> import lsst.afw.display.ds9 as ds9
>>> import lsst.afw.image as afwImage

# Display a CORR file with ds9
>>> corr = afwImage.ImageF("~/HSC/rerun/[rerun]/[pointing]/[filter]/corr/CORR-[visit]-[ccd].fits")
>>> ds9.mtv(corr)

# Loading catalog
>>> cat = pyfits.open("~/HSC/rerun/[rerun]/[pointing]/[filter]/output/SRC-[visit]-[ccd].fits")
>>> cat[1].data
# Output all data
#       FITS_rec([ ([ ....
#       ...
#       dtype=(numpy.record, [('flags', 'u1', (14,)), ('id', '>i8'), ('coord_ra', '>f8'), ('coord_dec', '>f8'), ('parent', '>i8'),
#       ... ('base_FootprintArea_value', '>i4'), ('base_ClassificationExtendedness_value', '>f8'), ('footprint', '>i4')]))

# Get source coordinates in the catalog
>>> xs = cat[1].data["base_SdssCentroid_x"]
>>> ys = cat[1].data["base_SdssCentroid_y"]

# Plot the sources on ds9
>>> for x,y in zip(xs,ys):
...     ds9.dot("o", x, y, size=30)

Figure 5：(left) Display CORR image with ds9. (right) Superpose sources in SRC catalog on the CORR image.

Single-visit processing¶