hscPipe Tutorial

HSC pipeline (hscPipe) has been devepoled by NAOJ (National Astronomical Observatory of Japan), Princeton Univsersity and Kavli IPMU (Institute for the Physics and Mathematics of the Universe, the University of Tokyo) and based on LSST (Large Synoptic Survey Telescope) pipeline. The all commands in hscPipe are performed with python script.

hscPipe can create calibrated images and object catalogs from HSC raw data. The basic processing step is like below;

1. Preparation (Download raw data and setup reduction environment)
2. Making data for detrending (Make Bias, Dark, Flat, and Fringe data)
3. Single-visit processing (CCD analysis)
4. Mosaicking
5. Coadding
6. Multiband analysis

Please read Structure of repositry and hscPipe basic information first, if you have never used hscPipe5. The link under Next Step in the side-bar is guidepost for the next process. The filters supported in hscPipe5 are shown in hscPipe5 support filter list

---

Structure of repositry

~/HSC                           # Directory for data reduction
|--_mapper
|--registry.sqlite3             # Registory for all data
|--DOMEFLAT
| └--[dateObs]
|   └--[pointing]
|     └--[filter]
|       └--HSC-[visit]-[ccd].fits
|
|--[field]
| └--[dateObs]
|   └--[pointing]
|     └--[filter]
|       └--HSC-[visit]-[ccd].fits
|
|--CALIB        # Directory of data for detrending
| |--calibRegistry.sqlite3      #
| |--BIAS       # BIAS data
| | └--[dateObs]
| |   └--NONE
| |     └--BIAS-[dateObs]-[ccd].fits
| |--DARK       # DARK data
| | └--[dateObs]
| |   └--NONE
| |     └--DARK-[dateObs]-[ccd].fits
| |--FLAT       # FLAT data
| | └--[dateObs]
| |   └--[filter]
| |     └--FLAT-[dateObs]-[filter]-[ccd].fits
| |--FRINGE     # FRINGE data
| | └--[dateObs]
| |   └--[filter]
| |     └--FRINGE-[dateObs]-[filter]-[ccd].fits
| └--BFKERNEL
|   └--brighter_fatter_kernel.pkl
|
|--ref_cats     # Directory for reference catalog
| └--ps1_pv3_3pi_20170110
|
└--rerun
  |--[rerun]
  | |--repositoryCfg.yaml
  | |--config           # Directory for analysis parameters
  | | |--packages.pickle
  | | |--singleFrameDriver.py
  | | |--mosaic.py
  | | |--coaddDriver.py
  | | |--multiBandDriver.py
  | | └--forcedPhotCcd.py
  | |--schema           # Directory for schema file
  | | |--src.fits
  | | |--icSrc.fits
  | | |--deepCoadd_det.fits
  | | |--deepCoadd_meas.fits
  | | |--deepCoadd_peak.fits
  | | |--deepCoadd_forced_src.fits
  | | |--deepCoadd_mergeDet.fits
  | | |--deepCoadd_ref.fits
  | | └--forced_src_schema.fits
  | |--[pointing]
  | | └--[filter]
  | |   |--thumbs
  | |   | |--oss-[visit]-[ccd].png
  | |   | └--flattened-[visit]-[ccd].png
  | |   |--output       # Directory for catalogs after detrending data of each CCD
  | |   | |--ICSRC-[visit]-[ccd].fits
  | |   | |--SRC-[visit]-[ccd].fits
  | |   | |--SRCMATCH-[visit]-[ccd].fits
  | |   | |--SRCMATCHFULL-[visit]-[ccd].fits
  | |   | └--[tract]
  | |   |   └--CALSRC-[visit]-[ccd].fits
  | |   |--corr # Directory for images after detrending data of each CCD
  | |   | |--CORR-[visit]-[ccd].fits
  | |   | |--BKGD-[visit]-[ccd].fits
  | |   | └--[tract]
  | |   |   └--CALEXP-[visit]-[ccd].fits
  | |   └--singleFrameDriver_metadata
  | |     └--[visit].boost
  | |--deepCoadd
  | | |--skyMap.pickle  # Defining tract
  | | └--[filter]
  | |   └--[tract]
  | |     └--[patch]
  | |       └--warp-[filter]-[tract]-[patch]-[visit].fits
  | |--jointcal-results # Directory for results of mosaic
  | | └--[tract]
  | |   |--wcs-[visit]-[ccd].fits
  | |   └--fcr-[visit]-[ccd].fits
  | └--deepCoadd-results        # Directory for results after coaddDriver and multiBandDriver
  |   |--[filter]
  |   | └--[tract]
  |   |   └--[patch]
  |   |     |--det_bkgd-[filter]-[tract]-[patch].fits
  |   |     |--det-[filter]-[tract]-[patch].fits
  |   |     |--calexp-[filter]-[tract]-[patch].fits
  |   |     |--srcMatch-[filter]-[tract]-[patch].fits
  |   |     |--srcMatchFull-[filter]-[tract]-[patch].fits
  |   |     |--meas-[filter]-[tract]-[patch].fits
  |   |     └--forced_src-[filter]-[tract]-[patch].fits
  |   └--merged
  |      └--[tract]
  |       └--[patch]
  |         |--mergeDet-[tract]-[patch].fits
  |         └--ref-[tract]-[patch].fits
  |--mosaic_diag_HSC-I  # Directory for validation of mosaic (need to add some options)
  | └--[tract]
  |   |--coeffs.dat
  |   |--ccd.dat
  |   |--jcont_[visit].png
  |   |--ResPosArrow2D_[visit].png
  |   |--dpos.dat
  |   |--ResPosScatter.png
  |   |--PosDPos.png
  |   |--ccdScale.dat
  |   |--fcont_[visit].png
  |   |--dmag.dat
  |   |--MdM.png
  |   |--ResFlux.png
  |   |--DFlux2D.png
  |   └--catalog.fits
  └--[detrend rerun]
      |--repositoryCfg.yaml
      |--config
      | |--packages.pickle
      | └--flat.py
      |--[pointing]
      | └--[filter]
      |   └--thumbs
      |     |--oss-[visit]-[ccd].png
      |     └--flattened-[visit]-[ccd].png
      └--postISRCCD
        └--v[visit]-f[filter]
          └--c[ccd].fits

hscPipe5 support filter list

Here is the supported filter information in hscPipe5. For more detailed filter data, please refer to Subaru Telescope - HSC filters . If you need to analyze unsupported filter data, please contact HSC pipeline Helpdesk.

Broad-band filters

Filter	Filter name in pipeline	hscPipe5
g	HSC-G	Yes
r	HSC-R	Yes
i	HSC-I	Yes
z	HSC-Z	Yes
y	HSC-Y	Yes
i2	HSC-I2	Yes
r2	HSC-R2	Yes

Narrow-band filters

Filter	Open from	hscPipe5
NB387	S16B	Yes
NB468	S16A	No
NB515	S15A	No (1)
NB527	S16A	No
NB656	S16A	No
NB718	S16B	Yes
NB816	S15B	Yes
NB921	S15A	Yes
NB926	S17A	No
IB945	S17A	No
NB973	S17A	Yes

(1) Refer to NB515 processing.