hscPipe Tutorial

HSC pipeline (hscPipe) has been developed by NAOJ (National Astronomical Observatory of Japan), Princeton Univsersity and Kavli IPMU (Institute for the Physics and Mathematics of the Universe, the University of Tokyo) based on LSST (Large Synoptic Survey Telescope) pipeline. All the commands in hscPipe are python scripts.

hscPipe can create calibrated images and object catalogs from raw data obtained with HSC. The basic steps to data reduction with hscpipe are the following;

  1. Preparation (Download raw data and setup reduction environment)

  2. Making data for detrending (Make Bias, Dark, Flat, and Fringe data)

  3. Single-visit processing (CCD analysis)

  4. Mosaicking

  5. Coadding images

  6. Multiband analysis

Please read Structure of repository and hscPipe basic information first, if you have never used hscPipe5. The link under Next Step in the side-bar is the guidepost for the next process. The filters supported in hscPipe8 are shown in hscPipe8 support filter list



Structure of repository

~/HSC   # Directory for data reduction
|--_mapper
|--registry.sqlite3   # Registory for all data
|--DOMEFLAT
| └--[dateObs]
|   └--[pointing]
|     └--[filter]
|       └--HSC-[visit]-[ccd].fits
|
|--[field]
| └--[dateObs]
|   └--[pointing]
|     └--[filter]
|       └--HSC-[visit]-[ccd].fits
|
|--CALIB   # Directory of data for detrending
| |--calibRegistry.sqlite3   # Registry for detrending
| |--BIAS   # BIAS data
| | └--[dateObs]
| |   └--NONE
| |     └--BIAS-[dateObs]-[ccd].fits
| |--DARK   # DARK data
| | └--[dateObs]
| |   └--NONE
| |     └--DARK-[dateObs]-[ccd].fits
| |--FLAT   # FLAT data
| | └--[dateObs]
| |   └--[filter]
| |     └--FLAT-[dateObs]-[filter]-[ccd].fits
| |--FRINGE   # FRINGE data
| | └--[dateObs]
| |   └--[filter]
| |     └--FRINGE-[dateObs]-[filter]-[ccd].fits
| |--SKY   # SKY data
| | └--[dateObs]
| |  └--[filter]
| |    └--SKY-[dateObs]-[filter]-[ccd].fits
| |--STRAY_LIGHT
| | └--ybackground-[ccd].fits
| |--BFKERNEL
| | └--brighter_fatter_kernel.pkl
| └--defects
|   └--[CalibDate]
|     └--defects-[CalibDate]-[ccd].fits
|
|--ref_cats   # Directory for astrometry reference catalog
| └--ps1_pv3_3pi_20170110
|--transmission   # Directory for transmission curve
| └--atmosphere.fits
| └--filter-[filter].fits
| └--optics.fits
| └--sensor-[ccd].fits
|
└--rerun
  |--[rerun for main analysis]
  | |--repositoryCfg.yaml
  | |--config   # Directory for analysis parameters
  | | |--packages.pickle
  | | |--singleFrameDriver.py
  | | |--mosaic.py or jointcal.py
  | | |--skyCorr.py
  | | |--coaddDriver.py
  | | |--multiBandDriver.py
  | | └--forcedPhotCcd.py
  | |--schema   # Directory for catalog schema file
  | | |--src.fits
  | | |--icSrc.fits
  | | |--deepCoadd_det.fits
  | | |--deepCoadd_meas.fits
  | | |--deepCoadd_peak.fits
  | | |--deepCoadd_forced_src.fits
  | | |--deepCoadd_mergeDet.fits
  | | |--deepCoadd_deblendedFlux.fits
  | | |--deepCoadd_deblendedModel.fits
  | | └--deepCoadd_ref.fits
  | |--[pointing]
  | | └--[filter]
  | |   |--output   # Directory for catalogs after detrending data of each CCD
  | |   | |--ICSRC-[visit]-[ccd].fits
  | |   | |--SRC-[visit]-[ccd].fits
  | |   | |--SRCMATCH-[visit]-[ccd].fits
  | |   | |--SRCMATCHFULL-[visit]-[ccd].fits
  | |   | └--[tract]
  | |   |   └--CALSRC-[visit]-[ccd].fits
  | |   |--corr   # Directory for images after detrending data of each CCD
  | |   | |--CORR-[visit]-[ccd].fits
  | |   | |--BKGD-[visit]-[ccd].fits
  | |   | |--skycorr-[visit]-[ccd].fits
  | |   | └--calexp-[visit].fits
  | |   └--singleFrameDriver_metadata
  | |     └--[visit].boost
  | |--deepCoadd
  | | |--skyMap.pickle   # tract definition file
  | | └--[filter]
  | |   └--[tract]
  | |      |--[patch].fits
  | |      |--[patch]_nImage.fits
  | |      └--[patch]
  | |        |--psfMatchedWarp-[filter]-[tract]-[patch]-[visit].fits
  | |        └--warp-[filter]-[tract]-[patch]-[visit].fits
  | |--jointcal-results   # Directory for results of jointcal.py
  | | └--[filter]
  | |   └--[tract]
  | |     └--jointcal_wcs-[visit]-[ccd].fits
  | |--fgcm-process   # Directory for a result of fgcmMakeLut.py
  | | └--fgcmLookUpTable.fits
  | |--fgcm-results   # Directory for results of fgcmCalibrateTract.py
  | | |--[filter]
  | | | └--[tract]
  | | |   └--fgcm_photoCalib-[visit]-[ccd].fits
  | | └--transmission
  | |   └--[tract]
  | |     └--atmosphere_[visit].fits
  | └--deepCoadd-results   # Directory for results after coaddDriver and multiBandDriver
  |   |--[filter]
  |   | └--[tract]
  |   |   └--[patch]
  |   |     |--det_bkgd-[filter]-[tract]-[patch].fits
  |   |     |--det-[filter]-[tract]-[patch].fits
  |   |     |--calexp-[filter]-[tract]-[patch].fits
  |   |     |--srcMatch-[filter]-[tract]-[patch].fits
  |   |     |--srcMatchFull-[filter]-[tract]-[patch].fits
  |   |     |--meas-[filter]-[tract]-[patch].fits
  |   |     └--forced_src-[filter]-[tract]-[patch].fits
  |   └--merged
  |      └--[tract]
  |       └--[patch]
  |         |--mergeDet-[tract]-[patch].fits
  |         └--ref-[tract]-[patch].fits
  |--mosaic_diag_HSC-I   # Directory for validation of mosaic, if it is outpted in rerun (need to add some option)
  | └--[tract]
  |   |--coeffs.dat
  |   |--ccd.dat
  |   |--jcont_[visit].png
  |   |--ResPosArrow2D_[visit].png
  |   |--dpos.dat
  |   |--ResPosScatter.png
  |   |--PosDPos.png
  |   |--ccdScale.dat
  |   |--fcont_[visit].png
  |   |--dmag.dat
  |   |--MdM.png
  |   |--ResFlux.png
  |   |--DFlux2D.png
  |   └--catalog.fits
  └--[rerun for data detrending]
      |--repositoryCfg.yaml
      |--config
      | |--packages.pickle
      | |--bias.py
      | |--dark.py
      | |--flat.py
      | |--fringe.py
      | └--sky.py
      |--[pointing]
      | └--[filter]
      |   └--corr
      |     └--ICEXPBKGD-[visit]-[ccd].png
      └--postISRCCD
      |└--v[visit]-f[filter]
      |    └--c[ccd].fits
      └--focalplane
        |--BIAS
        |└--NONE
        |  └--BIAS-[date].fits
        |--DARK
        |└--NONE
        |  └--DARK-[date].fits
        |--FLAT
        |└--NONE
        |  └--FLAT-[date].fits
        |--FRINGE
        |└--NONE
        |  └--FRINGE-[date].fits
        |--SKY
        └--NONE
          └--SKY-[date].fits

hscPipe8 support filter list

Here is the supported filter information in hscPipe8. For more detailed filter data, please refer to Subaru Telescope - HSC filters . If you need to analyze unsupported filter data, please contact HSC pipeline Helpdesk.

Broad-band filters

Filter

Filter name in pipeline

hscPipe8 support

g

HSC-G

Yes

r

HSC-R

Yes

i

HSC-I

Yes

z

HSC-Z

Yes

y

HSC-Y

Yes

i2

HSC-I2

Yes

r2

HSC-R2

Yes


Narrow-band filters

Filter

Open from

hscPipe8 support

note

NB387

S16B

Yes

NB400

S19A

Yes

NB430

S20A

No

you need to add manually for singleFrameDriver.py or later

NB468

S16A

Yes

NB497

S20B

No

you need to add manually for singleFrameDriver.py or later

NB515

S15A

Yes

NB527

S16A

Yes

NB656

S16A

Yes

NB718

S16B

Yes

NB816

S15B

Yes

NB921

S15A

Yes

NB926

S17A

Yes

IB945

S17A

Yes

NB973

S17A

Yes

NB1010

S18A

Yes