Backgrounds

HSC pipeline is based on the LSST pipeline, and has been developed by NAOJ (National Astronomical Observatory of Japan), Princeton Univsersity and Kavli IPMU (Institute for the Physics and Mathematics of the Universe, the University of Tokyo). All commands in HSC pipeline are performed with python script. You can obtain reduced images and catalogs through data reduction by HSC pipeline. Here we introduce useful background knowledge before you use HSC pipeline.

Repository, Registry

HSC pipeline requires raw and calib (bias/dark/flat) data to be located under a defined directory tree, which is called “Repository”. The meta information of the data in the Repository is registered in a dedicated database called “Registry”. You can generate both Repository and Registry with HSC pipeline commands. The structure of Repository is shown in here.


rerun

The rerun is the concept used in data reduction with HSC pipeline. We refer a single data reduction processing run as rerun. If you perform data reduction with different parameters, it will refer as different rerun.


dataID

The dataId is an unique ID to specify input data. Commonly-used keywords are following:

  • visit 　　: Name for each observing shot.
  • ccd 　　 : CCD ID.
  • tract 　　: ID to specify the region of observed sky.
  • pointing : Observing epoch.
  • field 　　: A target name which corresponds to OBJECT in FITS header.
  • dateObs : Date of observations which corresponds to DATE-OBS in FITS header.
  • filter 　　: Filter name which corresponds to FILTER01 in FITS header.

tract, patch

The “tract” and “patch” are IDs to specify the observed sky. The tract is the largest square region that is usually defined as to include all observed sky. The small split region of a tract is called patch. For instance of SSP data, tract size is about 2 × 2 square degree. In a tract, there are ~100 patches whose size is 4200 × 4200 square pix (1 pix corresponds to 0.168”). Although you can set both tract and patch sizes by yourself, you should choose appropriate tract size to avoid large distortion effect which is possible to appear near the edge of a tract.


PBS/TORQUE

Several HSC commands can run with batch processing called TORQUE (Tera-scale Open-source Resource and QUEue manager). The TORQUE is a popular variant of PBS (Portable Batch System), and manage jobs and queues. Because available resources depend on a computer environment, please check your computer status and submit your job to the smallest queue.

Here, we introduce some linux commands to manage jobs.

  • qstat ：Check a job status
  • qdel ：Kill a job. You can find job ID using qstat command.
  • qsub ：Submit a job to your computer.

Schema file

The column reference file for catalog data is called schema file which is created under [a directory for reduction]/rerun/[rerun]/schema.