hscPipe basic information

This page summarizes some important concepts which need to use hscPipe.

Repository, Registory

hscPipe creates a database for data reduction and it has to be located in a defined directory tree. This is called “repository” and the database itself “registory”. Both are generated by hscPipe command. The structure of repository is shown in Structure of repositry.


rerun

Rerun is the concept used in data reduction with hscPipe. We call the successive processing from raw data to images and catalogs with the common parameters as rerun. So if you create new images with different parameters, it is regarded as a processing in a different rerun. When you change processing parameters or region, we recommend that you make new rerun.


detaId

dataId is a unique ID to specify input data. The following keywords are often used;

  • visit : Observation shot
  • ccd : CCD chip ID
  • tract : ID specifing observed sky
  • pointing : Observing epoch
  • field : Target name of observation corresponding to OBJECT in FITS header
  • dateObs : Date of observations corresponding to DATE-OBS in FITS header
  • filter : Filter name corresponding to FILTER01 in FITS header

tract, patch

The tract and patch are IDs to specify the observed sky area. The tract is the largest square region that is usually defined as to include all observed sky. The small split region of a tract is called patch. In case of HSC SSP data, tract size is about 2 × 2 square degree. In a tract, there are ~100 patches whose size is 4200 × 4200 square pix (1 pix corresponds to 0.168”). Although you can set both tract and patch sizes by yourself, you should choose appropriate tract size to avoid large distortion effect which is possible to appear near the edge of a tract.


PBS/TORQUE

Some of hscPipe commands can execute with batch processing called TORQUE (Tera-scale Open-source Resource and QUEue manager). The TORQUE is a popular variant of PBS (Portable Batch System), and manage jobs and queues. Because available resources depend on a computer environment, please check your computer status and submit your job to the smallest queue.

Here, we introduce some linux commands to manage jobs.

  • qstat : Check job status
  • qdel : Kill a job. You can find job ID using qstat command.
  • qsub : Submit a job.

Schema file

The column reference for catalog data is called schema file which is created under [reduction directory]/rerun/[rerun]/schema