Setting a registry for raw data

The raw data downloaded from STARS should be moved to the repository for hscPipe processing and registered. The command “ingestImages.py” is used.

# Move the raw data to a directory for data reduction, create a registry and register the raw data.
# Assume that raw data are in ~/rawdata.
# Specify the directory with absolute path.
ingestImages.py ~/HSC ./rawdata/*.fits --mode=link

# ingestImages.py [directory for data reduction] [raw fits file name] --mode=[move,copy,link,skip]
# Options are,
#       --mode: Mode of delivering the files to their destination.
                If you choose "link", the link to raw data is created.

The new directories, such as [Object Name], DARK, and DOMEFLAT and a registry registry.sqlite3 are generated under ~/HSC. The structure of these directories should be ~/HSC/[Object Name, BIAS, DARK, DOMEFLAT]/[dateObs]/[opinting]/[filter]/HSCA*.fits.

The contents of registry.sqlite3 can be checked by SQL. If you are interested in SQL, please check other materials.

# Open registry.sqlite3 with SQLite.
sqlite3 registry.sqlite3

# Read header information.
sqlite> .header on
# Display data.
sqlite> SELECT visit, filter, field, count(visit)
...> FROM raw                                        # Select raw table in the registry.
...> GROUP BY visit, filter, field;                  # Order of display, visit > filter > field
# You can write it in one line, "select filter, field, visit, count(visit) from raw group by visit;"

# Results
visit|filter|field|count(visit)
17636|HSC-Z|DOMEFLAT|112
17638|HSC-Z|DOMEFLAT|112
17640|HSC-Z|DOMEFLAT|112
17642|HSC-Z|DOMEFLAT|112
17644|HSC-Z|DOMEFLAT|112
17646|HSC-Z|DOMEFLAT|112
18336|HSC-I|DARK|112
18350|HSC-Y|DARK|112
.....

# Quit
sqlite> .q

Please verify that the number of all “count(visit)” column is 112 which indicates all the 112 CCDs are included in 1 visit. You can also check the registered visit number of each filter and each field.

Warning

The task of hscPipe sometimes proceeds even if any CCD data is lacking. So please check that all visits have 112 CCDs data.