hscPipe command

General information about parameters in hscPipe command

  • ROOT
    Path to input data repository.
  • -h, –-help
    Show help messages.
  • -–calib <directory of data for detrending>
    Path to input calibration repository.
  • -–output <directory for output data>
    Path to output data repository. If you specify “rerun”, you do not have to set this option. This is useful when you want to store the output in a certain rerun to another directory.
  • -–rerun <rerun name>
    Name of a certain analysis process. Basically, hscPipe performs analysis referring the data under <rerun name> directory and creates post-data under the same directory. If you perform data reduction with different parameters, it will refer as different rerun.
  • -c <config item>=<parameter>, –-config <config item>=<parameter>
    Override configuration. e.g., -c foo=newfoo bar.baz=3. If you have many parameters to change, you can load config file which you have prepared using -C option.
  • -C <config file name>, –-configfile <config file name>
    Override configuration using a file. You need to write one parameter in one line in the file.
  • -L <log level> –-loglevel <log level>
    Specify the level of logging messages. Select DEBUG(ridiculous numbers of messages), INFO(basic information [default]), WARN(only warnings), or FATAL(only complete pipeline failures)
  • –-debug
    Enable debugging output.
  • –-doraise
    Raise an exception. If you want to pick up an exception from a log, use this option.
  • –-logdest LOGDEST
    Specify a file where the log messages are copied (they are still displayed on a terminal.)
  • –-show [{config,data,tasks,run} [{config,data,tasks,run} …]]
    Display the information which you specify individually. The most useful option is –show config. It is used to print the all config parameters on the terminal (refer to Arguments and parameters of hscPipe command). This can be also used to extract only a specific keyword. For example, in order to show only the things matching ’background‘, you can use –show config=*background*. –show tasks is also useful. This can print the tasks to be used by the command you are running.
  • -j <number of process>, –-processes <number of process>
    Number of processes to use. This can be used to perform multiprocessing on 1 node.
  • -t <process timeout [sec]>, –-process-timeout <process timeout [sec]>
    Timeout of multiprocessing; maximum wall time in sec.
  • –-clobber-output
    Remove and re-create the output directory if it already exists. Because the process specified by this parameter ends before multiprocessing, ‘-j’ parameter can be also added when you execute the command.
  • -–clobber-config
    Parameter to make a backup of your config (files in <data_repo>/config/ will be moved; <foo> –> <foo>~1), and overwrite existing config file. Every time you run the pipeline commands, all config parameters are stored in the repository. If you have changed something which would make your data inhomogeneous, the pipeline will refuse to execute. However, you can run the command due to adding –clobber-config.
  • –-id <Key>=<parameter value> <Key>=<parameter value> …
    Data Id that you want to run, e.g. –id visit=12345 ccd=1,2.

Output data to different rerun or directory

When you output the data to different rerun or directory (especially for processing in coaddDriver.py), there are some tips shown below.

If you run coaddDriver.py once, it uses /rerun/[rerun]/deepCoadd/[filter]/[tract]/[patch]/warp-*.fits and creates coadd images in the specified rerun in the following cases.

  • Specify both [rerun for data input] and [rerun for data output] using colon (:).

    # Input data from test1 directory, then output coadd data to test2 directory.
coaddDriver.py $home/hsc --rerun test1:test2 --id filter=HSC-I tract=0 --selectId visit=224608..224888:2 ccd=0..103

# Usage：
#   coaddDriver.py [data reduction directory] --rerun [rerun for data input]:[rerun for data output]

  • Specify full path of rerun for data input in [data reduction directory], and rerun in [rerun for data output]

    # Input data from test directory ($home/hsc/rerun/test) and output coadd images to rerun/test/rerun/test2
coaddDriver.py $home/hsc/rerun/test --rerun=test2 --id filter=HSC-I tract=0 --selectId visit=224608..224888:2 ccd=0..103

# Usage：
#   coaddDriver.py [full path to rerun for data input] --rerun [rerun for data output]
#
# !!Note!! In this case, new rerun is created under $home/hsc/rerun/test/rerun.

  • Use –output option

    # Load data from test directory, and output coadd images to $home/hsc/rerun/test2.
# You should execute coaddDriver.py in $home/hsc.
coaddDriver.py $home/hsc/rerun/test --output rerun/test2 --id filter=HSC-I tract=0 --selectId visit=224608..224888:2 ccd=0..103

# Usage：
#   coaddDriver.py [rerun for data input] --output [rerun for data output]
# Rerun for data output should be specified from the directory where coaddDriver.py is executed.

If you make warp images again, you have to move or rename the directories which have already created under deepCoadd, then run coaddDriver.py.

# Rename a directory in deepCoadd mv $home/hsc/rerun/test/deepCoadd/HSC-I $home/hsc/rerun/test/deepCoadd/tmp_HSC-I

Arguments and parameters of hscPipe command

You can check the arguments and parameters of hscPipe command from “help” or config files.

# Show help
# You can also use "-h"
mosaic.py --help

# Show config file
# Specify the directory on which _mapper file is as a first argument.
mosaic.py $home/HSC/ --show config

For example, the following messages are displayed for help.

usage: mosaic.py input [options]

positional arguments:
input                 path to input data repository, relative to
                      $PIPE_INPUT_ROOT

optional arguments:
  -h, --help            show this help message and exit
  --calib RAWCALIB      path to input calibration repository, relative to
                        $PIPE_CALIB_ROOT
  --output RAWOUTPUT    path to output data repository (need not exist),
                        relative to $PIPE_OUTPUT_ROOT
  --rerun [INPUT:]OUTPUT
                        rerun name: sets OUTPUT to ROOT/rerun/OUTPUT;
                        optionally sets ROOT to ROOT/rerun/INPUT
  -c [NAME=VALUE [NAME=VALUE ...]], --config [NAME=VALUE [NAME=VALUE ...]]
                        config override(s), e.g. -c foo=newfoo bar.baz=3
  -C [CONFIGFILE [CONFIGFILE ...]], --configfile [CONFIGFILE [CONFIGFILE ...]]
                        config override file(s)
  -L [LEVEL|COMPONENT=LEVEL [LEVEL|COMPONENT=LEVEL ...]], --loglevel [LEVEL|COMPONENT=LEVEL [LEVEL|COMPONENT=LEVEL ...]]
                        logging level; supported levels are
                        [trace|debug|info|warn|error|fatal]
  --longlog             use a more verbose format for the logging
  --debug               enable debugging output?
  --doraise             raise an exception on error (else log a message and
                        continue)?
  --profile PROFILE     Dump cProfile statistics to filename
  --show SHOW [SHOW ...]
                        display the specified information to stdout and quit
                        (unless run is specified).
  -j PROCESSES, --processes PROCESSES
                        Number of processes to use
  -t TIMEOUT, --timeout TIMEOUT
                        Timeout for multiprocessing; maximum wall time (sec)
  --clobber-output      remove and re-create the output directory if it
                        already exists (safe with -j, but not all other forms
                        of parallel execution)
  --clobber-config      backup and then overwrite existing config files
                        instead of checking them (safe with -j, but not all
                        other forms of parallel execution)
  --no-backup-config    Don't copy config to file~N backup.
  --clobber-versions    backup and then overwrite existing package versions
                        instead of checking them (safe with -j, but not all
                        other forms of parallel execution)
  --no-versions         don't check package versions; useful for development
  --id [KEY=VALUE1[^VALUE2[^VALUE3...] [KEY=VALUE1[^VALUE2[^VALUE3...] ...]]
                        data ID, with raw CCD keys + tract
  --diagDir DIAGDIR     Directory in which to dump diagnostics
  --diagnostics         Save diagnostics plots?
  --snapshots           Save snapshots of ObsVecs during iteration?
  --numCoresForReadSource NUMCORESFORREADSOURCE
                        Number of cores to be used for reading source catalog
  --readTimeout READTIMEOUT
                        Timeout (sec) for reading inputs with multiple
                        processes

Notes:
            * --config, --configfile, --id, --loglevel and @file may appear multiple times;
                all values are used, in order left to right
            * @file reads command-line options from the specified file:
                * data may be distributed among multiple lines (e.g. one option per line)
                * data after # is treated as a comment and ignored
                * blank lines and lines starting with # are ignored
            * To specify multiple values for an option, do not use = after the option name:
                * right: --configfile foo bar
                * wrong: --configfile=foo bar

When you use –show config option, you can see the following message.

root INFO: Loading config overrride file '/opt/hscpipe/6.7/lsst_home/miniconda3-4.3.21-10a4f6/stack/Linux64/obs_subaru/6.7-hsc+1/config/Mosaic.py'
root INFO: Loading config overrride file '/opt/hscpipe/6.7/lsst_home/miniconda3-4.3.21-10a4f6/stack/Linux64/obs_subaru/6.7-hsc+1/config/Mosaic.py'
import lsst.meas.mosaic.mosaicTask
assert type(config)==lsst.meas.mosaic.mosaicTask.MosaicConfig, 'config is of type %s.%s instead of lsst.meas.mosaic.mosaicTask.MosaicConfig' % (type(config).__module__, type(config).__name__)
# Use Chebyshev polynomials for flux fitting?
config.chebyshev=True

# Fit to catalog flux?
config.fluxFitAbsolute=True

# Minimum number of sources to be merged.
#       Valid Range = [0,inf)
config.numSourceMerge=2

# flux fitting order
#       Valid Range = [0,inf)
config.fluxFitOrder=7

# Solve flux correction?
config.doSolveFlux=True

# number of stars used for fitting per exposure
config.nBrightest=0

# Solve CCD alignment?
config.solveCcd=True

# If True, saturated objects will also be used for mosaicking.
config.includeSaturated=False
...

If you want to check a particular parameter,

# Display parameters only including "doColorterms".
mosaic.py ~/HSC --show config=*doColorterms*

# Output
root INFO: Loading config overrride file '/opt/hscpipe/6.7/lsst_home/miniconda3-4.3.21-10a4f6/stack/Linux64/obs_subaru/6.7-hsc+1/config/Mosaic.py'
root INFO: Loading config overrride file '/opt/hscpipe/6.7/lsst_home/miniconda3-4.3.21-10a4f6/stack/Linux64/obs_subaru/6.7-hsc+1/config/Mosaic.py'
Matching "*doColorterms*" without regard to case (append :NOIGNORECASE to prevent this)

# Apply color terms as part of solution?
config.doColorTerms=True

Command editting

If you want to edit the contents of a certain command, you can find the appropriate python script using the following way. For example, you will rewrite some contents of makeDiscreteSkyMap.py.

  1. Confirm the directory storing makeDiscreteSkyMap.py

    which makeDiscreteSkyMap.py

# Result
# >> $HOME/hscpipe/6.7/lsst_home/miniconda3-4.3.21-10a4f6/stack/Linux64/pipe_tasks/6.7-hsc/bin/makeDiscreteSkyMap.py

  2. Check the contents of makeDiscreteSkyMap.py

    cat $HOME/hscpipe/6.7/lsst_home/miniconda3-4.3.21-10a4f6/stack/Linux64/pipe_tasks/6.7-hsc/bin/makeDiscreteSkyMap.py

# Contents of makeDiscreteSkyMap.py
"""
#!/usr/bin/env python
#
# LSST Data Management System
# Copyright 2008, 2009, 2010 LSST Corporation.
#
# This product includes software developed by the
# LSST Project (http://www.lsst.org/).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.    See the
# GNU General Public License for more details.
#
# You should have received a copy of the LSST License Statement and
# the GNU General Public License along with this program.  If not,
# see <http://www.lsstcorp.org/LegalNotices/>.
#
from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask

MakeDiscreteSkyMapTask.parseAndRun()
"""
  3. Check which module is loaded by the command
    In case of makeDiscreteSkyMap.py, you can find that module ‘lsst.pipe.tasks.makeDiscreteSkyMap’ invokes class ‘MakeDiscreteSkyMapTask’.
  4. Move to directory storing the module.
    The path to the directory where the module is stored corresponds to the module name in command script, and the path is explicitly defined in the module name; 2 words following ‘lsst’. In the above case, the path to directory for the module is $PIPE_TASKS_DIR. The module and classes are stored in $PIPE_TASKS_DIR > python > lsst > pipe > tasks.
    cd $PIPE_TASKS_DIR/python/lsst/pipe/tasks
  5. Edit the target script.
    In case of makeDiscreteSkyMap.py, class “MakeDiscreteSkyMapTask” is used.
    vi makeDiscreteSkyMap.py

If you cannot edit the source file, please follow the steps shown below.

$ mkdir -p ~/opt/hscpipe/6.7/lsst_home/miniconda3-4.3.21-10a4f6/stack/Linux64/pipe_tasks
$ cp -r /opt/hscpipe/6.7/lsst_home/miniconda3-4.3.21-10a4f6/stack/Linux64/pipe_tasks/6.7-hsc ~/opt/hscpipe/6.7/lsst_home/stack/Linux64/pipe_tasks/
#
# Edit the source, then
$ setup -jr ~/opt/hscpipe/6.7/lsst_home/miniconda3-4.3.21-10a4f6/stack_6.7/Linux64/pipe_tasks/6.7-hsc

Perform the command ‘setup -jr …’ every time you execute hscPipe, then you setting is reflected. When you check the environment,

$ eups list
pipe_tasks            LOCAL:/home/<User name>/opt/hscpipe/6.7/lsst_home/miniconda3-4.3.21-10a4f6/stack/Linux64/pipe_tasks/6.7-hsc   setup