Machine environment for hscPipe8¶

We introduce several machines for hscPipe.

1. The Multi-wavelength Data Analysis System (MDAS) maintained by Astronomy Data Center (ADC)/NAOJ

2. Large-scale data analysis system (LSC)

4. HSC data analysis machine for open use（hanaco）

Warning

Now hanaco is closed. Researchers who analyze HSC data are encouraged to use the large-scale data analysis system (LSC) rather than the multi-wavelength data analysis system (MDAS).

1. The Multi-wavelength Data Analysis System (MDAS) maintained by Astronomy Data Center (ADC)/NAOJ¶

You can also use the open-use PCs in ADC/NAOJ, called Multi-wavelength Data Analysis System (MDAS) for HSC data reduction. Some tips to analyze data using the PCs are shown below. Please read Data Analysis System User’s Guide first.

Work Disk¶

We provide two types of interactive data analysis servers (kaim[01-20] and kaih[01-12]) and large disk spaces (/lfs[01-16], /wkm[01-20], and /wkh[01-12]) for your work. Please make your own directory under this disk and perform data reduction in it. You can check the amount of disk space or status Working (Operational) Status of Data Analysis System .

# Example
mkdir /lfs01/[user name]

Warning

In order to perform analysis via HSC pipeline, large capacity of disk is required. Please check the amount of disk space before data reduction Disk use status and then execute on the one with enough space.

Installation of HSC pipeline¶

OS Red Hat Enterprise Linux 7 is used in MDAS. Binary package of HSC pipeline for Red Hat Enterprise Linux 7 (see instruction in hscPipe8 installation) should be installed.

You can download the reference catalogs (the astrometry catalog and Y-band stray light patterns) from https://hscdata.mtk.nao.ac.jp/hsc_bin_dist/index-ja.html

However, you cannot access to Binary Distribution server from MDAS. So please download the package and catalog file for astrometry with the following way;

Download to your own PC, then copy them to MDAS by scp command, or
Download directly to MDAS via your own PC.

# For Case 2, you can use the following command;
#
# Using wget
wget https://[pipeline URL] --no-check-c -O - | ssh [user name]@kaim01.ana.nao.ac.jp:/lfs01/hoge/ 'cat > pipe.tar.xz'

# Using curl
curl https://[pipeline URL] --insecure https://[pipeline URL] | ssh [user name]@kaim01.ana.nao.ac.jp:/lfs01/hoge/ 'cat > pipe.tar.xz'

In case of MDAS, you need to unset LD_LIBRARY_PATH. Unless this command, hscpipe sets up but scripts do not run properly.

# set up hscpipe.
usnet LD_LIBRARY_PATH
source [your_path_for_hscpipe]/bashrc
setup-hscpipe

Server and Queue information for HSC pipeline execution¶

On MDAS, you can execute the hscPipe commands in either directly running the commands (smp mode) or batch processing mode (PBS script mode). The detailed specification of configuration or architecture is described here

PBS Batch Processing¶

We here explain how to execute the hscPipe commands in the batch processng mode. PBS batch processing is built in MDAS. Using q16 queue which has maximum number of core, batch processing is executed with 16 cores per node. You need to prepare PBS batch script to execute batch processing in HSC pipeline.

For HSC pipeline, batch processing is available for constructBias.py, constructDark.py, constructFlat.py, constructFringe.py, constructSky.py, singleFrameDriver.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py. The following example is PBS batch script for singleFrameDriver.py.

# Preparing batch script using dry-run.
singleFrameDriver.py /lfs01/hoge/hsc --calib /lfs01/[user name]/hsc/CALIB --id filter=HSC-I visit=902798..902808:2 --config processCcd.isr.doFringe=False --time 600 --nodes 1 --procs 16 --dry-run --clobber-config

# Options：
#   --dry-run 　　　　：Dry run to create PBS script.
#   --clobber-config ：Execute command without using same rerun information.

When you add –dry-run to the command, the result of batch script will be output under /var/tmp/. You can use the script after copying this result to your own directory and edit it.

# Copy (or move) the --dry-run result to your woking directory
cp /var/tmp/tmph0gE /lfs01/[user name]/work/


# Edit tmph0gE file (PBS batch sfcript).
# In the batch script, there are some default comments.
# Please delete them all, then add below comments.
:
:
#!/bin/bash
#PBS -m abe
#PBS -q q16
#PBS -N test
#PBS -l walltime=336:00:00
#PBS -o hoge.out
#PBS -e hoge.err
#PBS -M hoge@nao.ac.jp


# To make log file for tracking batch process,
# please add below line to the end of the above PBS commens.
:
:
{

        （pipeline commands）

} &> /lfs01/hoge/work/tmph0gE.log

Please refer to Data Analysis System User’s Guide for detailed PBS option.

Note that;

Specify queue to q16 with -q q16.
Set maximum of actual time in case that a job is in a run state to max by -l walltime=336:00:00.

After preparing PBS batch script, run the following commands to perform it.

# Run PBS batch script
qsub -V /lfs01/[user name]/work/tmphOgE

The progress of this script is logged on tmphOgE.log, and you can check it by the following command;

# Output appended data as the file grows
tail -f tmphOgE.log

2. Large-scale data analysis system (LSC)¶

LSC is an extension to the existing data analysis system (MDAS), which is developed by ADC and is cooperated by Subaru Telescope at NAOJ. This system is the main data analysis machine for HSC open-use (October in 2021).

The open-use PI/CoIs have a higher privilege to use LSC resources for one-year term from the start of the processing. After the priority period ends, the users are automatically transitioned to a general user privilege with smaller available resources.

We have started a test operation from 2019 autumn and every PIs recieve the user instruction.

Please see the information e-mail (only send to PIs) and the link below. https://www.adc.nao.ac.jp/LSC/ug/users_guide_e.html

If you have any questions about “hscPipe” or HSC data reduction, please contact following address: helpdesk [at-mark] hsc-software.mtk.nao.ac.jp ([at-mark] replaces “@”)

If you have any questions about the system except for “hscPipe”, please contact following address: lsc-consult [at-mark] ana.nao.ac.jp ([at-mark] replaces “@”)

3. Batch Processing other than PBS¶

There are some batch systems other than PBS. Though the default system is PBS in HSC pipeline, you can set other ones by specify it in option command. Please use the best system operating on your machine.

# In case of useing SLURM.
singleFrameDriver.py ~/hsc --calib ~/hsc/CALIB --rerun test --id filter=HSC-I visit=902798..902808:2 --config processCcd.isr.doFringe=False --batch-type=slurm

# Option：
#   --batch-type ：Specify batch system. You can select from {slurm,pbs, or smp}.

4. HSC data analysis machine for open use（hanaco）¶

Warning

Now hanaco is closed. We reccomend you to migrate to the large-scale data analysis system (LSC).

Machine environment for hscPipe8¶

1. The Multi-wavelength Data Analysis System (MDAS) maintained by Astronomy Data Center (ADC)/NAOJ¶

Work Disk¶

Installation of HSC pipeline¶

Server and Queue information for HSC pipeline execution¶

PBS Batch Processing¶

2. Large-scale data analysis system (LSC)¶

3. Batch Processing other than PBS¶

4. HSC data analysis machine for open use（hanaco）¶

Table Of Contents

Index of Pipeline