Machine environment for hscPipe8¶
We introduce several machines for hscPipe.
1. The Multi-wavelength Data Analysis System (MDAS) maintained by Astronomy Data Center (ADC)/NAOJ
2. Large-scale data analysis system (LSC)
3. Batch Processing other than PBS
4. HSC data analysis machine for open use（hanaco）
Warning
Now hanaco is closed. Researchers who analyze HSC data are encouraged to use the large-scale data analysis system (LSC) rather than the multi-wavelength data analysis system (MDAS).
1. The Multi-wavelength Data Analysis System (MDAS) maintained by Astronomy Data Center (ADC)/NAOJ¶
You can also use the open-use PCs in ADC/NAOJ, called Multi-wavelength Data Analysis System (MDAS) for HSC data reduction. Some tips to analyze data using the PCs are shown below. Please read Data Analysis System User’s Guide first.
Work Disk¶
We provide interactive data analysis servers (mana[00-07]), large disk spaces (/lwk), and extended disk area (/ext_nfs[1-3]) for your work. Please make your own directory under this disk and perform data reduction in it. You can check the amount of disk space or status Working (Operational) Status of Data Analysis System .
# Example
mkdir /lwk/[user name]
Warning
In order to perform analysis via HSC pipeline, large capacity of disk is required. Please check the amount of disk space before data reduction Disk use status and then execute with enough space. In addition, please be considerate of other users and avoid long-term occupancy of the lwk area, such as moving data once at a turning point of processing.
Installation of HSC pipeline¶
MDAS became a new system and hscPipe 8.5.3 is installed in /usr/local/hscpipe, so users do not need to install this version by themselves if they use this version (latest version at July 2024) from July 4, 2024.
OS Rocky Linux 8 (compatible with Red Hat Enterprise Linux 8) is used in MDAS. If you want to use a version other than 8.5.3, binary package of HSC pipeline for CentOS 7 (see instruction in hscPipe8 installation) should be installed.
The reference catalogs (the astrometry catalog and Y-band stray light patterns) are deployed in /lwk/software/data/hscpipe (the old system required users to download them by themselves, which is no longer necessary with the new system).
In case of MDAS, you need to unset LD_LIBRARY_PATH. Unless this command, scripts may not run properly.
# set up hscpipe.
unset LD_LIBRARY_PATH
source /usr/local/hscpipe/8.5.3/bashrc # If you installed it yourself, change the path accordingly
setup-hscpipe
Server and Queue information for HSC pipeline execution¶
On MDAS, you can execute the hscPipe commands in directly running the commands (smp mode). Batch processing mode (PBS script mode), which was possible on the old system, cannot be used on the new system. The detailed specification of configuration is described here
2. Large-scale data analysis system (LSC)¶
LSC is an extension to the existing data analysis system (MDAS), which is developed by ADC and is cooperated by Subaru Telescope at NAOJ. This system is the main data analysis machine for HSC open-use (October in 2021).
The open-use PI/CoIs have a higher privilege to use LSC resources for one-year term from the start of the processing. After the priority period ends, the users are automatically transitioned to a general user privilege with smaller available resources.
We have started a test operation from 2019 autumn and every PIs recieve the user instruction.
Please see the information e-mail (only send to PIs) and the link below. https://www.adc.nao.ac.jp/LSC/ug/users_guide_e.html
If you have any questions about “hscPipe” or HSC data reduction, please contact following address: helpdesk [at-mark] hsc-software.mtk.nao.ac.jp ([at-mark] replaces “@”)
If you have any questions about the system except for “hscPipe”, please contact following address: lsc-consult [at-mark] ana.nao.ac.jp ([at-mark] replaces “@”)
PBS Batch Processing¶
We here explain how to execute the hscPipe commands in the batch processng mode. PBS batch processing is built in the LSC. To execute batch processing for some of the HSC pipeline commands (commands with –batch-type in option), you need to prepare a PBS batch script.
For HSC pipeline, commands with –batch-type in option are constructBias.py, constructDark.py, constructFlat.py, constructFringe.py, constructSky.py, singleFrameDriver.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py. The following example is PBS batch script for singleFrameDriver.py.
# Preparing batch script using dry-run.
singleFrameDriver.py /lwk/hoge/hsc --calib /lwk/[user name]/hsc/CALIB --id filter=HSC-I visit=902798..902808:2 --config processCcd.isr.doFringe=False --time 600 --nodes 1 --procs 16 --dry-run --clobber-config
# Options：
# --dry-run ：Dry run to create PBS script.
# --clobber-config ：Execute command without using same rerun information.
When you add –dry-run to the command, the result of batch script will be output under /tmp/. You can use the script after copying this result to your own directory and edit it.
# Copy (or move) the --dry-run result to your woking directory
cp /tmp/tmph0gE /lwk/[user name]/work/
# Edit tmph0gE file (PBS batch sfcript).
# In the batch script, there are some default comments.
# Please delete them all, then add below comments.
:
:
#!/bin/bash
#PBS -m abe
#PBS -q qm
#PBS -N test
#PBS -l select=28:ncpus=1:mpiprocs=1:mem=16g
#PBS -l walltime=24:00:00
#PBS -o hoge.out
#PBS -e hoge.err
#PBS -M hoge@nao.ac.jp
# To make log file for tracking batch process,
# please add below line to the end of the above PBS commens.
:
:
{
（pipeline commands）
} &> /lwk/hoge/work/tmph0gE.log
Please refer to section 4.2.4 PBS Professional of Data Analysis System User’s Guide for the old system for detailed PBS option.
After preparing PBS batch script, run the following commands to perform it.
# Run PBS batch script
qsub -V /lwk/[user name]/work/tmphOgE
The progress of this script is logged on tmphOgE.log, and you can check it by the following command;
# Output appended data as the file grows
tail -f tmphOgE.log
3. Batch Processing other than PBS¶
There are some batch systems other than PBS. Though the default system is PBS in HSC pipeline, you can set other ones by specify it in option command. Please use the best system operating on your machine.
# In case of useing SLURM.
singleFrameDriver.py ~/hsc --calib ~/hsc/CALIB --rerun test --id filter=HSC-I visit=902798..902808:2 --config processCcd.isr.doFringe=False --batch-type=slurm
# Option：
# --batch-type ：Specify batch system. You can select from {slurm,pbs, or smp}.
4. HSC data analysis machine for open use（hanaco）¶
Warning
Now hanaco is closed. We reccomend you to migrate to the large-scale data analysis system (LSC).