User Tools

Site Tools


pipeline

KVN Pipeline

Current version: 1.0.


Jeff Hodgson - (KASI, jhodgson@kasi.re.kr)

<WRAP center round download 60%> KVNPipe.zip </WRAP>

Description

The KVN Pipeline is a ParselTongue based pipeline for the automatic reduction of VLBI data obtained using the KVN. Currently the KVN Pipeline operates in single polarisation only, but automatically handles simultaneous frequency data. For a full description of the Pipeline, see PAPER. If you use the pipeline, please add a citation to PAPER.

Installation and Requirements

The KVN Pipeline requires both ParselTongue and AIPS to be installed.

  • Installation instructions for ParselTongue can be found here.
  • Installation instructions for AIPS can be found here.

The KVN Pipeline consists of two files:

  • KVNpipeline_v1_0.py
  • KVN_Pipeline_Control_File.py

The KVN Pipeline can then be run from the command line as:

ParselTongue KVN_Pipeline_Control_File.py

Operation

The KVN Pipeline is contained within the KVNpipeline_v1_0.py file, which also contains many function based from the EVN Pipeline. This file is not directly run by the user but is imported into a separate script ( KVN_Pipeline_Control_File.py ) which is run. The Control File is only an example and you can easily create your own. The control file requires that you import the KVN Pipeline and some ParselTongue functions:

import KVNpipeline_v1_0 as KVNpipeline
from AIPSTask import AIPSTask, AIPSList

Due to the way this operates, if you upgrade to a newer version of the Pipeline, simply change the version imported here, as new versions should be backwards compatible.

The minimum required inputs are:

  • AIPS user number
  • Input FITS file
  • Experiment name
  • ANTAB files. The script should automatically handle changing station names. Files should be included within a single array (see example).
  • Directory to output files.

There are many optional parameters which can be specified. These can be useful for speeding up the pipeline when there are problems in the data etc. and the data must not need be re-loaded from scratch again.

Parameter Description
LOAD_DATA=“y” skip loading data
AMP_CAL=“y” skip amplitude calibrations
PLOT=“y” do not create plots
ACCOR=“y” do not perform autocorrelation corrections
BPASS=“y” do not perform bandpass corrections
UVSPLIT=“y” do not split the 'big' file into single frequencies
IFALIGN=“y” do not perform the manual phase calibrations
BIG_FRING=“y” do not perform the big fringe fit
PFT=“y” do not perform the frequency phase transfer
SPLIT=“y” do not split the data into individual sources and export them
EXPORT=“y” do not export the big unsplitted datasets
CLEAN_UP=“y” do not remove AIPS catalogs created by the script
SECOND_IFALIGN=“y” do no perform the second manual phase cal step to remove additional station based delays
FITLD_SOURCES=AIPSList([“ ”,“ ”])) Put sources in here if only want to load a subset of the observed sources
INITIAL_FRING=“n” Run an initial global FRING, this can sometimes help with the IF-Aligner, default is off
NO_PFT_MODE=“n” Run the scipt without performing Phase transfer. Default is off.
FRING_SOLINT=5 Set the solution interval for global FRING runs. Default is 6 mins (scan length for KVN).
FRING_SNR=6 Set the SNR limit for the global FRING. Default 5.
MAN_PCAL=“n” Do standard manual phase cal. If “y”, must also set the timerrange in MAN_PCAL_TIMERANGE. If MAN_PCAL is set, IF-Aligner will not be performed
MAN_PCAL_TIMERANGE=AIPSList([0,0,0,0,0,0,0,0]) Define the time-range used for Manual phase cal. Entered in standard AIPS format.

Operating via SSH

It is easy to run the script via SSH, but you must have ParselTongue and AIPS installed on the remote computer. If they are installed, make sure they are sourced with the following commands:

source /home/aips/LOGIN.SH

For your AIPS installation, and:

export PATH=$PATH:/opt/parseltongue-2.1/bin

For ParselTongue. The above examples will work for the earth shared cluster at KASI. The exact paths may differ on other machines.

Example

An example of how to code the Control File:

It is good practice to set up the inputs using arrays:

FITS = ["/home/test/AIPS/earth/AIPS/FITS/IMOGABA32.FITS"]
NAMES = ["IMO32Y"]
ANTAB = [["/home/test/AIPS/earth/AIPS/FITS/p16sl01bKTN.ANTAB.1",
"/home/test/AIPS/earth/AIPS/FITS/p16sl01bKUS.ANTAB.1",
"/home/test/AIPS/earth/AIPS/FITS/p16sl01bKYS.ANTAB.1"]]
EXPORT = ["/home/test/AIPS/earth/AIPS/FITS/TEST1/IMO32_TESTING1"]

With the KVN Pipeline then being run as:

KVNpipeline.KVN_pipeline(502,FITS[0],NAMES[0],ANTAB[0],EXPORT[0])

This will run the KVN Pipeline with default settings (performing the frequency phase transfer, IF-aligning and exporting data). Various parameters can be turned on and off at the users desire:

KVNpipeline.KVN_pipeline(502,FITS[0],NAMES[0],ANTAB[0],EXPORT[0],REFANT=2,LOAD_DATA="y",AMP_CAL="y",ACCOR="y",BPASS="y",UVSPLIT="y",PLOT="y",CLEAN_UP="y",INITIAL_FRING="n",NO_PFT_MODE="n",FRING_SOLINT=5.5,FITLD_SOURCES=AIPSList(["3C454.3","BLLAC"]),SPLIT="n",EXPORT="n",MAN_PCAL="y",MAN_PCAL_TIMERANGE=AIPSList([0,6,5,0,0,6,11,0]))

In this case, the KVN Pipeline will load data, perform amplitude calibrations, ACCOR and BPASS, split the data into smaller parts, but not perform an initial fringe-fit, only load two sources, and perform a manual phase calibration rather than using the IF-Aligner. It will also not export the data.

The advantage of putting the inputs into an array is that multiple datasets can then be easily run in a for-loop:

INPATH = "/home/test/AIPS/earth/AIPS/FITS"
OUTPATH = "/home/test/AIPS/earth/AIPS/FITS/TEST1"

FITS = [INPATH+"/IMOGABA24.FITS",INPATH+"/IMOGABA26.FITS",INPATH+"/IMOGABA27.FITS",INPATH+"/IMOGABA28.FITS",
        #INPATH+"/IMOGABA29.FITS",INPATH+"/IMOGABA30.FITS",INPATH+"/IMOGABA31.FITS",INPATH+"/IMOGABA32.FITS"]
NAMES = ["IMO24A","IMO26","IMO27","IMO28","IMO29","IMO30","IMO31","IMO32"]
ANTAB = [[INPATH+"/p15sl01c"+"KTN.ANTAB.1",INPATH+"/p15sl01c"+"KUS.ANTAB.1",INPATH+"/p15sl01c"+"KYS.ANTAB.1"],
         [INPATH+"/p15sl01g"+"KTN.ANTAB.1",INPATH+"/p15sl01g"+"KUS.ANTAB.1",INPATH+"/p15sl01g"+"KYS.ANTAB.1"],
         [INPATH+"/p15sl01h"+"KTN.ANTAB.1",INPATH+"/p15sl01h"+"KUS.ANTAB.1",INPATH+"/p15sl01h"+"KYS.ANTAB.1"],
         [INPATH+"/p15sl01i"+"KTN.ANTAB.1",INPATH+"/p15sl01i"+"KUS.ANTAB.1",INPATH+"/p15sl01i"+"KYS.ANTAB.1"],
         [INPATH+"/p15sl01j"+"KTN.ANTAB.1",INPATH+"/p15sl01j"+"KUS.ANTAB.1",INPATH+"/p15sl01j"+"KYS.ANTAB.1"],
         [INPATH+"/p15sl01k"+"KTN.ANTAB.1",INPATH+"/p15sl01k"+"KUS.ANTAB.1",INPATH+"/p15sl01k"+"KYS.ANTAB.1"],
         [INPATH+"/p16sl01a"+"KTN.ANTAB.1",INPATH+"/p16sl01a"+"KUS.ANTAB.1",INPATH+"/p16sl01a"+"KYS.ANTAB.1"],
         [INPATH+"/p16sl01b"+"KTN.ANTAB.1",INPATH+"/p16sl01b"+"KUS.ANTAB.1",INPATH+"/p16sl01b"+"KYS.ANTAB.1"]]
EXPORT = [OUTPATH+"/IMO24A",OUTPATH+"/IMO26",OUTPATH+"/IMO27",OUTPATH+"/IMO28",OUTPATH+"/IMO29",OUTPATH+"/IMO30",OUTPATH+"/IMO31",OUTPATH+"/IMO32"]

for i in range(len(FITS)):
   try:
      KVNpipeline.KVN_pipeline(502,FITS[i],NAMES[i],ANTAB[i],EXPORT[i],REFANT=2,LOAD_DATA="y",AMP_CAL="y",ACCOR="y",BPASS="y",UVSPLIT="y",PLOT="y",CLEAN_UP="y",
				    IFALIGN="y",INITIAL_FRING="n",NO_PFT_MODE="n",FRING_SOLINT=5.5,FITLD_SOURCES=AIPSList(["3C454.3","BLLAC"]),SPLIT="n",EXPORT="n",
				    MAN_PCAL="y",MAN_PCAL_TIMERANGE=AIPSList([0,6,5,0,0,6,11,0]))
   except Exception:
      print "Something went wrong with:",NAMES[i]
      pass

Then running:

ParselTongue KVN_Pipeline_Control_File.py

will execute the pipeline.

Outputs

By default, the pipeline outputs the following files (where <EXP> is the experiment name):

Output Desciption
<EXP>15MM.FITTP The pre “splitted” 15 mm dataset
<EXP>7MM.FITTP The pre “splitted” 7 mm dataset
<EXP>3MM.FITTP The pre “splitted” 3 mm dataset
<EXP>2MM.FITTP The pre “splitted” 2 mm dataset
<EXP>ScanSummary LISTR output
<ANTABFILE>.orig The original pre-editing ANTAB files

In addition, it creates two directories:

Directory Description
plots Output of plots at every stage of the pipeline
SPLIT Location of output source files after SPLIT

In the SPLIT directory, it is further subdivided into separate directories for each frequency.

Feedback and Contact

Feel free to use the pipeline and report any issues back to me. If you have any suggestions for new features, also send me an email.

jhodgson@kasi.re.kr

2016 - Jeff Hodgson

pipeline.txt · Last modified: 2017/09/27 10:58 (external edit)