ASDM import, weights, and MS concatenation

ASDM import, weights, and MS concatenation
Dirk Petry (ESO), November 2014
Outline:
→ special features of importasdm
→ The MS WEIGHT and WEIGHT_SPECTRUM columns
→ concat and virtualconcat
D. Petry, importasdm, WEIGHT, concat, November 2014
1
The ASDM to MS conversion
CASA works on MSs. Conversion of ASDMs to MSs done via the CASA task
importasdm
asdm
vis
singledish
ocorr_mode
= '' # Name of input asdm directory (on disk)
= '' # Root name of the ms to be created. Note the .ms is NOT added
= False # Set true to output single-dish data format
= 'ca' # output data for correlation mode AUTO_ONLY (ao)
# or CROSS_ONLY (co) or CROSS_AND_AUTO (ca)
wvr_corrected_data = 'no' # Specifies which values are considerd in the SDM binary data
# to fill the DATA.
#
no, for uncorrected data (default)
#
yes, for the corrected data
#
both, (two MSs are created)
scans
= '' # processes only the specified scans.
# This value is a semicolon separated list of scan specifications.
# exec bock index : followed by a comma separated list of scan indices
#
(0-based)
(1-based)
# typical example "0:1~23"
D. Petry, importasdm, WEIGHT, concat, November 2014
2
The ASDM to MS conversion
importasdm
asis
=
'' # Creates verbatim copies of the ASDMtables in the output MS
# Value given must be a string of table names separated by spaces
# A * wildcard is allowed.
# Typical example asis='Antenna Station CalWVR'
process_flags = True # Create online flags in the FLAG_CMD sub-table.
tbuff
= 0.0
# Time padding buffer (seconds)
applyflags = False # Apply the flags to the MS.
flagbackup
verbose
overwrite
showversion
bdfflags
= True
= False
= False
= False
= False
#
#
#
#
#
Back up flag column before applying flags.
Output lots of information while the filler is working
Over write an existing MS
Report the version of asdm2MS being used
Set the FLAGs according to the ASDM _binary_ flags
D. Petry, importasdm, WEIGHT, concat, November 2014
3
The ASDM to MS conversion
importasdm
- in principle a waste of time and space:
we are rewriting exactly the same information in a different format
- LOFAR had the same problem and came up with a solution:
read the native data format directly!
- Create an "MS view" on the ASDM bulk data
introducing the new parameter
lazy
= False # Make the MS DATA column read the ASDM Binary data directly
Fully available only starting with CASA 4.3.
One of the contributions of ESO to CASA.
D. Petry, importasdm, WEIGHT, concat, November 4444
4
The ASDM to MS conversion - "lazy filler"
Normal translation from ASDM to MS creates a complete copy of the visibilities
ASDM
Subtables
MainTable
MS
Subtables
normal MainTable
without DATA
BLOB0
BLOB1
DATA
BLOB2
...
BLOBn
normal MS
D. Petry, importasdm, WEIGHT, concat, November 2014
5
The ASDM to MS conversion - "lazy filler"
To permit on-the-fly conversion from ASDM visibility format to MS visibilty format,
introduce new ASDM Table Storage Manager (asdmstman)
MS
ASDM
Subtables
MainTable
traditional conversion during import
traditional conversion during import
BLOB0
BLOB1
BLOB2
...
BLOBn
Subtables
normal MainTable
without DATA
DATA
with
asdmstman
asdmstman
Lookup
Table
MS with "lazily filled" DATA column
D. Petry, importasdm, WEIGHT, concat, November 2014
6
The ASDM to MS conversion - "lazy filler"
To permit on-the-fly conversion from ASDM visibility format to MS visibilty format,
introduce new ASDM Table Storage Manager (asdmstman)
MS
ASDM
Subtables
MainTable
traditional conversion during import
traditional conversion during import
BLOB0
BLOB1
BLOB2
...
BLOBn
Subtables
normal MainTable
without DATA
DATA
with
asdmstman
asdmstman
Lookup
Table
< 5 % of the size of normal MS
D. Petry, importasdm, WEIGHT, concat, November 2014
7
The ASDM to MS conversion - "lazy filler"
To permit on-the-fly conversion from ASDM visibility format to MS visibilty format,
introduce new ASDM Table Storage Manager (asdmstman)
MS
ASDM
Subtables
MainTable
traditional conversion during import
traditional conversion during import
BLOB0
BLOB1
BLOB2
...
BLOBn
D. Petry, importasdm, WEIGHT, concat, November 2014
Subtables
normal MainTable
without DATA
normal
DATA
CORRECTED
with
DATA
asdmstman
applycal creates
asdmstman
normal corrected
Lookup
data column
Table
8
The ASDM to MS conversion - "lazy filler"
To permit on-the-fly conversion from ASDM visibility format to MS visibilty format,
introduce new ASDM Table Storage Manager (asdmstman)
MS
ASDM
Subtables
MainTable
traditional conversion during import
traditional conversion during import
BLOB0
Subtables
normal MainTable
without DATA
normal
DATA
CORRECTED
with
DATA
asdmstman
BLOB1
BLOB2
applycal creates
asdmstman
normal corrected
Lookup
data column
Table
...
BLOBn
50 % of the size of normal MS with CORR_DATA
D. Petry, importasdm, WEIGHT, concat, November 2014
9
The ASDM to MS conversion - "lazy filler"
Unexpected positive side effect: the read access to a lazily filled MS is faster!
In particular applycal becomes faster!
Bottom line: with CASA 4.3, you can use the lazy=True in importasdm
so save time (>10%) and disk space (>30%).
Only drawback: must keep the original ASDM available (MS references it)
Use the tool method
ms.asdmref()
- to check the path to the ASDM referenced by the MS
If you move the ASDM, you need to update the reference:
ms.open("myms.ms")
ms.asdmref("/new/path/to/the/ASDM")
ms.close()
D. Petry, importasdm, WEIGHT, concat, November 2014
10
Radio Data WEIGHTs
single-dish
Amplitudes
interferometer
Visibilities (Amp,Phase)
Antenna A12, Timestamp 2:49:45 h
Spectral Channel
1
2
3 ...
Amplitudes 0
Polarization
X
0.1 2.3 2.1 1.9
Y
1.2 3.3 8.9 6.5
.
...
Baseline (A3,A12), Timestamp 2:49:45 h
Spectral Channel
0
1
2
3 ...
Visibilities
Correlation
XX
(0.2,2.1) (3.2,2.0) (3.0,1.9) (2.8,1.8) ...
YY
(1.3,1.9) (2.8,2.0) (7.6,1.8) (5.5,1.9) ...
.
...
Each of these physical quantities has a statistical Error.
The average RMS over all channels is stored in SIGMA.
2
The imaging weight = 1/SIGMA is stored in the column WEIGHT.
Optionally, there is a spectrally resolved weight: WEIGHT_SPECTRUM
D. Petry, importasdm, WEIGHT, concat, November 2014
11
WEIGHTs: the full story
The columns WEIGHT and SIGMA would be redundant if the relation
2
WEIGHT=1/SIGMA would always hold. In fact, their usage in more involved:
SIGMA is the per-channel noise of DATA:
SIGMA = 1/sqrt(N ΔfΔt)
(where N = 1 for auto-correlations, 2 for cross-correlations,
Δf = channel width, Δt = integration time)
WEIGHT is the imaging weight
calculated from the per-channel noise of CORRECTED_DATA
Only if CORRECTED_DATA does not exist (i.e. before applycal), the relation
2
WEIGHT=1/SIGMA
holds. (WEIGHT is initialised from SIGMA in this way.)
During applycal, WEIGHT is modified by applying calibration factors:
WEIGHT_new = w w WEIGHT_old
i
j
D. Petry, importasdm, WEIGHT, concat, November 2014
12
WEIGHT modification in CASA 4.2.2
importasdm initialises SIGMA to 1/sqrt(N ΔfΔt) (N= 1 or 2, see above)
2
WEIGHT to 1/SIGMA = N ΔfΔt
applycal of a Tsys table modifies weight for baseline (i,j) to
WEIGHT_new(i,j) = 1/(<Tsys_i> <Tsys_j>)WEIGHT_old(i,j)
(< > : SPW average)
applycal of a Gain table modifies weight for baseline (i,j) to
WEIGHT_new(i,j) = abs(G_iG_j) WEIGHT_old(i,j)
applycal of a Bandpass table modifies weight for baseline (i,j) to
WEIGHT_new(i,j) = <abs(B_iB_j)> WEIGHT_old(i,j)
D. Petry, importasdm, WEIGHT, concat, November 2014
13
WEIGHT modification in CASA 4.2.2
importasdm initialises SIGMA to 1/sqrt(N ΔfΔt) (N= 1 or 2, see above)
2
WEIGHT to 1/SIGMA = N ΔfΔt
applycal of a Tsys table modifies weight for baseline (i,j) to
WEIGHT_new(i,j) = 1/(<Tsys_i> <Tsys_j>)WEIGHT_old(i,j)
(< > : SPW average)
applycal of a Gain table modifies weight for baseline (i,j) to
WEIGHT_new(i,j) = abs(G_iG_j) WEIGHT_old(i,j)
applycal of a Bandpass table modifies weight for baseline (i,j) to
WEIGHT_new(i,j) = <abs(B_iB_j)> WEIGHT_old(i,j)
split with timebin>1 and/or width>1 updates WEIGHT and SIGMA:
WEIGHT_new = WEIGHT_old * N
SIGMA_new = WEIGHT_old / sqrt(N)
cvel, when output channel width (in old ref frame) changes, updates WEIGHT and SIGMA:
WEIGHT_new = WEIGHT_old * q
SIGMA_new = SIGMA_old / sqrt(q)
where q = chanwidth_after / chanwith_before
D. Petry, importasdm, WEIGHT, concat, November 2 0 1 4
14
WEIGHT modification in CASA 4.2.2
If not happy with the weighting scheme or when importing data with corrupt weights,
can use task
statwt
to initialise weights.
statwt initialises SIGMA to StdDev of DATA (time-dependent)
2
WEIGHT to 1/SIGMA
instead of standard sample std dev can also use RMS
(parameter dorms, recommended when no outliers are present)
Should be applied after calibration!
D. Petry, importasdm, WEIGHT, concat, November 2014
15
WEIGHT_SPECTRUM in CASA 4.3
spectrally resolved weights
importasdm initialises SIGMA to 1/sqrt(N ΔfΔt) (N= 1 or 2, see above)
2
WEIGHT to 1/SIGMA = N ΔfΔt
WEIGHT_SPECTRUM to N ΔfΔt (for each channel)
applycal of a Tsys table modifies weight for baseline (i,j) to
WEIGHT_new(i,j) = 1/(<Tsys_i> <Tsys_j>)WEIGHT_old(i,j) (< > : SPW average)
WEIGHT_SPECTRUM_new(i,j) = 1/(Tsys_i Tsys_j)WEIGHT_SPECTRUM_old(i,j)
applycal of a Gain table modifies weight for baseline (i,j) to
WEIGHT_new(i,j) = abs(G_iG_j) WEIGHT_old(i,j)
WEIGHT_SPECTRUM_new(i,j) = abs(G_iG_j) WEIGHT_SPECTRUM_old(i,j)
applycal of a Bandpass table modifies weight for baseline (i,j) to
WEIGHT_new(i,j) = <abs(B_iB_j)> WEIGHT_old(i,j)
WEIGHT_SPECTRUM_new(i,j) = abs(B_iB_j) WEIGHT_SPECTRUM_old(i,j)
One float number per channel: will increase data volume of the MS considerably ...
D. Petry, importasdm, WEIGHT, concat, November 2014
16
MS concatenation
Problem of merging a number MSs can be reduced to
merging 2 MSs: MS0 and MS1
by subsequently merging each MS to the result of the previous mergers.
Merging is called "concatenation" in CASA: task concat
The MS is a relational database: a concatenation is a database merger
Not enough to append Main Table 1 to Main Table 0
All subtables and subsubtables have to be properly merged
(uniqueness conditions fulfilled)
and references to them properly reindexed.
If order of antennas in ANTENNA table changes, baseline order
may be reversed and visibilities my have to be conjugated.
D. Petry, importasdm, WEIGHT, concat, November 2014
17
MS concatenation
task concat
concat(vis=[''],
- list of the MSs to concatenate (concat will sort them in time)
concatvis='', - name of the output MS (can be an existing MS to which to append)
freqtol='',
- frequency tolerance when judging if two SPWs are the same
dirtol='',
- direction (angular) tolerance when judging if two FIELDs are the same
respectname=False, - regard two fields as different when their names are different?
If not, name will be taken from first field (in time) encountered
with the given direction.
timesort=False,
- sort the whole MS Main table in time?
copypointing=True,
- keep the POINTING table? (not keeping it saves space)
visweightscale=[]
- scale the weights of each MS in the output
(see talk on TP/ACA/12m combination later today)
)
D. Petry, importasdm, WEIGHT, concat, November 2014
18
MS concatenation
task virtualconcat
- same as concat but output is MulitMS: MS made up of several normal MSs
- not much tested (yet)
- much faster: moves data instead of copying it
- one additional parameter: "keepcopy" - if True, a copy of the original MSs is kept
(cancels some of the advantages)
- MultiMS may be the future of processing large datasets on HPC clusters
- break the monolithic main table
into several pieces of different regions in time and/or frequency
D. Petry, importasdm, WEIGHT, concat, November 2014
19

Download Report