job_params.cfg
- Job Parameters Configuration¶
The job parameters file provides the detailed, scientifically-motivated settings for each pipeline step. Providing the appropriate configuration here is essential for achieving scientifically valid results.
The default job_params.cfg
file is as follows:
[persistence]
description = "TRAP dataset"
dataset_id = -1
rms_est_sigma = 4 ; Sigma value used for iterative clipping in RMS estimation
rms_est_fraction = 8 ; Determines size of image subsection used for RMS estimation
[quality]
rms_est_history = 100 ; how many images used for calculating rms histogram
rms_est_max = 100 ; global maximum acceptable rms
rms_est_min = 0.0 ; global minimum acceptable rms
rms_rej_sigma = 3 ; threshold for rejecting images using rms histogram
bandwidth_max = 0.0 ; if non zero override bandwidth of image, determines which images fall in same band
oversampled_x = 30 ; threshold for oversampled check
elliptical_x = 2.0 ; threshold for elliptical check
[quality_lofar] ; LOFAR only checks for casa images
low_bound = 1 ; multiplied with noise to define lower threshold
high_bound = 80 ; multiplied with noise to define upper threshold
min_separation = 10 ; minimum distance to a bright source (in degrees)
[source_extraction]
detection_threshold = 8 ; extraction threshold (S/N)
analysis_threshold = 3
back_size_x = 50
back_size_y = 50
margin = 10
deblend_nthresh = 0 ; Number of subthresholds for deblending; 0 disables
extraction_radius_pix = 250
force_beam = True
box_in_beampix = 10
ew_sys_err = 10 ; Systematic errors on ra & decl (units in arcsec)
ns_sys_err = 10
expiration = 10 ; number of forced fits performed after a blind fit
[association]
deruiter_radius = 5.68
beamwidths_limit = 3.0
[transient_search]
new_source_sigma_margin = 1
[pipeline]
mode = 'batch' ; batch or stream
; below are the hosts and ports defined. Needs to be a string, if multiple
; hosts split by ,. Lengths need to match.
hosts = ',,,,,' ; if stream, the stream server
ports = '6666,6667,6668,6669,6670,6671' ; the port of the stream
The file follows the standard ConfigParser syntax.
The parameters in this file are defined as follows:
persistence
Section¶
(See also the Persistence stage.)
dataset_id
- Integer. Specifies the unique ID of a dataset to which the current pipeline
run should be appended. If
-1
, a new dataset is created. If you specify a specific data set ID the configuration of your job is retrieved from the database. This will override your job configuration. description
- String. The name under which the database will be stored in the database.
This value is only used if a new dataset is constructed (see
dataset_id
, below). rms_est_sigma
- Float. Sigma value used for iterative clipping.
rms_est_fraction
- Integer. Determines the size of the subsection used for RMS measurement: the central
1/f
of the image will be used (where f=rms_est_fraction).
quality
Section¶
These are the quality-checking parameters applied to all ingested data in fits format. See also Quality check stage.
rms_est_history
- Integer. How many images are used for calculating rms histogram. Currently only works for streaming mode.
rms_est_max
- Float. The global maximum acceptable rms.
rms_est_min
- Float. The global minimum acceptable rms.
rms_rej_sigma
- Float. Sigma threshold used for rejecting outlier images when using the rms histogram
bandwidth_max
- Float. Limit the maximum bandwidth used when determining if two images belong to the same ‘band’ grouping.
oversampled_x
- Integer. The maximum length of a beam axis.
elliptical_x
- Float. The maximum ratio of major to minor axis length.
quality_lofar
Section¶
These are the quality-checking parameters applied if the ingested data is from LOFAR. See also Quality check stage.
low_bound
- Float. Reject the image if the measured RMS is less than
low_bound
times the theoretical noise. high_bound
- Float. Reject the image if the measured RMS is greater than
high_bound
times the theoretical noise. min_separation
- The minimum allowed distance from the image centre to a bright radio source in degrees.
source_extraction
Section¶
Parameters used in source extraction. See also “Blind” source extraction stage and Forced source-fitting stage.
detection_threshold
- Float. The detection threshold as a multiple of the RMS noise.
analysis_threshold
- Float. The analysis threshold as a multiple of the RMS noise.
back_size_x
,back_size_y
- Integers. The size of the background grid parallel to the X and Y axes of the pixel grid.
margin
- Integer. Pixel data within
margin
pixels of the edge of the image will be excluded from the analysis. extraction_radius_pix
- Integer. Pixel data more than
extraction_radius_pix
pixels from the centre of the image will be excluded from the analysis. deblend_nthresh
- Integer. The number of subthresholds to use for deblending. Set to
0
to disable deblending. force_beam
- Boolean. If
True
, all detected sources are assumed to have the size and shape of the restoring beam (ie, to be unresolved point sources), and these parameters are held constant during fitting. IfFalse
, all parameters are allowed to vary freely. box_in_beampix
- The size of the masking aperture which determines which pixels are used
for forced fitting, as a multiple of the beam major axis length.
See
tkp.sourcefinder.image.ImageData.fit_to_point()
for details. ew_sys_err
,ns_sys_err
- Floats. Systematic errors in units of arcseconds which augment the sourcefinder-measured errors on source positions when performing source association. These variables refer to an absolute angular error along an east-west and north-south axis respectively. (NB Although these values are stored during the source-extraction process, they affect the source-association process.)
expiration
- The number of forced fits performed since the last blind fit. Used to ‘expire’ the runningcatalog - else said to stop monitoring a source of which the flux went below the detection threshold after a configurable amount of timesteps.
association
Section¶
Parameters used in source-association. See Source association stage for details.
NB the ew_sys_err
, ns_sys_err
parameters detailed above also affect
source-association.
deruiter_radius
- Float. Maximum DeRuiter radius for two sources to be considered candidates for association.
beamwidths_limit
- Float. Maximum separation for two sources to be considered candidates for association, as a multiple of the restoring-beam semimajor-axis length. Default is 1.0, which was the fixed default prior to TraP release 2.1. It may be necessary to use a larger number if your data has large systematic position errors, i.e. if the sources ‘jitter’ between images, but note that using a large value can cause slowdown of database operations.
transient_search
Section¶
Parameters used in transient-detection. See also the Variability and new-source detection stage.
new_source_sigma_margin
- Float. A newly detected source is considered transient if it is significantly above the best (lowest) previous detection limit for that point on-sky. ‘Significantly above’ is defined by a ‘margin of error,’ intended to screen out steady sources that just happen to be fluctuating around the detection threshold due to measurement noise. This value sets that margin as a multiple of the RMS of the previous-best image.
pipeline
Section¶
Parameters used to detemine if the pipeline is run in batch or streaming mode.
mode
- String. batch is for a set of images to be processed offline. stream is for streaming mode where new images are automatically fed into the pipeline
hosts
,ports
- String. These are the hosts and ports required for streaming mode. If multiple hosts are defined they need splitting by , and the length of the number of ports needs to match.