job_params.cfg - Job Parameters Configuration

The job parameters file provides the detailed, scientifically-motivated settings for each pipeline step. Providing the appropriate configuration here is essential for achieving scientifically valid results.

The default job_params.cfg file is as follows:

[persistence]
description = "TRAP dataset"
dataset_id = -1
rms_est_sigma = 4            ; Sigma value used for iterative clipping in RMS estimation
rms_est_fraction = 8         ; Determines size of image subsection used for RMS estimation

[quality]
rms_est_history = 100        ; how many images used for calculating rms histogram
rms_est_max = 100            ; global maximum acceptable rms
rms_est_min = 0.0            ; global minimum acceptable rms
rms_rej_sigma = 3	     ; threshold for rejecting images using rms histogram
bandwidth_max = 0.0          ; if non zero override bandwidth of image, determines which images fall in same band
oversampled_x = 30           ; threshold for oversampled check
elliptical_x = 2.0           ; threshold for elliptical check


[quality_lofar]		     ; LOFAR only checks for casa images
low_bound = 1                ; multiplied with noise to define lower threshold
high_bound = 80              ; multiplied with noise to define upper threshold
min_separation = 10          ; minimum distance to a bright source (in degrees)

[source_extraction]
detection_threshold = 8      ; extraction threshold (S/N)
analysis_threshold = 3
back_size_x = 50
back_size_y = 50
margin = 10
deblend_nthresh = 0          ; Number of subthresholds for deblending; 0 disables
extraction_radius_pix = 250
force_beam = True
box_in_beampix = 10
ew_sys_err = 10              ; Systematic errors on ra & decl (units in arcsec)
ns_sys_err = 10
expiration = 10              ; number of forced fits performed after a blind fit

[association]
deruiter_radius = 5.68
beamwidths_limit =  3.0

[transient_search]
new_source_sigma_margin = 1

[pipeline]
mode = 'batch'                            ; batch or stream
; below are the hosts and ports defined. Needs to be a string, if multiple
; hosts split by ,. Lengths need to match.
hosts = ',,,,,'                           ; if stream, the stream server
ports = '6666,6667,6668,6669,6670,6671'   ; the port of the stream

The file follows the standard ConfigParser syntax.

The parameters in this file are defined as follows:

persistence Section

(See also the Persistence stage.)

dataset_id

Integer. Specifies the unique ID of a dataset to which the current pipeline run should be appended. If -1, a new dataset is created. If you specify a specific data set ID the configuration of your job is retrieved from the database. This will override your job configuration.

description

String. The name under which the database will be stored in the database. This value is only used if a new dataset is constructed (see dataset_id, below).

rms_est_sigma

Float. Sigma value used for iterative clipping.

rms_est_fraction

Integer. Determines the size of the subsection used for RMS measurement: the central 1/f of the image will be used (where f=rms_est_fraction).

quality Section

These are the quality-checking parameters applied to all ingested data in fits format. See also Quality check stage.

rms_est_history

Integer. How many images are used for calculating rms histogram. Currently only works for streaming mode.

rms_est_max

Float. The global maximum acceptable rms.

rms_est_min

Float. The global minimum acceptable rms.

rms_rej_sigma

Float. Sigma threshold used for rejecting outlier images when using the rms histogram

bandwidth_max

Float. Limit the maximum bandwidth used when determining if two images belong to the same ‘band’ grouping.

oversampled_x

Integer. The maximum length of a beam axis.

elliptical_x

Float. The maximum ratio of major to minor axis length.

quality_lofar Section

These are the quality-checking parameters applied if the ingested data is from LOFAR. See also Quality check stage.

low_bound

Float. Reject the image if the measured RMS is less than low_bound times the theoretical noise.

high_bound

Float. Reject the image if the measured RMS is greater than high_bound times the theoretical noise.

min_separation

The minimum allowed distance from the image centre to a bright radio source in degrees.

source_extraction Section

Parameters used in source extraction. See also “Blind” source extraction stage and Forced source-fitting stage.

detection_threshold

Float. The detection threshold as a multiple of the RMS noise.

analysis_threshold

Float. The analysis threshold as a multiple of the RMS noise.

back_size_x, back_size_y

Integers. The size of the background grid parallel to the X and Y axes of the pixel grid.

margin

Integer. Pixel data within margin pixels of the edge of the image will be excluded from the analysis.

extraction_radius_pix

Integer. Pixel data more than extraction_radius_pix pixels from the centre of the image will be excluded from the analysis.

deblend_nthresh

Integer. The number of subthresholds to use for deblending. Set to 0 to disable deblending.

force_beam

Boolean. If True, all detected sources are assumed to have the size and shape of the restoring beam (ie, to be unresolved point sources), and these parameters are held constant during fitting. If False, all parameters are allowed to vary freely.

box_in_beampix

The size of the masking aperture which determines which pixels are used for forced fitting, as a multiple of the beam major axis length. See tkp.sourcefinder.image.ImageData.fit_to_point() for details.

ew_sys_err, ns_sys_err

Floats. Systematic errors in units of arcseconds which augment the sourcefinder-measured errors on source positions when performing source association. These variables refer to an absolute angular error along an east-west and north-south axis respectively. (NB Although these values are stored during the source-extraction process, they affect the source-association process.)

expiration

The number of forced fits performed since the last blind fit. Used to ‘expire’ the runningcatalog - else said to stop monitoring a source of which the flux went below the detection threshold after a configurable amount of timesteps.

association Section

Parameters used in source-association. See Source association stage for details. NB the ew_sys_err, ns_sys_err parameters detailed above also affect source-association.

deruiter_radius

Float. Maximum DeRuiter radius for two sources to be considered candidates for association.

beamwidths_limit

Float. Maximum separation for two sources to be considered candidates for association, as a multiple of the restoring-beam semimajor-axis length. Default is 1.0, which was the fixed default prior to TraP release 2.1. It may be necessary to use a larger number if your data has large systematic position errors, i.e. if the sources ‘jitter’ between images, but note that using a large value can cause slowdown of database operations.

transient_search Section

Parameters used in transient-detection. See also the Variability and new-source detection stage.

new_source_sigma_margin

Float. A newly detected source is considered transient if it is significantly above the best (lowest) previous detection limit for that point on-sky. ‘Significantly above’ is defined by a ‘margin of error,’ intended to screen out steady sources that just happen to be fluctuating around the detection threshold due to measurement noise. This value sets that margin as a multiple of the RMS of the previous-best image.

pipeline Section

Parameters used to detemine if the pipeline is run in batch or streaming mode.

mode

String. batch is for a set of images to be processed offline. stream is for streaming mode where new images are automatically fed into the pipeline

hosts, ports

String. These are the hosts and ports required for streaming mode. If multiple hosts are defined they need splitting by , and the length of the number of ports needs to match.