tkp.steps – Define logic for each pipeline stage

tkp.steps.forced_fitting

tkp.steps.forced_fitting.perform_forced_fits(fit_posns, fit_ids, image_path, extraction_params)

Perform forced source measurements on an image based on a list of positions.

Parameters:

• fit_posns (list) – List of (RA, Dec) tuples: Positions to be fit.
• fit_ids – List of identifiers for each requested fit position.
• image_path (str) – path to image for measurements.
• extraction_params (dict) – source extraction parameters, as a dictionary.

Returns:

A matched pair of lists (serialized_fits, ids), corresponding to successfully fitted positions. NB returned lists may be shorter than input lists if some fits are unsuccessful.

Return type:

tuple

tkp.steps.misc

Various subroutines used in the main pipeline flow.

We keep them separately to make the pipeline logic easier to read at a glance.

tkp.steps.misc.check_job_configs_match(job_config_1, job_config_2)

Check if job configs match, except dataset_id which we expect to change.

tkp.steps.misc.group_per_timestep(images)

groups a list of TRAP images per time step.

Per time step the images are order per frequency and then per stokes. The eventual order is:

(t1, f1, s1), (t1, f1, s2), (t1, f2, s1), (t1, f2, s2), (t2, f1, s1), …) where:

• t is time sorted by old to new
• f is frequency sorted from low to high
• s is stokes, sorted by ID as defined in the database schema

Parameters:images (list) – Images to group. Returns:

List of tuples. The list is sorted by timestamp.

Each tuple has the timestamp as a first element, and a list of images sorted by frequency and then stokes as the second element.

Return type:list

tkp.steps.misc.load_job_config(pipe_config)

Locates the job_params.cfg in ‘job_directory’ and loads via ConfigParser.

tkp.steps.misc.setup_log_file(log_dir, debug=False, basename='trap.log')

sets up a catch all logging handler which writes to log_file.

Parameters:

• log_file – log file to write
• debug – do we want debug level logging?
• basename – basename of the log file

tkp.steps.persistence

This step is used for the storing of images and metadata to the database and image cache (mongodb).

tkp.steps.persistence.create_dataset(dataset_id, description)

Creates a dataset if it doesn’t exists Note: Should only be used in a master recipe :returns: the database ID of this dataset

tkp.steps.persistence.extract_metadatas(images, rms_est_sigma, rms_est_fraction)

Extracts metadata and rms_qc values from the list of images.

Parameters:

• images – list of image urls
• rms_est_sigma – used for RMS calculation, see tkp.quality.statistics
• rms_est_fraction – used for RMS calculation, see tkp.quality.statistics

Returns:

a list of metadata’s. The metadata will be False if extraction failed.

tkp.steps.persistence.image_to_mongodb(filename, hostname, port, db)

Copy a file into mongodb

tkp.steps.persistence.node_steps(images, image_cache_config, rms_est_sigma, rms_est_fraction)

this function executes all persistence steps that should be executed on a node. Note: Should only be used in a node recipe

tkp.steps.persistence.store_images(images_metadata, extraction_radius_pix, dataset_id)

Add images to database. Note that all images in one dataset should be inserted in one go, since the order is very important here. If you don’t add them all in once, you should make sure they are added in the correct order e.g. sorted by observation time.

Note: Should only be used in a master recipe

Parameters:

• images_metadata – list of dicts containing image metadata
• extraction_radius_pix – (float) Used to calculate the ‘skyregion’
• dataset_id – dataset id to be used. don’t use value from parset file since this can be -1 (TraP way of setting auto increment)

Returns:

the database ID of this dataset

tkp.steps.prettyprint

Pretty print the classified transients

tkp.steps.quality

All generic quality checking routines.

tkp.steps.quality.reject_check(image_path, job_config)

Check if an image passes the quality checks.

If not, a rejection reason is returned.

Parameters:

• id – database ID of image. This is not used but kept as a reference for distributed computation!
• image_path – path to image
• parset_file – parset file location with quality check parameters

Returns:

(rejection ID, description) if rejected, else None

tkp.steps.quality.reject_image(image_id, reason, comment)

Adds a rejection for an image to the database

tkp.steps.source_extraction

class tkp.steps.source_extraction.ExtractionResults(sources, rms_min, rms_max)

Create new instance of ExtractionResults(sources, rms_min, rms_max)

rms_max

Alias for field number 2

rms_min

Alias for field number 1

sources

Alias for field number 0

tkp.steps.source_extraction.extract_sources(image_path, extraction_params)

Extract sources from an image.

Parameters:

• image_path – path to file from which to extract sources.
• extraction_params – dictionary containing at least the detection and analysis threshold and the association radius, the last one a multiplication factor of the de Ruiter radius.

Returns:

list of ExtractionResults named tuples containing source measurements, min RMS value and max RMS value