tkp.steps – Define logic for each pipeline stage


tkp.steps.forced_fitting.perform_forced_fits(fit_posns, fit_ids, image_path, extraction_params)[source]

Perform forced source measurements on an image based on a list of positions.

  • fit_posns (list) – List of (RA, Dec) tuples: Positions to be fit.
  • fit_ids – List of identifiers for each requested fit position.
  • image_path (str) – path to image for measurements.
  • extraction_params (dict) – source extraction parameters, as a dictionary.

A matched pair of lists (serialized_fits, ids), corresponding to successfully fitted positions. NB returned lists may be shorter than input lists if some fits are unsuccessful.

Return type:



Various subroutines used in the main pipeline flow.

We keep them separately to make the pipeline logic easier to read at a glance.

tkp.steps.misc.check_job_configs_match(job_config_1, job_config_2)[source]

Check if job configs match, except dataset_id which we expect to change.


groups a list of TRAP images per time step.

Per time step the images are order per frequency and then per stokes. The eventual order is:

(t1, f1, s1), (t1, f1, s2), (t1, f2, s1), (t1, f2, s2), (t2, f1, s1), ...) where:

  • t is time sorted by old to new
  • f is frequency sorted from low to high
  • s is stokes, sorted by ID as defined in the database schema
Parameters:images (list) – Images to group.
Returns:List of tuples. The list is sorted by timestamp. Each tuple has the timestamp as a first element, and a list of images sorted by frequency and then stokes as the second element.
Return type:list

Locates the job_params.cfg in ‘job_directory’ and loads via ConfigParser.

tkp.steps.misc.setup_log_file(log_dir, debug=False, basename='trap.log')[source]

sets up a catch all logging handler which writes to log_file.

  • log_file – log file to write
  • debug – do we want debug level logging?
  • basename – basename of the log file


This step is used for the storing of images and metadata to the database and image cache (mongodb).

tkp.steps.persistence.create_dataset(dataset_id, description)[source]

Creates a dataset if it doesn’t exists Note: Should only be used in a master recipe :returns: the database ID of this dataset

tkp.steps.persistence.extract_metadatas(images, rms_est_sigma, rms_est_fraction)[source]

Extracts metadata and rms_qc values from the list of images.

  • images – list of image urls
  • rms_est_sigma – used for RMS calculation, see tkp.quality.statistics
  • rms_est_fraction – used for RMS calculation, see tkp.quality.statistics

a list of metadata’s. The metadata will be False if extraction failed.

tkp.steps.persistence.image_to_mongodb(filename, hostname, port, db)[source]

Copy a file into mongodb

tkp.steps.persistence.node_steps(images, image_cache_config, rms_est_sigma, rms_est_fraction)[source]

this function executes all persistence steps that should be executed on a node. Note: Should only be used in a node recipe

tkp.steps.persistence.store_images(images_metadata, extraction_radius_pix, dataset_id)[source]

Add images to database. Note that all images in one dataset should be inserted in one go, since the order is very important here. If you don’t add them all in once, you should make sure they are added in the correct order e.g. sorted by observation time.

Note: Should only be used in a master recipe

  • images_metadata – list of dicts containing image metadata
  • extraction_radius_pix – (float) Used to calculate the ‘skyregion’
  • dataset_id – dataset id to be used. don’t use value from parset file since this can be -1 (TraP way of setting auto increment)

the database ID of this dataset


Pretty print the classified transients


All generic quality checking routines.

tkp.steps.quality.reject_check(image_path, job_config)[source]
checks if an image passes the quality check. If not, a rejection
tuple is returned.

NOTE: should only be used on a NODE

  • id – database ID of image. This is not used but kept as a reference for distributed computation!
  • image_path – path to image
  • parset_file – parset file location with quality check parameters

(rejection ID, description) if rejected, else None

tkp.steps.quality.reject_image(image_id, reason, comment)[source]

Adds a rejection for an image to the database

NOTE: should only be used on a MASTER node


class tkp.steps.source_extraction.ExtractionResults(sources, rms_min, rms_max)

Return self as a plain tuple. Used by copy and pickle.


Exclude the OrderedDict from pickling


Return a nicely formatted representation string


Alias for field number 2


Alias for field number 1


Alias for field number 0

tkp.steps.source_extraction.extract_sources(image_path, extraction_params)[source]

Extract sources from an image.

  • image_path – path to file from which to extract sources.
  • extraction_params – dictionary containing at least the detection and analysis threshold and the association radius, the last one a multiplication factor of the de Ruiter radius.

list of ExtractionResults named tuples containing source measurements, min RMS value and max RMS value