tkp.main
– Top-level pipeline logic flow¶
tkp.main
¶
The main pipeline logic, from where all other components are called.
-
tkp.main.
extract_metadata
(job_config, accessors, runner)[source]¶ Parameters: - job_config – a TKP config object
- accessors (tuple) – list of tkp.Accessor objects
- runner (tkp.distribute.Runner) – the runner to use
Returns: a list of metadata dicts
Return type:
-
tkp.main.
get_metadata_for_sorting
(runner, image_paths)[source]¶ Group images per timestamp. Will open all images in parallel using runner.
Parameters: - runner (tkp.distribute.Runner) – Runner to use for distribution
- image_paths (tuple) – list of image paths
Returns: list of tuples, (timestamp, [list_of_images])
Return type:
-
tkp.main.
get_runner
(pipe_config)[source]¶ get parallelise props. Defaults to multiproc with autodetect num cores. Wil initialise the distributor.
One should not mix threads and multiprocessing, but for example AstroPy uses threads internally. Best practice then is to first do multiprocessing, and then threading per process. This is the reason why this function should be called as one of the first functions the in the TraP pipeline lifespan.
-
tkp.main.
initialise_dataset
(job_config, supplied_mon_coords)[source]¶ sets up a dataset in the database.
if the dataset already exists it will return the job_config from the previous dataset run.
Parameters: - job_config – a job configuration object
- supplied_mon_coords (tuple) – a list of monitoring positions
Returns: job_config and dataset ID
Return type:
-
tkp.main.
load_images
(job_name, job_dir)[source]¶ Load all the images for a specific TraP job.
Returns: a list of paths Return type: tuple
-
tkp.main.
quality_check
(db_images, accessors, job_config, runner)[source]¶ Returns: a list of db_image and accessor tuples Return type: tuple
-
tkp.main.
run
(job_name, supplied_mon_coords=None)[source]¶ TKP pipeline main loop entry point.
Parameters:
-
tkp.main.
run_batch
(image_paths, job_config, runner, dataset_id, copy_images)[source]¶ Run the pipeline in batch mode.
Parameters: - job_name (str) – job name, used for locating images script
- pipe_config – the pipeline configuration object
- job_config – a job configuration object
- runner (tkp.distribute.Runner) – Runner to use for distribution
- dataset_id (int) – The dataset ID to use
-
tkp.main.
run_stream
(runner, job_config, dataset_id, copy_images)[source]¶ Run the pipeline in stream mode.
Daemon function, doesn’t return.
Parameters: - runner (tkp.distribute.Runner) – Runner to use for distribution
- job_config – a job configuration object
- dataset_id (int) – The dataset ID to use
-
tkp.main.
timestamp_step
(runner, images, job_config, dataset_id, copy_images)[source]¶ Called from the main loop with all images in a certain timestep
Parameters: - runner (tkp.distribute.Runner) – Runner to use for distribution
- images (tuple) – list of things tkp.accessors can handle, like image paths or fits objects
- job_config – a tkp job config object
- dataset_id (int) – The
tkp.db.model.Dataset
id
Returns: of tuples (rms_qc, band)
Return type: