Persistence stage

(See also the relevant configuration parameters.)

A record of all images to be processed is made in the database. Within the database, images are sorted into datasets, which group related images together for processing: searches for transients are performed between images in a single databset, for instance. All images being processed are added to the same dataset.

Optionally, a copy of the image pixel data may be stored to a MongoDB instance at the same time. This is configured in the image_cache section of the pipeline config.

Note that only images which meet the data accessor requirements are stored in the database. Any other data provided to the pipeline will be processed: an error will be logged, and that data will not be included in further processing.