tkp.config – Pipeline Configuration


The tkp.config folder contains the job configuration file templates, and code for handling config files.


tkp.config.get_database_config(pipeline_config=None, apply=None)[source]

Determine database config and (optionally) use to set up the Database.

Determines a database configuration using the settings defined in a dict (if supplied) and possibly overridden by environment variables. The config resulting from the combination of defaults, supplied dict, and environment variables is returned as a dict. If apply==True, the Database singleton is configured using these resulting settings.

The following environment variables are recognized, and take priority:

  • pipeline_config – Dict of db settings. Relevant keys: (engine, database, user, password, host, port, passphrase )
  • apply – apply settings (configure db connection) or not

containing the resulting combined settings

(resulting from defaults, config_passed and possibly environment variables.)

Return type:


tkp.config.initialize_pipeline_config(pipe_cfg_file, job_name=False)[source]

Initializes the default variables and loads the ConfigParser file.

Sets defaults for start_time, job_name and cwd; these can then be used via variable substitution in other config values.

tkp.config.parse – Magic ConfigParser to dict functionality

Utilities for loading parameters from config files, with automatic type conversion.


Loads and returns timestamp with microsecond precission


Loads the ConfigParser object as a nested dictionary.

Automatically converts strings representing ints and floats to their respective types, through the magic of ast.literal_eval. This functionality is extensible via the loads_methods list. Each loads (load string) method is tested in turn to see if it throws an exception. If all throw, we assume the value is meant to be string.

Any values that you don’t want to be converted should simply be surrounded with quote marks in the parset - then ast.literal_eval knows to load it as a string.

Parameters:config – A ConfigParser object.
Returns:Nested dict {sections -> keys -> values } representing parsed params.