tkp.config – Pipeline Configuration

Outline

The tkp.config folder contains the job configuration file templates, and code for handling config files.

API

tkp.config.get_database_config(pipeline_config=None, apply=None)[source]

Determine database config and (optionally) use to set up the Database.

Determines a database configuration using the settings defined in a dict (if supplied) and possibly overridden by environment variables. The config resulting from the combination of defaults, supplied dict, and environment variables is returned as a dict. If apply==True, the Database singleton is configured using these resulting settings.

The following environment variables are recognized, and take priority:

  • TKP_DBENGINE
  • TKP_DBNAME
  • TKP_DBUSER
  • TKP_DBPASSWORD
  • TKP_DBHOST
  • TKP_DBPORT
Parameters:
  • pipeline_config – Dict of db settings. Relevant keys: (engine, database, user, password, host, port, passphrase )
  • apply – apply settings (configure db connection) or not
Returns:

containing the resulting combined settings (resulting from defaults, config_passed and possibly environment variables.)

Return type:

dict

tkp.config.initialize_pipeline_config(pipe_cfg_file, job_name=False)[source]

Initializes the default variables and loads the ConfigParser file.

Sets defaults for start_time, job_name and cwd; these can then be used via variable substitution in other config values.

tkp.config.parse – Magic ConfigParser to dict functionality

Utilities for loading parameters from config files, with automatic type conversion.

tkp.config.parse.loads_timestamp_w_microseconds(dt_str)[source]

Loads and returns timestamp with microsecond precission

tkp.config.parse.parse_to_dict(config)[source]

Loads the ConfigParser object as a nested dictionary.

Automatically converts strings representing ints and floats to their respective types, through the magic of ast.literal_eval. This functionality is extensible via the loads_methods list. Each loads (load string) method is tested in turn to see if it throws an exception. If all throw, we assume the value is meant to be string.

Any values that you don’t want to be converted should simply be surrounded with quote marks in the parset - then ast.literal_eval knows to load it as a string.

Parameters:config – A ConfigParser object.
Returns:Nested dict {sections -> keys -> values } representing parsed params.