Skip to content

Gamma Config

python build cov

An opinionated way of setting up configuration for data science projects.

Overview

Gamma config provides a standard and flexible way of implementing application configuration primarily based on YAML files. It promotes best practices by:

  • simplify separation of code and config data;
  • breaking large, complex configuration into multiple files;
  • providing a way to write environment-aware config files;
  • facilitating security best-practices, like proper secrets management;
  • providing a immutable central global object storing all contextual data.

Another benefit of a standard config mechanism is allowing Gamma extension libraries to provide configuration in a common way.

Features

  • Configuration expressed as a set of YAML files (by defaul) inside a config folder in the root of the project.
  • Multiple YAML files merged following simple rules. Simple file ordering convention using two digit prefixes.
  • Builtin support for environment specific parameters (production, development, etc.)
  • Support for .env files via python-dotenv, including .local.env that can be added to .gitignore
  • Dynamic evaluation via YAML tags. For instance, the !ref tag allow you to reference other parameters in any config file.
  • Custom tag support via simple and cool multiple dispatch mechanism.
  • Round-trip dump of config back into YAML. Support for hiding sensitive data on dump.
  • Simplified key access via dot (.). Eg. for config: {foo: {bar: 100}}, this is True: config.foo.bar == 100

Click here to view the full documentation

Getting started

Using pip:

pip install gamma-config

In most cases, you'll want to use the !j2 tag to interpolate values using Jinja2. This requires manually installing the jinja2 package or using the jinja2 extras.

pip install gamma-config[jinja2]

You must install pydantic if using the structured configuration feature.

pip install gamma-config[pydantic]

Basic Usage

The package comes with "scaffolding" to help you get started. In your project folder:

   python -m gamma.config.scaffold

Remove the sample files, then create yourself a config/20-myconfig.yaml file with the contents:

foo: 1
user: !env USER

To access the config from within your Python program:

import os
from gamma.config import get_config

def run():

    # it's safe and efficient to call this multiple times
    config = get_config()

    # get static value using the dict keys or attribute access
    assert config["foo"] == 1
    assert config.foo == 1

    # get dynamic variables
    assert config["user"] == os.getenv("USER")
    assert config.user == os.getenv("USER")

Most of the magic happen via tags. Look at the documentation for info on the built-in tags available.

Changelog

Breaking in 0.8

  • We DEPRECATED !py and !obj tags, replacing them by the cleaner/simpler !call tag. Those are schedule to be removed by release 1.0.

  • We DEPRECATED "Structured configuration" support, We recommend directly using Pydantic's V2 discriminated unions as an alternative.

  • By default, !env, !expr and !call will not dump the contents on a to_yaml call. We provide !env:dump, !expr:dump, !call:dump to force dumping. Tag !env_secret is still supported but use is discouraged.

  • We have !j2:secret in addition to !j2_secret for consistency. Use of !j2_secret is discouraged.

Breaking in 0.7

  • We've DEPRECATED our homegrown multiple dispatch system gamma.dispatch, replacing it by plum. Unless you were using gamma.dispatch directly, or extending via custom tags, no need to worry. The gamma.dispatch package will be removed by release 1.0.

New in 0.7

  • We have a new home in https://github.com/cjalmeida/gamma-config !

Breaking in 0.6

  • Strict support for YAML 1.2 Core Schema. In practice, unquoted ISO8610 dates (eg. 2022-12-20) won't get converted to datetime.date or datetime.datetime objects. Use !date or !datetime if needed.
  • .env files are loaded automatically and get precedence over config.env and config.local.env.
  • Use of config.env and config.local.env is deprecated.
  • Default scaffolded include_folder interpret ENVIRONMENT variable string like foo bar as two separate environment subfolders.
  • (dispatch) Val arguments passed as class (eg. foo(Val['bar'])) will be converted to instance, as if it were called foo(Val['bar']())
  • The !py:<module>:<func> will no longer a single None argument

New in 0.6

Breaking changes in 0.5

  • When using the dot (.) syntax, missing values raise AttributeError instead of returning a false-y object.
  • Dropped support for Python 3.7

New in 0.5

  • We're now in PyPI!
  • Options for installing extra dependencies (eg. jinja2, pydantic)