Config¶

Config Types¶

The following types are used to describe the schema of configuration data via config. They are used in conjunction with the builtin types above.

class dagster.ConfigSchema[source]¶

This is a placeholder type. Any time that it appears in documentation, it means that any of the following types are acceptable:

1. A Python scalar type that resolves to a Dagster config type (int, float, bool, or str). For example:

   • @op(config_schema=int)

   • @op(config_schema=str)

2. A built-in python collection (list, or dict). list is exactly equivalent to Array [ Any ] and dict is equivalent to Permissive. For example:

   • @op(config_schema=list)

   • @op(config_schema=dict)

3. A Dagster config type:

   • Any

   • Array

   • Bool

   • Enum

   • Float

   • Int

   • IntSource

   • Noneable

   • Permissive

   • Map

   • ScalarUnion

   • Selector

   • Shape

   • String

   • StringSource

4. A bare python dictionary, which will be automatically wrapped in Shape. Values of the dictionary are resolved recursively according to the same rules. For example:

   • {'some_config': str} is equivalent to Shape({'some_config: str}).

   • {'some_config1': {'some_config2': str}} is equivalent to

     Shape({'some_config1: Shape({'some_config2: str})}).

5. A bare python list of length one, whose single element will be wrapped in a Array is resolved recursively according to the same rules. For example:

   • [str] is equivalent to Array[str].

   • [[str]] is equivalent to Array[Array[str]].

   • [{'some_config': str}] is equivalent to Array(Shape({'some_config: str})).

6. An instance of Field.

class dagster.Field(config, default_value=<class 'dagster.config.field_utils.__FieldValueSentinel'>, is_required=None, description=None)[source]¶

Defines the schema for a configuration field.

Fields are used in config schema instead of bare types when one wants to add a description, a default value, or to mark it as not required.

Config fields are parsed according to their schemas in order to yield values available at job execution time through the config system. Config fields can be set on ops, on loaders and materializers for custom, and on other pluggable components of the system, such as resources, loggers, and executors.

Parameters

• config (Any) –

  The schema for the config. This value can be any of:

  1. A Python primitive type that resolves to a Dagster config type (int, float, bool, str, or list).

  2. A Dagster config type:

     • Any

     • Array

     • Bool

     • Enum

     • Float

     • Int

     • IntSource

     • Noneable

     • Permissive

     • ScalarUnion

     • Selector

     • Shape

     • String

     • StringSource

  3. A bare python dictionary, which will be automatically wrapped in Shape. Values of the dictionary are resolved recursively according to the same rules.

  4. A bare python list of length one which itself is config type. Becomes Array with list element as an argument.

• default_value (Any) –

  A default value for this field, conformant to the schema set by the dagster_type argument. If a default value is provided, is_required should be False.

  Note: for config types that do post processing such as Enum, this value must be the pre processed version, ie use ExampleEnum.VALUE.name instead of ExampleEnum.VALUE

• is_required (bool) – Whether the presence of this field is required. Defaults to true. If is_required is True, no default value should be provided.

• description (str) – A human-readable description of this config field.

Examples:

@op(
    config_schema={
        'word': Field(str, description='I am a word.'),
        'repeats': Field(Int, default_value=1, is_required=False),
    }
)
def repeat_word(context):
    return context.op_config['word'] * context.op_config['repeats']

class dagster.Selector(fields, description=None)[source]¶

Define a config field requiring the user to select one option.

Selectors are used when you want to be able to present several different options in config but allow only one to be selected. For example, a single input might be read in from either a csv file or a parquet file, but not both at once.

Note that in some other type systems this might be called an ‘input union’.

Functionally, a selector is like a Dict, except that only one key from the dict can be specified in valid config.

Parameters

fields (Dict[str, Field]) – The fields from which the user must select.

Examples:

@op(
    config_schema=Field(
        Selector(
            {
                'haw': {'whom': Field(String, default_value='honua', is_required=False)},
                'cn': {'whom': Field(String, default_value='世界', is_required=False)},
                'en': {'whom': Field(String, default_value='world', is_required=False)},
            }
        ),
        is_required=False,
        default_value={'en': {'whom': 'world'}},
    )
)
def hello_world_with_default(context):
    if 'haw' in context.op_config:
        return 'Aloha {whom}!'.format(whom=context.op_config['haw']['whom'])
    if 'cn' in context.op_config:
        return '你好，{whom}!'.format(whom=context.op_config['cn']['whom'])
    if 'en' in context.op_config:
        return 'Hello, {whom}!'.format(whom=context.op_config['en']['whom'])

class dagster.Permissive(fields=None, description=None)[source]¶

Defines a config dict with a partially specified schema.

A permissive dict allows partial specification of the config schema. Any fields with a specified schema will be type checked. Other fields will be allowed, but will be ignored by the type checker.

Parameters

fields (Dict[str, Field]) – The partial specification of the config dict.

Examples:

@op(config_schema=Field(Permissive({'required': Field(String)})))
def map_config_op(context) -> List:
    return sorted(list(context.op_config.items()))

class dagster.Shape(fields, description=None, field_aliases=None)[source]¶

Schema for configuration data with string keys and typed values via Field.

Unlike Permissive, unspecified fields are not allowed and will throw a DagsterInvalidConfigError.

Parameters

• fields (Dict[str, Field]) – The specification of the config dict.

• field_aliases (Dict[str, str]) – Maps a string key to an alias that can be used instead of the original key. For example, an entry {“solids”: “ops”} means that someone could use “ops” instead of “solids” as a top level string key.

class dagster.Map(key_type, inner_type, key_label_name=None)[source]¶

Defines a config dict with arbitrary scalar keys and typed values.

A map can contrain arbitrary keys of the specified scalar type, each of which has type checked values. Unlike Shape and Permissive, scalar keys other than strings can be used, and unlike Permissive, all values are type checked. :param key_type: The type of keys this map can contain. Must be a scalar type. :type key_type: type :param inner_type: The type of the values that this map type can contain. :type inner_type: type :param key_label_name: Optional name which describes the role of keys in the map. :type key_label_name: string

Examples:

@op(config_schema=Field(Map({str: int})))
def partially_specified_config(context) -> List:
    return sorted(list(context.op_config.items()))

class dagster.Array(inner_type)[source]¶

Defines an array (list) configuration type that contains values of type inner_type.

Parameters

inner_type (type) – The type of the values that this configuration type can contain.

class dagster.Noneable(inner_type)[source]¶

Defines a configuration type that is the union of NoneType and the type inner_type.

Parameters

inner_type (type) – The type of the values that this configuration type can contain.

Examples:

config_schema={"name": Noneable(str)}

config={"name": "Hello"}  # Ok
config={"name": None}     # Ok
config={}                 # Error

class dagster.Enum(name, enum_values)[source]¶

Defines a enum configuration type that allows one of a defined set of possible values.

Parameters

• name (str) – The name of the enum configuration type.

• enum_values (List[EnumValue]) – The set of possible values for the enum configuration type.

Examples:

@op(
    config_schema=Field(
        Enum(
            'CowboyType',
            [
                EnumValue('good'),
                EnumValue('bad'),
                EnumValue('ugly'),
            ]
        )
    )
)
def resolve_standoff(context):
    # ...

classmethod from_python_enum(enum, name=None)[source]¶

Create a Dagster enum corresponding to an existing Python enum.

Parameters

• enum (enum.EnumMeta) – The class representing the enum.

• name (Optional[str]) – The name for the enum. If not present, enum.__name__ will be used.

Example:

class Color(enum.Enum):
    RED = enum.auto()
    GREEN = enum.auto()
    BLUE = enum.auto()

@op(
    config_schema={"color": Field(Enum.from_python_enum(Color))}
)
def select_color(context):
    # ...

class dagster.EnumValue(config_value, python_value=None, description=None)[source]¶

Define an entry in a Enum.

Parameters

• config_value (str) – The string representation of the config to accept when passed.

• python_value (Optional[Any]) – The python value to convert the enum entry in to. Defaults to the config_value.

• description (Optional[str]) – A human-readable description of the enum entry.

class dagster.ScalarUnion(scalar_type, non_scalar_schema, _key=None)[source]¶

Defines a configuration type that accepts a scalar value OR a non-scalar value like a List, Dict, or Selector.

This allows runtime scalars to be configured without a dictionary with the key value and instead just use the scalar value directly. However this still leaves the option to load scalars from a json or pickle file.

Parameters

• scalar_type (type) – The scalar type of values that this configuration type can hold. For example, int, float, bool, or str.

• non_scalar_schema (ConfigSchema) – The schema of a non-scalar Dagster configuration type. For example, List, Dict, or Selector.

• key (Optional[str]) – The configuation type’s unique key. If not set, then the key will be set to ScalarUnion.{scalar_type}-{non_scalar_schema}.

Examples:

graph:
  transform_word:
    inputs:
      word:
        value: foobar

becomes, optionally,

graph:
  transform_word:
    inputs:
      word: foobar

dagster.StringSource¶

Use this type when you want to read a string config value from an environment variable. The value passed to a config field of this type may either be a string literal, or a selector describing how to look up the value from the executing process’s environment variables.

Examples:

from dagster import job, op, StringSource

@op(config_schema=StringSource)
def secret_op(context) -> str:
    return context.op_config

@job
def secret_job():
    secret_op()

secret_job.execute_in_process(
    run_config={
        'ops': {'secret_op': {'config': 'test_value'}}
    }
)

secret_job.execute_in_process(
    run_config={
        'ops': {'secret_op': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE'}}}
    }
)

dagster.IntSource¶

Use this type when you want to read an integer config value from an environment variable. The value passed to a config field of this type may either be a integer literal, or a selector describing how to look up the value from the executing process’s environment variables.

Examples:

from dagster import job, op, IntSource

@op(config_schema=IntSource)
def secret_int_op(context) -> int:
    return context.op_config

@job
def secret_job():
    secret_int_op()

secret_job.execute_in_process(
    run_config={
        'ops': {'secret_int_op': {'config': 1234}}
    }
)

secret_job.execute_in_process(
    run_config={
        'ops': {'secret_int_op': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE_INT'}}}
    }
)

dagster.BoolSource¶

Use this type when you want to read an boolean config value from an environment variable. The value passed to a config field of this type may either be a boolean literal, or a selector describing how to look up the value from the executing process’s environment variables. Set the value of the corresponding environment variable to "" to indicate False.

Examples:

from dagster import job, op, BoolSource

@op(config_schema=BoolSource)
def secret_bool_op(context) -> bool:
    return context.op_config

@job
def secret_job():
    secret_bool_op()

secret_job.execute_in_process(
    run_config={
        'ops': {'secret_bool_op': {'config': False}}
    }
)

secret_job.execute_in_process(
    run_config={
        'ops': {'secret_bool_op': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE_BOOL'}}}
    }
)

Config Utilities¶

class dagster.ConfigMapping(config_fn, config_schema=None, receive_processed_config_values=None)[source]¶

Defines a config mapping for a graph (or job).

By specifying a config mapping function, you can override the configuration for the child ops and graphs contained within a graph.

Config mappings require the configuration schema to be specified as config_schema, which will be exposed as the configuration schema for the graph, as well as a configuration mapping function, config_fn, which maps the config provided to the composite solid to the config that will be provided to the child nodes.

Parameters

• config_fn (Callable[[dict], dict]) – The function that will be called to map the graph config to a config appropriate for the child nodes.

• config_schema (ConfigSchema) – The schema of the graph config.

• receive_processed_config_values (Optional[bool]) – If true, config values provided to the config_fn will be converted to their dagster types before being passed in. For example, if this value is true, enum config passed to config_fn will be actual enums, while if false, then enum config passed to config_fn will be strings.

@dagster.configured(configurable, config_schema=None, **kwargs)[source]¶

A decorator that makes it easy to create a function-configured version of an object. The following definition types can be configured using this function:

• CompositeSolidDefinition

• ExecutorDefinition

• LoggerDefinition

• ResourceDefinition

• SolidDefinition

If the config that will be supplied to the object is constant, you may alternatively invoke this and call the result with a dict of config values to be curried. Examples of both strategies below.

Parameters

• configurable (ConfigurableDefinition) – An object that can be configured.

• config_schema (ConfigSchema) – The config schema that the inputs to the decorated function must satisfy.

• **kwargs – Arbitrary keyword arguments that will be passed to the initializer of the returned object.

Returns

(Callable[[Union[Any, Callable[[Any], Any]]], ConfigurableDefinition])

Examples:

dev_s3 = configured(s3_resource, name="dev_s3")({'bucket': 'dev'})

@configured(s3_resource)
def dev_s3(_):
    return {'bucket': 'dev'}

@configured(s3_resource, {'bucket_prefix', str})
def dev_s3(config):
    return {'bucket': config['bucket_prefix'] + 'dev'}