Hierarchical Configuration Environments

[sbrugman]

Description

Imagine using Kedro for running pipelines both locally and remote, with mock and warehouse data.
Development takes place on mock data (shared between the environments), but computational configuration differs (e.g. spark local vs cluster mode). On the warehouse data, again the data definitions are shared, but some properties vary, e.g. database names.

The two shared configurations above make that the configuration could be far more concise if we would be able to inherit from multiple environments:

graph TD;
    base["Base (base)"];
    dev["Dev"];
    local["Local (default)"];
    remote["Remote"];
    cluster["Data Warehouse"];
    acc["Acceptance"];
    prd["Production"];
    
    base --> dev --> local;
    dev --> remote;
    base --> cluster --> acc;
    cluster --> prd;

Loading

The bottom four environments are used by the Kedro user.
Alternatively, we could reduce the number of environments:

graph TD;
    base["Base (base)"];
    dev["Dev + Remote"];
    local["Local (default)"];
    cluster["Data Warehouse + Acceptance"];
    prd["Production"];
    
    base --> dev --> local;
    base --> cluster --> prd;

Loading

This relies on the local environment overwriting the configuration set for remote in the combined dev + remote env.

Is this something that could be supported?

Context

As mentioned above, this would simplify the config and avoid duplication of entries.

Possible Implementation

Configuration the base environment takes place in settings.py. The base_env argument could be simply accepting a dict as well as a string to specify the base per environment.

Possible Alternatives

Something might be achieved through the advanced features of the OmegaConfigLoader that I am not aware of.

[noklam]

Sorry the graph doesn't rendered on my phone.

Does that mean local = base + local + dev
And prod = base + dev + remote?

[jccalvojackson]

this is a very common use case whenever you have a series independent binary (for example) options.
For example, you want to run something either locally or remotely and on top of that you want to either do a smoke or full run.
it would be great to just write a smoke and remote besides the base (local + full run) instead of having to write a remote_smoke, remote_full, local_smoke (on top of the base)

[astrojuanlu]

@MatthiasRoels mentioned the need for something like this, although excess of layers makes it complex

[astrojuanlu]

We were introduced to another group of users that want this cc @jonascristens

[jonascristens]

Including @philiproeleveld as well

[astrojuanlu]

Old discussion suggesting the same thing #1709

[astrojuanlu]

Another user asked about this (or rather, was confused about not being able to use more than 2 levels of environments)

[salemileandro]

I'd like to +1 the idea!

That would be useful to be able to use more than 2 levels of environments. Override beteen level 1->2 could distinguish between local/prod environment (e.g., root_path as s3, global parameters, ...) while second override between level 2->3 could be used to distinguish between slight variations of models (e.g., for some datasets, consume the same as local/prod for all model flavors but for other datasets consume a model-specific input).

I understand that there might be other means to achieving that, e.g., by using runtime_params, but it might be cleaner in some instances to have things clearly defined in config files.

From the user perspective, something like this (if we want to set that up in the settings.py, probably not the main use case tbh)

CONFIG_LOADER_ARGS = {
    "base_env": "base",
    "default_run_env": ["local", "model1"],
}

Where in terms of priority, from high to low model1, local and base. I guess we would need to load base, then load local and override if key was also defined in base, then continue with loading model1 and override if key already exist. One would need to be careful about interpolation but if interpolation happens at parameters loading (which I think it's the case) rather than at file read it should be fine?

From the CLI, something like (probably the main use case, I don't think setting the env priorities in settings.py is good practice)

# Run model1 flavor locally 
kedro run --env local --env model1 

# Run model2 flavor locally 
kedro run --env local --env model2

# Run model1 flavor in prod 
kedro run --env prod --env model1 

# Run model2 flavor in prod 
kedro run --env prod --env model2 

---

I'm open to other ideas of how to handle this use case btw, please let me know!

[pierrejeden]

+1, and pitching in: I ran into this single env override limitation immediately in adopting Kedro. We used it in a pipeline where we had a base_env config, and then a complex config set up by users via a UI. A specific UI config was then selected via the env argument. However, this exhausted the possible overrides, i.e., we could not define an "experiment" in the intended way, by overriding a base + UI-config + experiement.

Since we also relied on OmegaConf (Kedro used anyconf at the time), I implemented a custom loader supporting multiple env:s. I used double-underscore as a logical separator:

kedro run -p main -e ui-1234567__new-params__DUMP-ALL-PKL

here ui_1234567 is one config ID among many, new_params is some overrides large enough to be cumbersome to handle with runtime_params and DUMP-ALL-PKL was a "functional" env creating default pickle datasets for all IO not present in the yaml catalog (this was before dataset-factories). I also exposed the base_env in a custom run command, but this could also be implemented as a base: directive in the env, e.g.,

kedro run -p main -e base:new-base__my-env # sets base_env to conf/new-base

I have hesitated to port my old config loader to my new team's codebase, since I want to keep maintenance in check. However, all that seems to be needed (naively) now, is to do for single_env in run_env.split('__'): wrapping this block in OmegaConfigLoader.

Like this, breaking out a new method that could also be used in loading the base_env.

    def __getitem__(self, key: str) -> dict[str, Any]:  # noqa: PLR0912
           
        ...

        for _run_env in run_env.split('__'):
            config = self._load_and_merge_single_env(
                config=config,
                env=_run_env,
                patterns=patterns,
                key=key,
                processed_files=processed_files,
                read_environment_variables=read_environment_variables,
            )

        if not processed_files and key != "globals":
            raise MissingConfigException(
                f"No files of YAML or JSON format found in {base_path} or {env_path} matching"
                f" the glob pattern(s): {[*self.config_patterns[key]]}"
            )

        return resulting_config  # type: ignore[no-any-return]
    
    def _load_and_merge_single_env(
        self,
        config,
        env,
        patterns,
        key,
        processed_files,
        read_environment_variables
    ):
        # Handle remote paths
        if self._protocol in CLOUD_PROTOCOLS or self._protocol in HTTP_PROTOCOLS:
            env_path = f"{self._remote_root_path}/{env}"
        elif self._protocol == "file":
            env_path = str(Path(self.conf_source) / env)
        else:
            env_path = str(Path(self._fs.ls("", detail=False)[-1]) / env)
        try:
            env_config = self.load_and_merge_dir_config(  # type: ignore[no-untyped-call]
                env_path, patterns, key, processed_files, read_environment_variables
            )
        except UnsupportedInterpolationType as exc:
            if "runtime_params" in str(exc):
                raise UnsupportedInterpolationType(
                    "The `runtime_params:` resolver is not supported for globals."
                )
            else:
                raise exc

        resulting_config = self._merge_configs(config, env_config, key, env_path)
        
        return resulting_config

(As an aside I'd love for the OmegaConfigLoader to have a more modular (and functional) implementation, currently I find it a bit hard to extend or modify it without duplicating a lot of the existing code.)