question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

YAML roundtrip not working correctly

See original GitHub issue

Hello, I am using hydra-zen to save config yaml files, ship them over to another system, and then load them back in and instantiate them. However, load_from_yaml is not working as expected - it loads back in a DictConfig rather than a hydrazen type and thus none of the type validation + coercion works when instantiating the loaded yaml. This is particularly an issue for enum types.

Repro:


import omegaconf
from omegaconf import MISSING, OmegaConf
import dataclasses
from enum import Enum

from hydra_zen import instantiate, builds, save_as_yaml, load_from_yaml, to_yaml

class Color(Enum):
    red = "RED"
    blue = "BLUE"

@dataclasses.dataclass
class Example:
    color: Color =  MISSING
    number: int = MISSING
    
    
built = builds(Example, color="blue", number="3")


print(type(built))
print(built)
print(instantiate(built))

save_as_yaml(built, "saved.yaml")
loaded = load_from_yaml("saved.yaml")

print()
print(type(loaded))
print(loaded)
print(instantiate(loaded))

The first 3 prints make sense:

<class 'type'>
<class 'types.Builds_Example'>
Example(color=<Color.blue: 'BLUE'>, number=3)

Proper type validation and coercion is done. When the yaml is saved and loaded back in however, type validation + coercion does not happen, since load_from_yaml returns a DictConfig rather than a type. The results of instantiate do not match. From the last 3 prints:

<class 'omegaconf.dictconfig.DictConfig'>
{'_target_': '__main__.Example', 'color': 'blue', 'number': 3}
Example(color='blue', number=3)

As you can see, Example is instantiated with the string blue rather than the enum <Color.blue: 'BLUE'>.

Issue Analytics

  • State:closed
  • Created 10 months ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
petosacommented, Dec 7, 2022

@Jasha10 thank you for this suggestion - it does seem to address my cases. I guess the only inconvenience is that I need to have special handling via builds for enums. I will try it out in my project and re-open if I run into more errors. Appreciate all the help!

0reactions
Jasha10commented, Dec 3, 2022

One workaround, which does not rely on OmegaConf’s implicit conversion of str to Enum, is to call builds(Color, value="BLUE") so as to create an object whose _target_ points to Color:

import dataclasses
from enum import Enum
from hydra_zen import builds, instantiate, load_from_yaml, save_as_yaml
from omegaconf import MISSING

def pp(x):
    print(x, "::", type(x))

class Color(Enum):
    red = "RED"
    blue = "BLUE"

@dataclasses.dataclass
class Example:
    color: Color = MISSING
    number: int = MISSING

built = builds(Example, color=builds(Color, value="BLUE"), number="3")

pp(instantiate(built))

save_as_yaml(built, "saved.yaml")
loaded = load_from_yaml("saved.yaml")

pp(instantiate(loaded))

print()
pp(loaded)
$ python script.py
Example(color=<Color.blue: 'BLUE'>, number=3) :: <class '__main__.Example'>
Example(color=<Color.blue: 'BLUE'>, number=3) :: <class '__main__.Example'>

{'_target_': '__main__.Example', 'color': {'_target_': '__main__.Color', 'value': 'BLUE'}, 'number': 3} :: <class 'omegaconf.dictconfig.DictConfig'>

Here are 3 specific situations where the proposed solution does not work (can attach some code snippets later if that’d help).

I believe my proposal above should work around the 3 situations you mentioned, @petosa.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to do round trip yaml while preserving tags without ...
Everything is working as expected, but the dump function is trying to resolve tags. I can't seem to find anything on how to...
Read more >
Objects do not round trip properly between YAML ... - GitHub
The Load() of both YAML and YAML::XS throw exceptions when confronted with a dump produced by the other module. Please run the attached...
Read more >
ruamel.yaml / Tickets / #390 Roundtripping regression
As this has always been working properly before, I highly doubt that our cython layer is causing any issue, and I highly suspect...
Read more >
ruamel.yaml 0.16.5 - PyPI
ruamel.yaml is a YAML parser/emitter that supports roundtrip preservation of comments, seq/map flow style, and map key order.
Read more >
Details — Python YAML package documentation
If your input is inconsistently indented, such indentation cannot be preserved. The first round-trip will make it consistent/normalize it. Here are some ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found