question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Hydra composition writes scientific float without decimal point which is parsed by YAML as str

See original GitHub issue

Bug Report

Description

When using the Hydra Composition, float numbers in the input config files in scientific format, e.g., 1.0e-5 are written to the generated params.yaml file as 1e-5. The latter is parsed as a str instead of a float. The decimal point is required.

Reproduce

conf/config.yaml

number: 1.0e-5

test.py

from pathlib import Path
import yaml

text = Path("params.yaml").read_text()
config = yaml.load(text, Loader=yaml.SafeLoader)
assert isinstance(config["number"], float)

dvc.yaml

stages:
  test:
    cmd: python test.py
dvc exp run test

Generated params.yaml:

number: 1e-5

Expected

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.30.0 (rpm)
---------------------------------
Platform: Python 3.8.3 on Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.14
Subprojects:

Supports:
        azure (adlfs = None, knack = 0.10.0, azure-identity = None),
        gdrive (pydrive2 = 1.14.0),
        gs (gcsfs = None),
        hdfs (fsspec = None, pyarrow = 9.0.0),
        http (aiohttp = None, aiohttp-retry = 2.8.3),
        https (aiohttp = None, aiohttp-retry = 2.8.3),
        oss (ossfs = 2021.8.0),
        s3 (s3fs = None, boto3 = None),
        ssh (sshfs = 2022.6.0),
        webdav (webdav4 = 0.9.7),
        webdavs (webdav4 = 0.9.7),
        webhdfs (fsspec = None)
Cache types: hardlink, symlink
Cache directory: xfs on /dev/md124
Caches: local, s3
Remotes: s3, s3
Workspace directory: xfs on /dev/md124
Repo: dvc (subdir), git

Additional Information (if any):

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
aschuh-hfcommented, Oct 25, 2022

OK, I think I understand now. The conclusion is that the params.yaml you produce requires a YAML 1.2 compatible reader. And PyYAML is not, which is what I use in my code currently. Switching to ruaml.yaml will fix it. Got it 😃

0reactions
aschuh-hfcommented, Oct 25, 2022

But when I install DVC from YUM repo, does it not use its own virtual environment / libraries in /usr/lib/dvc?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Numbers in scientific notation without dot are parsed as string
Hi, it appears that PyYAML requires a dot in the mantissa as well as a sign in the exponent in order to parse...
Read more >
YAML loads 5e-6 as string and not a number - Stack Overflow
The matching of floats can be fixed so it will accept floating point values with an e / E but without decimal dot...
Read more >
Config composition - Hydra
We use the same solution in configuration as in writing the underlying software: composition. To do this in Hydra, we first add a...
Read more >
See raw diff - Hugging Face
droptol : float - tolerance for dropping near-zero values when writing compressed binary. - Default is None - chunk : int - chunk...
Read more >
Floating point numbers (Float) - HitchDev
StrictYAML parses to a YAML object representing a decimal - e.g. ... require precision, but float is better for values for which precision...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found