Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

jsonschema `$ref` parsed as yaml?

See original GitHub issue

Describe the bug $refs in jsonschemas seems to be parsed as yaml? I get the following error when the parser tries to follow the $ref:

Traceback (most recent call last):
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/__main__.py", line 495, in main
    generate(
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/__init__.py", line 353, in generate
    results = parser.parse()
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/base.py", line 426, in parse
    self.parse_raw()
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1177, in parse_raw
    self._parse_file(self.raw_obj, obj_name, path_parts)
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1250, in _parse_file
    self.parse_obj(obj_name, root_obj, path_parts or ['#'])
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1142, in parse_obj
    self.parse_ref(obj, path)
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1084, in parse_ref
    self.parse_ref(value, path)
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1064, in parse_ref
    self.resolve_ref(obj.ref)
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1054, in resolve_ref
    self._get_ref_body(relative_path),
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1005, in _get_ref_body
    return self._get_ref_body_from_url(resolved_ref)
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1010, in _get_ref_body_from_url
    return self.remote_object_cache.get_or_put(
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/__init__.py", line 26, in get_or_put
    value = self[key] = default_factory(key)
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1011, in <lambda>
    ref, default_factory=lambda key: load_yaml(self._get_text_from_url(key))
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/datamodel_code_generator/__init__.py", line 85, in load_yaml
    return yaml.load(stream, Loader=SafeLoader)
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/home/freol/dev/brefv/venv/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "yaml/_yaml.pyx", line 707, in yaml._yaml.CParser.get_single_node
  File "yaml/_yaml.pyx", line 726, in yaml._yaml.CParser._compose_document
  File "yaml/_yaml.pyx", line 905, in yaml._yaml.CParser._parse_next_event
yaml.scanner.ScannerError: mapping values are not allowed in this context
  in "<unicode string>", line 9, column 25

To Reproduce When trying to generate code for the following example jsonschema (including a reference):

Example schema:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "title": "Position",
    "description": "A reported position as a GeoJSON Point",
    "required": [
      "position"
    ],
    "properties": {
      "position": {
        "$ref": "http://geojson.org/schema/Point.json"
      }
    },
    "additionalProperties": false
}

Used commandline:

$ datamodel-codegen --input position.json --input-file-type jsonschema --output model.py

Expected behavior Successful generation of a pydantic model.

Version:

OS: Ubuntu 20.04 (WSL2 under win10)
Python version: 3.8
datamodel-code-generator version: 0.11.14

Additional context Add any other context about the problem here.

Issue Analytics

State:
Created 2 years ago
Reactions:1
Comments:5 (2 by maintainers)

Top GitHub Comments

2reactions

freol35241commented, Dec 7, 2021

@koxudaxi @kylebarron thanks, just tested 0.11.15 and can confirm this solves my issue 👍

1reaction

kylebarroncommented, Nov 24, 2021

I also hit this today. A simple example is a list of GeoJSON Features.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://example.com/list-of-features.schema.json",
  "title": "ListOfFeatures",
  "type": "object",
  "required": ["features"],
  "additionalProperties": false,
  "properties": {
    "features": {
      "type": "array",
      "items": {
        "$ref": "https://geojson.org/schema/Feature.json"
      }
    }
  }
}

If I run datamodel-codegen on this, it fails:

> datamodel-codegen \
  --input list-of-features.schema.json \
  --input-file-type jsonschema \
  --output test.py
Traceback (most recent call last):
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/__main__.py", line 495, in main
    generate(
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/__init__.py", line 353, in generate
    results = parser.parse()
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/base.py", line 426, in parse
    self.parse_raw()
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1177, in parse_raw
    self._parse_file(self.raw_obj, obj_name, path_parts)
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1250, in _parse_file
    self.parse_obj(obj_name, root_obj, path_parts or ['#'])
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1142, in parse_obj
    self.parse_ref(obj, path)
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1084, in parse_ref
    self.parse_ref(value, path)
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1067, in parse_ref
    self.parse_ref(obj.items, path)
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1064, in parse_ref
    self.resolve_ref(obj.ref)
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1054, in resolve_ref
    self._get_ref_body(relative_path),
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1005, in _get_ref_body
    return self._get_ref_body_from_url(resolved_ref)
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1010, in _get_ref_body_from_url
    return self.remote_object_cache.get_or_put(
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/__init__.py", line 26, in get_or_put
    value = self[key] = default_factory(key)
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1011, in <lambda>
    ref, default_factory=lambda key: load_yaml(self._get_text_from_url(key))
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/datamodel_code_generator/__init__.py", line 85, in load_yaml
    return yaml.load(stream, Loader=SafeLoader)
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/Users/kbarron/.pyenv/versions/3.8.10/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "yaml/_yaml.pyx", line 707, in yaml._yaml.CParser.get_single_node
  File "yaml/_yaml.pyx", line 726, in yaml._yaml.CParser._compose_document
  File "yaml/_yaml.pyx", line 905, in yaml._yaml.CParser._parse_next_event
yaml.scanner.ScannerError: mapping values are not allowed in this context
  in "<unicode string>", line 11, column 25

If I apply the diff:

+   "$id": "blank",
-   "$id": "https://example.com/list-of-features.schema.json",

Then datamodel-codegen works as expected. My read of the JSON Schema docs only says that an $id must be unique, not that it actually exists. It’s hard to find explicit, official documentation on this, but for example this SO answer says

Note, these are NOT URLs, and are not nessecerily relating to a location of any physical files, nor are they nessecerily network-addressable. $id is an identifier only.

Top Results From Across the Web

APIDevTools/json-schema-ref-parser: Parse, Resolve ... - GitHub

JSON Schema $Ref Parser is a full JSON Reference and JSON Pointer implementation that crawls even the most complex JSON Schemas and gives...

Options | JSON Schema $Ref Parser - API Dev Tools

The parse options determine how different types of files will be parsed. JSON Schema $Ref Parser comes with built-in JSON, YAML, plain-text, and...

json-schema-ref-parser/README.md - UNPKG

The CDN for json-schema-ref-parser. ... 3, #### Parse, Resolve, and Dereference JSON Schema $ref pointers. 4. 5, [![

How to use the json-schema-ref-parser.dereference function ...

To help you get started, we've selected a few json-schema-ref-parser examples, ... var json = yaml.parse(data); refParser.dereference(json, function(err, ...

json-schema-ref-parser | Yarn - Package Manager

The Solution: · Use JSON or YAML schemas — or even a mix of both! · Supports $ref pointers to external files and...