needextract issues

needextract leads to a sphinx exception with sphinx-needs 1.1.0. The same document used to work with sphinx 1.0.3.

Example, file 1:

.. needextract::
  :filter: id == "REQ_SYS_ADSys_FL_check_EB1_0"

file 2:

.. req-sys:: Some title
    :id: REQ_SYS_ADSys_FL_check_EB1_0
    :parent_artifact: EB_FL_check_1, EB_approach_1
    :status: assumed
    :verification-criteria: ...
    :asil:
    :tags: SYSDES_SOI
    :collapse: false

    Lorem ipsum bla bla bla. This includes:

    - :need:`PAR_SYS_acc_long_comfort_max`
    - :need:`PAR_SYS_acc_lat_comfort_max_abs`

NOTE: It doesn’t matter whether subsitutions, need references or parent_artifact option is included or removed, the error remains.

Sphinx-build output (options: -j 1 -k):

Exception occurred:
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/docutils/writers/_html_base.py", line 1614, in unimplemented_visit
    raise NotImplementedError('visiting unimplemented node type: %s'
NotImplementedError: visiting unimplemented node type: substitution_reference

Full traceback:

# Sphinx version: 5.3.0
# Python version: 3.8.10 (CPython)
# Docutils version: 0.17.1 release
# Jinja2 version: 3.0.3
# Last messages:
#   [app] emitting event: 'doctree-resolved'(<container: <Need...>>, 'documentation/02_soi/navint/requirements/partial_situations/ps_1+2+4_cross
#   [app] emitting event: 'build-finished'(NotImplementedError('visiting unimplemented node type: substitution_reference'),)
#   
#   ====================== slowest reading durations =======================
#   3.933 documentation/cpp_developer_guide/cpp_coding_guidelines
#   3.439 common/interfaces/aracom_interfaces/aracom_interface_fusion/doc/fusion_dynamic_world_interface
#   2.336 documentation/tools/test_automation/dt_launcher_l4_modules/dt_launcher_l4.helper
#   1.936 documentation/tools/test_automation/dt_launcher_l4_modules/dt_launcher_l4.nodes
#   1.339 documentation/04_domain_level/video_perception/requirements/video_perception_requirements_viewpoint
#   Needs successfully exported
# Loaded extensions:
#   sphinx.ext.mathjax (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (2.0.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.5) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/alabaster/__init__.py
#   sphinx_copybutton (0.5.1) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx_copybutton/__init__.py
#   sphinx_rtd_theme (unknown version) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx_rtd_theme/__init__.py
#   sphinx_warnings_filter (0.2.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx_warnings_filter/__init__.py
#   sphinx.ext.duration (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/duration.py
#   sphinx.ext.extlinks (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/extlinks.py
#   sphinx.ext.ifconfig (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/ifconfig.py
#   sphinx.ext.intersphinx (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.todo (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/todo.py
#   sphinx_data_viewer (0.1.2) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx_data_viewer/__init__.py
#   sphinx_needs (1.1.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx_needs/__init__.py
#   myst_parser (0.18.1) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/myst_parser/__init__.py
#   sphinx.ext.autodoc.preserve_defaults (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/autodoc/preserve_defaults.py
#   sphinx.ext.autodoc.type_comment (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc.typehints (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/autodoc/typehints.py
#   sphinx.ext.autodoc (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.autosummary (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/autosummary/__init__.py
#   sphinx.ext.graphviz (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/graphviz.py
#   sphinx.ext.inheritance_diagram (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/inheritance_diagram.py
#   autoapi.extension (unknown version) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/autoapi/extension.py
#   sphinx.ext.autosectionlabel (5.3.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/ext/autosectionlabel.py
#   breathe (4.34.0) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/breathe/__init__.py
#   matplotlib.sphinxext.plot_directive (3.6.2) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/matplotlib/sphinxext/plot_directive.py
#   sim_scenario_integration (unknown version) from /home/arwedus/src/documentation/_ext/sim_scenario_integration.py
#   dt_launcher_test_case_integration (unknown version) from /home/arwedus/src/documentation/_ext/dt_launcher_test_case_integration.py
#   sphinxcontrib.plantuml (unknown version) from /home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinxcontrib/plantuml.py
#   driving_usecase_integration (unknown version) from /home/arwedus/src/documentation/_ext/driving_usecase_integration.py
#   yaaa_needimport (0.3.0) from /home/str2lr/projects/docs-as-code/sphinx-contrib/yaaa_needimport/src/yaaa_needimport/__init__.py
Traceback (most recent call last):
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/cmd/build.py", line 281, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/application.py", line 347, in build
    self.builder.build_update()
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 310, in build_update
    self.build(to_build,
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 376, in build
    self.write(docnames, list(updated_docnames), method)
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 571, in write
    self._write_serial(sorted(docnames))
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 581, in _write_serial
    self.write_doc(docname, doctree)
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/builders/html/__init__.py", line 666, in write_doc
    self.docwriter.write(doctree, destination)
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/docutils/writers/__init__.py", line 78, in write
    self.translate()
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/writers/html.py", line 59, in translate
    self.document.walkabout(visitor)
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/docutils/nodes.py", line 227, in walkabout
    if child.walkabout(visitor):
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/docutils/nodes.py", line 227, in walkabout
    if child.walkabout(visitor):
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/docutils/nodes.py", line 227, in walkabout
    if child.walkabout(visitor):
  [Previous line repeated 8 more times]
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/docutils/nodes.py", line 219, in walkabout
    visitor.dispatch_visit(self)
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/sphinx/util/docutils.py", line 565, in dispatch_visit
    method(node)
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/docutils/writers/_html_base.py", line 1423, in visit_substitution_reference
    self.unimplemented_visit(node)
  File "/home/arwedus/.sphinx-venv/lib/python3.8/site-packages/docutils/writers/_html_base.py", line 1614, in unimplemented_visit
    raise NotImplementedError('visiting unimplemented node type: %s'
NotImplementedError: visiting unimplemented node type: substitution_reference

@danwos: If you need to debug this issue, please contact me directly for accessing the actual project where this occurs.

[danwos]

Thanks for checking it again. I was able to reproduce it. So what is needed to get the error:

1. a need with a substitution in its content. Like this is an awesome |TEST|
2. a substitution definition somewhere in the same file. No matter if part of need content or not. Like .. |TEST| replace:: System Test
3. a needextract, which is creating a need-representation with the same content (so it is containing a substitution). It doesn’t matter, if this needextract file contains also a valid substitution definition.

Whenever a substitution is used in need_content and copied by needextract, the exception will be raised.

I will need to take a look into the Sphinx code and figure out how substitutions are handled by Sphinx and if I’m able to retrigger this mechanism for the copied need_content.

[danwos]

Found a solution, and PR #841 will be merged soon.

However, the problem is based on a general problem, where content related handling by Sphinx and docutils is already done, but our later copied need-content by needextract still needs some handling. Especially, if the content is related to the document, because of image paths or substitutions. So I’m pretty sure there are other scenarios, for which a Sphinx feature is not supported by needextract out of the box. So feel free to open new issues 😃

Thanks everybody for the detailed error reports and checks 👍 They helped a lot identifying the bug.