Include directive: Allow separate includes for reference links and reference definition

Describe the bug

foo.md

<!-- contents-begin-md -->

Hello world [Test][test].

<!-- contents-end-md -->

MyST-Parser is awesome!!!

<!-- references-begin-md -->

[test]: http://example.com/

<!-- references-end-md -->

bar.rst

.. contents-begin-rst

Hello world `Test <test_>`__

.. contents-end-rst

.. references-begin-rst

.. _test: http://example.com/

.. references-end-rst

1.

When I include parts of foo.md in another md.

```{include} foo.md
---
start-after: <!-- contents-begin-md -->
end-before: <!-- contents-end-md -->
---
```

```{include} foo.md
---
start-after: <!-- references-begin-md -->
end-before: <!-- references-end-md -->
---
```

When rendering you get where the reference is lost:

Hello world [Test][test].

2.

When I include parts of bar.rst in an md.

```{eval-rst}
.. include:: bar.rst
   :start-after: contents-begin-rst
   :end-before: contents-end-rst

.. include:: bar.rst
   :start-after: references-begin-rst
   :end-before: references-end-rst
```

So it renders as a link but points to: host/#test instead of to http://example.com/

3.

If I include foo.md in an .rst:

.. include:: foo.md
   :parser: myst_parser.sphinx_
   :start-after: <!-- contents-begin-md -->
   :end-before: <!-- contents-end-md -->

.. include:: foo.md
   :parser: myst_parser.sphinx_
   :start-after: <!-- references-begin-md -->
   :end-before: <!-- references-end-md -->

Hello world [Test][test]. is obtained. The link is not rendered.

4.

When I include parts of bar.rst in another .rst

.. include:: bar.rst
   :start-after: contents-begin-rst
   :end-before: contents-end-rst

.. include:: bar.rst
   :start-after: references-begin-rst
   :end-before: references-end-rst

It renders correctly.

Reproduce the bug

Source code: https://github.com/eyllanesc/myst_bug Output: https://myst-bug.readthedocs.io/en/latest/index.html

List your environment

• sphinx==4.4.0
• myst_parser==0.16.1, 0.17.0

[thht]

Ok, I did some digging.

The problem is basically the order in which things are done:

1. Convert the markdown in a file to a list of tokens. References are already resolved here and references defined within the same file are resolved correctly.
2. Evaluate all the directives. This includes the {include} directive.

This means that if the included file defines some references, they are not available, when the references in the original file are resolved.

There would be a few ways to potentially solve this:

1. Have MystParser.parse intercept any include directives before calling render and just add that content to the inputstring. You would need to make sure to handle recursion correctly.
2. Write a markdown_it plugin that provides the include file functionality. I think, this might be doable but at the moment, myst-parser comes with a set of pre-activated plugins and I could not find a way to tell it to activate more…
3. Anything else?

For me, having a collection of references in an included file is really something that makes my life easier…

@chrisjsewell and maybe @choldgraf : is there any chance of this getting included? I would be happy and capable of contributing code and time. But I would need some feedback from you:

1. Would you be willing to include this feature? If yes, under what conditions (except for the obvious like code quality, flake, unit tests)?
2. What strategy / approach would you prefer?

[mawieland]

I might have found a “workaround” for the first problem . Using the MySt directive with syntax from rst works in my case:

(note the extra colon before “start-after” and missing dashes)

```{include} ./my_file.md
:start-after: dummy-label start
:end-before: dummy-label end
```

myst-parser 0.18.0 Sphinx 5.1.1