Nested footnote declaration causes crash

Describe the bug + To Reproduce Take the following piece of code:

# Hello
World. Foo [^1].
- [^1]: BAM!

It fails with the following (full) traceback:

$ make html
# Sphinx version: 2.4.4
# Python version: 3.8.3 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 2.11.2
# Last messages:
#   reading sources... // many other files
#   reading sources... [ 93%] index
#   reading sources... [100%] test
# Loaded extensions:
#   sphinx.ext.mathjax (2.4.4) from <home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx\ext\mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from <home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinxcontrib\applehelp\__init__.py
#   sphinxcontrib.devhelp (1.0.2) from <home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinxcontrib\devhelp\__init__.py
#   sphinxcontrib.htmlhelp (1.0.3) from <home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinxcontrib\htmlhelp\__init__.py
#   sphinxcontrib.serializinghtml (1.1.4) from <home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinxcontrib\serializinghtml\__init__.py
#   sphinxcontrib.qthelp (1.0.3) from <home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinxcontrib\qthelp\__init__.py
#   alabaster (0.7.12) from <home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\alabaster\__init__.py
#   sphinx_rtd_theme (unknown version) from <home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx_rtd_theme\__init__.py
#   myst_parser (0.9.1) from <home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\myst_parser\__init__.py
Traceback (most recent call last):
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx\cmd\build.py", line 276, in build_main
    app.build(args.force_all, filenames)
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx\application.py", line 349, in build
    self.builder.build_update()
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx\builders\__init__.py", line 297, in build_update
    self.build(to_build,
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx\builders\__init__.py", line 311, in build
    updated_docnames = set(self.read())
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx\builders\__init__.py", line 418, in read
    self._read_serial(docnames)
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx\builders\__init__.py", line 439, in _read_serial
    self.read_doc(docname)
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx\builders\__init__.py", line 479, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx\io.py", line 316, in read_doc
    pub.publish()
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\docutils\core.py", line 217, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\sphinx\io.py", line 130, in read
    self.parse()
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\docutils\readers\__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\myst_parser\sphinx_parser.py", line 203, in parse
    to_docutils(
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\myst_parser\main.py", line 93, in to_docutils
    return md.render(text, env)
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\markdown_it\main.py", line 245, in render
    return self.renderer.render(self.parse(src, env), self.options, env)
  File "<home>\Anaconda3\envs\py3.8-sphinx2-myst\lib\site-packages\myst_parser\docutils_renderer.py", line 138, in render
    self.render_footnote_reference_open(self.env["foot_refs"][footref][0])
KeyError: '1'

If, however, anything is written before the footnote declaration, like:

- Prefix [^1]: BAM!

then the [^1] is printed verbatim in the rendered HTML and no footnote is generated.

Finally, related error:

# Hello back
One [^1].
- [^1]: Two

[^1]: Three

Then, a footnote is produced, the “One [^1]” appears verbatim, but with the “[^1]” inside a <a> that points to an incorrect id (and inside a <span class="problematic">), and for the other two appearances, it looks like a footnote was created, but without any links, and with different ids.

The console output:

<main_dir>\test2.md:5: WARNING: Duplicate explicit target name: "1".
<main_dir>\test2.md:2: WARNING: Too many autonumbered footnote references: only 0 corresponding footnotes available.
<main_dir>\test2.md:2: WARNING: Duplicate target name, cannot be used as a unique reference: "1".

None of this happens if these “false declarations” are done inline inside a paragraph, and there are no warning.

(Also, if one declares two times a footnote, only the first declaration is used, and the second one is completely absent. No warning is printed.)

Expected behavior This is obviously not a supported syntax for a footnote declaration, but the parser should not crash for this.

I guess the “wanted” syntax for footnote declarations should be with the [^<number>]: at the beginning of a line. So every other use of [^<number>]: should be considered as composed of [^<number>] followed by a colon.

Environment:

• Anaconda installation on Windows, with conda 4.8.3
• Python 3.8.3
• Sphinx 2.4.4
• MyST-Parser 0.9.1
• Markdown-it-py 0.4.8
• docutils 0.16

(Sorry if any information is missing.)

[chrisjsewell]

Thanks @Jimvy, firstly this may or may not have some link to https://github.com/executablebooks/MyST-Parser/issues/195. Perhaps try using [^a] instead and see if the same error occurs. Secondly, footnotes are parsed using the markdown-it-footnote, which implements the pandoc footnote logic. This states that:

The footnotes themselves… may appear anywhere except inside other block elements (lists, block quotes, tables, etc.).

Thus it is not possible to use them in a list item, and anyway they get extracted from the document and placed at the end of it (see https://github.com/executablebooks/MyST-Parser/issues/179).

As you mention though, it would be preferable though that this behaviour raised a warning rather than an exception. Also perhaps the documentation should be more specific about this.

[chrisjsewell]

Edit: There’s maybe some room for interpretation. markdown-it-footnote README says…

yeh that’s what I was going to mention: the plugin from mdit-py-plugins is a “faithful” port of that, so you would need to ask for it to be changed upstream first. Anyhow this is now fixed