How to respect hidden paragraphs?

Describe the bug

context

I’m migrating a Sphinx project from recommonmark to MyST, and noticed a few unexpected changes in formatting. One is the addition of <p> paragraph tags in list items. Take the following Markdown:

* A Python library
* A Sphinx extension

MyST converts this to:

<ul class="simple">
<li><p>A Python library</p></li>
<li><p>A Sphinx extension</p></li>
</ul>

This example comes from the getting started guide.

expectation

I would have expected:

<ul class="simple">
<li>A Python library</li>
<li>A Sphinx extension</li>
</ul>

This is the output of markdown-it-py, and the markdown-it live demo (not counting the simple class name).

bug

Instead, the <p> tags are added, which means extra vertical spacing and odd content semantics.

problem

I can’t think of a scenario where the extra vertical space, or the semantics, are desirable.

For example, for the MyST documentation, I can see this was worked around of with CSS: https://github.com/executablebooks/sphinx-book-theme/blob/1b5c3889bce036c4116a7e35aed83668ff810357/src/sphinx_book_theme/assets/styles/base/_typography.scss#L46-L52

Reproduce the bug

1. Create a Markdown document with an unordered or ordered list item
2. Convert to HTML

List your environment

myst-parser==0.17.0

[rowanc1]

Another stopgap alternative is to change the CSS styling so that visually the outcome is as expected. Semantically it is a bit different (is it data or text). I believe the spread flag in mdast handles this for listItems.

li p, table p {
    margin: 0px;
}

[thibaudcolas]

Looks like this got reported for sphinx 2.0b1, which they decided to fix by adding CSS in one theme 😐.

Testing this with rst2pseudoxml and rst2html – I get the same output as you for the AST, but can confirm the HTML does not have the paragraph tags.

So this increasingly looks like a clear sphinx-specific issue? It’s unclear to me why this came up specifically as part of our switch from recommonmark to MyST, but I assume it’s a dependencies management problem on our side.

Edit: yes, since MyST-Parser has an explicit dependency on sphinx>=3.1,<5, this overwrites the sphinx==1.8.6 we’ve explicitly installed previously, and leads to this.