Review available Markdown Syntax

A list of markdown items that we should review in addition to discussions on Jmarkdown, RST support etc.

• http://scholarlymarkdown.com/ - seems to be behind planned schedule fall 2015 but may have some useful ideas for consideration.
• https://spec.commonmark.org/0.29/

[choldgraf]

I’ve spent a little bit of time looking into Markdown flavors that might be of interest and wanted to update this issue with some more perspective.

First off, I think this is the main takeaway (it is a suggestion, not a directive): our build system should support a strict subset of Pandoc markdown or RMarkdown. It could do this in addition to another language like rST.

Why support Pandoc Markdown?

I spoke with a few folks at RStudio which is the main driver behind RMarkdown. This is the flavor of markdown supported by bookdown. It has proven resilient and quite popular in the R community, and supports many of the features that we’d need for publishing. RMarkdown is a subset of Pandoc markdown (here is a summary of pandoc markdown and here is the announcement that RMarkdown is a subset of pandoc markdown). This means that if we were to support the same subset of Pandoc Markdown, then we’d be supporting a language that is already utilized by a huge community of people.

How would we support this?

Here is a useful post with information on what we’d need to do to support this.

I think there are two options to supporting pandoc markdown.

• First is to write a direct parser that goes from Pandoc markdown -> a docutils AST. This could be, for example, by building on top of the recommonmark project, which does this for the base “commonmark” flavor of markdown.
• Second is to build a bridge between markdown and rST, such as the m2r package. This is also what nbsphinx does to support reading Jupyter notebooks.

In either case, we’d want to define a subset of Pandoc markdown syntax that we wish to support, and then create a mapping from that subset onto either docutils objects or rST. This could be a standalone Sphinx extension that would be really useful for the community outside of just this project.

As a side note, here’s an interesting post about the differences between pandoc MD vs rST

@chrisjsewell has already made a really interesting implementation of this approach here: https://github.com/ExecutableBookProject/meta/pull/12

A few caveats

There are a few potential pitfalls to this…here are some that I can think of:

• This could get edge-casey, depending on how much of the Pandoc or RMarkdown world we want to support.
• Perhaps we should think purely in form of the within-cell markdown flavor we wish to support, and then farm out the representation of a notebook in that markdown flavor to something like jupytext.
• We should do some profiling to see if this would add significant overhead to the build times

What about directives and roles?

The biggest question to my mind is what to do about directives and roles (if we are using Sphinx under the hood). These are one of the most powerful features in Sphinx, and something we could take advantage of to extend new features for books. But, there are no native “directives and roles” features in markdown.

One idea would be to piggy-back on Pandoc markdown syntax for these.

Directives

In rST directives look like this:

.. mydirective:
   :myparam1: myval1
   :myparam2: myval2

In markdown, this might be utilized with Pandoc syntax. For example, Pandoc allows you to separate <div> elements with fences like so:

::: mydiv
# Some markdown
inside the div
:::

and optionally:

::: myotherdiv {.myattribute}
:::

Perhaps the pattern of ::: something could be mapped onto directives in rST. For example, something like:

::: toctree {maxdepth=1}
* page1
* page2
:::

Roles

For in-line markup, we could use the “bracketed spans” syntax from Pandoc markdown. This is intended to making custom “span” elements in your text like so:

[This is *some text*]{.class key="val"}

However, we could piggy-back on this by defining some specific roles, e.g.:

I'm now linking to a [different document]{doc=anotherPage} which contains [this equation]{eq=myeqID}. And also for [references]{ref=mybibtexref}. 

Curious what folks think about that…

I’ll update this issue if I can think of some other things to consider…

[choldgraf]

Was going through the CommonMark forums and found an interesting comment from JGM re: extension syntax in markdown: https://talk.commonmark.org/t/support-for-extension-token/2771/7