AST is incomplete?

Hello,

I was interested in getting the AST from mistune so that I could manipulate it, then have mistune render the modified AST.

I created a couple of functions to do this:

#!/usr/bin/env python3

import mistune
assert mistune.__version__ == '2.0.0a1'

def md_to_ast(md_text):
    return mistune.markdown(md_text, renderer='ast')

def ast_to_html(ast):
    r = mistune.HTMLRenderer()
    md = mistune.Markdown(r)
    def inject_ast(self, tokens, _):
        return ast
    md.before_render_hooks.append(inject_ast)
    html = md.parse('')
    return html

This works for simple cases:

assert mistune.markdown('hello') == ast_to_html(md_to_ast('hello'))
assert mistune.markdown('hello *world*') == ast_to_html(md_to_ast('hello *world*'))

but falls down in other cases:

assert mistune.markdown('# title\nfoo') == ast_to_html(md_to_ast('# title\nfoo'))

Traceback (most recent call last):
  File "./test.py", line 20, in <module>
    assert mistune.markdown('# title\nfoo') == ast_to_html(md_to_ast('# title\nfoo'))
  File "./test.py", line 15, in ast_to_html
    html = md.parse('')
  File "/home/cell/github/cellularmitosis/junk/retainn/mistune-v2/mistune/markdown.py", line 54, in parse
    result = self.block.render(tokens, self.inline, state)
  File "/home/cell/github/cellularmitosis/junk/retainn/mistune-v2/mistune/block_parser.py", line 246, in render
    return ''.join(data)
  File "/home/cell/github/cellularmitosis/junk/retainn/mistune-v2/mistune/block_parser.py", line 265, in _iter_render
    yield method(children)
TypeError: heading() missing 1 required positional argument: 'level'

It looks like the AstRenderer might be dropping some information?

Ideally, it would be great to have a markdown -> AST -> markdown workflow supported.

[sunflowr]

@lepture I had a similar idea using markdown -> AST -> markdown. I would like to use the AST to search for something in the markdown, and after finding it, returning parts of the original markdown. Therefore, it would help a lot to convert (parts of) the AST back to markdown again. Do you have a suggestion of how to accomplish that?

Been looking for something like this too for a while and was happy when I found out mistune have MD -> AST but then I couldn’t find a way of going AST -> MD again. Is this a planned feature or maybe there’s a way of doing this?

On another note: Would be nice if the AST was tree based instead of a long list of objects and had a API for searching the tree. So I could for instance easily grab the content of “header1/subheader2”.

My use-case is that I have a document and I need to extract everything within a specific section denoted by header level (e.g. I’d like to grab the content of “header1/subheader2” and output this in a new markdown file)

[soerface]

@lepture I had a similar idea using markdown -> AST -> markdown. I would like to use the AST to search for something in the markdown, and after finding it, returning parts of the original markdown. Therefore, it would help a lot to convert (parts of) the AST back to markdown again. Do you have a suggestion of how to accomplish that?