Some general questions/suggestions

Hi, great and interesting project, just few questions/suggestions: 1- Why you don’t use the default python tokenizer module converted to Javascript because it is much simpler and battle tested, the current implementation is a bit bloated, plus there are many bugs which I can of course contribute/report (mainly related with error tokens and the start/end range). 2- Why not using ANTLR to generate the parse tree (ANTLR python grammar is already defined and battle tested and the Javascript or python runtimes are also available+ possibility to interface any python version like 2.x which is out the scope of this project + possibility to get inspired from other projects like Jython) and thus the whole project will consist of a visitor which will have all the transform methods migrated to. 3- Implementing 2 will give an easier out-of-the-box implementation of the ast module which is unimplemented currently. 4- Why not using a safer typed language like Typescript and divide the transpiler into smaller chunks using better OOP paradigms(an ultimate goal is the whole project written in python and which it can transpile itself to Javascript)

[PierreQuentel]

I am not too sure what you mean by “separating code into modules” but if you mean different Python modules that can import each other, possibly in a package, yes this is supported.

I am currently working on an engine that generates Javascript from the AST representation instead of the Brython-specific tree, in module ast_to_js.js. It is progressing very well, still a few weeks of work before it can replace the current implementation but I am very optimistic.

If it succeeds, it will make the Python-to-JS engine much easier to understand : since version 3.9.4 the tokenizer produces the same tokens as CPython, since 3.10.4 the AST tree is also the same, and both are documented in Python docs. The generation engine will also be much shorter (notably, no need for the .transform() methods, which are currently very obscure and poorly documented).

[PierreQuentel]

@deadlocklogic

Thanks for the pointers to the Skulpt scripts, I didn’t think of looking for this.

What you suggest makes sense, obviously, but it means a very radical change in Brython code - probably an almost complete rewriting of py2js.js. I would suggest that you start a separate project, relying on the new PEG parser, with Javascript code to generate a similar AST to that of CPython 3.10. I and other contributors would follow the project and we could discuss its progress and a possible integration in Brython.

What do you think ?