Demo: Incremental Parser for Web IDE

Dear Colleagues,

I made a demo web application , clearly illustrating what an incremental parser is and how it works. Watch it, please. I will be glad to hear your feedback and criticism.
')
And under the cut, I'll tell you about how modern IDEs work. And as a project I'm working on, it can help bring the web editor industry to a new level.

Web IDE

Recently, IDEs and code editors, made entirely on the basis of Web technologies, are gaining popularity. Surely you've heard of products like Cloud9, Koding, or the recently released Atom from GitHub. And maybe even work with them.

These editors are convenient because they provide a remote development server and do not require special steps for installation: you simply open the browser tab and get the environment ready for work for the programmer. Not a bad decision for a web developer working with dynamic languages, such as JavaScript, PHP, Ruby, Python.

However, for languages ​​with static typing, for example, for Java, C #, Objective-C, the competitiveness of web editors compared to such giants of the market as IntelliJ Idea and Eclipse is not yet relevant. A key feature of the latter is a deep understanding of the syntax and semantics of source codes, giving us tools such as code completion, jump to definition, refactoring, and error highlighting. This is quite important, especially if we work with a large code base.

The work of these IDEs is based on indexing the source code of the entire project: the editor uses special parsers, called "incremental parsers / compilers" , for continuous scanning and searching for definitions of functions, variables, and places where they are used in the code.

An incremental compiler differs from a regular compiler in a number of properties:

1. Continuous indexing of changes . A regular compiler runs the entire code, and ends up with its work. If we changed something in the code, then we run the compiler again. For the IDE, this approach is unacceptable, since a complete recompilation of the source codes after each character entered by the user, will create delays for seconds, and even minutes.
2. Tolerance to syntax errors . In the process of typing a program, the source code is often in a syntactically incorrect state. However, the IDE must understand the remaining, correct portions of the code, for example, for the code completion function to be available at any time.
3. The relationship between the structure of the tree parse and the text of the program . All nodes of the parse tree must keep the boundaries of the code fragments from which they were obtained. For example, so that when you click on the function name in code, the IDE moved the cursor to the definition of this function.

The development of such a compiler differs significantly from the development techniques of conventional compilers. Unfortunately, today in our industry there are practically no tools for creating incremental parsers and compilers. Most of the existing ones are written more or less manually, at great cost and time.

For this reason, I developed a library — Papa Carlo — for building incremental parsers. With this library you can create an incremental parser with all the above properties. Moreover, the library API looks like the API of ordinary parser combinators that most compiler developers are used to working with: you just describe the grammar of the language in PEG notation.

The demo above is written entirely using the Papa Carlo library, and works entirely on the client side, in the browser. This approach could be used to develop plug-ins for languages ​​with static typing, for example, to existing Web IDEs. Or even for code editors like Sublime Text, turning them into fully-fledged IDEs.

A little bit about the fact that under the hood of this demo

The Papa Carlo library itself is written in Scala, and compiled into JavaScript using the ScalaJS compiler. Thus, all calculations are performed entirely on the client side, in your browser. Only static content is sent from the server: JS and HTML files.

The parser runs in a web worker (if supported by the browser). This allows you to remove the load from the user interface, as well as to perform parallel computing.

All graphics are made using SVG and the d3.js library. CodeMirror is used as a code editor widget.

---

Friends, if you like this demo, please support the project, put a star on GitHub: https://github.com/Eliah-Lakhin/papa-carlo .