How does the JetBrains MPS system allow you to achieve a wider use of DSLs (domain-specific languages)?
DSL-i (domain specific languages or languages for specific areas) programmers have known for a long time. Despite this, they are rarely used in real systems. This article will discuss what DSLs are and why they are not widely used. It will also describe how the JetBrains MPS system solves problems that prevent their widespread use.
So what is DSL? DSL is a language designed to solve problems in a particular subject area. DSLs are most declarative languages that solve problems in narrow subject areas. For example, SQL, regular expressions, XPath, Prolog, formulas in Excel. Unfortunately, this list of widely used DSL ends. The main advantage of such languages is that due to the proximity of their structures to the subject area, the code in these languages is very clear and short. Moreover, to edit the code in such languages, it is not necessary to be a programmer. If a person understands the subject area, then he can easily write code in such languages thanks to his knowledge.
In theory, everything looks simple: we take the subject area, say, accounting, write the language and give it to the experts or use it ourselves for a more brief description of the subject area. Unfortunately, this approach is not widespread. Let's see why.
Why DSL and not widely spread?
')
Let's look at the reasons why DSLs are not widely used. The first reason is that the authors of such languages focus on closed languages for only one specific problem, instead of expanding existing general-purpose programming languages, such as Java, PHP, JavaScript. There is a reason for this: the possibility of ambiguities when combining such extensions with each other. Another reason is the difficulty of creating the language infrastructure necessary for the implementation of the language, and comfortable work with it.
Most of the efforts in the DSL community are aimed at working with closed languages. In many cases it would be more useful to add new constructions to existing languages, for example in Java. Imagine that you can use language extensions, just like you are using libraries now. With this approach, developers will be able to simultaneously use the expressive power of DSL and the universality of languages like Java, which is impossible with the use of existing technologies.
When creating extensions, in order to use them in the same way as we currently use libraries, it is necessary to make languages compatible with each other. This means that if we add one extension to our language, for example, money support: a type for money, literals like $ 10 or $ 100, and another extension that adds mathematical symbols to the language: sums, products, etc., then we can use them together, even if they were created by different authors.
Unfortunately, all popular general-purpose programming languages are based on text grammars. These grammars have one unpleasant property: they can be ambiguous, they may have several interpretations of the same line. Moreover, if we add new constructions to Java with the help of the extension A, and we achieve the uniqueness of the grammar, and then we do the same with the extension B, it may turn out that if we take Java and both extensions, the resulting grammar will be ambiguous.
Let's take an example. Let's say 2 companies decided to add support for string interpolation (string interpolation allows you to write expressions inside string literals) in Java. Suppose the first company uses the following syntax:
"{2 + 3}"
And the second one is:
"$ {2 + 3}"
If we use both extensions at the same time, and introduce such a program:
"Account balance is $ {account.getBalance ()}"
its interpretation is ambiguous. Is $ part of the interpolation syntax, or part of a string literal? The example is somewhat artificial, but it allows us to understand the general problem of ambiguity that arises when there is a similar syntax for different constructions.
To achieve high developer productivity, intelligent development tools are needed. With the advent of intelligent editors, such as in IntelliJ IDEA or Eclipse, it can be difficult for developers to switch to editing text in regular editors. Text editors do not highlight errors, do not provide context-sensitive help, do not show menus with available options, they do not support refactorings. There are frameworks for creating intelligent editors, for example, IntelliJ IDEA Language API, XText, Oslo, but none of these frameworks support extensible languages at the proper level. Even if we do not need extensibility, building language support using these tools requires good knowledge of programming languages and takes a lot of time. Apparently, instrumental support is very important, but it is not easy to implement.
Let's summarize: people are doing the wrong type of DSL; to achieve increased productivity, it is necessary to extend existing programming languages of general purpose. It is difficult to create such extensions due to the fact that widespread technologies do not support the compatibility of extensions with each other.
How JetBrains MPS solves these problems
Let's now look at how MPS solves these problems. In order to maintain compatibility of extensions with each other, MPS does not work with programs as with text. Instead, MPS stores them as a syntax tree, and editing is done directly, without intermediate use of the text. This approach allows you to significantly simplify the creation of IDE, since the constant presence of the syntax tree makes it easy to implement error highlighting, automatic addition, contextual prompts and so on.
MPS solves the problem of ambiguity in a radical way: if we do not have text grammar, then we do not have ambiguity. Such an approach, however, does not mean that grammars are not used in MPS. Instead of a specific syntax, the abstract syntax (syntax tree structure) is defined in the language definition in MPS. If you are familiar with XML, you probably know about XML Schema, which resembles the syntax used in MPS.
Since this approach does not allow ambiguities, languages can easily be combined with each other. You can extend the syntax of the language with new constructions. You can insert a code in one language inside a code in another language, or even insert a code in a general-purpose programming language inside a relatively closed DSL-i. This means that languages are compatible with each other, and this makes it possible to reuse languages and their parts, which is possible with big problems in the case of traditional technologies. We at JetBrains have experimented a lot with such reuse. The MPS distribution includes a large number of Java extensions:
Most languages that we use to define languages simultaneously extend and contain Java. For example, one of the language constructs for type systems, the output rule, looks like a regular DSL. However, inside this rule, you can write in Java, extended constructs specific to type systems.
Since we got rid of the text view, we cannot use a plain text editor. To work with the code, we use a special projection editor. For each node of the syntactic tree, it creates a projection - a part of the screen with which the user can interact. During the development of MPS, great efforts were made to ensure that such an editor behaved as close to the test editor as possible. For example, if you enter 1 + 2 + 3 in MPS, you will also receive a syntax tree, which would be obtained when parsing this line in Java. Of course, the projection editor is different from the text editor, and there are things that are possible in one and impossible in the other, and vice versa. Despite this, you can get used to these differences without losing performance. In our experience, it takes about 2 weeks to become productive in a projection editor.
Creating support for intelligent editing when working with a syntax tree is greatly simplified directly. Moreover, in many places, intellectual possibilities are provided by MPS IDE without any efforts from the author of the language. Features such as automatic addition, search for uses, rename, work automatically. When developing IntelliJ IDEA, intelligent editing support was implemented for many languages. The implementation of such support required a lot of effort: several people per month. With MPS, similar capabilities can be realized in a matter of days. This is possible because special languages are used to develop languages that configure the existing language infrastructure. MPS is not just an editor. You can create a full IDE with it.
Inside JetBrains, we use MPS to develop commercial projects. Our new error tracking system, codenamed Charisma, is created entirely on the MPS, and this is just the beginning.
Conclusion
Widespread use of DSL is hampered by two problems: the impossibility of their reuse in systems based on text grammars, and the difficulty of creating intelligent tools for working with them. MPS solves both of these problems by working with the syntax tree directly, without an intermediate textual representation, and providing the infrastructure for creating intelligent tools for working with such languages.
MPS 1.0 was released in July. Most of the code is available under the Apache 2.0 license (with the exception of the JetBreains IDE Framework, the license of which allows the use of MPS in MPS-based products without buying any licenses from JetBrains).
You can download the MPS from here: www.jetbrains.com/mps and start creating languages right today.
PS We are looking for a senior developer in the project. Details in the vacancy in my profile.
So what is DSL? DSL is a language designed to solve problems in a particular subject area. DSLs are most declarative languages that solve problems in narrow subject areas. For example, SQL, regular expressions, XPath, Prolog, formulas in Excel. Unfortunately, this list of widely used DSL ends. The main advantage of such languages is that due to the proximity of their structures to the subject area, the code in these languages is very clear and short. Moreover, to edit the code in such languages, it is not necessary to be a programmer. If a person understands the subject area, then he can easily write code in such languages thanks to his knowledge.
In theory, everything looks simple: we take the subject area, say, accounting, write the language and give it to the experts or use it ourselves for a more brief description of the subject area. Unfortunately, this approach is not widespread. Let's see why.
Why DSL and not widely spread?
')
Let's look at the reasons why DSLs are not widely used. The first reason is that the authors of such languages focus on closed languages for only one specific problem, instead of expanding existing general-purpose programming languages, such as Java, PHP, JavaScript. There is a reason for this: the possibility of ambiguities when combining such extensions with each other. Another reason is the difficulty of creating the language infrastructure necessary for the implementation of the language, and comfortable work with it.
Most of the efforts in the DSL community are aimed at working with closed languages. In many cases it would be more useful to add new constructions to existing languages, for example in Java. Imagine that you can use language extensions, just like you are using libraries now. With this approach, developers will be able to simultaneously use the expressive power of DSL and the universality of languages like Java, which is impossible with the use of existing technologies.
When creating extensions, in order to use them in the same way as we currently use libraries, it is necessary to make languages compatible with each other. This means that if we add one extension to our language, for example, money support: a type for money, literals like $ 10 or $ 100, and another extension that adds mathematical symbols to the language: sums, products, etc., then we can use them together, even if they were created by different authors.
Unfortunately, all popular general-purpose programming languages are based on text grammars. These grammars have one unpleasant property: they can be ambiguous, they may have several interpretations of the same line. Moreover, if we add new constructions to Java with the help of the extension A, and we achieve the uniqueness of the grammar, and then we do the same with the extension B, it may turn out that if we take Java and both extensions, the resulting grammar will be ambiguous.
Let's take an example. Let's say 2 companies decided to add support for string interpolation (string interpolation allows you to write expressions inside string literals) in Java. Suppose the first company uses the following syntax:
"{2 + 3}"
And the second one is:
"$ {2 + 3}"
If we use both extensions at the same time, and introduce such a program:
"Account balance is $ {account.getBalance ()}"
its interpretation is ambiguous. Is $ part of the interpolation syntax, or part of a string literal? The example is somewhat artificial, but it allows us to understand the general problem of ambiguity that arises when there is a similar syntax for different constructions.
To achieve high developer productivity, intelligent development tools are needed. With the advent of intelligent editors, such as in IntelliJ IDEA or Eclipse, it can be difficult for developers to switch to editing text in regular editors. Text editors do not highlight errors, do not provide context-sensitive help, do not show menus with available options, they do not support refactorings. There are frameworks for creating intelligent editors, for example, IntelliJ IDEA Language API, XText, Oslo, but none of these frameworks support extensible languages at the proper level. Even if we do not need extensibility, building language support using these tools requires good knowledge of programming languages and takes a lot of time. Apparently, instrumental support is very important, but it is not easy to implement.
Let's summarize: people are doing the wrong type of DSL; to achieve increased productivity, it is necessary to extend existing programming languages of general purpose. It is difficult to create such extensions due to the fact that widespread technologies do not support the compatibility of extensions with each other.
How JetBrains MPS solves these problems
Let's now look at how MPS solves these problems. In order to maintain compatibility of extensions with each other, MPS does not work with programs as with text. Instead, MPS stores them as a syntax tree, and editing is done directly, without intermediate use of the text. This approach allows you to significantly simplify the creation of IDE, since the constant presence of the syntax tree makes it easy to implement error highlighting, automatic addition, contextual prompts and so on.
MPS solves the problem of ambiguity in a radical way: if we do not have text grammar, then we do not have ambiguity. Such an approach, however, does not mean that grammars are not used in MPS. Instead of a specific syntax, the abstract syntax (syntax tree structure) is defined in the language definition in MPS. If you are familiar with XML, you probably know about XML Schema, which resembles the syntax used in MPS.
Since this approach does not allow ambiguities, languages can easily be combined with each other. You can extend the syntax of the language with new constructions. You can insert a code in one language inside a code in another language, or even insert a code in a general-purpose programming language inside a relatively closed DSL-i. This means that languages are compatible with each other, and this makes it possible to reuse languages and their parts, which is possible with big problems in the case of traditional technologies. We at JetBrains have experimented a lot with such reuse. The MPS distribution includes a large number of Java extensions:
- collections language, which makes it easier to work with collections
- dates language that adds date support directly to java
- math language, which allows you to write sums, works, and so forth mathematically constructions
Most languages that we use to define languages simultaneously extend and contain Java. For example, one of the language constructs for type systems, the output rule, looks like a regular DSL. However, inside this rule, you can write in Java, extended constructs specific to type systems.
Since we got rid of the text view, we cannot use a plain text editor. To work with the code, we use a special projection editor. For each node of the syntactic tree, it creates a projection - a part of the screen with which the user can interact. During the development of MPS, great efforts were made to ensure that such an editor behaved as close to the test editor as possible. For example, if you enter 1 + 2 + 3 in MPS, you will also receive a syntax tree, which would be obtained when parsing this line in Java. Of course, the projection editor is different from the text editor, and there are things that are possible in one and impossible in the other, and vice versa. Despite this, you can get used to these differences without losing performance. In our experience, it takes about 2 weeks to become productive in a projection editor.
Creating support for intelligent editing when working with a syntax tree is greatly simplified directly. Moreover, in many places, intellectual possibilities are provided by MPS IDE without any efforts from the author of the language. Features such as automatic addition, search for uses, rename, work automatically. When developing IntelliJ IDEA, intelligent editing support was implemented for many languages. The implementation of such support required a lot of effort: several people per month. With MPS, similar capabilities can be realized in a matter of days. This is possible because special languages are used to develop languages that configure the existing language infrastructure. MPS is not just an editor. You can create a full IDE with it.
Inside JetBrains, we use MPS to develop commercial projects. Our new error tracking system, codenamed Charisma, is created entirely on the MPS, and this is just the beginning.
Conclusion
Widespread use of DSL is hampered by two problems: the impossibility of their reuse in systems based on text grammars, and the difficulty of creating intelligent tools for working with them. MPS solves both of these problems by working with the syntax tree directly, without an intermediate textual representation, and providing the infrastructure for creating intelligent tools for working with such languages.
MPS 1.0 was released in July. Most of the code is available under the Apache 2.0 license (with the exception of the JetBreains IDE Framework, the license of which allows the use of MPS in MPS-based products without buying any licenses from JetBrains).
You can download the MPS from here: www.jetbrains.com/mps and start creating languages right today.
PS We are looking for a senior developer in the project. Details in the vacancy in my profile.
Source: https://habr.com/ru/post/66094/
All Articles