Language Oriented Programming (LOP) in action

In the continuation of the previous publication on Domain Driven Design, where Nikolay Grebnev consistently brought the design topic using DDD to the need to use the domain language, this publication will discuss the practice of designing and developing both the languages ​​themselves and programming them (the experience of JetBrains ).

Report smax Maxim Mazin last year's conference of software architects Application Developers Days
')

Video report:

Download

ftp.linux.kiev.ua/pub/conference/peers/addconf/2011/1a1-language-oriented-programming-mazin.avs.avi

Presentation

docs.google.com/present/view?id=dccwwvbq_729dxjj82gc

Text report (done by Belonesox )

Today I have the first report, I will talk about the language topic in programming.
My name is Maxim Mazin, I work at JetBrains as an engineer in the YouTrack project.
YouTrack is a bugtracker, and the most exciting thing about YouTrack is that it is written using Language Record programming.
What it is? This is a concept that assumes that when writing programs, first of all you create a special subject-oriented language, and then, in this subject-oriented language, you begin to write the actual program code.

What is this for? What is the use of this, and what prevents you from doing this to everyone - I will talk about it today.

Look - here there are universal programming languages. When programming using the Java, PHP or C # language, a problem arises - when programming a huge amount of boilerplate code arises; such code, which will be repeated from time to time, but not because it does some meaningful work, but simply so that the compiler can compile this code.
Write round brackets, dots, ... and so on.
In the case when you use some kind of framework, then you still have additional pain from the framework.
You need to initialize the library, open windows, set some properties, depending on what library you have.
In principle, in many modern programming languages, for example, C #, there is a practice to add object-oriented constructions to the language.

In the Java language, there is a synchronized construct, which is not a general purpose construct, it is a special, object-oriented construct designed for parallel programming.

Similarly, in the C ++ language, there is an operator overload. Here, compare.
Normally, if you didn’t have a synchronized construct, you would have to write this much code here.
The “finally” block and blocking block, this is all a boilerplate code that is not always interesting to write.
Instead, there is a synchronized DSL construction that allows you to lock the lock and does not worry about releasing it.

Likewise, in C ++, there is a mechanism for operator overloading that allows you to pretend as if you have defined special operations, for example, on complex numbers.

This is all very good, but as you know, read-write locks appeared in the Java language, starting from version 5, and this means for us that we need to write again the code of such a style as shown on the left:

Those. it is necessary to seize locks, lock locks, ... this all completely levels convenience.
If it were possible to write ... independently expand languages, we would be able to solve various problems.
For example, those constructions, language extensions that are, in the Java language, they allow you to solve a specific set of problems. But we face other challenges in life. For example, if we write a web application, then we need to work with databases, write a web level, and so on. There is no DSL for this, there is only a framework.

Similarly, if we use some kind of concept like dependency injection, then we also need to use frameworks for this, and even worse, use mixtures of languages, write simultaneously in Java, in XML, and all this is not very fun.
And the main thing is that we cannot do anything about it, because the creation of language expansion threatens us with various risks.

What problems arise when creating a DSL and why, if there is any need, do not immediately rush to create a DSL?

The biggest problem that everyone faces when trying to create a DSLay is the compatibility of language extensions.
Compatibility refers to the ability to simultaneously use two language extensions that are created by independent developers.

For example. If you have some libraries and frameworks, in our case for Java, for example
Hibernate, Spring or Joda Time, they are all compatible with each other, in principle, at the compilation level.
Those. You can use both Hibernate and Spring components in your project at the same time, and nothing can stop you.
But if you have language add-ins, in the form of some macros, or some other techniques that add object-oriented constructs to your Java language, then these add-ins are created independently, all the time there is a risk that these add-ins will clash with each other.


For example, such a very simple example.
Suppose we have an extension from somewhere that does something useful, and in particular supports the interpolation of expressions in strings.
You can write such a construct, and the value of this expression will be substituted for “resultCount”.
At the same time, another language extension comes to our friend, which allows us to do something useful and, in particular, also interpolate strings. In Java, there is no interpolation of strings, so it is expected that different people will release it in different language extensions.
And let's say the syntax is slightly different. Then, when sharing language extensions A and B, you instantly have a fatal ambiguity, how to interpret this buzz.
This example is very simple, but in fact, everyone who creates a DSL based on textual information, one way or another, is faced with a problem, they probably have no confidence that the extensions will not be contradictory.

Another problem that prevents us from creating DSLi. Assume nonetheless, despite the difficulties and problems with the compatibility of different extensions, you decide to write a DSL.
Write some preprocessor, set this preprocessor to your program.
your DSL constructs, it compiles and issues some kind of code, bytecode, for example,
or some executable code.
This is all very good, but in our time, no one writes code without using an IDE.
This is not done simply, because without the use of IDE, developer productivity is dramatically reduced.
Therefore, in most cases, when you create such a DSL, there will be situations where your developers, your colleagues, will prefer to use just the Java language in Idea, just the C # language together with ReSharper, rather than take your share, and write something with your help. because there is no IDE support.

IDE support means all features that modern IDEs provide to a developer. Those. This syntax highlighting is natural, error highlighting before compilation, refactoring, integration with Version Controls, everything else.
And yes, it can be noted that the creation of a language is in itself
This is a solved math problem.
If you have an example for syntax, you can create a compiler.
But it is nonetheless time consuming, even if you use a compiler compiler.

And if you immerse yourself in language-oriented programming very deeply, i.e. If you start building different extensions, then your compatibility problem is reaching a new level. Those. you have not only compatibility at the grammar level, but also for example at the type system level.
Those. if you have two independent extensions, then it is necessary that the system types of these two extensions work out normally and lead to each other normally.

There are a lot of different attempts to do this, to support DSL in one form or another, the topic is very hot, here you can probably add Nemerle, and maybe OCaml, now there are a lot of different means to create DSLs.
As part of Eclipsa, there is an XText framework that allows you to write a grammar and get something like DSL with IDE support.

They all have problems. The main problem stems from the fact that they are focused on text grammar.
And since they are focused on text grammar, then they have no chance to avoid the problem of compatibility of structures.

But fortunately, in our company JetBrains, the MPS environment was created.

MPS stands for meta-programming system, mainly intended for creating and extending languages, and such an extension, that immediately after its creation, IDE support appeared.

I have said many times that text grammar inevitably leads to problems. Therefore, the natural solution is to work directly with the AST → abstract syntax tree, and force the programmer to directly, directly create an abstract syntax tree.

Attempts such were made earlier, and as a rule such attempts are reduced to editing diagrams.
A diagram is drawn, in a sense ... my child programs on properties to a greater extent, you draw diagrams, you get pictures, then these pictures are assembled, you get code.
There is a problem - to draw these pictures is inconvenient - to grab with a mouse, to drag, it is inconvenient.
We, programmers, have a habit of writing code by hand.
Therefore, the concept of projection editors is implemented in MPS.

What is it about? Here you have an abstract syntax tree. Those. this is the presentation of the program, which is obtained, in any case, it turns out when the parser parses the program.
At the same time, you have a projection of it, into something like a text.

For each node of this syntactic tree there is a corresponding cell.
Those of you who are familiar with TeX should roughly present, roughly similar concepts.
Similarly, for each element you have a cell, the cells are merged into even larger cells, and so on.
At the same time, the impact on these cells, when you write to these cells, or in some way influence them, immediately immediately affects this piece of wood.
At the same time, our goal, as MPS developers, is to ensure that you do not feel at all that you are working not with text, but directly with wood. Here, of course, the problem is, but those of you who use IDEA should understand that when you edit text in IDEA, Eclipse or somewhere else, you also do not edit the text - you tap something that is absorbed by IDEA, and it’s turns into something.
In the case of Eclipse, this is more text editing, in the case of IDEA, this is even less text editing.

Such a projection editor has positive sides, and there are negative sides.
The positive side is clear. We have no text grammars, no text, no problems with text grammars, no conflicts, everything is compatible, everything is very good.
The negative side, is that after all there is some difference from text editing. At the same time, the problem. The fact is that addictive ... in our experience, people who start working with MPS, they get used to the programming style inside MPS, for about two weeks.
Those. during these two weeks, they can give some positive feedback, that they are uncomfortable, unusual, something different from programming in a text editor.
After two weeks, they get used to it and start working at the same speed as they did before.
I hope that if you try to download and start using it, we will get more feedback and we can make the projection editor even more like a text editor.

The idea is this. When creating languages, you write out a meta-model of languages, its abstract syntax, set the type system, describe the specific syntax, and automatically get the output language, along with the compiler, plus the MPS test for working with code in that language.

Inside MPS, along with MPS, a number of ready-made extensions are supplied, since we write projects in the end in Java, i.e. we are all generic to java, then we have a stack of languages ​​that is intended, finally, for java.
Together with MPS, inside it, different language extensions are supplied, for working with closures, with collections ... and so on. And a number of languages ​​for working with XML, stupid for work. Those. You can enter XML, and most importantly, generate XML from your code.

Well, there are still a number of languages ​​that are designed to create languages.
MPS in terms of creating languages ​​is quite fair (??? 16:20)
Instead of a demonstration, it looks something like this ↑.
I’m not showing it from my computer, so everyone who is interested in the demonstration is invited to the stand in the hallway.
See here how easy it is to take and support the support of a language construct.

In fact, this is all I wanted to tell, there are a few important things left.

MSP is not only suitable for creating language extensions, i.e. Not only in Java, you can add constructs that are missing in them, but you can also create some kind of your own DSLs.
For example, Martin Fowler created a language to describe his financial statements.

In addition, with the help of MPS, we wrote YouTrack, such a big bugtracker is big, in the sense of having a lot of code (generated). But my previous development experience tells me that the code is informative, which we write and edit, this code is quite small.
Again, we are interested, we can show at the output what the code looks like in the editor, how many tables there are.
The code is quite small, due to the fact that the level of abstraction is raised quite high.
For the development of YouTrack, a set of languages ​​was created, some of these languages ​​are open, and some are closed.
We do not give them away yet, because they have not yet achieved selling quality.

Here, the application is real, alive.

MPS has a positive property. It is “Apache 2.0” licensed, absolutely free, even for commercial use.

Download it here.
www.jetbrains.com/mps

MPS blog:
blogs.jetbrains.com/mps

Now if you have any questions, I suggest you ask them. (18:46)