Lua in Moscow 2019: an interview with Roberto of Jerusalem

Some time ago, our Moscow office was visited by Roberto of Jerusalem, the author of the Lua language. We interviewed him, during which asked questions from Habr's readers. Finally, we can share with you the entire recording of the conversation.

- Let's start to philosophize. If you created Lua from scratch again, what three things would you change in this language?

- Wow! Hard question. Behind the creation and development of the language is a whole story. This was not one big decision. There were decisions that I regretted and which over the years have been able to correct. And programmers complain about it all the time because compatibility has broken. We did this several times. And I think only about some small tasks.
')
- Global default (Global-by-default)? Think this is the right way?

- May be. But this path is very difficult for dynamic languages. Perhaps you should completely abandon the concept of "default", but then it will be difficult to use variables.
For example, you have to somehow declare all the standard libraries. You need one-liner , print(sin(x)) , and then you need to declare “print” and also “sin”. It is strange to have ads for such short scripts.

Anything larger should have nothing by default, as it seems to me. The local locality (Local-by-default) is not a solution, it does not exist at all. It is for assignment only, not for use. We assign something, then use and assign it again, and a completely inexplicable error occurs.

Perhaps globality by default is not perfect, but locality by default is definitely not an option. I think some kind of announcement may have been optional ... Many times we intended to make some kind of global announcement. But if we blindly introduce everything that we are asked for, nothing good will come of it.

(sarcastically) Yes, we are going to implement a global announcement, add this and that, fifth, tenth, pull out another, and as a result realize that the final solution does not satisfy most of the programmers, and we will not add all the features that we are asked for, and therefore we do not implement anything. After all, strict mode is a reasonable compromise.

There is a problem: most often we use, for example, the fields inside the modules, and then you again have the same problems. This is only one very specific case of errors, which should probably be taken into account in the overall solution. So if you really need it, then use a statically typed language.

- Global default is still convenient for small configuration files.

- Yes, that's right, for small scripts and the like.

- And no compromise in this case?

- There are always compromises. In this case, a compromise between small scripts and real programs, something like that.

“And here we return to the first big question: what three things would you change if there was such an opportunity?” It seems to me that you are completely satisfied with the status quo, right?

“Well, I would not call it a big change, and yet ... One of the unsuccessful decisions that became a big change is nil in tables (nils in tables). I regret it. I made this hack ... you know what I'm talking about? C half a year or a year ago I released a version of Lua, in which there were nil in the tables.

- Nil values?

- Exactly. I think this is called nil in the tables — that is, null. We made a grammar hack to ensure compatibility.

- Why did it take?

“I’m really convinced that this is a whole problem ... I think most of the problems with nil in arrays would disappear if we could have [nil in tables] ... More precisely, the problem is not with nil in arrays. They tell me that arrays cannot be nil, so arrays should be separated from the tables. The real problem is that we can't keep nil in the tables! Therefore, the problem is not in how we represent arrays, but in tables. If we could have nil in tables, we would have nil in arrays without everything else. So I’m very sorry about this, and many people don’t understand how things would change if Lua allowed to place nil in tables.

- Can I tell a story about Tarantool? We have our own implementation of null, which is a CDATA on a null pointer. We use it in those cases when it is necessary to create empty fragments in memory to insert positioning arguments for remote calls, and so on. But usually this causes problems, because CDATA is always converted to "true". And nil in arrays would solve many problems.

- Yes I know. That's what I’m talking about - it would solve a lot of problems for many, but this leads to an even greater compatibility problem. We don’t have enough courage to release a version that is so incompatible, and then destroy the community and release other documentation for Lua 5, Lua 6, etc. But maybe one day we will release such a version. It would be a very big change. I think that it was necessary to do so from the very beginning, and then we would talk about a trivial change in the language, if not counting compatibility. And today it will break a lot of programs.

- What are the disadvantages you see besides the compatibility issue?

- Two new primitive functions will be needed. Something like the "Delete key", because assigning nil will not delete the key, so a primitive operation is needed to actually remove it from the table. And we need “test” to check where exactly there is a difference between nil and the lack of data.

- Have you analyzed the impact of these innovations in practical implementations?

- Yes, we released this version of Lua. As I said, it imperceptibly broke the code. Some use the function call table.insert(f(x)) . And intentionally done so that if the function does not want to insert anything, it returns nil. So instead of a separate check for “do I want to insert?” I call table.insert , and if I get nil, I know that I will not insert it. As in any other language, the bug has become a feature, and people use it. But if you change the feature, it will break the code.

“What about the void type?” How nil, only void?

- Well, no, this is a nightmare. You simply postpone the solution. If you enter one “crutch”, then you will need another one, and another, and another. This is not a solution. One of the problems is that nil is already ingrained in many places in the language. Here is a typical example: we say “avoid nil in arrays, i.e. holes”. And then we write a function that returns nil and something after it, and we get the error code. Such a construction itself implies that it is nil ... For example, if I want to make a list of the results returned by a function, just to catch them all.

- For this you have a hack :)

“Quite right, but you don’t have to use hacks to solve such simple and obvious tasks.” However, libraries do this ... Somehow I thought about it: perhaps the libraries should return false instead of nil, although this is a crude option, it will solve only a small part of the problem. The real problem, as I said, is that we need to have nil in the tables. Otherwise, we should not use nil as often as it seems to me. In general, it is not clear. So if you create a void, these functions will still return nil, and we will not solve the problem until we create a new type and the functions will not return void instead of nil.

- With the help of void, you can explicitly say that the key should be stored in a table, a key with the value void. And nil can work as before.

- Yes, that is what I mean. All these functions in libraries must return void or nil.

- They can also return and nil, why not?

- Because then for some cases we will not solve the problem of the impossibility of capturing all the values ​​returned by the function.

- But we will not have the first key, only the second.

- No, there will be no second, because the counting will be conducted incorrectly and a hole will appear in the array.

- So you want to say you need a fake metamethod?

- Yes. I dream of this:

{f(x)}

It is necessary to intercept all the results returned by the function f(x) . And then I can say %x or #x , and thus find out the number of results returned. This is how a normal language should work. So creating a void problem will not solve until it is very strict that functions never return nil. But then why do we need nil at all? Perhaps, it is necessary to refuse it.

- Roberto, will there be much stronger support for static analysis in Lua? Like "Lua Check on Steroids"? Of course, I understand, it will not solve all the problems. You said that this feature will appear in version 6.0, if it appears at all, right? And if in 5.x there will be a powerful static analysis tool - if man-hours are embedded in it - will it help?

- No, I believe that a truly powerful static analysis tool is called a ... type system! If you need a really powerful tool, use a statically typed language, like Haskell, or some kind of language with dependent types. Then you will definitely have a powerful analysis tool.

- But then I won't have Lua.

“True, Lua is needed for ...”

- Uncertainties? I like your picture with a giraffe about static and dynamic types.

- Yeah, this is my last slide from the presentation.


The last slide of the presentation for the speech of Roberto Jerusalem “Why Lua? (and why not) at the conference Lua in Moscow 2019.

- To ask the next question, back to this picture. If I understand correctly, you think that Lua is a small convenient tool for solving not very large tasks.

- No, I believe that you can solve some big tasks, but not implying static analysis. I believe in tests with all my heart. By the way, I do not agree with you regarding the coverage that we should not chase the coating ... I want to say, I completely agree with the opinion that the coating does not provide complete testing, but the lack of coverage does not allow testing at all. I spoke in Stockholm about the test room, you were there. I started testing with [several] bugs - this is the strangest thing - one of them was widely known, and the others are completely unknown. Something broke down completely in the Microsoft, C and C ++ header file. I searched the net, but no one was worried about it, or did not even notice it.

For example, there is a mathematical function modf () , which needs to pass a pointer to a double value, because it returns two doubles. We convert the integer or fractional part of a number. This function has long been part of the standard library. Then C 99 appeared, and this function is now needed for float numbers. And the Microsoft header file just saved this function and declared another one as a macro. That is, the first function was subjected to implicit type conversion. Double was converted to float, and then a pointer to a double was converted to a pointer to float!

- There is something wrong with this picture.

- This is a header file from Visual C ++ and Visual C 2007. If you call this function once, with any parameters, and then check the result, it will be wrong, unless it is 0. Any other value will not be correct. You can never use this feature. Zero coverage. And then there was a lot of discussion about testing ... Just calling the function once and checking the result! This problem has existed for a long time, for many years it did not bother anyone. Apple had a very famous bug, something like " if… what… goto… ok ". Someone put in another expression here, and everything became normal. And then they argued a lot about whether rules were needed, whether curly braces should be used in code style, and so on and so forth. No one mentioned that there are many other "ifs". No one has done this ...

- There is also a security issue, as I recall.

- Absolutely. After all, they only test confirmed situations. They do not test everything because everything will be confirmed. This means that in an application that checks security, there is not a single test that would check the failure of any connections, or what should be denied there. And everyone is arguing about whether brackets are needed ... We need tests, at least tests! After all, no one even tested what I mean by "coating." It's incredible, people don't even get rid of basic tests. Of course, it would be very difficult to provide coverage with basic tests, to get rid of all lines of code. People avoid even basic tests, so coverage is minimal. I want to encourage you to pay attention to those parts of the program that you forgot about. Something like hints on how to improve your tests a little.

- Do you know what test coverage in Tarantool? 83%! And what is the coverage in Lua?

- About 99.6%. How many lines of code do you have? Million, hundreds of thousands? This is a huge amount. 1% of one hundred thousand is a thousand lines of code that have never been tested. You did not perform them at all. Your users are not testing anything.

- So, about 17% of Tarantool functions are not used now?

“You hardly want to go back to what we said ... I think one of the problems of dynamic languages ​​(and static too) is that people do not test their code.” Even using a static language - not like Haskell, but something like Coq - then as long as you don’t have a verification system, you’ll change it to that. And no static analysis tool can catch these errors, so you need tests. And if you have them, you can identify global problems, typos in names, and so on, all sorts of errors. You definitely need tests. Sometimes it will be difficult to debug, otherwise it is simple - it depends on the language and type of the bug. But the bottom line is that no static analysis tool can replace tests. On the other hand, they do not guarantee the absence of errors, but with them I feel much more confident.

- We have a question about testing Lua-modules. I, as a developer, want to test some local functions that I can use later. Question: Coverage should be at the level of 99%, but the number of functional situations that the module for the API should generate is much less than the amount of functionality that it supports internally.

- Excuse me, why?

- There is functionality that is not available for public interfaces.

- It should not be there, just delete this code.

- Just delete?

- Yes, I sometimes do this in Lua. There was some kind of code coverage, I could not get here, there or there, I decided that this was impossible and just deleted the code. Rarely, but it happens. If any situations are impossible, just write in the comments why it is impossible. If you cannot get from a public API into a function, it should not be. You need to write public APIs with erroneous input data, this is important for tests.

- Delete the code well, it reduces the complexity. Less complexity - higher stability and ease of maintenance. Do not complicate.

- Yes, in extreme programming there is such a rule. If something cannot be tested, it does not exist.

- What languages ​​were you inspired when creating Lua? What paradigms, or functional features, or parts of which languages ​​do you like?

- I designed Lua for a very narrow purpose, it was not an academic project. So when you asked me about re-creating the language, I replied that there is a whole story behind this. I did not start developing Lua with “I will create a language that I like, or I want to use, or that everyone needs.” I had a problem: a configuration language for geologists and engineers is needed here, it should be small and with a simple interface so that they can use it. Therefore, the API has always been an integral part of the language. That was the task. As for inspiration, at that time I was familiar with about ten different languages.

“I’m wondering what languages ​​you wanted to include in Lua

- I borrowed ideas from many languages, all that was suitable for solving my problem. The synula of the Modula language had the greatest impact, although it is difficult to say, because there are so many languages. Something took from awk. Of course, from Scheme and Lisp ... I've been partial to Lisp since I started programming.

“And there are still no macros in Lua!”

- Yes, the syntax is completely different. Probably the first language was Fortran ... no, my first language was assembler, and then Fortran. I studied, but never used CLU. Many programmed on Smalltalk and SNOBOL. I also studied, but did not use Icon, also a very interesting language. I borrowed a lot from Pascal and C. When I created Lua, C ++ was too complicated for me, it was before templates and so on. I started work in 1991 and released Lua in 1993.

- The USSR collapsed and you started creating Lua :) You didn’t like semicolons and objects when you started Lua? It seemed to me that its syntax would be similar to C, because Lua is integrated into it. But...

- I believe that the syntax should be different, then you will not get confused, because these are two different languages.

This is really funny, and is connected with the answer about the arrays beginning with the unit, which you did not allow me to express [at the conference]. My answer was too long.

When Lua was released, the world was different. Not all languages ​​were similar to C. There was no Java and JavaScript yet, Python was a baby and did not pass for version 1.0. So not all languages ​​were similar to C, its syntax was one of many.

The same with arrays. It’s funny that most of them don’t realize it. Arrays, starting with 0 and 1, have their merits.

It is thanks to C in most popular languages ​​today arrays are used, starting with 0. Their creators were inspired by C. And it’s funny that there is no indexing in C. , , . , — , (offset). , , , , .

, , . Java, JavaScript — . 0, . , « ».

— , , . -, , , - , , . , Lua ? Why?

— ?

— .

— . , , . . ? , , . . .

— Lua C.

— , , . , , … , , - . : , … .

, . , ? , - … ?

— .

— . ? C , , … ?

— 16 ?

— , .

— , , .

— , . , , … , , . — , . , , . , … : ?

— C++ .

— , C++ .

— ? ptrdiff_t ?

— ptrdiff_t — (signed type). , , . ?

, diff , . . , ? No , , . 2 , .

, . , . diff , .

, ++.

— Lua ?

— , C++, - , . , - ++, … .

— , ?

— . , . , , . , .

— JVM?

— , JVM. , … — , . JVM , .NET, . JVM , Lua. , . JVM, . JVM . Java-, 10 . , JVM , .

— JVM, , Mobile JVM?

— , JVM. - Java, Java.

— , Go Oberon? Lua, ?

— Oberon… … Go , runtime- Lua. Oberon , . , , . , , const Pascal Oberon. , . .
, 1994- Oberon Self. Self? JIT- . Self , , . - , — ! — , - . , …

Oberon, , 10 Self, . Oberon , , .

, .

— Haskell?

— Haskell, , Lua.

— Python, R Julia Lua?

— , .

R . , .

Python , . , . , .

Python , , . , , - . API… , Python, . , , « ».

, -: , , , -. . , API , - . , . , , …

- , (pattern matching). , , , . , , .

: Perl. Lua, . , Python . , , . - .

— ?

— Python. Perl : $1, $2, $3. Perl, …

— Python, , ( Tarantool).

— , , , API, . Python , .

Julia, LuaJIT, , . , , . , . , , , . , , . : , - . , , - , .

Julia, : , , . , - . , double, []… . .

() «, , , , , . , ». , , .

. R, Python.

— Erlang?

— . , . , , , - . Erlang , , . , .

, . , . ? , . .

— , Erlang , Python . Lua?

— , . Lua , , Lua , .

— ?

— Forth, .

— Lua?

— , . , . - , . Lua, Lua, .

Java. Java, ? Not. (reflection)? Not. ? Java, , Java. , , Java, .

, Lua, … , , FFI.

— Lua?

— , .

— Lua ? ?

— , . , Haskell. , , … Lua, , , , , .

— , Lua.

— , , . . , .

— , . Lua ?

— , , …

— « »? , ?

— , . , , , . , .

— Lua?

— . , , LaTex DocBook. , … LaTex, . @, . Gsub , , - - . , , , .

— LaTeX?

— LaTeX? -, , . , , LaTex. , inline-. /, . — . , , , . , . , .

— LaTeX?

— , . , , . , 3+1, . , , . , . , , . , 1 «and», . . , .

— ?

— , git . 2html . HTML… , . , , . , , . , TeX. TeX- TeX.

— ?

— , . , TeX. , . DocBook, . , .

— 2html DocBook?

— , DocBook.

— , , !

— .

---

- , Lua Mailing List .