In the first part, I argued that it was time to think about how to replace a modern web platform for applications. The reasons are its low productivity and, in principle, unsolvable security problems.

Some have decided that I am writing too negatively and not paying attention to the positive aspects of the web. So it is: the first part was in the style of “Let's discuss the fact that we fell into a deep hole”, and the second part - “How to develop something better?” This is a huge topic, so in fact it will not be limited to two parts.

Let's call our competitor the web NewWeb (er, you can do branding later). First you need to understand why the web initially became successful. The web has bypassed other application creation technologies with the best GUI development tools, so it clearly has some qualities that outweigh the disadvantages. If we do not meet these qualities, we are doomed.

GUI creation in Matisse, UI editor with auto-alignment and delimiters. Long live the (new stylish) king?
')
It is also necessary to concentrate on cheapness. The web has numerous development teams. Most of their work is duplicated or discarded, some can be reused. A new development in a small amount is also possible ... but by and large any NewWeb technology will have to be assembled from pieces of existing software. The poor do not have to choose.

---

Here is my personal list of the top five web properties:

• Sandbox Deployment and Isolation
• Easy to learn users and developers
• Eliminate document / application dichotomy
• Advanced styling and branding
• Open source and free use

This is not exactly what is generally considered the main principles of the architecture: if you interview the developers, most of them are likely to be called the essence of the architecture of the Web URL, viewing the source code, cross-platform, HTTP, and so on. But these are all just implementation details. The Web has got the upper hand over Delphi and Visual Basic not because JavaScript is so cool. There are deeper reasons.

The above things are, in principle, quite understandable. I will analyze them in more detail during the article. To defeat the web, it is not enough just to repeat its strengths, we must go further and offer unique improvements that the web is not able to integrate easily. Otherwise it makes no sense: instead of our project, we will simply work on improving the web.

In this article, I propose two things: certain principles of architecture, which, in my opinion, should be followed by any serious competitor of the web, and a concrete example compiled from various open source projects. Many readers will not like a specific example, because they like other projects or programming languages, there is nothing to worry about. I do not care. This is just an illustration - the principles are really important. I call specific code bases just to illustrate that these dreams are not completely unrealistic.

Principles of architecture

The web lacks an intelligible philosophy of architecture, at least in terms of applications. Here are some of the necessary things he lacks, in my opinion:

1. Clear concept of application ids.
2. Single view of data from backend to frontend.
3. Binary protocols and APIs.
4. User authentication at the platform level.
5. IDE oriented development.
6. Components, modules, and a great UI layout system - just like in the usual development for desktop computers or mobile devices.
7. You will also need things that the web does well: instantly launching streaming applications without installation or the need for manual updates, isolating these applications in the sandbox, complicated UI styling, a combination of and alignment of “documentary” and “software” material (such as the ability to link certain types of applications), the possibility of gradual learning and architecture, which does not make it difficult for developers to create too complex UIs.

The first four points highlighted in bold are related to security, because it was security issues that forced me to take such a radical position. Perhaps the poor performance of the web can be fixed with a new JavaScript framework - maybe. But I do not think that you can fix security problems. In subsequent articles, I will consider items that are not bold.

Application IDs and Links

To get the benefits of the web in deploying and isolating in a sandbox, you need some kind of browser for applications. It would be nice to be able to refer to parts of the application as parts of a hypertext document. This is a key element of web success: the Amazon search results page looks like an application, but we can link to it, so this is also a document.

But our application browser should not physically resemble a web browser. Take a fresh look: Web browser UI is not perfect.



The URL is the main part of the design of any browser, but sometimes it causes confusion!

The first problem is that URLs get into the UI everywhere. The address bar constantly contains random bits from the browser's memory, encoded in a form that is difficult for both people and machines to understand, which leads to a parade of exploits. If the desktop application would unload random internal variables into the header row, we would consider this a serious bug that threatens the company's reputation, so why do we have to endure it here?

The second problem is that it is also difficult for machines to perceive URLs ( it is possible to create crazy exploits here ). Without even considering clever coding tricks like this, there are various ways on the web to establish the identity of an application. The browser requires a certain notion of the identity of the application for separating cookie stores and for tracking issued permissions. This is the "origin". Over time, the concept of the source on the web has evolved, and now it is essentially impossible to formulate. RFC 6454 attempts to do this, but the document itself says:

Over time, many technologies converged on the concept of a source as a convenient isolation unit. However, many of the technologies currently used, such as cookies [RFC6265], were created earlier than the modern concept of a source on the web. These technologies often have different isolation units, leading to vulnerabilities.

For example, consider what prevents your server from setting cookies for a .com domain. What about kamagaya.chiba.jp? (this is not a website, but just a section of the hierarchy , like. com!)

The ability to unexpectedly put a link to a part of an application that does not receive any benefit from it and does not expect this is one of the sources of the “dangerous links” problem. Strongly URL-based design puts your entire application in danger of data injection by unauthorized intruders. This is a design limitation that is known to be almost impossible to fight against it, but it is carelessly encouraged by the web architecture itself.

However, the ability to put a link to the middle of an application from documents and vice versa would be very pleasant if everything works as intended by the developers. So, we formulate a couple of requirements:

Requirement: The application's identity (isolation) must be simple and predictable.

Requirement: we should be able to put a deep link to the application, but not always.

To their credit, Android architects understood these requirements and offered solutions. From here we can start:

• The identity of the application determines the public key. All documents signed with one key are isolated in the same domain. To determine isolation, no line breaks are used.
• Applications can subscribe to receive strongly typed data packets that ask them to open in some state. Android calls this concept “intents” (intents). Intentions are not really very strongly typed in Android, but they could be. They can also be used for internal navigation, but to allow links from some other application, the intent must be published in the application manifest. By default, applications do not respond to links.
• Where to find the application for download? A domain name is a good enough position to start with. Although we can support and download via HTTP [S], but we find the NewWeb server on a well-known port and retrieve the code from there. So the domain name becomes the starting point for getting the application, but otherwise not so important.

To distinguish between URLs and other people's intentions from our own, let's call someone else's link packages .

Disabling the reception of external links by default will lower the link connectivity of NewWeb, but will improve security. Maybe this is a bad compromise and it will be fatal. But many links that people can theoretically create in the modern web are essentially useless either because of authentication requirements, or because they do not contain any meaningful initial state (many SPAs). Therefore, applications minimize the attack area. Malicious link, which is the starting point of such a large number of exploits, immediately becomes less dangerous. This seems like a valuable accomplishment.

But there are other advantages - the importance of a domain name for web applications makes it tempting for providers to confiscate domains that their leadership does not like . Although simple sites can be transferred to another domain without any problems, more complex sites can value the established reputation of postal addresses, OAuth tokens, and so on, for them this is a more painful procedure. Secret keys can be lost or stolen, but the same with domain names. If you have lost the secret key - at least it is only your fault.

As for the rest of the browser UI, you can probably get rid of them. Tabs can be useful, but the page reload button should never be used, and the return button to the previous state can be embedded in the application itself, if it makes sense, a la iOS.

How to create an application browser? Although NewWeb is quite different from the web, the basic UI of full-screen applications is quite standard, and users will want to switch back and forth. So why not fork Chromium and add a tab in the new mode?

(Edit: some have understood the above as “using Chrome for everything listed here” - this is not what I meant. I meant using its tabbed UI implementation so that you can have NewWeb and OldWeb open next to you. But a tabbed UI is easy to do, and it’s not necessary to use Chromium (the application browser can also be an application created from scratch).

Single view of data

Maybe the link package concept sounds a bit vague. What it is? For a clearer definition, we need data structures.

There are many ways to do this on the Web, but they are all text based. I recall the thesis from my previous article: text protocols (not only JSON) have a fundamental vulnerability: this is “in-band” signaling of the buffer, that is, to find out where the data ends, you need to read them all in search of a specific sequence of characters. These sequences may be a legal part of the data, that is, you need a screening mechanism. And then, since these protocols are supposed to be human-readable or at least unreadable, we often have strange borderline cases with handling spaces or canonized Unicode, which leads to exploits, like HTTP header splitting attacks.

In the old days, text protocols helped web developers. Browsing the source definitely helped me so many times that I can't count. But now the average web page size is over 2 megabytes , not to mention web applications. Even boring web pages with static text often contain a lot of minified scripts that you cannot even begin to understand without machine assistance. The benefits of text protocols seem less than in past times.


Primitive decompiler

To be honest, in recent years, the web has been slowly rejecting text protocols. HTTP / 2 binary. And the advertised “WebAssembly” is a binary way of expressing code, although it doesn't really solve the problems we talked about. But nonetheless.

Requirement: data serialization should be automatic typed, binary and unchanged from data storage to fronted.

Serialization code is not only tedious to write, but it is also a serious attack vector. A good platform should take on the challenge. Let's define in the most primitive form the syntax for expressing data structures:

enum SortBy { Featured, Relevance, PriceLowToHigh, PriceHighToLow, Reviews } @PermazenType class AmazonSearch( val query: String, val sortBy: SortBy? ) : LinkPacket 

On the web, the equivalent of this is the URL on amazon.com. It defines the data structure of an immutable type to indicate a request to open an application in a certain state.

This data structure is marked as @PermazenType . What does it mean?

If we are serious about the code with the prefixed length, the strict typification of protection against injections throughout the stack, then we have to do something with SQL. Structured query language is a pleasant and well-understood way to express complex queries to a variety of extremely powerful database engines, so it is a pity. But SQL is a text API. Although SQL injections are one of the easiest types of exploits that are easy to understand and fix, they are also one of the most common bugs on websites. No wonder: if you use SQL on the web server in the most obvious way, it will work fine, but quietly make the server vulnerable to hacking. Parameterized queries help, but this is a screwed-up solution from the side, which not every situation can be used. The technological stack creates a well-disguised bear trap for all of us, and our goal is to minimize the balance of risks and functionality.

SQL has some other problems. You will quickly encounter the problem of object-relational mapping. The results of executing a SQL query cannot be natively sent over an HTTP connection or embedded in a web page, so you always need some kind of transformation to another format. SQL hides the performance of the underlying queries from you. Backend is difficult to scale. Schema changes often require table freezing, so that they cannot be deployed without unacceptable downtime.

NoSQL engines are not much better: they usually fix one or two of these problems, but at the expense of discarding SQL solutions for everything else. In the end, you often get stuck with a solution that is different, but not necessarily the best. As a result, the largest companies like Google and Bloomberg spend a lot of time trying to find a way to scale SQL databases ( F1 , ComDB2 ).

Permazen is a new data storage approach that delights me at the moment. Quote from their site:

Permazen is a completely new approach to robust programming. Instead of starting development from the storage technology side, he starts from the programming language side, asking a simple question: “What problems are inherent in robust programming, regardless of the programming language or DBMS technology, and how can they be solved at the language level with the simplest, most correct? and in the most natural way from the point of view of language? "

Permazen is a Java library. However, its architecture and technology can be used on any platform or language. She needs a sorted key-value repository of the type that many cloud providers can provide (she can also use RDBMS as K / V storage), but everything else is done inside the library. It does not have tables or SQL. Instead, she:

• Uses host language operators to implement joins and intersections (connections).
• Schemas are directly defined by classes; you do not need to set an object-relational mapping.
• It can produce a step-by-step evolution of the “just-in-time” scheme, eliminating the need to freeze the table to change the presentation of data in the repository.
• Transactions are established unequivocally, and copying data from a transaction and back is clearly controlled by the developer.
• Tracking changes so that you can receive callbacks when the underlying data changes, including between processes (if the K / V repository supports it, and often it does).
• There is a command line interface that allows you to query and work with the data store (for example, to initiate data migration if you are not satisfied with the schedule migration), and the GUI.

In fact, you just need to read a great report or watch the slides (the slides use the old library name, but this is the same software). Permazen is not very well known, but it is the smartest and clever approach to closely integrating a data warehouse with an object-oriented language I have ever seen.

One of the interesting features of Permazen is that you can use “transaction snapshots” to serialize and deserialize the object graph. This snapshot even includes indexes, that is, you can broadcast data to the local storage if the memory is low and you can already perform queries on the index. It should now become clear how this library unifies data storage with support for offline synchronization. Conflict resolution can be done transactionally, since all Permazen operations can be performed inside serializable transactions (if you want conflict-free work in the style of Google Docs, which requires an operational conversion library).

It is not necessary to use this approach in NewWeb. You can choose something a bit more traditional, like protobuf, but then you have to use a special IDL, which is equally inconvenient in all languages, and tag the fields. You could use CBOR or ASN1. In my current project, Corda, we created our own object serialization engine built on AMQP / 1.0. In it, by default, the messages describe themselves, so for each particular binary package you always get a schema - and therefore the source view function is available there, as in XML or JSON. AMQP is preferred because it is an open standard, and the basic type system is pretty good (for example, it recognizes dates and annotations).

The point is that it is risky to take data from potentially harmful sources, so that within the format it is desirable to get as many degrees of freedom as possible and the maximum number of integrity checks. All buffers must be prefixed with length, so users cannot attempt to inject malicious data into the field — such attempts will stop at an early stage. The data must be carefully checked at boot time, and the best place for such a check is the type system and the constructors of the data structures themselves, a way that you might have forgotten about. Schemes help to understand what the structure should be, preventing intruders from using type mixing.

Despite the absolutely binary nature of the proposed scheme, it is possible to make a one-way conversion to text for debugging and educational purposes. In fact, Permazen and it can.

Simplicity and phased learning

Type systems have one problem - they add extra complexity. To developers who have not encountered types, they seem to be some sort of meaningless bureaucracy. And without the proper tools, binary protocols can be more difficult to learn and debug.

Requirement: easy to learn

I am sure that a significant part of the success of the web is explained by the fact that it is not typed in essence. Everything here is just strings. Bad for safety, performance and ease of maintenance, but very good for training.

One way to deal with such a structure in the world of the web is a gradual typing in JavaScript versions. This is a good and valuable research work, but such dialects are not widely used, and most new languages ​​are at least partially strictly typed (Rust, Swift, Go, Kotlin, Ceylon, Idris ...).

Another way to fulfill this requirement can be a smart IDE: if you allow developers to first work with untyped structures (all defined as Any ), the runtime can try to determine what the original types should be and translate this information back to the IDE. She can then suggest a replacement for the best type annotations. If the developer encounters type conversion errors while the program is running, then the IDE may offer to relax the restriction again.

Of course, this approach provides less security and ease of maintenance than if you force the developer to think over the types in advance, but even relatively weak heuristically exposed types, which often lead to runtime errors, still protect against a large number of exploits. Runtimes like JVM and V8 already collect information about these types. Java 9 has a new API that allows you to control the virtual machine at a low level (JVMCI), it is engaged in such profiling - now it would be much easier to experiment with such tools.

Languages ​​and virtual machines

What language should NewWeb use? Of course, this is a difficult question: the platform should not become fanciful.

Over the past years, there have been many attempts to introduce other languages ​​to the web, except JavaScript, but all of them were unsuccessful without receiving consensus from browser developers (mostly from Mozilla). In fact, in this respect, the web goes back in time - you could run Java, ActionScript, VBScript, and other languages, but browser vendors systematically removed all non-JavaScript plugins from the platform. This is a shame. Of course, it was possible to keep the competition. The sad verdict to the web platform is that WebAssembly is the only attempt to add a new language, and that language is C ... in which I hope you don’t want to write web applications! XSS is enough for us to add top-level vulnerabilities like double-freeing the same memory.

It is always easier to agree with a problem than with a solution, but that's okay - the time has come to put forward a (more) contradictory thesis. The core of my personal NewWeb design would be the JVM. It will not surprise those who know me. Why JVM?

• HotSpot supports more different programming languages ​​than any other virtual machine, and the low cost principle dictates the maximum possible use of open source code.
  
  Yes, it means that I want to leave the possibility to use JavaScript (at high speed), in addition to Ruby , Python , Haskell , and yes - C, C ++ and Rust will also remain in the game . All this is possible thanks to two really cool projects called Graal and Truffle, which I talked about in detail . These projects allow the JVM to even run code in unexpected languages ​​(such as Rust) in a virtualized environment, quickly and interoperably. I do not know of any other virtual machine that organically blends together so many languages ​​and does so with such high performance.
• Over the years, the OpenJDK sandbox has undergone stubborn attacks, and browser developers have learned a number of painful lessons. The last 0-day exploit is dated 2015, and the previous one was in 2013. Only two 0-day sandbox exploits in five years are not bad, in my opinion, especially given the fact that after each of them lessons were learned and fundamental safety improvements were made over time. No sandbox has an ideal reputation, so in reality it all comes down to personal preferences - what level of security should be considered sufficient ?
• There are a huge number of high-quality libraries that stack key puzzle pieces, such as Permazen and JavaFX.
• It so happened that I know the JVM quite well and I like Kotlin, which is suitable for the JVM backend.

I understand that many will disagree with my choice. No problem - the same ideas of architecture can be implemented on the basis of Python, or V8, or Go, or Haskell, or whatever you are swimming into. It is better to choose something with open specifications and the presence of competing implementations (like the JVM).

Oracle policy does not bother me, because Java has open source. Among the high-quality and high-performance open source runtimes there is a small selection. Existing projects are created by large corporations that have messed themselves with controversial decisions in the past. At various stages of its development, the web was influenced or directly controlled by Microsoft, Google, Mozilla and Apple. All of them committed acts that I consider reprehensible: this is typical of large companies. You do not want to always agree with their actions. Oracle is unlikely to win the popularity contest, but the advantage of open source is that it will not have to participate in it.

Specifically, from Google Chrome, I would have borrowed something. My application browser should support the equivalent of WebGL (i.e. eGL), and the Chromium ANGLE project is suitable for this purpose. The application browser should be automatically updated invisibly, as Chrome does, and Google's automatic update engines are also distributed under a free license. In the end, although the Pack200 format strongly compresses the code, using quality codecs like Zopfli would not hurt.

RPC

Binary data structures are not enough. Binary protocols are also needed. The standard way to associate a client with a server is RPC.

Currently, one of the coolest British startups is Improbable . Recently, developers have published an excellent blog post about how they switched from REST + JSON to gRPC + protobuf from browser to server . Improbable describes the result as "safe typing nirvana." Browsers do not do it as easily as with HTTP + XML or JSON, but using the necessary JavaScript libraries, you can tie the required stack on top. It's a good idea. If you return to the real world, where we all write web applications, you definitely should consider this option.

Existing experience should be used in the development of RPC protocols. My perfect RPC supports:

• Return of futures (promises) and ReactiveX Observables , so that you can broadcast push events in a natural way.
• Returning large or infinite byte streams (for example, for live video feeds).
• Flow control when transmitting traffic with low / high priority.
• Versioning, interface evolution, type checking.

Again, in the Corda project, we design the RPC stack with exactly these properties. We have not released it as a separate library yet, but we hope to do it in the future.
HTTP / 2 is one of the most recent and most developed parts of the web; it is quite a decent frame transport protocol. But he inherited a lot of trash from HTTP. Just imagine what hacks become possible if you refuse HTTP methods that are never used anyway . HTTP's weird approach to state descriptions is not needed. HTTP itself does not change the state in the execution process, but applications need state-storing sessions, so application developers have to add their own implementation over the protocol, using a mix of easily stolen cookies, tokens, and so on. This leads to problems like session commit attacks .

It is better to divide everything by layers more clearly:

• Encryption, authentication and session management. TLS copes well. You can just use it.
• Transport and flow control. HTTP / 2 does this, but messaging protocols like AMQP also do the job, and without the burden of legacy of the past.
• RPC

Obviously, the RPC stack is integrated with the data structure framework, so all data transfers are typed. Developers will never need to parse manually. Naming in RPC is a simple string comparison. Therefore, there are no vulnerabilities going beyond the current directory (path traversal).

User Authentication and Sessions

NewWeb will not use cookies. For identification of sessions, pairs of public and private keys are better suited. The web is moving in this direction , but to maintain compatibility with existing web applications a complicated “binding” stage is required, in which the owner's tokens, such as cookies, are connected to the carrier cryptographic session, and this adds additional complexity to an important security component - this stage can be abandoned. full redesign Sessions are identified on the server side only by the public key.

User authentication is one of the most difficult things that is difficult to implement correctly in a web application. I previously expressed a few thoughts on this matter , so I will not repeat. Suffice it to say that the binding of the session to the postal address is better to perform on the side of the basic platform, and not to constantly reinvent the wheel at the application level. The TLS certificate on the client side is enough to implement a basic single sign-on system, where the workflow like “send a letter and sign a certificate if received” is so cheap that providers of free certificates in Let's Encrypt style will become quite real.

This approach is based on existing technologies, but it will seriously reduce the number of phishing, password cracking, brute-force, and many other security-related issues.

Conclusion

A platform that wants to compete with the web should seriously bother with security issues, since this is the most important rationale for its creation and a competitive advantage. This is the main thing that cannot be easily fixed on the web. That is, we need: binary data structures with secure typing and binary APIs, RPC, cryptographic sessions and user identification.

The new web also has code isolation in the sandbox and streaming of content, a UI with good prototyping, but at the same time well responsive and with style support, some way to mix documents and applications together, as the web does, and a very productive development environment.We should also consider the new web policy.

We will discuss these issues in the next article.