At the beginning of last year, I had the idea to write my own interface language (IDL), which would be similar to Protobuf or Thrift, but would be intended for the web. I was hoping to finish it somewhere in three months. A little more than a year passed before the first stable version.

Pdef (protocol definition language) is a statically typed interface description language that supports JSON and HTTP RPC. It allows you to once describe the interfaces and data structures, and then generate code for specific programming languages. Pidef is suitable for public api, internal services, distributed systems, configuration files, as a format for storing data, cache and message queues.

Main functionality:

• A developed system of packages, modules and namespaces.
• Support for cyclic imports and type dependencies (with some limitations).
• A simple type system based on a clear separation of interfaces and data structures.
• Inheritance of messages (analog structs) and interfaces.
• Support for call chains, for example, github.user(1).repos().all() .
• JSON as data format and HTTP RPC for data transfer.
• Ability to use other formats and RPC.
• Plug-in code generators (officially supported by Java , Python and Objective-C ).
• Optional code generation, i.e. Pidef allows you to serialize data and send requests by hand.

Why do I need Pidef? Primarily to increase productivity and simplify the development and support of client-server, service-oriented and distributed code. But it also combines the documentation and description of api and allows you to build vertically integrated systems in which the overhead costs of the interaction of individual components are reduced.
')
Message description example:

 message Human { id int64; name string; birthday datetime; sex Sex; continent ContinentName; } 

Examples of use ( examples of generated code ):

Installation

Pidef consists of a compiler, plug-in code generators and bayindings specific to specific programming languages. The compiler and code generators are written in Python.

Installing the compiler as a Python package with PyPI :

 pip install pdef-compiler 

Or, you can download an archive with a specific version from the project's release page , unzip it and execute:

 python setup.py install 

Installing code generators (download links are on specific language pages):

 pip install pdef-java pip install pdef-python pip install pdef-objc 

That's it, the compiler is ready to use. You can run the following command to make sure everything is set correctly. She will download the sample package and check it out.

 pdefc -v check https://raw.github.com/pdef/pdef/master/example/world.yaml 

Each code generator during installation adds its own commands to the compiler, you can see them in the help:

 pdefc -h ... generate-python Python code generator. generate-objc Objective-C code generator. generate-java Java code generator. pdefc generate-python -h 

Using

Create a package file myproject.yaml :

 package: name: myproject modules: - posts - photos 

Create the module files:

 //  posts.pdef namespace myproject; import myproject.photos; interface Posts { get(id int64) Post; @post create(title string @post, text string @post) Post; } message Post { id int64; title string; text string; photos list<Photo>; } 

 //  photos.pdef namespace myproject; message Photo { id int64; url string; } 

Run code generation:

 pdefc generate-java myproject.yaml --out generated-java/ pdefc generate-python myproject.yaml --out generated-python/ pdefc generate-objc myproject.yaml --out generated-objc/ 

Code generators support the mapping of Pidef’s modules and namespaces to specific programming languages. More details can be found in the command description

Guide to Pidef 1.1

Syntax

Pidef syntax is similar to Java / C ++ with an inverted order of types and fields / arguments. All identifiers must begin with a Latin character and contain only Latin characters, numbers and underscores. Grammar Description (BNF).

Example:

 namespace example; interface MyInterface { method( arg0 int32, arg1 string ) list<string>; } message MyMessage { field0 int32; field1 string; } enum Number { ONE, TWO; } 

Comments

There are two types of comments: single-line and multi-line for documentation. Documentation comments can be placed at the very beginning of the module, before determining the type (by message, interface, or enumeration) or before the method. Single-line comments are cut when parsing, multi-line ones are saved and used by code generators for documentation.

 /** * This is a multi-line module docstring. * It is a available to the code generators. * * Start each line with a star because it is used * as line whitespace/text delimiter when * the docstring is indented (as method docstrings). */ namespace example; // This is a one line comment, it is stripped from the source code. /** Interface docstring. */ interface ExampleInterface { /** * Method docstring. */ hello(name string) string; } 

Packages and modules

Packages

Pidef files should be organized into packages. Each package is described by one yaml file, which contains the name of the package and lists the modules and dependencies. Cyclic dependencies between packages are prohibited. Module names are automatically mapped to files. For this, the points are replaced with the system directory separator and the extension .pdef . For example, users.events corresponds to the users/events.pdef . Dependencies indicate the package name and an optional path to its yaml file yaml a space. Dependency paths can be set and overridden when executing console commands.

Sample package file:

 package: # Package name name: example # Additional information version: 1.1 url: https://github.com/pdef/pdef/ author: Ivan Korobkov <ivan.korobkov@gmail.com> description: Example application # Module files. modules: - example - photos - users - users.profile # Other packages this package depends on. dependencies: - common - pdef_test https://raw.github.com/pdef/pdef/1.1/test/test.yaml 

And its file structure (the api directory is optional):

 api/ example.yaml example.pdef photos.pdef users.pdef users/profile.pdef 

Modules and namespaces

A module is a separate *.pdef file with a description of messages, interfaces, and enumerations. Each module immediately after the optional documentation should contain an indication of the namespace. All types in the same namespace must have unique names. Different packages can use the same namespace.

Namespaces in pidef are wider than in Java / C # / C ++ and do not have to follow the structure of files and directories. For the latter, there are module names. Usually, one or more packages use the same namespace. Possible examples: twitter , github , etc.

 /** Module with a namespace. */ namespace myproject; message Hello { text string; } 

Imports

Imports are similar to include in other languages, they allow you to access types from another module in one module. Imports are placed immediately after specifying the module namespace. Modules are imported using the package name and file path without the .pdef extension, with a .pdef instead of a directory separator. When the module name matches the package name, the module can be imported only by the package name.

Separate imports:

 namespace example; import package; // Equivalent to "import package.package" when package/module names match. import package.module; 

Batch Imports:

 namespace example; from package.module import submodule0, submodule1; 

Cyclic imports and dependencies

Cyclic imports are possible as long as the types of one module do not inherit the types of another module and vice versa. Otherwise, you can try to divide the modules into smaller ones or merge them into one file. Cyclic dependencies between types are allowed.

Such restrictions are sufficient to support most programming languages. Interpreted languages ​​like Ruby or Python are also supported, since the Pidef compiler ensures that the modules will have a clear tree order of execution when inheriting, otherwise the modules can be executed in any order. More information about the implementation of circular dependencies in specific languages ​​can be found in the Pdef Generated and Language-Specific Code.

Example of cyclic imports and dependencies:

 // users.pdef namespace example; from example import photos; // Circular import. message User { bestFriend User; // References a declaring type. photo Photo; // References a type from another module. } 

 // photos.pdef namespace example; from example import users; // Circular import. message Photo { user User; // References a user from another module. } 

Name resolution

Within the same namespace, a local type name is used, for example, MyMessage ; within a different namespace.MyMessag full name is namespace.MyMessag .

Type system

Pidef has a simple static type system built on the principle of separation of interfaces and data structures.

Void

void is a special type that indicates that the method does not return a result.

Data types

Primitive types

• bool : boolean value (true / false)
• int16 : signed 16-bit number
• int32 : a significant 32-bit number
• int64 : signed 64-bit number
• float : 32-bit floating point number
• double : 64-bit floating point number
• string : Unicode string
• datetime : date to time without specifying the time zone

Containers

• list : an ordered list whose elements can be any data type.
• set : an unordered set of unique values, the elements of which can be any data type.
• map : container is key-value, keys must be primitive types, values ​​can be any data types.

 message Containers { numbers list<int32>; tweets list<Tweet>; ids set<int64>; colors set<Color>; userNames map<int64, string>; photos map<string, list<Photo>>; } 

Transfers

An enumeration is a collection of unique string values. Enums are also used to specify discriminators for inheritance.

 enum Sex { MALE, FEMALE; } enum EventType { USER_REGISTERED, USER_BANNED, PHOTO_UPLOADED, PHOTO_DELETED, MESSAGE_RECEIVED; } 

Messages and Exceptions

A message (similar to struct 'a) is a collection of statically typed named fields. Messages support simple and polymorphic inheritance. Messages defined as exceptions can optionally be used to specify exceptions in interest.

• All message fields must have unique names.
• The field type must be a data type.
• The field may indicate the message in which it is defined (self-referencing).

 /** Example message. */ message User { id int64; name string; age int32; profile Profile; friends set<User>; // Self-referencing. } /** Example exception. */ exception UserNotFound { userId int64; } 

Inheritance

Inheritance allows one message to inherit the fields of another message or exception. In simple inheritance, descendants cannot be unpacked from the parent, for this there is polymorphic inheritance.

• Cyclic inheritance is prohibited.
• A message can have only one parent.
• Redefinition of fields in descendants is prohibited.
• The child and the parent must be either messages or exceptions, i.e. they can not be mixed.
• A parent must be defined before its descendants, and also cannot be imported from dependent modules (more details in cyclic imports).

Example of inheritance:

 message EditableUser { name string; sex Sex; birthday datetime; } message User : EditableUser { id int32; lastSeen datetime; friendsCount int32; likesCount int32; photosCount int32; } message UserWithDetails : User { photos list<Photo>; friends list<User>; } 

Polymorphic inheritance

Polymorphic inheritance allows you to unpack descendants based on the value of the discriminator field. The parent with all descendants is the inheritance tree. One descendant can inherit another (and not just the parent), but only within the same tree.

For polymorphic inheritance you need:

• Create an enumeration that will serve as a set of values ​​for the discriminator.
• Add a field with the type of this enumeration to the parent message and mark it as @discriminator .
• Specify the discriminator value of each of the descendants as message Subtype : Base(DiscriminatorEnum.VALUE) .

Limitations:

• Parent and all descendants must be defined in one package.
• The type of discriminator must be defined before the parent and cannot be imported from the dependent module.
• In one inheritance tree there can be only one discriminator field.
• You can not inherit a polymorphic message without specifying the value of the discriminator.

An example of polymorphic inheritance:

 /** Discriminator enum. */ enum EventType { USER_EVENT, USER_REGISTERED, USER_BANNED, PHOTO_UPLOADED, } /** Base event with a discriminator field. */ message Event { type EventType @discriminator; // The type field marked as @discriminator time datetime; } /** Base user event. */ message UserEvent : Event(EventType.USER_EVENT) { user User; } message UserRegistered : UserEvent(EventType.USER_REGISTERED) { ip string; browser string; device string; } message UserBanned : UserEvent(EventType.USER_BANNED) { moderatorId int64; reason string; } message PhotoUploaded : Event(EventType.PHOTO_UPLOADED) { photo Photo; userId int64; } 

Interfaces

An interface is a collection of statically typed methods. Each method has a unique name, named arguments and result. The result can be any type of data, including other interfaces.

A method is called terminal when it returns a data type or void . The method is called interface when it returns the interface. A sequential method call must end with a terminal method, for example, app.users().register("John Doe") .

Terminal methods can be labeled @post to separate methods that modify data. Their arguments can also be marked as @post . HTTP RPC sends these methods as POST requests, and @post adds arguments to the request body.

Terminal methods not marked with @post can have @query arguments that are sent as an HTTP query string.

• Interface methods must have unique names.
• Arguments must have unique names.
• Arguments must be data types.
• Only terminal methods can be marked as @post .
• Only terminal methods not marked as @post can have @query arguments.
• The last method in the call chain must be terminal.

Interface example:

 interface Application { /** Void method. */ void0() void; /** Interface method. */ service(arg int32) Service; /** Method with 3 args. */ method(arg0 int32, arg1 string, arg2 list<string>) string; } interface Service { /** Terminal method with @query args. */ query(limit int32 @query, offset int32 @query) list<string>; /** Terminal post method with one of args marked as @post. */ @post mutator(arg0 int32, postArg string @post) string; } 

Interface Inheritance

Interfaces can inherit other interfaces.

• Redefinition of methods is prohibited.
• A heir can have only one parent.
• If the parent has an exception, then the heirs must either not specify the exception, or specify the exception of the parent.

An example of interface inheritance:

 interface BaseInterface { method() void; } interface SubInterface : BaseInterface { anotherMethod() void; } 

Exceptions

Exceptions are specified in root interfaces using @throws(Exception) . The root interface is the interface from which all calls begin. Exceptions to other interfaces in the call chain are ignored. To support multiple exceptions, polymorphic inheritance or composition is used. Usually there is one root interface, for example, Github or Twitter , and one exception.

An example of polymorphic exceptions:

 @throws(AppException) interface Application { users() Users; photos() Photos; search() Search; } enum AppExceptionCode { AUTH_EXC, VALIDATION_EXC, FORBIDDEN_EXC } exception AppException { type AppExceptionCode @discriminator; } exception AuthExc : AppException(AppExceptionCode.AUTH_EXC) {} exception ValidationExc : AppException(AppExceptionCode.VALIDATION_EXC) {} exception ForbiddenExc : AppException(AppExceptionCode.FORBIDDEN_EXC) {} 

Conclusion

It was pretty easy to write a draft compiler, I think it was ready in about a month of free time work. The rest of the year was spent on making the device relatively simple, unambiguous and convenient to use. Generics were not included in the stable version of the language, polymorphic inheritance with multiple discriminators, redefinition of exceptions in call chains, an open type system (which allowed you to use your own native types, such as native mytype ), weak typing (when the field or the result of the method was of object type, and had to unpack it themselves), and much more. As a result, I hope, we have a simple, easy-to-read and easy-to-use language.

Why is there no full support for REST? Initially, it was planned, but the specification and functionality turned out to be quite voluminous, so REST was replaced with a simpler implementation of HTTP RPC. In future versions, it may appear. More information about RPC can be found in the specification, and examples can be found on the buyding village pages of specific languages. Links are at the end of the article.

I would like to share my feelings about the use of language from the point of view of the user, and not the author. Over the past year I have used it in several projects, in some of them even the alpha version. I like Pidef. It enhances the weak connectivity of components, unifies types, interfaces, and documentation, and frees programmers from routine code duplication in different languages.

I think, as I already wrote at the beginning of the article, it greatly reduces the overhead of organizing the interaction of various systems, including mobile clients, websites, api servers, internal services, distributed systems, push notification servers, queues, and data storage systems. All of them, as a result, can use the same available data types and interfaces. At the same time there is not any technological lock-in, because inside by default it is the same JSON and HTTP.

Links

• Project site pdef.io (redirects to GitHab)
• Sample package
• Language guide
• Style guide
• JSON format specs
• HTTP RPC specs
• Grammar in BNF
• Generated and language specific code with examples
• How to write a code generator
• Google group (pdef@googlegroups.com)
• Java , Python , Objective-C code generators and banding (available at Central Maven, PyPI and CocoaPods, respectively)