About three years ago, I had the idea of ​​creating a small framework for developing small services that could somehow interact with each other, provide an API to the outside, work with databases and a little more. During the solution of some work tasks, the idea of ​​a project close to the solution of work tasks was finally formed. About a year ago, all this was formed in the MIF project (MetaInfo Framework) . It was supposed that with its help it would be possible to solve such tasks as:

• Development of lightweight HTTP services
• Microservice communication via inter-process interfaces
• Serialization and deserialization based on the reflection of data structures in different formats
• Work with databases
• Some auxiliary components for creating service frameworks

All of this is focused on developing backend services for the web, but can be used in other systems.

Introduction

The release of the new standard C ++, the preparation of the following. Years pass, but there is no reflection in C ++. The possibility of the appearance of reflection in C ++ was discussed, and it seemed that it would be included in the next standard. No ... Maybe reflection is not needed? Maybe ... But there are tasks where it could be useful: interprocess communication (IPC, RPC, REST API, microservices), work with databases (ORM), various (de) serializers, etc. - In general, there is space for use.

The compiler has all the information about the types, and why not to share it with the developers? I will try to assume that there is just no document on which compiler developers should release this information to honest developers of other software. Such a document is a C ++ standard, in which the “merging” regulation will not appear.
')
In many languages ​​there is reflection. C ++ is considered a language for software development, where performance is important and, it would seem, one of its applications is web development. Development of backend services. In this industry, there are REST APIs, and ORM, and all sorts of serialization into anything (often in json). To solve these problems, you have to use ready-made libraries or write your own, in which C ++ data is manually linked with other entities, for example, mapping structures in C ++ to json format. Sometimes using macros adds meta information about the type, which is used later in building the REST API, ORM, etc. There are even solutions with plugins for the compiler, for example, odb .

I would like to have something more natural, without code generation by external utilities, cluttering up macros and templates, or manual “mapping”. This is not yet there, and to solve the above problems, you need to choose one of the above approaches.

The proposed solution is based on the addition of meta-information to C ++ types and its subsequent use in solving various problems. We can say that “the type needs to be shod in meta-information about it”, after which it will be able to safely step across the boundaries of processes and data centers and be able to present itself in different ways (binary, json, xml, text, etc) with minimal developer intervention.

Often, such tasks are solved using code generation, for example, thrift , gSOAP , protobuf , etc. I wanted to get my solution, which will eliminate external code generation, and everything that is necessary will be added to the type at the time of compilation, and at the same time there is a desire to preserve the natural syntax of C ++ language as much as possible without creating a new language in the existing language.

MIF in examples

I would like to show some features of the MIF project. And by feedback on the post, perhaps I will prepare a post about the implementation with all its tediousness, subtleties and explanations why one or another solution was chosen.

The tasks listed at the beginning of the post suggest that there will be some possibility to display C ++ data types for serialization, object-oriented interprocess communication, implementation of the REST API, etc. With this offer and start ...

Reflection

Reflection is the basis of the whole project, allowing to solve my applied tasks.

An example of how you can get the name of a C ++ data structure, the number of its fields and access one of the fields by its number.

struct Data { int field1 = 0; std::string field2; }; 

The solution of the problem could look like this:

 int main() { Data data; data.field1 = 100500; data.field2 = "Text"; using Meta = Mif::Reflection::Reflect<Data>; std::cout << "Struct name: " << Meta::Name::GetString() << std::endl; std::cout << "Field count: " << Meta::Fields::Count << std::endl; std::cout << "Field1 value: " << data.field1 << std::endl; std::cout << "Field2 value: " << data.field2 << std::endl; std::cout << "Modify fields." << std::endl; data.*Meta::Fields::Field<0>::Access() = 500100; data.*Meta::Fields::Field<1>::Access() = "New Text."; std::cout << "Field1 value: " << data.field1 << std::endl; std::cout << "Field2 value: " << data.field2 << std::endl; return 0; } 

Everything would have been like this, it was written a little differently, but the standard does not make it possible to simply display C ++ data types. And in order for the code to work, you need to add meta information to the Data structure.

 MIF_REFLECT_BEGIN(Data) MIF_REFLECT_FIELD(field1) MIF_REFLECT_FIELD(field2) MIF_REFLECT_END() MIF_REGISTER_REFLECTED_TYPE(Data) 

Meta-information about the type could be added to the type itself, expanding it. I wanted to refuse such a decision in order to be able to display types that cannot be interfered with in the code (types of third-party libraries). Several macros allow you to add all the necessary information for further work. There is also the possibility of inheritance, but more on that later ...


A slightly complicated example: try to write a generalized code of output to the console of all fields of the structure and its basic structures.


Result

 Struct name: Base1 field1 = 1 field2 = true Struct name: Base2 field3 = Text Struct name: Data field4 = 100 field5 = String field6 = key1 Struct name: Nested field = 100 key2 Struct name: Nested field = 200 

An example is a prototype for a full serializer, because based on the added meta information, it serializes the structure into a stream (in the example, to standard output stream). To determine whether a type is a container, use a function from the Serialization namespace. In this namespace are assembled ready-made serializers in json and boost.archives (xml, text, binary). They are built according to the principle close to the one given in the example. If there is no need to extend the framework with its serializer, then there is no need to write such code.

Instead of using the Printer class, you can use a ready serializer, for example, in json, and the amount of code will be greatly reduced.

 #include <mif/reflection/reflect_type.h> #include <mif/serialization/json.h> // Data and meta int main() { Data data; // Fill data auto const buffer = Mif::Serialization::Json::Serialize(data); //   json std::cout << buffer.data() << std::endl; return 0; } 

For variety, you can try to use serialization boost.archives in xml format.



From the example it can be seen that, apart from calling a specific serializer, nothing changes. The same metainformation is used. It will also be used in other places when implementing interprocess communication.

Serialization and deserializers of several formats are implemented in the frame. And if necessary, you can add support for the new format using the example of the Printer class or using the (de) serializer of json format using the proposed API to bypass the C ++ data structures. There are some restrictions on the types (where without them), but more on that later.

Next, I propose to move on to more interesting things - the implementation of interprocess communication based on interfaces transferred between processes (C ++ data structures with purely virtual methods).

Interprocess communication

What started the project MIF - is the implementation of interprocess communication. It was a primary need. At one of the first implementations of this mechanism, one of the services was developed, which at the time of writing these lines of the post has worked steadily for more than six months without crashing and reloading.

There was a need to make communication services, located on multiple machines, using interfaces. I wanted to work with services as if they were all in the same process.

The example shows how close it was to get closer to the desired result.

Objective: to develop a server and client to exchange information about employees of a fictional company.

The solution of such problems is reduced to several steps of the same type.

• Identify the interface (s) for component interaction
• Define custom data structures (if necessary) for method parameters or as return values
• Add meta information
• Implement server application
• Implement the client application

Example complex_type

a common part

Data structures

 // data.h namespace Service { namespace Data { using ID = std::string; struct Human { std::string name; std::string lastName; std::uint32_t age = 0; }; enum class Position { Unknown, Developer, Manager }; struct Employee : public Human { Position position = Position::Unknown; }; using Employees = std::map<ID, Employee>; } // namespace Data } // namespace Service 

Meta-information

 // meta/data.h namespace Service { namespace Data { namespace Meta { using namespace ::Service::Data; MIF_REFLECT_BEGIN(Human) MIF_REFLECT_FIELD(name) MIF_REFLECT_FIELD(lastName) MIF_REFLECT_FIELD(age) MIF_REFLECT_END() MIF_REFLECT_BEGIN(Position) MIF_REFLECT_FIELD(Unknown) MIF_REFLECT_FIELD(Developer) MIF_REFLECT_FIELD(Manager) MIF_REFLECT_END() MIF_REFLECT_BEGIN(Employee, Human) MIF_REFLECT_FIELD(position) MIF_REFLECT_END() } // namespace Meta } // namespace Data } // namespace Service MIF_REGISTER_REFLECTED_TYPE(::Service::Data::Meta::Human) MIF_REGISTER_REFLECTED_TYPE(::Service::Data::Meta::Position) MIF_REGISTER_REFLECTED_TYPE(::Service::Data::Meta::Employee) 

Interface

 // imy_company.h namespace Service { struct IMyCompany : public Mif::Service::Inherit<Mif::Service::IService> { virtual Data::ID AddEmployee(Data::Employee const &employee) = 0; virtual void RemoveAccount(Data::ID const &id) = 0; virtual Data::Employees GetEmployees() const = 0; }; } // namespace Service 

Meta-information

 // ps/imy_company.h namespace Service { namespace Meta { using namespace ::Service; MIF_REMOTE_PS_BEGIN(IMyCompany) MIF_REMOTE_METHOD(AddEmployee) MIF_REMOTE_METHOD(RemoveAccount) MIF_REMOTE_METHOD(GetEmployees) MIF_REMOTE_PS_END() } // namespace Meta } // namespace Service MIF_REMOTE_REGISTER_PS(Service::Meta::IMyCompany) 

The definition of the data structure and the addition of meta-information to it are the same as in the reflection examples, except that everything is separated by namespaces.

Interface definition is the definition of a C ++ data structure containing only pure virtual methods.

For interfaces, there was another wish - the ability to request from one interface the others contained in the implementation, and, possibly, not connected into a single hierarchy. Therefore, the interface being defined must always inherit Mif :: Service :: IService or any other that inherits from Mif :: Service :: IService. There is multiple inheritance. Inheritance is done through the intermediate entity Mif :: Service :: Inherit. This is a template with a variable number of parameters. Its parameters are inherited interfaces or implementations ( inheritance ). This is necessary for the implementation of the interface request mechanism that is the same as dynamic_cast, but operating outside the boundaries of the process.

Adding meta information to an interface is not much different from adding meta information to data structures. This is another, but similar, set of macros. Perhaps later, everything will be reduced to a single set of macros for defining data structures and interfaces. While they are different. This happened during the development of the project.

When adding meta information to the interface, you do not need to specify its basic interfaces. The entire hierarchy will be found at compile time. Here, the small auxiliary entity Mif :: Service :: Inherit plays its main role in the search for heirs and related meta-information.

When adding meta information to interfaces, only the interface and its methods are specified without specifying parameters, return values ​​and cv qualifiers. There was a desire to make adding meta-information to the interfaces in the spirit of minimalism. The lack of overload has become a pay for minimalism. I think this is a small price for the opportunity not to list all the parameters and return types of values ​​for each method and not to edit them with minor changes in the interface.

Having identified common entities, it remains to implement the server and client applications.

Each interface can have multiple implementations. They somehow need to be distinguished. When creating an object, you need to explicitly specify the desired implementation. This requires identifiers of interface implementations and their connection with implementations.

For convenience and rejection of the “magic values ​​in the code”, the identifiers of implementations are better placed in one or several header files. The MIF project uses numbers as an identifier. In order to somehow give uniqueness while not building any counters or not putting everything into a single enum and being able to logically separate IDs across different files and namespaces, it is proposed to use crc32 from the string as an identifier, with the creation of unique problems the developer should have less.

To implement the interface IMyCompany need an identifier

 // id/service.h namespace Service { namespace Id { enum { MyCompany = Mif::Common::Crc32("MyCompany") }; } // namespace Id } // namespace Service 

Server application

IMyCompany implementation

 // service.cpp // MIF #include <mif/common/log.h> #include <mif/reflection/reflection.h> #include <mif/service/creator.h> // COMMON #include "common/id/service.h" #include "common/interface/imy_company.h" #include "common/meta/data.h" namespace Service { namespace Detail { namespace { class MyCompany : public Mif::Service::Inherit<IMyCompany> { public: // … private: // … // IMyCompany virtual Data::ID AddEmployee(Data::Employee const &employee) override final { // ... } virtual void RemoveAccount(Data::ID const &id) override final { // ... } } virtual Data::Employees GetEmployees() const override final { // ... } }; } // namespace } // namespace Detail } // namespace Service MIF_SERVICE_CREATOR ( ::Service::Id::MyCompany, ::Service::Detail::MyCompany ) 

There are several points that I would like to draw attention to:

• Inheritance in the implementation is also via Mif :: Service :: Inherit. This is not necessary, but it can be considered a good form and will be useful when implementing several interfaces with the inheritance of a part of previously implemented interfaces.
• The whole implementation can and, preferably, should be made in one cpp-file, without separation into h and cpp files. That allows to increase the encapsulation and in large projects to reduce the compilation time due to the fact that all necessary include implementation files are in the implementation file. When they are modified, a smaller number of dependent cpp-files undergo recompilation.
• Each implementation has an entry point - MIF_SERVICE_CREATOR, which is a factory implementation. Parameters are implementation class, identifier and, if necessary, a variable number of parameters passed to the implementation constructor.

To complete the server application, it remains to add an entry point - the main function.

 // MIF #include <mif/application/tcp_service.h> // COMMON #include "common/id/service.h" #include "common/ps/imy_company.h" class Application : public Mif::Application::TcpService { public: using TcpService::TcpService; private: // Mif.Application.Application virtual void Init(Mif::Service::FactoryPtr factory) override final { factory->AddClass<::Service::Id::MyCompany>(); } }; int main(int argc, char const **argv) { return Mif::Application::Run<Application>(argc, argv); } 

When creating an entry point, you need to implement your application class — a successor from the base application class or from one of the predefined application templates. In the overridden Init method, you need to add to the factory all existing implementations of the interfaces that the service will export (factory-> AddClass). In the AddClass method, you can pass parameters to the implementation constructor.

The service uses a predefined transport tcp, serialized on the basis of boost.archive in binary format with gzip data compression to exchange information about interfaces, methods, parameters, returned results, exceptions and instances of objects.

You can use another type of transport (for example, http, which is also available in MIF or implement your own), serialize and assemble your own unique data processing chain (defining package boundaries, compression, encryption, multi-thread processing, etc.). To do this, you no longer need to use the application template, but the base application class (Mif :: Application :: Application), determine the necessary parts of the data processing chain or the transport yourself.

In the first version of the MIF project, there were no predefined application templates. The examples did not look so short, but they showed the entire path that needs to be done in order to have complete control over the data processing flow. The entire chain is shown in the examples of the first version of the project ( MIF 1.0 ).

Client application

On the client side, everything that was defined in the common part is used.
The client is the same framework of the application (in the example the predefined application template is used) in which a remote class / service factory is requested, through which the necessary object is created and its methods are called.

 // MIF #include <mif/application/tcp_service_client.h> #include <mif/common/log.h> // COMMON #include "common/id/service.h" #include "common/ps/imy_company.h" class Application : public Mif::Application::TcpServiceClient { public: using TcpServiceClient::TcpServiceClient; private: void ShowEmployees(Service::Data::Employees const &employees) const { // ... } // Mif.Application.TcpServiceClient virtual void Init(Mif::Service::IFactoryPtr factory) override final { auto service = factory->Create<Service::IMyCompany>(Service::Id::MyCompany); { Service::Data::Employee e; e.name = "Ivan"; e.lastName = "Ivanov"; e.age = 25; e.position = Service::Data::Position::Manager; auto const eId = service->AddEmployee(e); MIF_LOG(Info) << "Employee Id: " << eId; } { Service::Data::Employee e; e.name = "Petr"; e.lastName = "Petrov"; e.age = 30; e.position = Service::Data::Position::Developer; auto const eId = service->AddEmployee(e); MIF_LOG(Info) << "Employee Id: " << eId; } auto const &employees = service->GetEmployees(); ShowEmployees(employees); if (!employees.empty()) { auto id = std::begin(employees)->first; service->RemoveAccount(id); MIF_LOG(Info) << "Removed account " << id; auto const &employees = service->GetEmployees(); ShowEmployees(employees); try { MIF_LOG(Info) << "Removed again account " << id; service->RemoveAccount(id); } catch (std::exception const &e) { MIF_LOG(Warning) << "Error: " << e.what(); } } } }; int main(int argc, char const **argv) { return Mif::Application::Run<Application>(argc, argv); } 

Result

Yes, exceptions also overcome the boundaries of processes ...

 [WARNING]: Error: [Mif::Remote::Proxy::RemoteCall] Failed to call remote method "IMyCompany::RemoveAccount" for instance with id "411bdde0-f186-402e-a170-4f899311a33d". Error: RemoveAccount. Employee with id 0 not found. 

From the message you can see that the method of deleting information about an employee with identifier 0 was recalled. On the server side, there is already no such record, as indicated by the server with the exception “Employee with id 0 not found” /

The example demonstrated the interprocess communication of the client with the server, hiding as much as possible all the details related to the transport and the format of the transmitted data.

This example completes the presentation of the base underlying the MIF project. Additional features include

• The ability to request interfaces from an implementation not merged into a single hierarchy (not counting the inheritance from Mif :: Service :: IService).
• Pass pointers and smart pointers to interfaces between services. What can be useful for different implementations of services with callbacks. , publish/subscribe. visitor , tcp.

HTTP

TCP -, HTTP -. , , curl .

HTTP , web- - json REST API, C++ .

. HTTP- , . HTTP- publish / subscribe. , HTTP «-». .

MIF TCP- . , . . , , . publish / subscribe, .

MIF HTTP , .. backend . HTTP -, HTTP HTTP . web-, , web- .

HTTP web-

 // MIF #include <mif/application/http_server.h> #include <mif/common/log.h> #include <mif/net/http/constants.h> class Application : public Mif::Application::HttpServer { public: using HttpServer::HttpServer; private: // Mif.Application.HttpServer virtual void Init(Mif::Net::Http::ServerHandlers &handlers) override final { handlers["/"] = [] (Mif::Net::Http::IInputPack const &request, Mif::Net::Http::IOutputPack &response) { auto data = request.GetData(); MIF_LOG(Info) << "Process request \"" << request.GetPath() << request.GetQuery() << "\"\t Data: " << (data.empty() ? std::string{"null"} : std::string{std::begin(data), std::end(data)}); response.SetCode(Mif::Net::Http::Code::Ok); response.SetHeader( Mif::Net::Http::Constants::Header::Connection::GetString(), Mif::Net::Http::Constants::Value::Connection::Close::GetString()); response.SetData(std::move(data)); }; } }; int main(int argc, char const **argv) { return Mif::Application::Run<Application>(argc, argv); } 

 curl -iv -X POST "http://localhost:55555/" -d 'Test data' 

HTTP - MIF . backend libevent. ab 160K . «», , , .. - Python Go. Python 2 , Go 10% . , C++ Python Go , , C++, …

HTTP

Mif::Net::Http::Connection . HTTP- C++ MIF. … MIF-, , , , , , ..

-, :

 // STD #include <cstdlib> #include <future> #include <iostream> #include <string> // MIF #include <mif/net/http/connection.h> #include <mif/net/http/constants.h> int main() { try { std::string const host = "localhost"; std::string const port = "55555"; std::string const resource = "/"; std::promise<std::string> promise; auto future = promise.get_future(); Mif::Net::Http::Connection connection{host, port, [&promise] (Mif::Net::Http::IInputPack const &pack) { if (pack.GetCode() == Mif::Net::Http::Code::Ok) { auto const data = pack.GetData(); promise.set_value({std::begin(data), std::end(data)}); } else { promise.set_exception(std::make_exception_ptr( std::runtime_error{ "Failed to get response from server. Error: " + pack.GetReason() })); } } }; auto request = connection.CreateRequest(); request->SetHeader(Mif::Net::Http::Constants::Header::Connection::GetString(), Mif::Net::Http::Constants::Value::Connection::Close::GetString()); std::string data = "Test data!"; request->SetData({std::begin(data), std::end(data)}); connection.MakeRequest(Mif::Net::Http::Method::Type::Post, resource, std::move(request)); std::cout << "Response from server: " << future.get() << std::endl; } catch (std::exception const &e) { std::cerr << "Error: " << e.what() << std::endl; return EXIT_FAILURE; } return EXIT_SUCCESS; } 

. . - , , curl, .

HTTP web-

HTTP MIF. HTTP web-, curl, , C++ . Mif::Application::HttpServer, . . github http .

a common part

Interface

 namespace Service { struct IAdmin : public Mif::Service::Inherit<Mif::Service::IService> { virtual void SetTitle(std::string const &title) = 0; virtual void SetBody(std::string const &body) = 0; virtual std::string GetPage() const = 0; }; } // namespace Service 

, - — , .

Server part

 class Application : public Mif::Application::HttpServer { //... private: // Mif.Application.HttpService virtual void Init(Mif::Net::Http::ServerHandlers &handlers) override final { std::string const adminLocation = "/admin"; std::string const viewLocation = "/view"; auto service = Mif::Service::Create<Service::Id::Service>(viewLocation); auto webService = Mif::Service::Cast<Mif::Net::Http::IWebService>(service); auto factory = Mif::Service::Make<Mif::Service::Factory, Mif::Service::Factory>(); factory->AddInstance(Service::Id::Service, service); std::chrono::microseconds const timeout{10000000}; auto clientFactory = Service::Ipc::MakeClientFactory(timeout, factory); handlers.emplace(adminLocation, Mif::Net::Http::MakeServlet(clientFactory)); handlers.emplace(viewLocation, Mif::Net::Http::MakeWebService(webService)); } }; 

, , :

• : HTTP , , REST API C++

Mif::Net::Http::WebService. . Since , , , - — . HTTP web- Mif::Net::Http::MakeServlet.

MIF - — . MIF - , HTTP- . - . MIF .

-

 namespace Service { namespace Detail { namespace { class WebService : public Mif::Service::Inherit < IAdmin, Mif::Net::Http::WebService > { public: WebService(std::string const &pathPrefix) { AddHandler(pathPrefix + "/stat", this, &WebService::Stat); AddHandler(pathPrefix + "/main-page", this, &WebService::MainPage); } private: // … // IAdmin virtual void SetTitle(std::string const &title) override final { // ... } // … // Web hadlers Result<Mif::Net::Http::JsonSerializer> Stat() { // ... std::map<std::string, std::int64_t> resp; // Fill resp return resp; } Result<Mif::Net::Http::PlainTextSerializer> MainPage(Prm<std::string, Name("format")> const &format) { // ... } }; } // namespace } // namespace Detail } // namespace Service MIF_SERVICE_CREATOR ( ::Service::Id::Service, ::Service::Detail::WebService, std::string ) 

:

• Mif::Service::Inherit . Mif::Service::Inherit . IAdmin , IWebService Mif::Net::Http::WebService, HTTP .
• . , - Mif::Net::Http::WebService , AddHandler. , /view/stat
• , . -. . .
• - . Prm. Prm , . Prm. , , ( stl , , list, vector, set). , timestamp , boost::posix_time. Content. . , Headers / Params. . http_crud
• MIF_SERVICE_CREATOR , . MIF_SERVICE_CREATOR .

Customer

. . . HTTP MIF . … , , - . — http .

results

, , curl

 curl "http://localhost:55555/view/main-page?format=text" curl "http://localhost:55555/view/main-page?format=html" curl "http://localhost:55555/view/main-page?format=json" 

, curl . , HTTP .

Backedn ? How without them? , …

MIF . . , - . , ORM, , . ORM - . . , . , ORM , .. - , , SQL-.

ORM , …

:

• -
• ,

- , JDBC ( Java) - C++ .

db_client : PostgreSQL SQLite.

HTTP CRUD

http_crud HTTP — , .

web- CRUD. . . , — . NoSQL . . . — C++ . . , , .

Conclusion

, : , , , , , HTTP — . , , . , Json API . http_crud () , .

MIF, , , db_client , . ( ) . MIF, , HTTP . , , . , . «-». , visitor , .

, , , , , , « » MIF, , , thrift , .. , «home project» , , , , , .

, MIF on C++ .

.