DBMS Cache + Erlang
In this article I will talk about how we made friends with Cache + Erlang , and why we need it. DBMS Cache was selected as a data warehouse. We also created and operated MCA (Middleware for Cache Applications) , an intermediate software that provides a competitive model for the interaction of Erlang and Cache.
For the interaction of Erlang and Intersystems Cache the following features are implemented:
Developed by MCA consists of three main components:
MD receives requests from the yaws web server (recall: yaws is also written in Erlang). MD checks if there is such a request in the queue currently being processed or in the response cache. If everywhere is empty - the request is added to the queue. When the answer is ready, MD will return it to anyone who has requested it during this time. The MD settings allow you to store the cache of answers at this level, the memkesh is used at higher levels. In fairness - I’ll say that we don’t use caching at the MD level - but there is such a possibility. The number of simultaneously processed MD requests is equal to the number of requests received from the web server yaws - that is, the value is random.
')
C-Node provides waiting for messages from EN, sending requests to connected libraries and to Cache, sending back the result of the request. For each running C-Node - there is one running PJ (process in our mess processing request). In PJ, the request data is stored in a unique global node (by means of CallIn). In this case, the request parameters in the form of Erlang-tuples are converted into global branches. Immediately after writing to the global begins, the processing of the request begins. In fact, due to the presence of C-Node, each PJ, in relation to Erlang, looks like a set of ordinary EN (Erlang-node). Tazhke using C-node, in particular, is the XSLT transformation call for an XML file generated in ache. After executing XSLT, it turns out ready HTML, which is returned back to MD - and then to the web server.
Porte is a process running directly in Cache. The request is processed according to the type of simplified logical output engine. There is a global request and a global rule - on the basis of the global request, certain rules are selected - certain actions are executed in Cache. Also at this level there is a definition of the set of transformations that must be performed after processing inside Cache (as a rule, this is the choice of one or another XSLT transformation).
Middleware is used to communicate Cache with requests from the outside world. With it, the system can cope with large loads under conditions of limited computing resources or lack of the number of PJ. Also, there is a convenient opportunity to use for the development of interfaces not only the csp / zen built-in tools in Cache - but all the other methods of implementing interfaces (in our case, this is the xml-xslt-html chain). Erlang is chosen because the internal data format in it, as in Cache is tree-like, the exchange of trees does not require serious intermediate transformations. Yaws is also chosen because it is written in Erlang - thus a high degree of homogeneity of the entire system is achieved.
The data storage itself is implemented, in Cache, on the principle of global rules. That is, even internal processes that do not directly interact with the outside world receive a global data (“facts”) as input and, based on the global rules, select a set of necessary actions and transformations.
PS: Some parts of the article were copied from our internal documents, links to which I can not. I did not take a direct part in the development of middleware, but I will try to answer all the questions that have arisen, in which case I will turn to senior comrades for advice. I took part in the design and implementation of the data warehouse, that is, everything that happens after the data got into Porte, and before the answer is returned to EN. If it is interesting, I can describe in detail how the storage is implemented, what subsystems it uses (dictionaries, indexing, output), and so on.
For the interaction of Erlang and Intersystems Cache the following features are implemented:
- Cache messages from Erlang in Cache, translating Erlang tuples (the internal tree format of Erlang data) to Cache globals.
- Send messages from Cache to Erlang processes, translating Cache globals to Erlang tuples.
Developed by MCA consists of three main components:
- Message Dispatcher (MD) - controls the exchange of messages in a competitive environment between different Erlang-node (EN) and Cache-processes, provides caching of messages according to certain rules. Runs in the appropriate EN.
- C-node - provides for loading C / C ++ libraries and exchanging messages between them, system interaction with shared-memory, EN, CallIn / CallOut (functionality implemented in Cache in C), etc. At the moment, to C-node, for web applications, using Cache, we have connected libraries to support XSLT transformation, regular expression processing.
- Porte - Messaging Gateway with MD for Cache. Runs as a separate background job, which we will call Porte-job (PJ) .
MD
MD receives requests from the yaws web server (recall: yaws is also written in Erlang). MD checks if there is such a request in the queue currently being processed or in the response cache. If everywhere is empty - the request is added to the queue. When the answer is ready, MD will return it to anyone who has requested it during this time. The MD settings allow you to store the cache of answers at this level, the memkesh is used at higher levels. In fairness - I’ll say that we don’t use caching at the MD level - but there is such a possibility. The number of simultaneously processed MD requests is equal to the number of requests received from the web server yaws - that is, the value is random.
')
C-Node
C-Node provides waiting for messages from EN, sending requests to connected libraries and to Cache, sending back the result of the request. For each running C-Node - there is one running PJ (process in our mess processing request). In PJ, the request data is stored in a unique global node (by means of CallIn). In this case, the request parameters in the form of Erlang-tuples are converted into global branches. Immediately after writing to the global begins, the processing of the request begins. In fact, due to the presence of C-Node, each PJ, in relation to Erlang, looks like a set of ordinary EN (Erlang-node). Tazhke using C-node, in particular, is the XSLT transformation call for an XML file generated in ache. After executing XSLT, it turns out ready HTML, which is returned back to MD - and then to the web server.
Porte
Porte is a process running directly in Cache. The request is processed according to the type of simplified logical output engine. There is a global request and a global rule - on the basis of the global request, certain rules are selected - certain actions are executed in Cache. Also at this level there is a definition of the set of transformations that must be performed after processing inside Cache (as a rule, this is the choice of one or another XSLT transformation).
Why do we need it
Middleware is used to communicate Cache with requests from the outside world. With it, the system can cope with large loads under conditions of limited computing resources or lack of the number of PJ. Also, there is a convenient opportunity to use for the development of interfaces not only the csp / zen built-in tools in Cache - but all the other methods of implementing interfaces (in our case, this is the xml-xslt-html chain). Erlang is chosen because the internal data format in it, as in Cache is tree-like, the exchange of trees does not require serious intermediate transformations. Yaws is also chosen because it is written in Erlang - thus a high degree of homogeneity of the entire system is achieved.
The data storage itself is implemented, in Cache, on the principle of global rules. That is, even internal processes that do not directly interact with the outside world receive a global data (“facts”) as input and, based on the global rules, select a set of necessary actions and transformations.
PS: Some parts of the article were copied from our internal documents, links to which I can not. I did not take a direct part in the development of middleware, but I will try to answer all the questions that have arisen, in which case I will turn to senior comrades for advice. I took part in the design and implementation of the data warehouse, that is, everything that happens after the data got into Porte, and before the answer is returned to EN. If it is interesting, I can describe in detail how the storage is implemented, what subsystems it uses (dictionaries, indexing, output), and so on.
Source: https://habr.com/ru/post/151443/
All Articles