Introducing Apache Ignite: First Steps

I would venture to suggest that the average reader of this article is not familiar with the Apache Ignite product. Though, probably, heard or even read article on Habré in which one of possible scenarios of use of this product is described. I recently wrote about using Ignite as an L2 cache for Activiti. Perhaps learning that this is a Java open source product, positioning itself as a “high-performance, integrated and distributed in-memory platform for computing and processing large amounts of data in real time,” which, among other things, has the ability to automatically deploy your project to all nodes are complex topology, you want to meet him. Having experienced such a desire, you will find that Ignite is not documented so badly, but not very well. There is a tutorial , some javadoc, but a complete and holistic impression of familiarization with these sources does not arise. In this article, I will try to fill this gap based on my own experience of learning Ignite, obtained mainly by debugging. Perhaps in my conclusions and impressions I will not always be right, but these are the costs of the method. From the reader and those who want to repeat my path, not so much is required, namely, knowledge of Java 8 core, multithreading and Spring core.



The article will consider and prepare an example of the “Hello World!” Class using this technology.

Install and Run

The latest version of Ignite at the time of this writing was 1.7.0 and it was the one that was investigated (although already 1.8.0-SNAPSHOT is on GitHub). There are two ways to get Ignite. First, add a Maven dependency to the application on org.apache.ignite: ignite-core; LATEST and additionally on org.apache.ignite: ignite-spring: LATEST. You can also download from the manufacturer's website the assembly release , which consists mainly of the same libraries that Maven connects to, or the Docker image. Since I do my research on Windows 7, the docker version is not available to me, and I downloaded the binary distribution. It must be downloaded and unpacked, the folder where it was unpacked will be called IGNITE_HOME. Further, I will generally follow the order of presentation of the original tutorial, inevitably duplicating it in places, but only for the convenience of the reader.



First of all, it should be noted that the Ignite topology consists of two types of nodes, clients and servers. In a typical case, the load is performed on the servers, and clients running on weak machines connect to them and initiate tasks. Client and server nodes can be run inside the same JVM, but more often the nodes belong to JVM 1: 1. You can run any number of nodes on one physical (or virtual machine). Next, we analyze this difference more deeply. In this terminology, our “Hello World!” - the application will consist of a server and a client, which will send its famous message to the server.

')

To get the Ignite node, the utility class Ignition is used . Of his many methods, we are still interested in five overloaded start methods. One of them is without parameters and starts the node with default parameters; it does not suit us. The second one receives the generated configuration object of the IgniteConfiguration type, and the other three want to receive a spring configuration file describing the same IgniteConfiguration object as a path to the resource with the xml configuration, the URL to the xml configuration, or it is also an InputStream. From personal experience, I do not recommend using the option with manual configuration generation via new IgniteConfiguration. The fact is that the IgniteConfiguration object is composite, it has many nested objects that must also be initialized. And here the catch may be hiding, since some classes contain private fields that are initialized solely by injection. For example, in the TcpDiscoveryJdbcIpFinder class, a logger is thus injected. As you know, when creating objects through new, no injection occurs, and the logger remains uninitialized, which obviously results in a NullPointerException at the most inappropriate moment. So, regardless of your preferences, it is safer to write the xml configuration and use it. This option is also good because this config can be used to launch Ignite from the command line. Examples of kofigov can be seen in the distribution, in the $ {IGNITE_HOME} folder \ examples \ config \. The simplest config is below:





Here we say that we are creating a node with the name “testGrid-client”, that this is a client, and that it will search for a server in the address range 127.0.0.1:47500..47509, that is, locally. For the server we will prepare a similar config:





Save the server config to the test.xml file and put it in $ {IGNITE_HOME} \ examples \ config. To start the server, go to the $ {IGNITE_HOME} \ bin folder and execute the command ignite. (Bat | sh) examples \ config \ test.xml. If not done eksepshenov, the config is valid, and in the end should appear something like:







The executed command file is useful to study. In addition to the standard features for setting JVM variables, you can learn from it about the existence of the IGNITE_QUIET system variable, which controls logging details. A complete list of system variables is given in the IgniteSystemProperties class with decoding; it makes sense to read (it turns out, Ignite even knows how to check the appearance of their new versions). Then you can find out that the CommandLineStartup class is responsible for starting from the command line. He is also not without interest. You can see that if you are working on OSX, then you will pop up a pop-up window when you start it. A trifle, but not nice - for what is it only to them such happiness? From the interesting it is clear that if you get into this class without parameters, then the interactive mode will turn on and you will be offered a choice of available configs, which GridConfigurationFinder will find; he can search in $ {IGNITE_HOME}. Since we cannot start without parameters through a batch file, this feature is not available here. But do not worry, you can run the command $ {IGNITE_HOME} \ bin \ ignitevisorcmd.bat - this is the interactive monitoring of Ignite, execute the open command in it, and it will output something like this:







Here we can specify our config, in this list it is numbered 6. Entering 6, we will connect to our server and see







Next, we can enter the top command on the console and see our topology:

Look deeper

Returning to the CommandLineStartup class, you can find a longing for aliases for classes that Scala likes so much: for the sake of brevity, they created class G, an empty heir to the Ignition class. Well, ok, we started the server, what next? Next, run the client. A sample code for running an instance looks like this:





Here we check if a node with that name is already running in this JVM, if it is running, then it is stored in not anyhow, not in java.util.concurrent.ConcurrentHashMap, as someone probably thought, but in org. jsr166.ConcurrentHashMap8. What are their differences even afraid to assume, I hope that someone in the comments enlighten. And if there is no node yet, it is created on the basis of the config. Since we are connecting as a client, we need to find a server. TcpDiscoverySpi and TcpDiscoveryMulticastIpFinder are specified as a detection method in the config, these classes are initialized and perform their own search manipulations. The main ones are as follows.



In accordance with our instructions, the choice between two implementations of the TcpDiscoveryImpl interface is made in favor of ClientImpl . Then, if you specified the ssl configuration, the ssl context would be raised - it would later be useful for creating sockets. It is very important for the TcpDiscoverySpi object to identify itself, for this we set the "localAddress" property in the config. If we did not install it, we would get org.apache.ignite.spi.IgniteSpiException: Failed to resolve local host to addresses: 0.0.0.0/0.0.0.0 Next, MBeans are registered for internal self-diagnostics, that is, they can be used to product monitoring. Then, in the spiStart method, the selected implementation starts. Both the client and server must connect to the topology, but the client is blocked until the connection is established. In the config, we specified a range of ports for a local host, and Ignite tries to kill each one of them. The client sends a joinRequest to each of these port addresses. Here in this place I was personally disappointed, because interaction is provided only through sockets and, for example, it is impossible to build a topology based on JMS. It's a shame. But okay, on port 47500, which is the default port for Ignite, I have unzipped the server. In response, we receive the first hearthbeat of the server and, on its basis, update the corresponding diagnostic metrics. In the future, this process of searching for a server and receiving hearthbeats will occur continuously. We return to our visor and ask about the state of the topology, and the answer meets our expectations:







Notice the output from the server console:



[15:36:11] Topology snapshot [ver = 7, servers = 1, clients = 1, CPUs = 8, heap = 7.1GB]

[15:37:11] Topology snapshot [ver = 8, servers = 1, clients = 0, CPUs = 8, heap = 3.6GB]

[15:42:15] Topology snapshot [ver = 9, servers = 1, clients = 1, CPUs = 8, heap = 7.1GB]

[15:42:24] Topology snapshot [ver = 10, servers = 1, clients = 0, CPUs = 8, heap = 3.6GB]



It can be seen that at some point the client connected, and then fell off - this is because I was in debug, and he fell off on timeout. Fine. Now you can say hello to the world. For this guide offers to use the code type





What is he doing? The ignite object represents our node. The compute () method for our client, in accordance with his knowledge of topology, and taking into account his affinity, creates an object for distributed computing. The broadcast method asynchronously performs the job that it constructed from the System.out.println ("Hello World!") Command. The result we get is quite unexpected:





We will see the same server side. This is not exactly what I would like. This happened because we did not include the amazing steepness of the feature, P2P class loading or Zero Deployment. This point is well explained in the authentic guide , so I will not repeat. The point is that all our classes, and lambda-closures, too, must be propagated to all nodes. The alternative is to lay jar with classes in the $ {IGNITE_HOME} \ libs folder. But let's enable the feature by adding a line to the configs

 <property name="peerClassLoadingEnabled" value="true"/> 

We make a change, restart the server. And cheers!



[16:21:11] Topology snapshot [ver = 6, servers = 1, clients = 0, CPUs = 8, heap = 3.6GB]

[16:21:48] Topology snapshot [ver = 7, servers = 1, clients = 1, CPUs = 8, heap = 7.1GB]

Hello World!

Hello World!

Hello World!

findings

Predictably, a simple example revealed not so much the abysses, but interesting details. I think it will be possible to tell about them in the next series, as well as other Ignite features that are not yet affected.

Links

" GitHub Test Case Code