Apache Ignite - calculations in grid
Calculations in grid or mining of “beautiful” hashes, I decided to check this problem for calculations in the Apache Ignite grid. Earlier, I tried and wrote Ignite as Sql DB, but for myself I realized that this is still a convenient option in this computing system (I’ll come back to SQL on Ignite), just as I imagine it with the possibility of fast and inexpensive scalability . So let's see how you can quickly and inexpensively increase computations, or not, for example, to increase the computations of 1 powerful computer by adding a few weak ones to it.
The task is to calculate a hash for data transaction blocks, but not simple, but with some complexity, for example, containing seven consecutive 'A' characters. In order for this to be possible, we will paste the number constantly increasing in the loop to the data block until a hash of a given complexity is obtained. Yes, it looks like miners do mining cryptocurrency. Since I have several transactions, I will send them to the computational grid. Computational grid is Ignite nodes, running instances on different computers, they discover themselves and form a grid. These calculations will be distributed between the nodes evenly and automatically.
And so my computing power at home is:
')
Intel Core I5-7400 3.5 GHz 8 GB. Ram
Intel Core I3-6100 3.7 GHz 8 GB. Ram
Intel Core 2Duo E6550 2.3 GHz 8 GB. Ram
Ignite will be launched on each of them.
The result of calculations for 11 transactions on one node (one computer), the code is higher
It's nice that the power since 2Duo has grown. And this is how it looked in the pictures with I5
And now we will begin to increase the computing power, run on multiple computers Ignite server. Since the example is implemented in such a way that it waits for all tasks to be completed, the end will be a calculation on the last node, and if we connect weak computers to strong ones, the result will fall (at least at the beginning).
The result of calculations on several nodes.
It can be seen, for example, that adding to I5, I3 the result became lower than for one I5, but better than one I3. Thus, the calculation time for this implementation will be measured by a weak node and is equal to the time over which the node will process its portion of tasks . To get the calculation time in a combination of I5, I3 is better than on one I5, you need to understand on how many transactions I3 time will show better than I5 for all. Experimentally quickly found that I3 a portion of 4-5 transactions processes as well or better as I5 all 11 transactions, so such portions are possible when there are 3 nodes in the grid - I5 + I3 + I3, my expectations that my grid will calculate this task ~ 30 sec. (Against 40 sec. On one I5), such is scalability.
Well, adding to the weak computers in the grid are powerful, of course, we immediately get an increase. One 2Duo counted 140 seconds, and in the grid with others for 68 seconds.
This is one of the Ignite consoles for three nodes in the grid.
It shows 3 servers, one client who sends tasks to them, the CPU shows how the sum of three computers, well, the memory also as a sum. It can be seen that the node received 4 tasks from 11 (transactions), at the end there were three servers.
In general, distributed tasks are solved coolly, various templates are offered, with different capabilities. Next, I want to return to SQL in Ignite and work with caches, I will write ...
Materials:
Ignite Getting Started
The task is to calculate a hash for data transaction blocks, but not simple, but with some complexity, for example, containing seven consecutive 'A' characters. In order for this to be possible, we will paste the number constantly increasing in the loop to the data block until a hash of a given complexity is obtained. Yes, it looks like miners do mining cryptocurrency. Since I have several transactions, I will send them to the computational grid. Computational grid is Ignite nodes, running instances on different computers, they discover themselves and form a grid. These calculations will be distributed between the nodes evenly and automatically.
And so my computing power at home is:
')
Intel Core I5-7400 3.5 GHz 8 GB. Ram
Intel Core I3-6100 3.7 GHz 8 GB. Ram
Intel Core 2Duo E6550 2.3 GHz 8 GB. Ram
Ignite will be launched on each of them.
Here is one of the templates that Ignite offers for computing in the grid.
// Ignition.setClientMode(true); try (Ignite ignite = Ignition.start("examples/config/example-ignite.xml")) { // , // , for (final T item: list) { calls.add(new IgniteCallable<String>() { @Override public String call() throws Exception { // , return result; } }); } // Collection<String> res = ignite.compute().call(calls); // for (String hash : res) { System.out.println(">>> Check all nodes for output : " + result); } } Here is the code that will be calculated on the grid nodes (public String call ())
calls.add(new IgniteCallable<String>() { @Override public String call() throws Exception { System.out.println(); System.out.println(">>> Id transaction=#" + transaction.getId() + " on this node from ignite job."); MessageDigest md = MessageDigest.getInstance("MD5"); String transactHash; // do { md.update(transaction.getDifficultyContent().getBytes()); byte[] digest = md.digest(); transactHash = DatatypeConverter.printHexBinary(digest).toUpperCase(); // transaction.setDifficulty(transaction.getDifficulty() + 1); } while (!transactHash.contains("AAAAAAA")); return transactHash; } });
Full code
public class MyComputeCallable { // public static final String LOREM_IPSUM = "Lorem ipsum dolor sit amet, consetetur sadipscing elitr,# " + "sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua.#" + "At vero eos et accusam et justo duo dolores et ea rebum.#" + "Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet."; /** * Executes example. * * @param args Command line arguments, none required. * @throws IgniteException If example execution failed. */ public static void main(String[] args) throws IgniteException { String[] loremIpsum = LOREM_IPSUM.split("#"); List<Transaction> transactionList = new ArrayList<>(); for (int i= 0; i <= 10; i++) { transactionList.add(i, new Transaction(i, loremIpsum[i % 4])); } Ignition.setClientMode(true); try (Ignite ignite = Ignition.start("examples/config/example-ignite.xml")) { System.out.println(); System.out.println(">>> Compute callable example started."); Collection<IgniteCallable<String>> calls = new ArrayList<>(); // Iterate through all words in the sentence and create callable jobs. for (final Transaction transaction : transactionList) { calls.add(new IgniteCallable<String>() { @Override public String call() throws Exception { System.out.println(); System.out.println(">>> Id transaction=#" + transaction.getId() + " on this node from ignite job."); MessageDigest md = MessageDigest.getInstance("MD5"); String transactHash; do { md.update(transaction.getDifficultyContent().getBytes()); byte[] digest = md.digest(); transactHash = DatatypeConverter.printHexBinary(digest).toUpperCase(); // transaction.setDifficulty(transaction.getDifficulty() + 1); } while (!transactHash.contains("AAAAAAA")); return transactHash; } }); } // Execute collection of callables on the ignite. long millis = System.currentTimeMillis(); Collection<String> res = ignite.compute().call(calls); System.out.println(); // individual received from remote nodes. for (String hash : res) { System.out.println(">>> Check all nodes for output hash: " + hash); } System.out.println(">>> Total msec: " + (System.currentTimeMillis() - millis)); } } } //----------------------- Transaction --------------------------------- public class Transaction { private int difficulty; private int id; private String content; public Transaction(int id, String content) { this.id = id; this.content = content; } public int getId() { return id; } public void setId(int id) { this.id = id; } public int getDifficulty() { return difficulty; } public void setDifficulty(int difficulty) { this.difficulty = difficulty; } public String getContent() { return content; } public void setContent(String content) { this.content = content; } public String getDifficultyContent() { return "" + difficulty + content; } }
The result of calculations for 11 transactions on one node (one computer), the code is higher
| Computer
| Result msec.
|
|---|---|
| I5
| 40,909
|
| I3
| 57 162
|
| 2 Duo
| 140 673
|
It's nice that the power since 2Duo has grown. And this is how it looked in the pictures with I5
Ignite console
It can be seen that the client connected to the server (srevers = 1, clients = 1), transactions arrived at the node for calculations (11 total) >>> Id transaction ...
at the end of the client disconnected (srevers = 1, clients = 0)
It can be seen that the client connected to the server (srevers = 1, clients = 1), transactions arrived at the node for calculations (11 total) >>> Id transaction ...
at the end of the client disconnected (srevers = 1, clients = 0)
The output of the program (beautiful hashes)
But the calculated "beautiful" hashes
But the calculated "beautiful" hashes
And now we will begin to increase the computing power, run on multiple computers Ignite server. Since the example is implemented in such a way that it waits for all tasks to be completed, the end will be a calculation on the last node, and if we connect weak computers to strong ones, the result will fall (at least at the beginning).
The result of calculations on several nodes.
| Nodes
| Result msec.
|
|---|---|
| I5 + i3
| 44 389
|
| I5 + I3 + 2Duo
| 68,892
|
It can be seen, for example, that adding to I5, I3 the result became lower than for one I5, but better than one I3. Thus, the calculation time for this implementation will be measured by a weak node and is equal to the time over which the node will process its portion of tasks . To get the calculation time in a combination of I5, I3 is better than on one I5, you need to understand on how many transactions I3 time will show better than I5 for all. Experimentally quickly found that I3 a portion of 4-5 transactions processes as well or better as I5 all 11 transactions, so such portions are possible when there are 3 nodes in the grid - I5 + I3 + I3, my expectations that my grid will calculate this task ~ 30 sec. (Against 40 sec. On one I5), such is scalability.
Well, adding to the weak computers in the grid are powerful, of course, we immediately get an increase. One 2Duo counted 140 seconds, and in the grid with others for 68 seconds.
This is one of the Ignite consoles for three nodes in the grid.
It shows 3 servers, one client who sends tasks to them, the CPU shows how the sum of three computers, well, the memory also as a sum. It can be seen that the node received 4 tasks from 11 (transactions), at the end there were three servers.
In general, distributed tasks are solved coolly, various templates are offered, with different capabilities. Next, I want to return to SQL in Ignite and work with caches, I will write ...
Topology
Materials:
Ignite Getting Started
Source: https://habr.com/ru/post/348202/
All Articles