Parsing tasks from Odnoklassniki on JPoint 2018
Aloha!
Probably the most interesting event this week in the Java world was the JPoint conference, which was held at the World Trade Center in Moscow. Classmates suggested that visitors also participate in the development of the most highly loaded system in Java and help our developers in solving practical problems that they face in their work.
The eleven people who best solved these puzzles have already received prizes from us, but for the rest, we wrote down this post with an analysis of solutions.
To make it easier for you to first try to solve the puzzles yourself, we hid the correct answers under the spoiler. Chur, open only after they themselves have thought of a solution ;-)
')
Go!
Vadim wrote the following code for partitioning music tracks across servers. What did he do wrong? Correct the Vadim error in the code:
Leonid wrote such code to clean up temporary files from the directory. What did he do wrong?
Dasha is migrating code from Java 10 to a legacy system running under Java 8. What should she replace
Choose the appropriate answers:
Anton limits the number of requests to the face detector on user photos with the following code. How can he implement a limit change on the fly in a thread-safe way?
That's all! Thank you all for taking part in solving our problems. Special thanks to those who wrote us their opinion about our tasks, came to our booth and argued with us about solutions!
Or maybe you know the best solutions? Welcome to the comments!
Probably the most interesting event this week in the Java world was the JPoint conference, which was held at the World Trade Center in Moscow. Classmates suggested that visitors also participate in the development of the most highly loaded system in Java and help our developers in solving practical problems that they face in their work.
The eleven people who best solved these puzzles have already received prizes from us, but for the rest, we wrote down this post with an analysis of solutions.
To make it easier for you to first try to solve the puzzles yourself, we hid the correct answers under the spoiler. Chur, open only after they themselves have thought of a solution ;-)
')
Go!
The Vadim and score task
Vadim wrote the following code for partitioning music tracks across servers. What did he do wrong? Correct the Vadim error in the code:
/** * @return partition between 0 inclusive * and partitions exclusive */ int partition(Track track, int partitions) { assert track != null; assert partitions > 0; return track.hashCode() % partitions; } Decision
The obvious problem in Vadim's code is that track.hashCode () can return both positive and negative values.
Vadim is a neat person (you see how many asserts!), It means that he also wrote a comment to the method carefully, and there it was indicated that the method would return a number in the interval from 0 to partitions exclusively.
An obvious way to fix the problem would be a slightly different ending of the method, for example, like this:
It would seem - cheers and in production! But there is one less obvious problem - according to the hashCode () specification, it can easily return Integer.MIN_VALUE to itself, and an attempt to take Math.abs from such a number will not lead to anything good.
Oh, my mother said, watch the brackets, son! It will be right like this:
Another, more serious problem of Vadim is much less obvious. The method he chose to distribute data across servers is inconsistent: when the number of partitions changes (for example, when servers are added) all tracks are redistributed throughout the cluster almost completely. If there are few tracks, there are few problems, and if the cluster is large and there are hundreds of terabytes of tracks, all of them need to be moved between servers. And Vadim has already flooded terabytes of tracks for a hundred — another server ...
Oh, Vadim, Vadim! You would use any of the variants of consistent hashes , you would not have a headache now.
Vadim is a neat person (you see how many asserts!), It means that he also wrote a comment to the method carefully, and there it was indicated that the method would return a number in the interval from 0 to partitions exclusively.
An obvious way to fix the problem would be a slightly different ending of the method, for example, like this:
return Math.abs( track.hashCode() ) % partitions;
It would seem - cheers and in production! But there is one less obvious problem - according to the hashCode () specification, it can easily return Integer.MIN_VALUE to itself, and an attempt to take Math.abs from such a number will not lead to anything good.
Oh, my mother said, watch the brackets, son! It will be right like this:
return Math.abs( track.hashCode() % partitions );
Another, more serious problem of Vadim is much less obvious. The method he chose to distribute data across servers is inconsistent: when the number of partitions changes (for example, when servers are added) all tracks are redistributed throughout the cluster almost completely. If there are few tracks, there are few problems, and if the cluster is large and there are hundreds of terabytes of tracks, all of them need to be moved between servers. And Vadim has already flooded terabytes of tracks for a hundred — another server ...
Oh, Vadim, Vadim! You would use any of the variants of consistent hashes , you would not have a headache now.
Leonid works as a janitor
Leonid wrote such code to clean up temporary files from the directory. What did he do wrong?
void cleanup(Path dir) throws IOException { for (Path file : Files.newDirectoryStream(dir)) { if (file.endsWith(".tmp")) { Files.delete(file); } } }
Decision
Leonid was in a hurry to roll out the garbage collection in production and inattentively studied the documentation for the DirectoryStream. DirectoryStream implements AutoCloseable. That is, it has a close () method. And if something has such a method, it should be called. The consequences were not long in coming - rolling out a new version in production, he found a leak of native memory in the java process.
Since the release included not only this code, the source of the leak could not be unambiguously determined, and Leonid had to learn how to identify native leaks . When profiling, a leak was found when calling the native method __alloc_dir, which is called as a result of calling Files.newDirectoryStream. Leonid slapped his forehead and corrected the problem:
- “Hooray!”, Said Leonid and rolled out the release. And hurried again.
After all, it would be nice to check that the file from the stream is a file and not a directory, for example, by adding a condition with Files.isRegularFile (file).
Leonid also forgot that Files.delete can throw an IOException, and the cleaning process will be interrupted - something needs to be done with this too.
"In vain I went to the wipers," Leonid thought to himself ...
Since the release included not only this code, the source of the leak could not be unambiguously determined, and Leonid had to learn how to identify native leaks . When profiling, a leak was found when calling the native method __alloc_dir, which is called as a result of calling Files.newDirectoryStream. Leonid slapped his forehead and corrected the problem:
void cleanup(Path dir) throws IOException { try ( DirectoryStream<Path> stream = Files.newDirectoryStream(dir) ) { for (Path file : stream) { if (file.endsWith(".tmp")) { Files.delete(file); } } } }
- “Hooray!”, Said Leonid and rolled out the release. And hurried again.
After all, it would be nice to check that the file from the stream is a file and not a directory, for example, by adding a condition with Files.isRegularFile (file).
Leonid also forgot that Files.delete can throw an IOException, and the cleaning process will be interrupted - something needs to be done with this too.
"In vain I went to the wipers," Leonid thought to himself ...
Poor dasha
Dasha is migrating code from Java 10 to a legacy system running under Java 8. What should she replace
var in order for the program to compile?
var list = Arrays.asList("1", 2, 3.0);
Choose the appropriate answers:
List<?>List<? super Comparable>List<? extends Comparable>List<? extends Comparable & Serializable>List<? super Comparable & Serializable>
Decision
“God, what do I spend my life on,” Dasha thought and quickly found the right options: of course, first, second and third.
Anton, soup and limits
Anton limits the number of requests to the face detector on user photos with the following code. How can he implement a limit change on the fly in a thread-safe way?
class ConcurrencyLimiter { static final int DEFAULT_LIMIT = 10; final Semaphore semaphore = new Semaphore(DEFAULT_LIMIT); void runTask(Runnable task) { semaphore.acquireUninterruptibly(); try { task.run(); } finally { semaphore.release(); } } // Implement me void setLimit(int newLimit) { assert newLimit > 0; // TODO: Implementation pending } }
Decision
“Multithreading is difficult” - Anton once again read the wisdom of the ancients from a poster and composed the following code:
Naturally, it is not the only true one; different variations are possible here, for example, without CAS, but with synchronization of the entire setLimit, this is not fundamental.
But Anton did not take into account that he announced the semaphore unfair at the very beginning, and an attempt to get many permits at once with the expression semaphore.acquireUninterruptibly (previous - newLimit) may never be completed, especially if there are a lot of tasks for identifying people, and the semaphore never not enough permit for 1 time.
Anton can solve this problem with the help of such a cycle, which takes the characters one by one:
“Or maybe this is not the best solution?” Anton wondered, because “Multithreading is difficult” - Anton once again read the wisdom of the ancients from a poster hanging over his table ...
// Implemented final AtomicInteger limit = new AtomicInteger(DEFAULT_LIMIT); void setLimit(int newLimit) { assert newLimit > 0; final int previous = limit.getAndSet(newLimit); if (newLimit >= previous) { semaphore.release(newLimit - previous); } else { semaphore.acquireUninterruptibly(previous - newLimit); } }
Naturally, it is not the only true one; different variations are possible here, for example, without CAS, but with synchronization of the entire setLimit, this is not fundamental.
But Anton did not take into account that he announced the semaphore unfair at the very beginning, and an attempt to get many permits at once with the expression semaphore.acquireUninterruptibly (previous - newLimit) may never be completed, especially if there are a lot of tasks for identifying people, and the semaphore never not enough permit for 1 time.
Anton can solve this problem with the help of such a cycle, which takes the characters one by one:
// Implemented final AtomicInteger limit = new AtomicInteger(DEFAULT_LIMIT); void setLimit(int newLimit) { assert newLimit > 0; final int previous = limit.getAndSet(newLimit); if (newLimit >= previous) { semaphore.release(newLimit - previous); } else { // magic goes here: for ( int i = 0; i < previous - newLimit; i++) semaphore.acquireUninterruptibly(1); } }
“Or maybe this is not the best solution?” Anton wondered, because “Multithreading is difficult” - Anton once again read the wisdom of the ancients from a poster hanging over his table ...
That's all! Thank you all for taking part in solving our problems. Special thanks to those who wrote us their opinion about our tasks, came to our booth and argued with us about solutions!
Or maybe you know the best solutions? Welcome to the comments!
Source: https://habr.com/ru/post/353006/
All Articles