How to write a beautiful code and fill up the project

“We wandered into a zone with a strong magic index,” he explained. “Once upon a time, a powerful magic field formed here.
“Exactly,” answered a bush passing by.
Terry Pratchett, The Color of Magic

Maintaining ugly code is unpleasant. The ugly code is more difficult to understand, it is often obsolete and often contains errors. However, this is an honest nuisance - you immediately know that the code is not okay and you write additional tests before the change, check it several times, put the time in the assessments to fix everything.

Beautiful code is different in this respect: you read it easily, it usually uses new technologies and you readily believe that it works optimally and there are no errors in it. Although this may just not be true.
')


In this article I will show that you can’t believe in any code (everyone lies) and demonstrate some interesting errors.

Guava: how to write less code and spend more resources

Guava is a library of basic methods and objects, created by Google as an alternative to standard libraries and possessing many useful functions. In particular, Guava has implemented libraries for working with collections, which make it easier to manipulate data. This story will show that this simplicity is very easy to use for evil.

Let us need to implement a service that deals mainly with data transformations: it requests data from an external system, notifies another external system, and gives data to the one who requested it. The implementation might look something like this:

private final static Predicate<Entity> checkPredicate = entity -> entity.performCheck(); private final static Function<Entity, Integer> mapToIdFunction = entity -> entity.getId(); private final Function<Integer, Entity> lookupFunction = id -> storageService.lookupEntity(id); @Override public ImmutableList<Entity> getCheckedEntityByIds(List<Integer> ids) { logger.info("Received " + ids.size() + " ids."); List<Entity> uncheckedEntities = Lists.transform(ids, lookupFunction); Collection<Entity> filteredEntities = Collections2.filter(uncheckedEntities, checkPredicate); notificationService.sendUpdate(Collections2.transform(filteredEntities, mapToIdFunction)); logger.info("Got " + filteredEntities.size() + " entities."); return ImmutableList.copyOf(filteredEntities); } 

Agree, it is written quite concisely and you can easily understand what is happening, just by reading the source code. There is only one problem - this beautiful and concise code contains an error, which leads to a waste of resources and can lead to internal errors in the data. The error is easy to demonstrate with this test:

  @Before public void setup() { storageService = new StorageService() { @Override public Entity lookupEntity(int id) { lookupCounter++; return new Entity(id); } }; notificationService = new NotificationService() { @Override public void sendUpdate(Collection<Integer> ids) { System.out.println(ids); } }; guavaServiceImpl = new GuavaServiceImpl(storageService, notificationService); } @Test public void testGuava() { List<Integer> ids = new ArrayList<>(); for (int i = 0; i < 100; ++i) { ids.add(i); } guavaServiceImpl.getCheckedEntityByIds(ids); Assert.assertEquals(100, lookupCounter); } 

java.lang.AssertionError: expected: <100> but was: <300>
at org.junit.Assert.fail (Assert.java:88)

As it is easy to see, the test result is disappointing - we make three times more requests than reasonable. This is due to the fact that the corresponding methods do not actually transform \ filter \ etc, but create the corresponding wrapper. Those. for information, for example, about the length of the collection, we will have to recheck all the elements from the upper collection (transforming), which in turn will transform the elements each time it is requested.

So at best, we will get a performance loss due to the hidden cost of simple operations ~ O (n * m), where n is the number of elements and m is nesting of collections, and at worst this will result in memory leaks and uncontrolled requests to other services. because even a filtered collection with several elements will store references to all the original elements.

Advice on using Guava: even if the results of transformation and filtering look like collections and quack as collections, you should not consider them as collections, they are one-time and before almost any use it is worth copying their contents into another collection. For example, a good practice to use ImmutableList.copyOf () - this can protect against other problems.

Google App Engine: why it’s helpful not to believe documentation

Everybody lies.
Dr. House

Google App Engine is a fairly convenient platform for developing and deploying cloud applications, with capabilities to automatically allocate and distribute resources under high load and quickly work with various services, such as a database and file storage. This story will show what pitfalls are the ease of integration between the various components.

The protagonist of this story is a medium-sized App Engine-based project with a single Cloud SQL database (Maria DB), which he mapped into a business entity using Hibernate.

The project was developing, the base was growing and at one point, our future patient just fell. The drop was accompanied by emptiness in the logs and a huge number of connections to the database (with only a few copies of the application). Also in the database logs found several unsuccessful backups. After the restart of the database, the application began to work as if nothing had happened.

The most obvious diagnosis is lupus - someone locks the database. Since no long write transactions were found in the logs, the culprit was considered a backup of the database, which was reconfigured to work without locks. Binary logs were also included so that in case of repetition it was possible to find transactions.



However, neither the treatment nor the tests helped - the fall was repeated, and the binary logs did not show any suspicious transactions.

After a thorough rechecking of everything related to the database, the cause of the crashes was found and it was caused by following the recommendations from the App Engine documentation. In particular, in the case of an application and a database on App Engine, the documentation advises opening a new connection to the database each time a new request is received, since opening a new connection is cheaper than maintaining the old one:

How to best manage your database case. For example, if you’re trying to use a new database connection, then you’re using a database connection. Conversely, if you need to do so, you can . In particular, you can connect it with Google App Engine to Google Cloud SQL.
Google Cloud SQL FAQ

However, in conjunction with Hibernate, this approach led to rather sad results: in this application there are several rather heavy queries for retrieving data from a table, which are usually worked out normally. But in the case of several parallel requests, they can run longer than a minute, which leads to an automatic termination of the request by means of App Engine. The code that is waiting for the results of the query on the Java side ceases to be executed, but the session on the database side continues processing the result, slowing down the following queries, which results in a kind of “snowball”.

For the sake of completeness, it’s worth telling that everything ended well thanks to setting up a pool of connections. In the case of Hibernate, this can be done by replacing the property

 <connection.pool_size>0</connection.pool_size> 

on settings of specific implementation of a pool of connections. In the case of App Engine, the preferred options are DBCP and HikariCP.

Small conclusion

These two stories symbolize that no simplicity and beauty of the code is given for free. Remember - everyone is lying and in order for your application to work as well as you want, you need to understand exactly what your beautiful code actually does and the magic it uses.