Google Hadoop Review (dataproc)

Some time ago I activated a free trial for Google under their cloud, I didn’t solve my problem, it turned out Google gives $ 300 for 12 months for a trial, but contrary to my expectations, other limits are imposed as well as the budget limit. For example, I did not allow virtual users to use more than 8 vcpu in one region. After half a year, I decided to use the trial budget for acquaintance with dataproc, the pre-installed Hadup cluster from Google. The task is to try to estimate how easy it would be for me to start a project on Google’s Hadup, is there any sense in it, or is it better to focus on your hardware and think through administration. I have a vague feeling that the modern iron and bigdat stack should already easily adapt to small databases of tens or hundreds of GB, brutally loading if not all, then the overwhelming part of the cluster memory. Some separate subd data marts may not be necessary.

In short, dataproc was impressed with the ease of launching and configuration, against the background of Oracle and Cloudera. At the first stage, I played with one node cluster at 8 vCpu, the maximum that a completely free trial allows. If you look at simplicity, then their technologies already allow a Hindu in 15 minutes to start a cluster, load sample data and prepare an ordinary BI tool, without any intermediate sub-windows. Any deep knowledge about Khadup is no longer required.

In principle, I saw that a wonderful thing for a quick start and for the sane money you can run a prototype, evaluate what kind of hardware you need for the task. However, a larger cluster, in dozens of nodes, will obviously eat significantly more than rent + a couple of admins looking after the cluster. Far from the fact that cloud will look cost effective. At the first stage I tried to evaluate a completely micro version with one node cluster 8 vCpu and 0.5 TB of raw data. In principle, the spark + hadoop tests on clusters are larger and more complete on the Internet, but I plan to test a bit larger version later.

Literally in an hour, I nagulit scripts creating a cluster of hadup, setting up its firewall and setting up a thrift server, which allowed jdbc from a home Windows to connect to the spark sql. I spent another two or three hours optimizing the default spark settings and loading a couple of small tables about 10 GB in size (the size of the data files in Oracle). I pushed the tables into memory (alter table cache;) and it was possible to work with them from my Windows machine from Dbeaver and Tableau (via spark sql connector).
')
By default, Spark used only 1 executor on 4 vCpu, I edited spark-defaults.conf, put 3 executers, 2 vCpu each, and for a long time could not understand why I really only have 1 executer. It turned out that I had not edited the memory, the other two yarn simply could not allocate memory. I put 6.5 GB on the executer, after that, as expected, all three began to rise.
Then I decided to play with a bit more serious volumes and a task closer to DWH from the TPC-DS tests. To begin with, I officially generated the tables with a scale factor of 500. It turned out to be about 480 GB of raw data (text with separator). The TPC-DS test is a typical DWH, with facts and dimensions. I did not understand how to generate the data immediately on google storage, I had to generate a virtual disk on disk and then copy it to google storage. Google, as I understand it, thinks that Hadup works fine with google storage and the speed there promises a little better than if the data were inside a cluster on HDFS. At the same time, part of the load is transferred from HDFS to google storage.

Having connected via Dbeaver, I converted text files into SQL commands into parquet-based tablets with snappy packaging. 480 GB of text data were packed into 187 GB parquet files. The process took about two hours, the largest table took 188 GB in the text, 3 spark executers in the parquet turned them in 74 minutes, the size of the SUV was 66.8 GB. On my desktop with about the same 8 vCpu (i7-3770k) I think “insert into table select * ...” into an 8k block table would take a day, and how many data files would be even scary to imagine.



Next, I checked the performance of BI tools on such a config, built a simple report in Tableua



As for queries, Query1 from the TPC-DS test


executed in 1:08, Query2 with the participation of the largest tables (catalog_sales, web_sales)


executed in 4:33 minutes, Query3 in 3.6, Query4 in 32 minutes.

If someone is interested in the settings, under the cut are my notes on creating a cluster. In principle, there are only a couple of gcloud commands and setting up HIVE_SERVER2_THRIFT_PORT.


To be continued ...