Comparing taking from the cache of the same file using fs.readFileSync and fs.readFile (and reading multiple files)
After reading the article Episode 8: Interview with Ryan Dahl, Creator of Node.js and comments on the translation , I decided to test the effectiveness of blocking and non-blocking reading of the file in Node.js, under the cut of the table and graphics.
UPD: Under the cut is a bad benchmark. As correctly indicated in the comments, essentially taking the same file from the cache is compared using fs.readFileSync and fs.readFile.
UPD2: The article has been edited, the benchmark has been corrected, the results have been added.
The blocking operation (fs.readFileSync is one of these) assumes that the execution of the entire application will be suspended until non-JS-related direct operations are performed.
Non-blocking operations allow you to perform non-JS operations asynchronously in parallel threads (for example, fs.readFile).
More about blocking vs non-blocking here .
Although Node.js runs in one thread, using child_process or cluster, you can distribute the execution of the code into several threads.
Tests were conducted with parallel and sequential reading of the cached file (large and small), as well as reading of uncached files.
All tests took place on the same computer, with one HDD, and immeno:
OS: Ubuntu 16.04
Node.js version: 8.4.0
Processor: AMD Phenom (tm) 9750 Quad-Core Processor
Physical cores: 4
HDD: 2TB 7200rpm 64MB
File system type: ext4
file.txt size: 3.3 kB
bigFile.txt size: 6.5 MB
Results for cached file.
When reading 3.3 kB file 10,000 times
| Symbol | Name | ops / sec | Percents |
|---|---|---|---|
| A | Loop readFileSync | 7.4 | 100% |
| B | Promise chain readFileSync | 4.47 | 60% |
| C | Promise chain readFile | 1.09 | 15% |
| D | Promise.all readFileSync | 4.58 | 62% |
| E | Promise.all readFile | 1.69 | 23% |
| F | Multithread loop readFileSync | 20.05 | 271% |
| G | Multithread promise.all readFile | 4.98 | 67% |
When reading 3.3 kB file 100 times
| Symbol | Name | ops / sec | Percents |
|---|---|---|---|
| A | Loop readFileSync | 747 | 100% |
| B | Promise chain readFileSync | 641 | 86% |
| C | Promise chain readFile | 120 | sixteen% |
| D | Promise.all readFileSync | 664 | 89% |
| E | Promise.all readFile | 238 | 32% |
| F | Multithread loop readFileSync | 1050 | 140% |
| G | Multithread promise.all readFile | 372 | 50% |
When reading 6.5 MB file 100 times
| Symbol | Name | ops / sec | Percents |
|---|---|---|---|
| A | Loop readFileSync | 0.63 | 83% |
| B | Promise chain readFileSync | 0.66 | 87% |
| C | Promise chain readFile | 0.61 | 80% |
| D | Promise.all readFileSync | 0.66 | 87% |
| E | Promise.all readFile | 0.76 | 100% |
| F | Multithread loop readFileSync | 0.83 | 109% |
| G | Multithread promise.all readFile | 0.81 | 107% |
CPU load when reading 3.3 kB file 10,000 times
CPU usage when reading 6.5 MB file 100 times
As you can see, fs.readFileSync is always executed in one thread on one core. fs.readFile uses several threads in its work, but the kernels are not loaded at full capacity. For a small file, fs.readFileSync is faster than fs.readFile, and only when reading a large file when a node is running in one stream, fs.readFile runs faster than fs.readFileSync.
Therefore, reading small files is best done using fs.readFileSync, and large files using fs.readFile (how big the file should be depends on the computer and software).
For some tasks, fs.readFileSync may be preferable for reading large files. For example, with a long reading and processing of multiple files. In this case, the load between the cores must be distributed using child_process. Roughly speaking, run the node itself, and not operations in multiple threads.
UPD2
Below is the data obtained for reading a lot of un-cached files of the same size (3.3kB).
When reading 1000 files
| Symbol | Name | ops / sec | Percents |
|---|---|---|---|
| A | Loop readFileSync | 8.47 | 74% |
| B | Promise chain readFileSync | 6.28 | 55% |
| C | Promise chain readFile | 5.49 | 48% |
| D | Promise.all readFileSync | 8.06 | 70% |
| E | Promise.all readFile | 11.05 | 100% |
| F | Multithread loop readFileSync | 3.71 | 32% |
| G | Multithread promise.all readFile | 5.11 | 44% |
When reading 100 files
| Symbol | Name | ops / sec | Percents |
|---|---|---|---|
| A | Loop readFileSync | 79.19 | 85% |
| B | Promise chain readFileSync | 50.17 | 54% |
| C | Promise chain readFile | 48.46 | 52% |
| D | Promise.all readFileSync | 54.7 | 58% |
| E | Promise.all readFile | 92.87 | 100% |
| F | Multithread loop readFileSync | 80.46 | 86% |
| G | Multithread promise.all readFile | 92.19 | 99% |
CPU usage when reading unzipped files is small, about 20%. Results vary ± 30%.
The results show that using non-blocking fs.readFile is more profitable.
An example of a file reading situation.
Suppose we have a web server on a node in a single T1 thread. Two requests come at the server at the same time (P1 and P2) for reading and processing small files (one per request). When using fs.readFileSync, the sequence of execution of the code in the T1 stream will look like this:
P1 -> P2
When using fs.readFile, the sequence of code execution in the T1 stream will look like this:
P1-1 -> P2-1 -> P1-2 -> P2-2
Where P1-1, P2-1 is a delegation of reading to another thread, P1-2, P2-2 is getting reading results and processing data.
')
Source: https://habr.com/ru/post/337746/
All Articles