Performance analysis of block devices with blktrace

I / O operations are known to be among the critical resources in terms of performance in modern linux systems. Identifying and analyzing performance bottlenecks in Linux systems is quite complicated. Usually, specialized utilities are used for this purpose. Among the most well-known performance analysis tools you should first of all mention the utilities included in the sysstat package (iostat, sar, etc.). However, in some situations, information obtained through these utilities is not enough. For example, using iostat, it is impossible to know which process is performing a particular operation. Meanwhile, such information is necessary for solving some specific tasks: for example, for searching and analyzing bottlenecks in data storage systems.

Renowned Linux kernel developer Jens Axbo created blktrace in 2007, a special utility that traces I / O operations and provides the user with detailed information about them. In this article we would like to talk in detail about the possibilities of blktrace.

Opportunities

C using blktrace can solve the following tasks:

• analyze the performance of block devices;
• calculate the potential cost of resources (for example, when connecting software RAID);
• analyze the performance of various hardware configurations;
• determine the optimal configuration for a specific software environment;
• evaluate the performance of different file systems: different file systems (ext4, JFS, XFS, Btrfs) interact differently with the block I / O subsystem, and using blktrace you can determine which file system can provide the best performance when working with a particular application or hardware configuration.

')

Installation and Getting Started

Blktrace is available in most common Linux distributions, so you don’t need to build anything from source to install it. Installation is carried out in the standard way using the package manager. Together with blktrace, the blkparse utility utility is also installed, representing the output in a more convenient, human-readable form.

Now execute the following command:

  blktrace -w 30 -d / dev / sdf -o- 

The command line arguments in this case are as follows:

• “W” means the period of time during which the observation will be carried out (in this case it is 30 seconds);
• after the “d” argument, the device is specified for which statistics on I / O operations will be collected;
• "---" is an indication that all statistics will be displayed on the console, and not stored in a special text file.

More details about the command line arguments and command syntax can be found in the official documentation .

The following table will appear on the screen:

 === sdd ===
 CPU 0: 34 events, 2 KiB data
 CPU 1: 27 events, 2 KiB data
 CPU 2: 41 events, 2 KiB data
 CPU 3: 46 events, 3 KiB data
 CPU 4: 2769 events, 130 KiB data
 CPU 5: 1718 events, 81 KiB data
 CPU 6: 1326 events, 63 KiB data
 CPU 7: 2279 events, 107 KiB data
 CPU 8: 14 events, 1 KiB data
 CPU 9: 12 events, 1 KiB data
 CPU 10: 22 events, 2 KiB data
 CPU 11: 50 events, 3 KiB data
 CPU 12: 455 events, 22 KiB data
 CPU 13: 184 events, 9 KiB data
 CPU 14: 508 events, 24 KiB data
 CPU 15: 1100 events, 52 KiB data
 Total: 10585 events (dropped 0), 497 KiB data

It displays information about the loading of processor cores, which has little practical value: based on it, no conclusions can be drawn about the performance of I / O operations.

To get more detailed information, presented in an understandable form, let's use the blkparse utility:

  blktrace -w 1 -d / dev / sdf -o - |  blkparse -i - 

Now the output will look like this:

 8.32 0 19190 28.774795629 2039 DR 94229760 + 32 [fio]
 8.32 0 19191 29.927624071 0 CR 94229760 + 32 [0]

/ then displays statistics on I / O operations for all involved processor cores; we give an example of such statistics for one core /

  CPU15 (8,32):
 Reads Queued: 0, 0 KiB Writes Queued: 64, 354 KiB         
 Read Dispatches: 0 0KiB Write Dispatches: 33, 276 KiB
 Reads Requeued: 0 Writes Requeued: 0
 Reads Completed: 0,0KiB Writes Completed: 0.0 KiB          
 Read Merges: 0, 0KiB Write Merges: 0.0KiB        
 Read depth: 0 Write depth: 68
 IO unplugs: 22 Timer unplugs: 16

 Total (8.32)
 Reads Queued: 0, 0KiB Writes Queued: 1908, 7665KiB
 Read Dispatches: 0, 0KiB Write Dispatches: 1009.7665KiB
 Reads Requeued: 0 Writes Requeued: 0
 Reads Completed: 0, 0KiB Writes Completed: 1954,7655KiB
 Read Merges: 0.0KiB Write Merges: 0, 0KiB
 IO unplugs: 612 Timer unplugs: 382

 Throughput (R / W): 0KiB / s / 7701KiB / s
 Events (8.32): 11684 entries
 Skips: 0 forward (0 - 0.0%) 

At the beginning is a table consisting of the following columns:

1. Major and minor device numbers (in our case, 8, 32);
2. the core involved in the operation;
3. sequence number of the operation;
4. operation time (in milliseconds);
5. process identifier (PID);
6. event (blktrace monitors the life cycle events of all I / O operations, including its own);
7. RWBS (R —Read, W — Write, B — Barrier Operation, S — Synchronous Operation);
8. the block that started the operation + the number of blocks;
9. the name of the process that performed the operation (indicated in square brackets).

Basic operations are indicated as:

• A - I / O operation has been transferred to another device;
• C - operation completed;
• F - the operation is combined with the adjacent operation in the queue;
• I - the request to perform the operation is queued;
• M - the operation is combined with the adjacent operation in the queue;
• Q - the operation is queued;
• T - disabled due to timeout;
• X - operation is divided into several operations.

Next, blkparse displays summary information about all I / O operations and compares the load level of read operations and write operations.

Auxiliary tools

Blktrace receives data and presents it in human readable form, but does not analyze it. Specialized utilities are intended for analyzing these data and plotting graphs based on them - here, first of all, btt and seekwatcher / iowatcher should be called.

Btt

The name of this utility is an abbreviation of the expression blktrace timeline, which can be translated as "the blktrace chronicle." It is designed to analyze files in which blktrace pins, processed by blkparse, are stored, extracting information from them:

• the time spent on processing the operation before it is queued;
• the time spent waiting in the queue;
• about the time spent directly on the operation.

To get a btt report, you must first trace the I / O operations using blktrace and save it in a separate file:

  blktrace -d / dev / sda -o-> trace 

Now we will process this file using blkparse; we will save the processed result in a separate file:

  blkparse -i trace -d trace1 

(The -d argument in this case points to the file to which the processed data will be saved).

Now we process the output with btt by executing the following command:

  btt -i trace1 

A report will be displayed in tabular form. For more information about the structure of the findings of btt and their interpretation can be found in the official documentation.

Seekwatcher / Iowatcher

The seekwatcher utility was created in 2007 by Chris Mason. It was intended for processing blktrace reports and plotting graphs, including animated ones. The site of the project seekwatcher still exists, but it is more of a memorial character.

Today, Chris Mason is developing a new data visualization tool blktrace - iowatcher. You can install iowatcher from the repository . iowatcher requires a minimum of dependencies: to create animated graphics, you only need to install the program ffmpeg or librsvg.

With the help of iowatcher, you can build graphs (including animated) based on the findings of blktrace, as well as utilities btt, fio and mstat.
To build a graph, you must first run blktrace and save the output to a text file:

  blktrace -w 30 -d / dev / sdf -o-> trace.dump 

Then enter the following command:

  iowatcher -t trace.dump -o trace.svg 

iowatcher presents blktrace data as a graph.

Animated schedule can be obtained using the command:

  iowatcher -t trace.dump --movie -o trace.mp4 / 

More details about the command syntax can be found here .

Where is blktrace used

Blktrace is used as an auxiliary tool in software solutions designed to analyze and diagnose performance problems in storage systems.

For example, the company LSI produces several models of SSD-drives, made in the form of PCI-express cards. To help the user choose the most optimal model, the company has developed a special software product - Nytro Predictor . Nytro Predictor collects information about the use of repositories by applications and, on their basis, formulates recommendations for improving response time. Blktrace is used as a data collection tool in Linux systems. Then, these data are processed using special algorithms, after which the selection of a hardware solution is performed, which allows to ensure the optimal speed.

Intel is launching a similar product that uses software components from LSI - Intel RAID SSD Cache Sizing and Performance Prediction Tool . It also uses blktrace as a statistics collection tool.

The problem of speeding up access to data and reducing response time is very relevant for social networks with a large number of users. Facebook programmers are actively working to solve this problem. In 2010, they created Flashcache, a module for the Linux kernel that allows you to use one block device to cache access to other block devices. The product is distributed under the GPL license (here is the GitHub repository ). Flashcache developers used blktrace to analyze disk accesses made by database applications.

For those who can not comment on posts on Habré, we invite to our blog .