Memcached: Statistics, Debugging and RPC
A series of posts about “Web, caching and memcached” continues. Start here: 1 , 2 , 3 , 4 and 5 .
In these posts we talked about memcached, its architecture, possible use, caching key selection, clustering, atomic operations and the implementation of counters in memcached , as well as the problem of simultaneously rebuilding caches and tagging caches.
Today's post completes this series, in it we will talk about technical “trifles”:
The full text of all sections as a single large PDF can be downloaded and viewed here (in the “Materials” section).
In addition to the need to implement mechanisms for working with memcached, you need to constantly monitor the cluster of memcached servers in order to be sure that we have achieved optimal performance. Memcached provides a set of commands for getting information about its work.
The simplest command,
')
To allocate memory for key values, memcached uses the slab-allocator option . This type of allocator tends to reduce internal fragmentation when allocating memory, and also ensures good efficiency of memory allocation operations.
The mechanism of its work is that all available memcached memory is divided into slabs (blocks), each of which will store items of a certain size. For example, slab for storing objects with a size of 256 bytes, while the slab itself is 1 MB in size, so it can save 4,096 such objects. The memory inside such a slab is allocated only 256 bytes each. If we have slabs for objects of 64, 128, 256, 1024 and 2048 bytes in size, then the maximum size of the object that we can save is 2048 bytes (in the last slab). If we want to save an object of 65 bytes in size, memory in slab'e-128 will be allocated for it, 1 byte in slab'e 64.
In order to achieve efficient memory utilization of memcached to store our keys and values, we must be sure that the sizes of the slabs that memcached has allocated are in the right size, as well as their reasonable content. To do this, we can ask memcached to provide statistics on slabs, which can, for example, be visualized in the form of such a graph:
Here, the slab sizes are plotted on the horizontal axis, and the amount of memory used by slabs of a given size is plotted on the vertical axis. At the moment, all memcached memory is occupied by keys and their values, so this graph represents the current distribution of values in server memory. It is easy to see that most of all slab's are allocated for keys with relatively small values - up to 20 Kb, for much larger keys, slabs are much smaller. This distribution is adequate to our task: we have most of all just small keys (counters, locks, small caches). At the same time, the same keys occupy most of the memory, with local allocation peaks for keys of 300 bytes, 8 Kb in size. If the schedule is different from what is expected by the logic of the problem, this is a cause for concern.
We wrote a large subsystem for working with memcached, implemented various mechanisms for solving problems associated with high loads. How to check that everything really works as we would like it? High load, network delays, etc. It is almost impossible to reproduce in the local environment, it is not easy to do in the test environment. On servers in production, only those debugging mechanisms are available that do not affect the normal operation of the application itself. The debugging method should not introduce noticeable time delays, otherwise it will change the behavior of the application, and debugging will become meaningless.
We can offer the following "trick" that can help in this situation: for each cache (key in memcached) or for a group of caches (keys) we create a separate file in the local file system. In the append mode, we add one character to this file in response to each logical action that occurred with the cache. To view in real-time the behavior of the caching subsystem, it suffices to tail-f to this file:
Let the letters have the following meaning:
Then, using the given sequence, you can tell what happened with this cache: at first it was absent, we did not find the cache (
A complex project consists of separate components, services that must interact with each other using RPC mechanisms, API calls, exchanging information through a database or in some other way. Sometimes memcached can be used for this exchange of information.
As an example, consider the user broadcasting service: there is a certain number of broadcasts, each of which has a certain number of viewers at a given time. The popularity of broadcasting is determined by the number of viewers. Actual information about the number of viewers has only the broadcast server, and the list of broadcasts on the broadcast page forms the frontend. Of course, it would be possible to make the broadcast server periodically flush information about the number of viewers to the database or via the API in the frontend, or the frontend could receive the latest information through the broadcast server API. However, the number of viewers is a very rapidly changing characteristic, and in this situation, you can periodically (once every few seconds) save memcached information about the number of viewers in each of the broadcasts from the broadcast server, and referring to memcached can receive information at any convenient time. moment. This can be interprocess communication implemented with memcached.
In these posts we talked about memcached, its architecture, possible use, caching key selection, clustering, atomic operations and the implementation of counters in memcached , as well as the problem of simultaneously rebuilding caches and tagging caches.
Today's post completes this series, in it we will talk about technical “trifles”:
- statistics analysis memcached;
- debugging memcached;
- “RPC” with memcached.
The full text of all sections as a single large PDF can be downloaded and viewed here (in the “Materials” section).
Job statistics memcached
In addition to the need to implement mechanisms for working with memcached, you need to constantly monitor the cluster of memcached servers in order to be sure that we have achieved optimal performance. Memcached provides a set of commands for getting information about its work.
The simplest command,
stats , allows you to get elementary statistics: server uptime (uptime), amount of memory used, number of get requests and number of hits (hits), i.e. cache hits. Their ratio allows us to judge caching efficiency as a whole, although it must be taken into account that not only cached samples are keys in memcached, but counters, locks, tags, etc., so this value needs to be adjusted to calculate the net caching efficiency. From the general statistics, we can also find out how many keys were deleted before the expiration date (evictions), this parameter can signal the lack of memory memcached.')
Slab-allocator
To allocate memory for key values, memcached uses the slab-allocator option . This type of allocator tends to reduce internal fragmentation when allocating memory, and also ensures good efficiency of memory allocation operations.
The mechanism of its work is that all available memcached memory is divided into slabs (blocks), each of which will store items of a certain size. For example, slab for storing objects with a size of 256 bytes, while the slab itself is 1 MB in size, so it can save 4,096 such objects. The memory inside such a slab is allocated only 256 bytes each. If we have slabs for objects of 64, 128, 256, 1024 and 2048 bytes in size, then the maximum size of the object that we can save is 2048 bytes (in the last slab). If we want to save an object of 65 bytes in size, memory in slab'e-128 will be allocated for it, 1 byte in slab'e 64.
In order to achieve efficient memory utilization of memcached to store our keys and values, we must be sure that the sizes of the slabs that memcached has allocated are in the right size, as well as their reasonable content. To do this, we can ask memcached to provide statistics on slabs, which can, for example, be visualized in the form of such a graph:
Here, the slab sizes are plotted on the horizontal axis, and the amount of memory used by slabs of a given size is plotted on the vertical axis. At the moment, all memcached memory is occupied by keys and their values, so this graph represents the current distribution of values in server memory. It is easy to see that most of all slab's are allocated for keys with relatively small values - up to 20 Kb, for much larger keys, slabs are much smaller. This distribution is adequate to our task: we have most of all just small keys (counters, locks, small caches). At the same time, the same keys occupy most of the memory, with local allocation peaks for keys of 300 bytes, 8 Kb in size. If the schedule is different from what is expected by the logic of the problem, this is a cause for concern.
Debugging projects using memcached
We wrote a large subsystem for working with memcached, implemented various mechanisms for solving problems associated with high loads. How to check that everything really works as we would like it? High load, network delays, etc. It is almost impossible to reproduce in the local environment, it is not easy to do in the test environment. On servers in production, only those debugging mechanisms are available that do not affect the normal operation of the application itself. The debugging method should not introduce noticeable time delays, otherwise it will change the behavior of the application, and debugging will become meaningless.
We can offer the following "trick" that can help in this situation: for each cache (key in memcached) or for a group of caches (keys) we create a separate file in the local file system. In the append mode, we add one character to this file in response to each logical action that occurred with the cache. To view in real-time the behavior of the caching subsystem, it suffices to tail-f to this file:
MLWUHHHHHHHHHHHHHHHMLLHHHHHHHHHH Let the letters have the following meaning:
M- cache is outdated (or not found);L- attempt to block;W- write (and build) a new cache;U- blocking removal;H- successful cache request.
Then, using the given sequence, you can tell what happened with this cache: at first it was absent, we did not find the cache (
M ), tried to block ( L ) to build it, blocked, built the cache ( W ), removed the lock ( U ), then for some time the cache worked successfully, returning the cached data ( H ). Then at some point the cache was outdated or was reset ( M ), we tried to block, did not work ( L ), tried again ( L ), the lock was removed, someone else built a new cache, we read it ( H ) and then they used it.Interprocess communication with memcached
A complex project consists of separate components, services that must interact with each other using RPC mechanisms, API calls, exchanging information through a database or in some other way. Sometimes memcached can be used for this exchange of information.
As an example, consider the user broadcasting service: there is a certain number of broadcasts, each of which has a certain number of viewers at a given time. The popularity of broadcasting is determined by the number of viewers. Actual information about the number of viewers has only the broadcast server, and the list of broadcasts on the broadcast page forms the frontend. Of course, it would be possible to make the broadcast server periodically flush information about the number of viewers to the database or via the API in the frontend, or the frontend could receive the latest information through the broadcast server API. However, the number of viewers is a very rapidly changing characteristic, and in this situation, you can periodically (once every few seconds) save memcached information about the number of viewers in each of the broadcasts from the broadcast server, and referring to memcached can receive information at any convenient time. moment. This can be interprocess communication implemented with memcached.
Source: https://habr.com/ru/post/43735/
All Articles