Sphinx: increase the maximum size of MVA
Good day to all.
Recently I encountered an interesting problem related to the MVA update (multi value attributes).
Initial conditions:
As part of the task, I needed to implement on-a-fly updates of mva attributes. From the beginning everything seemed simple enough - we take the UpdateAttributes () function from the php-shnoy API and write the necessary updates. Wrote the necessary wrapper, began to test - everything works fine. Even as it did not believe that it happened so quickly, it means that somewhere there is a catch. I began to test, so to speak, with fanaticism - and the trick came up almost immediately.
')
When trying to stuff an array containing more than 1022 values into the MVA, the client fell off with the error “ out of pool memory on MVA update ”. Began to google. It turned out that this problem is not new, and as a possible solution they suggest changing the mva_updates_pool parameter in sphinx.conf .
Tried with different meanings. But whatever value I put there, the error appeared again and again.
I decided to dig deeper and look at the source code - the benefit when I wrote it in C.
I invited a friend (I decided to use the practice of extreme programming), took a beer, and sat down at the source debug. After 2 hours of debugging and some beer drunk, the problem was found.
It turned out that in sphinx.cpp there is another check, the parameters of which cannot be changed from the config, namely:
iBytes is the actual number of bytes that will be allocated in our case under the MVA attribute. And the iBytes> ((1 << MAX_BITS) - (int) sizeof (int)) condition is actually a constraint from above. Simply put - the variable MAX_BITS is responsible for the shift. Those. IBytes is compared with 2 ^ MAX_BITS - 4 (for 32-bit) and s with 2 ^ MAX_BITS - 2 for 16-bit. The value of MAX_BITS is hard and equal to 12. Thus, you can calculate the maximum number of int-values that can be pushed into the MVA attribute. We assume that int takes 4 bytes. 4092/4 = 1023. So we got our limit.
Thus, by increasing the value of MAX_BITS, we can increase the maximum length of the MVA attribute. It is a pity that it is declared as a constant in the code, i.e. we can not specify it in the config without governing the source. Those. I had to increase this constant and rebuild again.
But most importantly - the problem is solved.
Recently I encountered an interesting problem related to the MVA update (multi value attributes).
Initial conditions:
- sphinx-1.10-beta
- sphinx php api
As part of the task, I needed to implement on-a-fly updates of mva attributes. From the beginning everything seemed simple enough - we take the UpdateAttributes () function from the php-shnoy API and write the necessary updates. Wrote the necessary wrapper, began to test - everything works fine. Even as it did not believe that it happened so quickly, it means that somewhere there is a catch. I began to test, so to speak, with fanaticism - and the trick came up almost immediately.
')
When trying to stuff an array containing more than 1022 values into the MVA, the client fell off with the error “ out of pool memory on MVA update ”. Began to google. It turned out that this problem is not new, and as a possible solution they suggest changing the mva_updates_pool parameter in sphinx.conf .
Tried with different meanings. But whatever value I put there, the error appeared again and again.
I decided to dig deeper and look at the source code - the benefit when I wrote it in C.
I invited a friend (I decided to use the practice of extreme programming), took a beer, and sat down at the source debug. After 2 hours of debugging and some beer drunk, the problem was found.
It turned out that in sphinx.cpp there is another check, the parameters of which cannot be changed from the config, namely:
int CSphArena::RawAlloc ( int iBytes ) { CheckFreelists (); if ( iBytes<=0 || iBytes>( ( 1 << MAX_BITS ) - (int)sizeof(int) ) ) { return -1; } … } iBytes is the actual number of bytes that will be allocated in our case under the MVA attribute. And the iBytes> ((1 << MAX_BITS) - (int) sizeof (int)) condition is actually a constraint from above. Simply put - the variable MAX_BITS is responsible for the shift. Those. IBytes is compared with 2 ^ MAX_BITS - 4 (for 32-bit) and s with 2 ^ MAX_BITS - 2 for 16-bit. The value of MAX_BITS is hard and equal to 12. Thus, you can calculate the maximum number of int-values that can be pushed into the MVA attribute. We assume that int takes 4 bytes. 4092/4 = 1023. So we got our limit.
Thus, by increasing the value of MAX_BITS, we can increase the maximum length of the MVA attribute. It is a pity that it is declared as a constant in the code, i.e. we can not specify it in the config without governing the source. Those. I had to increase this constant and rebuild again.
But most importantly - the problem is solved.
Source: https://habr.com/ru/post/134864/
All Articles