Fast full text search in Redmine

We use the Redmine project and task management system. We continue to finish it to fit our needs in order to improve usability and expand functionality. The next task was to speed up the search.

Since the number of tasks in Redmine has grown to several hundred thousand, the time to process a search query has taken tens of seconds, which is unacceptable for us. Therefore, we decided to implement full-text search based on Elasticsearch . About this will be this post.

I will not describe what Elasticsearch is, what it is for, how to install and configure it. There are many articles on Habré (for example, one , two , three ). Let us dwell on the implementation.

Among the list of plug-ins, a seemingly completely suitable redmine_elasticsearch was found. But it turned out that everything is not so simple: this plugin has not been supported for a couple of years and is very outdated (the latest compatibility is specified in the changelog with Redmine 2.5.x). However, the task has been set, and it is necessary to solve it.
')

I will describe the implementation procedure

The plugin requires a number of dependencies. First, redmine_elasticsearch uses Resque to perform background tasks for indexing entities. But in our Redmine several plug-ins have already used Sidekiq for these tasks, and there was no point in connecting Resque yet. Therefore, the first step was to replace Resque with Sidekiq.

Secondly, for work with Elasticsearch used gem 'tire' , which is outdated. Moreover, it is increasingly recommended to use elasticsearch-rails instead. This imposes a hard limit on the version of the elastic. We managed to work, but only on version 1.7.X. We did not carry out the work on tire replacement and left it for the future.

Thirdly, for Lasastika two additional plugins are needed:

• Morphological Analysis Plugin for ElasticSearch
• Mapper Attachments Type for Elasticsearch

There are no problems with their installation.

Now about the installation

Put the plugin as standard. We clone a code from a repository in a folder with plug-ins. In our case, it looks like this:

git clone https://github.com/centosadmin/redmine_elasticsearch.git /opt/redmine/plugins 

Put the necessary gems:

 bundle install 

Now we need to index all Redmine entities. This is done by the command:

 bundle exec rake redmine_elasticsearch:reindex_all RAILS_ENV=production 

Please note: the indexing process is quite long, in our case it took ~ 40 minutes.

We start Sidekiq for the queue with the name 'index_queue':

 sidekiq -q index_queue 

Restart redmine.

Everything. The plugin is ready to go and replaces the standard Redmine search with full text.

Summarize

Of course, we understand that there is still a lot of work to replace the tire with elasticsearch-rails to eliminate the hard dependence on the elastic version in the future. But the solution found has the right to exist and has already proved its efficiency. After the introduction of the plugin, the search speed decreased to ~ 0.11–0.16 seconds against a dozen seconds without it. We think this is a great result. And grateful feedback from our staff confirms this opinion.

I hope the article will be useful to you. Questions / comments and suggestions write in the comments. Thanks for attention!

Link to the fork with our modifications: redmine_elasticsearch