Handling a large number of tasks using delayed_job
I’m a big resque fan that uses Redis as storage, but if there is a need to quickly perform a large number of background tasks, in some cases delayed_job can work more efficiently because it does not call fork () every time a new task is performed.
Linux fork () is a rather slow operation despite all the advantages of Copy on Write . In the case of a large number of homogeneous tasks, there is no point in generating a new process to complete each task. Therefore, we decided to use delayed_job , which performs all the tasks in one cycle one after the other.
The advantage is clear, I will try here to present a few tips on how you can eliminate the shortcomings, thereby speeding up the implementation. I think these are fairly well-known facts, but for beginners it should be useful.
delayed_job inserts a delayed_jobs table each time a new task is added.
')
In ActiveRecord, save calls by default are wrapped in a transaction. Therefore, in order not to get a huge number of transactions during the mass creation of background tasks, it is recommended to wrap everything in one or several transactions:
Learn more here: www.coffeepowered.net/2009/01/23/mass-inserting-data-in-rails-with-killing-your-performance
delayed_job serializes data when it creates a task and deserializes it just before execution. As parameters for background tasks, it is always better to transfer objects of the Fixnum , String or Hash class instead of heavy ActiveRecord objects.
Much better than:
Ruby's multithreading leaves much to be desired, so usually everything is parallelized by creating several processes.
No exception and delayed_job . For example, if your top / htop shows 4 threads, then to achieve the best performance, you need to run 4 workers:
Sometimes you can do more, but here you need to monitor the use of the processor and take into account the specifics of specific tasks.
By the way, the delayed_job has a configuration for monitoring several workers with monit (see the file contrib / delayed_job_multiple.monitrc).
Also, you should pay attention to the following additions:
It is worth noting that we did not manage to start delayed_job_redis with Rails 3.2.
Linux fork () is a rather slow operation despite all the advantages of Copy on Write . In the case of a large number of homogeneous tasks, there is no point in generating a new process to complete each task. Therefore, we decided to use delayed_job , which performs all the tasks in one cycle one after the other.
The advantage is clear, I will try here to present a few tips on how you can eliminate the shortcomings, thereby speeding up the implementation. I think these are fairly well-known facts, but for beginners it should be useful.
Wrap a lot of INSERT in one transaction.
delayed_job inserts a delayed_jobs table each time a new task is added.
')
In ActiveRecord, save calls by default are wrapped in a transaction. Therefore, in order not to get a huge number of transactions during the mass creation of background tasks, it is recommended to wrap everything in one or several transactions:
job_items.in_groups_of(500, false) do |group| ActiveRecord::Base.transaction do group.each do |job_item_id| JobRunner.delay.execute_job(job_item_id) end end end Learn more here: www.coffeepowered.net/2009/01/23/mass-inserting-data-in-rails-with-killing-your-performance
Avoid serialization / deserialization overhead
delayed_job serializes data when it creates a task and deserializes it just before execution. As parameters for background tasks, it is always better to transfer objects of the Fixnum , String or Hash class instead of heavy ActiveRecord objects.
JobRunner.delay.send_activation_instructions(user.id) Much better than:
JobRunner.delay.send_activation_instructions(user) Run delayed_job workers by number of cores
Ruby's multithreading leaves much to be desired, so usually everything is parallelized by creating several processes.
No exception and delayed_job . For example, if your top / htop shows 4 threads, then to achieve the best performance, you need to run 4 workers:
script/delayed_job -i 0 start script/delayed_job -i 1 start script/delayed_job -i 2 start script/delayed_job -i 3 start Sometimes you can do more, but here you need to monitor the use of the processor and take into account the specifics of specific tasks.
By the way, the delayed_job has a configuration for monitoring several workers with monit (see the file contrib / delayed_job_multiple.monitrc).
Additionally
Also, you should pay attention to the following additions:
- github.com/SponsorPay/em-resque (allows work asynchronously to workers)
- github.com/Viximo/delayed_job_redis (experimental backend for working with Redis)
It is worth noting that we did not manage to start delayed_job_redis with Rails 3.2.
Source: https://habr.com/ru/post/145899/
All Articles