Hangfire - Task Scheduler for .NET

Image from hangfire.io

Hangfire is a multi-threaded and scalable task scheduler built on a client-server architecture on the .NET technology stack (primarily Task Parallel Library and Reflection), with intermediate storage of tasks in the database. Fully functional in the free (LGPL v3) open source version. This article explains how to use Hangfire.

Article layout:

• Work principles
• The insides and capabilities of the Hangfire server
• Repeatable and deferred tasks
• Quartz.NET micro review
• Pro (un) load testing
• Findings. Poll.

Work principles

What is the point? As you can see on the KDPV, which I honestly copied from the official documentation, the client process adds the task to the database, the server process periodically polls the database and performs tasks. Important points:

• All that binds the client and server is access to a common database and shared assemblies in which task classes are declared.
  
• Load scaling (increasing the number of servers) - there is!
• Without a database (task storage) Hangfire does not work and cannot work. SQL Server is supported by default, there are extensions for a number of popular DBMSs . The paid version adds support for Redis.
  
• Anything can act as a host for Hangfire: an ASP.NET application, a Windows Service, a console application, etc. up to the Azure Worker Role.
  

From the client’s point of view, work with the task takes place on the principle of “fire-and-forget”, or rather, “added to the queue and forgot” - nothing happens on the client, except for saving the task to the database. For example, we want to run the MethodToRun method in a separate process:

BackgroundJob.Enqueue(() => MethodToRun(42, "foo")); 

This task will be serialized along with the values ​​of the input parameters and saved to the database:

 { "Type": "HangClient.BackgroundJobClient_Tests, HangClient, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null", "Method": "MethodToRun", "ParameterTypes": "(\"System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\",\"System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\")", "Arguments": "(\"42\",\"\\\"foo\\\"\")" } 

This information is enough to call the MethodToRun method in a separate process via Reflection, subject to access to the HangClient assembly in which it is declared. Naturally, it is not necessary to keep the code for background execution in the same assembly with the client, in general, the dependency scheme is as follows:

The client and the server must have access to the common assembly, while access is not necessary for the embedded web interface (below). If necessary, it is possible to replace the implementation of a task already stored in the database - by replacing the assembly referenced by the server application. This is convenient for repeatable tasks, but, of course, it works on condition that the MethodToRun contract fully matches in the old and new builds. The only limitation on the method is the presence of a public modifier.
Need to create an object and call its method? Hangfire will do it for us:

  BackgroundJob.Enqueue<EmailSender>(x => x.Send(13, "Hello!")); 

And even get a copy of EmailSender via a DI container if necessary.
')
It is easier to deploy a server (for example, in a separate Windows Service):

 public partial class Service1 : ServiceBase { private BackgroundJobServer _server; public Service1() { InitializeComponent(); GlobalConfiguration.Configuration.UseSqlServerStorage("connection_string"); } protected override void OnStart(string() args) { _server = new BackgroundJobServer(); } protected override void OnStop() { _server.Dispose(); } } 

After starting the service, our Hangfire server will begin to pull tasks from the database and execute them.

Optional for use, but useful and very pleasant is the built-in web dashboard, which allows you to manage the processing of tasks:

The insides and capabilities of the Hangfire server

First of all, the server contains its own thread pool, implemented through the Task Parallel Library. And it is based on the well-known Task.WaitAll (see the BackgroundProcessingServer class).

Horizontal scaling? Web Farm? Web Garden? Supported by:

The server uses a separate, separate and limited thread pool.
Do you want to be able to use your computer?

We can create an arbitrary number of Hangfire-servers and not think about their synchronization - Hangfire guarantees that one task will be completed by one and only one server. An example implementation is using sp_getapplock (see the SqlServerDistributedLock class).
As already noted, the Hangfire server is not picky about the host process and can be deployed anywhere from the Console App to the Azure Web Site. However, it is not all-powerful, so when hosting in ASP.NET one should take into account a number of common features of IIS, such as process recycling , auto-start (startMode = "AlwaysRunning"), etc. However, the scheduler documentation provides comprehensive information in this case.
By the way! I can not fail to note the quality of the documentation - it is beyond all praise and is somewhere in the region of the ideal. The source code of Hangfire is open and well-designed, there are no obstacles to raising the local server and going around the code as a debugger.

Repeatable and deferred tasks

Hangfire allows you to create repeatable tasks with a minimum interval of a minute:

 RecurringJob.AddOrUpdate(() => MethodToRun(42, "foo"), Cron.Minutely); 

Start the task manually or delete:

 RecurringJob.Trigger("task-id"); RecurringJob.RemoveIfExists("task-id"); 

Postpone the task:

 BackgroundJob.Schedule(() => MethodToRun(42, "foo"), TimeSpan.FromDays(7)); 

Creation of a recurring and deferred task is possible with the help of CRON expressions (support implemented through the NCrontab project). For example, the following task will be performed every day at 2:15 am:

 RecurringJob.AddOrUpdate("task-id", () => MethodToRun(42, "foo"), "15 2 * * *"); 

Quartz.NET micro review

A story about a specific task scheduler would be incomplete without mentioning worthy alternatives. On the .NET platform, this alternative is Quartz.NET — the Quartz scheduler port from the Java world. Quartz.NET solves similar problems, as does Hangfire — it supports an arbitrary number of “clients” (adding a task) and “servers” (performing a task) using a common database. But the execution is different.
My first acquaintance with Quartz.NET could not be called successful - the source code taken from the official GitHub repository simply did not compile until I manually corrected the links to several missing files and assemblies (disclaimer: just telling how it was). There is no separation into client and server parts in the project - Quartz.NET is distributed as a single DLL. In order for a specific instance of the application to allow only add tasks, and not execute them - you need to configure it .
Quartz.NET is completely free, out of the box offers storage of tasks both in-memory, and using many popular DBMSs (SQL Server, Oracle, MySQL, SQLite, etc.). In-memory storage is essentially a regular dictionary in the memory of a single server process that performs tasks. Implementing multiple server processes becomes possible only when tasks are saved in the database. For synchronization, Quartz.NET does not rely on the specific features of the implementation of a specific DBMS (the same Application Lock in SQL Server), but uses one generalized algorithm. For example, by registering in the QRTZ_LOCKS table, a one-time operation of no more than one process scheduler with a specific unique id is guaranteed, issuing the task “for execution” is performed by simply changing the status in the QRTZ_TRIGGERS table.

The class task in Quartz.NET should implement the IJob interface:

 public interface IJob { void Execute(IJobExecutionContext context); } 

With such a restriction, it is very easy to serialize the task: the full name of the class is stored in the database, which is sufficient for later retrieving the type of the task class via Type.GetType (name). To transfer the parameters to the task, the JobDataMap class is used, and the parameters of an already saved task are allowed to be modified.
As for multithreading, Quartz.NET uses classes from the System.Threading: new Thread () namespace (see the QuartzThread class), its own thread pools, synchronization via Monitor.Wait / Monitor.PulseAll.
A good fly in the ointment is the quality of official documentation. For example, here is the clustering material: Lesson 11: Advanced (Enterprise) Features . Yes, yes, this is all that is on the official site on this topic. Somewhere in the open spaces of SO, there was enchanting advice to look through also the guides on the original Quartz , where the topic was revealed in more detail. The desire of developers to maintain a similar API in both worlds — Java and .NET — cannot but affect the speed of development. Quartz.NET releases and updates are rare.

The main characteristics of the two considered planners are summarized in the table:

Characteristic	Hangfire	Quartz.NET
Unlimited clients and servers	Yes	Yes
Source	github.com/HangfireIO	github.com/quartznet/quartznet
NuGet Package	Hangfire	Quartz
License	LGPL v3	Apache License 2.0
Where is the host	Web, Windows, Azure	Web, Windows, Azure
Task store	SQL Server (by default), a number of DBMS through extensions , Redis (in the paid version)	In-memory, database row (SQL Server, MySQL, Oracle ...)
Multithreading implementation	Tpl	Thread Monitor
Web interface	Yes	Not. Planned in future versions.
Deferred tasks	Yes	Yes
Repeatable tasks	Yes (minimum interval 1 minute)	Yes (minimum interval 1 millisecond)
Cron expressions	Yes	Yes

UPDATE: As ShurikEv rightly noted in the comments, the web interface for Quartz.NET exists: github.com/guryanovev/CrystalQuartz

Pro (un) load testing

It was necessary to check how Hangfire will cope with a large number of tasks. Said and done, and I wrote the simplest client, adding tasks at intervals of 0.2 seconds. Each task writes a string with debug information in the database. Putting a 100K task limit on the client, I launched 2 client instances and one server, with the server running with the profiler (dotMemory). After 6 hours, 200K of successfully completed tasks in the Hangfire and 200K of added rows in the database already awaited me. The screenshot shows the results of profiling - 2 snapshots of the memory status “before” and “after” execution:

At the next stages, 20 client processes and 20 server processes were running, and the task execution time was increased and became a random variable. That's just Hangfire is not reflected at all:

Findings. Poll.

I personally liked the Hangfire. A free, open source product that reduces the cost of developing and supporting distributed systems. Do you use something like that? I invite you to take part in the survey and tell your point of view in the comments.