Launch queues and the Laravel scheduler in the Elastic Beanstalk environment

The official Laravel documentation describes in some detail the installation of the web application and the associated worker processes, but what if I want to deploy the product in AWS Elastic Beanstalk?

As it turned out, there are practically no articles on the Internet, no packagist packages at the ready, no mention in the documentation.

This article will not only show how you can easily start the scheduler and queue handler in AWS, but also prove once again that Laravel is expanding very easily.

What is Elastic Beanstalk?

For those who are not familiar with the EB service from AWS, I will try to explain in two sentences. Elastic Beanstalk is a ready-made bundle of services (virtual servers, load balancers, monitoring) for automatic scaling of applications. Thanks to EB, the team doesn't have to have DevOps, and the application will automatically adapt to any load.

Elastic Beanstalk: Features

Amazon offers a separate, special type of environment for application workers — the 'worker' environment. And despite the fact that AWS allows you to run both scheduled tasks and tasks from queues, the process differs from the standard one:

In a standard process, Laravel inserts tasks into a queue, and another copy of the same application polls the queue periodically, hoping to get the task. Scheduled tasks are handled by the internal Laravel scheduler, which in turn is run every minute through the standard UNIX cron tab.

But in AWS EB, we can no longer install our cron files or work with the queue directly:

Instead, the internal AWS process will send us POST requests, notifying our copies of applications about scheduled tasks, ready for execution, or about new tasks in the queue. It sounds simple enough, but Laravel (the current version is 5.2) does not support either, the scheduler is started only from the console, and the queue handler wants access to the queue directly.

Implementation

Scheduler

Let's start with the scheduler. We want to happen the same thing that happens when running in the php artisan schedule:run console, but from a web request (web hook). There is no wish to create separate hooks (some developers choose this path), since:

• I would like to rely on the built-in Laravel scheduler - the syntax is easier to read, developers do not need knowledge of UNIX, business logic remains in the application, and not outside it;
• Other environments in which the application runs (local, development) may not be in AWS, and we do not want to have two different ways of working for AWS and non-AWS options;
• I do not want to create a bunch of hook methods that will be used only by AWS.

This is the final version of the controller method that runs the scheduler. The method is very similar to the built-in Laravel ScheduleRunCommand :: class:

 /** * @param Container $laravel * @param Kernel $kernel * @param Schedule $schedule * @return array */ public function schedule(Container $laravel, Kernel $kernel, Schedule $schedule) { $events = $schedule->dueEvents($laravel); $eventsRan = 0; $messages = []; foreach ($events as $event) { if (! $event->filtersPass($laravel)) { continue; } $messages[] = 'Running: '.$event->getSummaryForDisplay(); $event->run($laravel); ++$eventsRan; } if (count($events) === 0 || $eventsRan === 0) { $messages[] = 'No scheduled commands are ready to run.'; } return $this->response($messages); } 

Perhaps the most important line. As we know, Laravel will try to provide all the dependencies in the parameter list, but in this case we need a side effect from this:

 public function schedule(Container $laravel, Kernel $kernel, Schedule $schedule) 

The Laravel web application uses its own kernel class, which does not load the list of scheduled tasks, but we are asked to provide us with a console kernel (Illuminate \ Contracts \ Console \ Kernel) - Laravel will have to download it for us. During the download, a 'side effect' will occur - scheduled tasks will be downloaded from the App / Console, and the application will finally know about them. When Laravel provides the following dependency, the Schedule class will have an application.

An important detail: swap Kernel and Schedule in the parameter list and the method will stop working. The kernel needs to be loaded before Schedule, because we need a side effect of loading it.

What happens next is quite simple and understandable, almost repeats ScheduleRunCommand. It would be nice to use an existing class, but unfortunately it cannot be expanded or redefined.

Queues

One of the goals was to keep the number of new classes to a minimum, so I did not enter my queues or connections — I managed to manage only one job-class, which will be transferred to the standard queue handler.

The method turned out like this:

 /** * @param Request $request * @param Worker $worker * @param Container $laravel * @return array */ public function queue(Request $request, Worker $worker, Container $laravel) { $this->validateHeaders($request); $body = $this->validateBody($request, $laravel); $job = new AwsJob($laravel, $request->header('X-Aws-Sqsd-Queue'), [ 'Body' => $body, 'MessageId' => $request->header('X-Aws-Sqsd-Msgid'), 'ReceiptHandle' => false, 'Attributes' => [ 'ApproximateReceiveCount' => $request->header('X-Aws-Sqsd-Receive-Count') ] ]); try { $worker->process( $request->header('X-Aws-Sqsd-Queue'), $job, 0, 0 ); } catch (\Exception $e) { return $this->response([ 'Couldn\'t process ' . $job->getJobId() ], 500); } return $this->response([ 'Processed ' . $job->getJobId() ]); } 

All that had to be done was to pull out the SQS metadata from the HTTP headers, and insert them into the job class. It turned out a kind of adapter with HTTP to SQS. We don’t need to remove the work from the queue ourselves or mark it as unsuccessful, AWS will do it all. If we do not return the HTTP code 200 (for example, we caught the error), then AWS will do the rest.

That's all! It remains to add a couple of routes (only two routes for any number of tasks) and the application is ready for battle!

AWS Setup

Do not forget to subscribe the AWS worker environment to the corresponding SQS queue (or SNS topic).

In order for AWS to start “pestering” every minute, at the moment of submitting a new version of the application, the cron.yaml file must be in the root. You can add it to the repository, or you can add it in the last step. File contents:

 version: 1 cron: - name: "schedule" url: "/worker/schedule" schedule: "* * * * *" 

findings

Laravel has once again proved its flexibility and extensibility.

The full source code, a working package with integration for Laravel and Lumen has already uploaded to GitHub (and Packagist): https://github.com/dusterio/laravel-aws-worker