Huginn: a simple integration platform
In the world of API, service architectures and cloud solutions, much is available without any programming at all. However, companies are still wasting valuable developer time on routine integration tasks. We want to talk about one of the platforms, which allows us to connect a new-fangled marketing service or test the hypothesis as quickly as possible and without development.
Hello! My name is Yura Builov, and I am responsible for the development in CarPrice. The company is developing dynamically, so the speed of the experiments plays an important role for us. Today I want to talk about a simple tool that saves developers from the endless stream of simple tasks of the same type for integration with external services.
The market is replete with services that allow managers to build simple chains of actions, for example:
')
We have already actively used Zapier for such tasks, but self-hosting, scalability and the absence of restrictions on the number of tasks to be processed are important to us. So we began to look for a tool for simple and fast integrations that would be useful to both managers and developers. The approach itself is not new and resembles a simplified implementation of EIPs (Enterprise Integration Patterns). Naturally, we tried several solutions from the “bloody enterprise” world (Mulesoft, Jboss Fuse, WSO2, Servicemix, Corezoid). We negotiated with vendors of proprietary software. It's not that: long, expensive, difficult or uncomfortable.
So we came to Huginn.
For capital integration, we use RabbitMQ with adapters for each service that is critical for system performance.
Monitoring of queue size, service availability and component health is configured. However, to make changes and keep the documentation up to date in such a scheme is expensive.
We use Huginn for experiments and simple fast integrations without high reliability requirements:
Initially, Huginn is a platform that was designed to launch agents who perform your tasks on the Internet. Parsing pages, track events and perform actions on your behalf. Agents create and accept events, transferring them to each other in a chain.
Huginn provides many built-in agents that we can use as custom elements of our Pipeline. Agents can run on a schedule or receive events from other agents, process and transmit the result further down the chain.
For complete integration, we lacked only an agent who knows how to track changes in the database and we wrote it ourselves, since in Huginn it is very convenient to write our agents.
huginn_mysql2_agent
As it turned out, we cover most cases with three agents:
Below is one of the first implemented chains. The agent continually checks the new auctions in the database by created_at, then by the internal API receives all the information on the car and sends a message to the Telegram.
The most remarkable thing is that the interfaces are intuitive, and the actual configuration is visualized as a graph. Therefore, this integration does not require development resources and is done by managers and internet marketers.
Below is an example of the implementation of the experiment on the delivery of personal recommendations to dealers on sms. For each new auction, we receive a list of dealers via the API of the internal recommendation system (LSTM RNN), from the database we get the phone number, shorten the link to the auction using urlshortener and send sms via the external mailing service.
Below is a part of the integration scheme with marketing tools - exponea service. In the course of its implementation, no developer was hurt - implementation and support by a single manager.
The easiest way to try Huginn is to deploy it to docker.
You can use the external database MySQL and separately run different instances for web interfaces and event handlers, which makes it easy to scale.
Thanks for attention. I hope this post will be useful, and Huginn will help someone to free up some time for cool projects!
→ Of. Repository
→ Installation in docker
→ Production environment
→ List of agents
Hello! My name is Yura Builov, and I am responsible for the development in CarPrice. The company is developing dynamically, so the speed of the experiments plays an important role for us. Today I want to talk about a simple tool that saves developers from the endless stream of simple tasks of the same type for integration with external services.
Motivation. Why huginn
The market is replete with services that allow managers to build simple chains of actions, for example:
- customer complaint
- fix fact in a journal
- send a message to Telegram
')
We have already actively used Zapier for such tasks, but self-hosting, scalability and the absence of restrictions on the number of tasks to be processed are important to us. So we began to look for a tool for simple and fast integrations that would be useful to both managers and developers. The approach itself is not new and resembles a simplified implementation of EIPs (Enterprise Integration Patterns). Naturally, we tried several solutions from the “bloody enterprise” world (Mulesoft, Jboss Fuse, WSO2, Servicemix, Corezoid). We negotiated with vendors of proprietary software. It's not that: long, expensive, difficult or uncomfortable.
So we came to Huginn.
- OpenSource - more than 16k stars on github.
- Written on RoR - you can refine and write your components.
- Easy to deploy and scale.
- It has a low threshold of entry.
Where to use huginn
For capital integration, we use RabbitMQ with adapters for each service that is critical for system performance.
Monitoring of queue size, service availability and component health is configured. However, to make changes and keep the documentation up to date in such a scheme is expensive.
We use Huginn for experiments and simple fast integrations without high reliability requirements:
- the publication of auctions in the internal Telegram channel,
- trigger notifications: complaints, violations, spikes and drawdowns in metrics,
- Push data to external systems such as exponea, expertsender or piwik.
What is Huginn
Initially, Huginn is a platform that was designed to launch agents who perform your tasks on the Internet. Parsing pages, track events and perform actions on your behalf. Agents create and accept events, transferring them to each other in a chain.
Huginn provides many built-in agents that we can use as custom elements of our Pipeline. Agents can run on a schedule or receive events from other agents, process and transmit the result further down the chain.
- Webhook Agent - accepts POST requests from external sources.
- Post Agent, Website Agent - sends requests to the specified URL, Parsit JSON, HTML or XML.
- DeDuplication Agent - do not miss duplicate events using a unique key.
- Event Formatting Agent - allows you to remap fields in the message.
- JavaScript Agent - executes arbitrary JS code (V8, therubyracer).
For complete integration, we lacked only an agent who knows how to track changes in the database and we wrote it ourselves, since in Huginn it is very convenient to write our agents.
huginn_mysql2_agent
As it turned out, we cover most cases with three agents:
- Mysql2Agent - Pollim database for changes. For each row of the sample, an event will be generated and passed down the chain.
- EventFormatting - format the incoming message using liquid templates .
- JsonAPIAgent - call external or internal API for receiving / sending data.
Examples of using
Publication of new auctions in the Telegram channel
Below is one of the first implemented chains. The agent continually checks the new auctions in the database by created_at, then by the internal API receives all the information on the car and sends a message to the Telegram.
The most remarkable thing is that the interfaces are intuitive, and the actual configuration is visualized as a graph. Therefore, this integration does not require development resources and is done by managers and internet marketers.
Personal sms recommendations
Below is an example of the implementation of the experiment on the delivery of personal recommendations to dealers on sms. For each new auction, we receive a list of dealers via the API of the internal recommendation system (LSTM RNN), from the database we get the phone number, shorten the link to the auction using urlshortener and send sms via the external mailing service.
Integration with marketing tools
Below is a part of the integration scheme with marketing tools - exponea service. In the course of its implementation, no developer was hurt - implementation and support by a single manager.
Expand and try
The easiest way to try Huginn is to deploy it to docker.
docker run -it --name huginn \ -p 3000:3000 \ -e ADDITIONAL_GEMS="huginn_mysql2_agent(git: https://github.com/yubuylov/huginn_mysql2_agent.git),huginn_jsonapi_agent(git: https://github.com/yubuylov/huginn_jsonapi_agent.git)" \ huginn/huginn http: // localhost: 3000 /
Login: admin, Password: password
You can use the external database MySQL and separately run different instances for web interfaces and event handlers, which makes it easy to scale.
Thanks for attention. I hope this post will be useful, and Huginn will help someone to free up some time for cool projects!
useful links
→ Of. Repository
→ Installation in docker
→ Production environment
→ List of agents
Source: https://habr.com/ru/post/330382/
All Articles