Consul.io Part 1
When developing applications, special attention should be paid to architecture. If this is not done initially, scaling problems may appear suddenly (and sometimes they may not have a solution). The scaling of the application and the efficient use of resources at the initial stage are the saved months of work in the future.
To prevent such problems, a distributed architecture is often used, that is, an architecture with the possibility of horizontal scaling of all components. But unfortunately, with the implementation of SOA, new problems arise, namely, the coherence and complexity of service configuration.
In this article we will talk about one of the discovery services called Consul, with which you can solve the above problems and make the architecture more transparent and understandable.
When designing an application as a set of loosely coupled components, we initially get the ability to scale any component.
SOA ( Service Oriented Architecture ) is an architectural pattern that describes the architecture of an application as autonomous components with weak connectivity, communicating with each other using standard protocols (for example, REST). The logical continuation (or subset) of SOA is the microservice architecture. It is based on increasing the number of services instead of increasing the functionality of a specific service (a reflection of the principle of Single Responsibility in the architecture) and deep integration with continuous processes.
If we are engaged in the implementation of the microservice architecture, then undoubtedly, the responsibility for the interaction of our services moves towards the infrastructure solution of our application. Services that comply with the principle of Single Responsibility, can only accept the request and return the answer. At the same time, it is necessary to balance the traffic between the nodes of the system, you need to connect, albeit weakly, but all the same, the services dependent on each other. And of course, we need to respond to changes in the configuration of our system:
In any system, this is a continuous and complex process, but with the help of the discovery service we will be able to control it and make it more transparent.
')
Discovery is a tool (or toolkit) for connecting the components of an architecture. Using discovery, we provide connectivity between application components, but not connectivity . Discovery can be viewed as a registry of meta-information about a distributed architecture, which stores all the data about components. This allows for the interaction of components with minimal manual intervention (that is, in accordance with the principle of ZeroConf ).
Discovery service provides three main functions on which connectivity is based within a distributed architecture:
Expand the values of each item in detail:
Consistency
Distributed architecture implies that components can be scaled horizontally, while they must have relevant information on the status of the cluster. Discovery-service provides (de) centralized storage and access to it for any node. Components can save their data and information will be delivered to all interested members of the cluster.
Registration and monitoring
Newly added services must inform about themselves, and already running must pass a constant check for accessibility. This is a prerequisite for automatic cluster configuration. Traffic balancers and dependent nodes must have information about the current cluster configuration in order to use resources efficiently.
Detection
Detection means a search engine for services, for example, by the roles they perform. We can request a location for all services of a specific role, not knowing their exact number and specific addresses, but knowing only the address of the discovery service.
This article discusses the implementation of discovery based on Consul.
Consul is a decentralized, fault-tolerant discovery service from HashiCorp (which develops products such as Vagrant, TerraForm, Otto, Atlas, and others).
Consul is a decentralized service, that is, the Consul agent is installed on each host and is a full member of the cluster. Thus, services do not need to know the discovery address in our network; all discovery requests are made to the local address 127.0.0.1.
To disseminate information, it uses algorithms that are based on the eventual consistency model.
Agents use the gossip protocol to distribute information.
Servers to select a leader use the Raft algorithm .
A leader is a server that accepts all requests to change information. If we draw an analogy with the database, then this is master in the context of master / slave replication. All other servers replicate data from the leader. The key difference from DB replication is that if a leader fails, all other servers launch the mechanism for electing a new leader and after the elections automatically begin to replicate from it. The switching mechanism is fully automatic and does not require administrator intervention.
Each instance can work in two modes: agent and server. The difference is that the agent is a distribution point, and the server is a registration point. Those. agents accept requests only for reading, and the server can perform changes to existing information (registration and deletion of services). In fact, we, in any case, make a request to the local address, the only difference is that the read request will be processed by the agent on the local host, and the data change request will be forwarded to the leader, who will save and distribute the data throughout the cluster. If our local agent is not the leader at the moment, then our change request will be fully processed locally and distributed across the cluster.
Consul cluster is a network of connected nodes that are running services registered in discovery. Consul guarantees that cluster information will be distributed to all cluster members and is available upon request. Also, support is provided not only for a peer-to-peer, but also for a multi-rank, divided by zones, cluster, which in the terminology of Consul are called data centers. With Consul, you can work with a specific data center, and perform actions on any other. Data centers are not isolated from each other within the discovery. An agent in one DC can get information from another DC, which can help in building an effective solution for a distributed system.
Consul agents running in server mode, in addition to their primary role, also get the role of a potential cluster leader. It is recommended to use at least three agents in a cluster in server mode for fault tolerance. Using server mode does not impose any restrictions on the main functionality of the agent.
When entering a new node into the cluster, we need to know the address of any node in the cluster. Running the command:
we register a new node in the cluster and, after a short time, information about the state of the entire cluster will be available to this node. Accordingly, the new node will be available for requests from other nodes.
In Consul, we can register our service in two ways:
Consider both cases in more detail.
Using the HTTP API provided by Consul, it is possible to make a correct registration of the component and remove the service in discovery. In addition to these two states, you can use the
Consider an example of a component registration request (JSON must be transmitted in a PUT request):
An example of a request to remove a component from the catalog:
An example of a request to put a service into maintenance mode:
Using the enable flag, we can switch the state and add an optional parameter, reason, which contains a text description of the reason for switching the component to maintenance mode.
If we need to register any external service and we do not have the opportunity to “teach” it to register with Consul on our own, then we can register it not as a service provider, but as an external service. After registration, we will be able to obtain external service data via DNS:
In addition to the HTTP API, you can use agent configuration files with a description of services.
In the second part we will complete the story about the Consul service, namely, we will tell about its following functions:
And of course, let's summarize the work with Consul.
To prevent such problems, a distributed architecture is often used, that is, an architecture with the possibility of horizontal scaling of all components. But unfortunately, with the implementation of SOA, new problems arise, namely, the coherence and complexity of service configuration.
In this article we will talk about one of the discovery services called Consul, with which you can solve the above problems and make the architecture more transparent and understandable.
Distributed architectures (SOA) and problems of their construction
When designing an application as a set of loosely coupled components, we initially get the ability to scale any component.
SOA ( Service Oriented Architecture ) is an architectural pattern that describes the architecture of an application as autonomous components with weak connectivity, communicating with each other using standard protocols (for example, REST). The logical continuation (or subset) of SOA is the microservice architecture. It is based on increasing the number of services instead of increasing the functionality of a specific service (a reflection of the principle of Single Responsibility in the architecture) and deep integration with continuous processes.
If we are engaged in the implementation of the microservice architecture, then undoubtedly, the responsibility for the interaction of our services moves towards the infrastructure solution of our application. Services that comply with the principle of Single Responsibility, can only accept the request and return the answer. At the same time, it is necessary to balance the traffic between the nodes of the system, you need to connect, albeit weakly, but all the same, the services dependent on each other. And of course, we need to respond to changes in the configuration of our system:
- Services are constantly being added.
- Some services stop responding to healthcheck.
- New components appear in the system and you need to spread information about them as efficiently as possible.
- Versions of individual components are updated - backward compatibility breaks.
In any system, this is a continuous and complex process, but with the help of the discovery service we will be able to control it and make it more transparent.
')
What is discovery?
Discovery is a tool (or toolkit) for connecting the components of an architecture. Using discovery, we provide connectivity between application components, but not connectivity . Discovery can be viewed as a registry of meta-information about a distributed architecture, which stores all the data about components. This allows for the interaction of components with minimal manual intervention (that is, in accordance with the principle of ZeroConf ).
The role of discovery in the process of building a distributed architecture
Discovery service provides three main functions on which connectivity is based within a distributed architecture:
- The consistency of the meta information about the services within the cluster.
- A mechanism for registering and monitoring component availability.
- A mechanism for detecting components.
Expand the values of each item in detail:
Consistency
Distributed architecture implies that components can be scaled horizontally, while they must have relevant information on the status of the cluster. Discovery-service provides (de) centralized storage and access to it for any node. Components can save their data and information will be delivered to all interested members of the cluster.
Registration and monitoring
Newly added services must inform about themselves, and already running must pass a constant check for accessibility. This is a prerequisite for automatic cluster configuration. Traffic balancers and dependent nodes must have information about the current cluster configuration in order to use resources efficiently.
Detection
Detection means a search engine for services, for example, by the roles they perform. We can request a location for all services of a specific role, not knowing their exact number and specific addresses, but knowing only the address of the discovery service.
Consul.io as discovery implementation
This article discusses the implementation of discovery based on Consul.
Consul is a decentralized, fault-tolerant discovery service from HashiCorp (which develops products such as Vagrant, TerraForm, Otto, Atlas, and others).
Consul is a decentralized service, that is, the Consul agent is installed on each host and is a full member of the cluster. Thus, services do not need to know the discovery address in our network; all discovery requests are made to the local address 127.0.0.1.
What else you need to know about Consul:
To disseminate information, it uses algorithms that are based on the eventual consistency model.
Agents use the gossip protocol to distribute information.
Servers to select a leader use the Raft algorithm .
A leader is a server that accepts all requests to change information. If we draw an analogy with the database, then this is master in the context of master / slave replication. All other servers replicate data from the leader. The key difference from DB replication is that if a leader fails, all other servers launch the mechanism for electing a new leader and after the elections automatically begin to replicate from it. The switching mechanism is fully automatic and does not require administrator intervention.
Each instance can work in two modes: agent and server. The difference is that the agent is a distribution point, and the server is a registration point. Those. agents accept requests only for reading, and the server can perform changes to existing information (registration and deletion of services). In fact, we, in any case, make a request to the local address, the only difference is that the read request will be processed by the agent on the local host, and the data change request will be forwarded to the leader, who will save and distribute the data throughout the cluster. If our local agent is not the leader at the moment, then our change request will be fully processed locally and distributed across the cluster.
Using Consul in a cluster
Consul cluster is a network of connected nodes that are running services registered in discovery. Consul guarantees that cluster information will be distributed to all cluster members and is available upon request. Also, support is provided not only for a peer-to-peer, but also for a multi-rank, divided by zones, cluster, which in the terminology of Consul are called data centers. With Consul, you can work with a specific data center, and perform actions on any other. Data centers are not isolated from each other within the discovery. An agent in one DC can get information from another DC, which can help in building an effective solution for a distributed system.
Consul agents running in server mode, in addition to their primary role, also get the role of a potential cluster leader. It is recommended to use at least three agents in a cluster in server mode for fault tolerance. Using server mode does not impose any restrictions on the main functionality of the agent.
When entering a new node into the cluster, we need to know the address of any node in the cluster. Running the command:
consul join node_ip_addresswe register a new node in the cluster and, after a short time, information about the state of the entire cluster will be available to this node. Accordingly, the new node will be available for requests from other nodes.
Types of nodes: internal, external
In Consul, we can register our service in two ways:
- Use HTTP API or agent configuration file, but only if your service can communicate with Consul on its own.
- Register the service as a 3d-party component, in case the service cannot communicate with Consul.
Consider both cases in more detail.
Using the HTTP API provided by Consul, it is possible to make a correct registration of the component and remove the service in discovery. In addition to these two states, you can use the
maintenance state. In this mode, the service is marked as inaccessible and is no longer displayed in DNS and API requests.Consider an example of a component registration request (JSON must be transmitted in a PUT request):
http://localhost:8500/v1/agent/service/register { "ID": "redis1", "Name": "redis", "Tags": [ "master", "v1" ], "Address": "127.0.0.1", "Port": 8000, "Check": { "Script": "/usr/local/bin/check_redis.py", "HTTP": "http://localhost:5000/health", "Interval": "10s", "TTL": "15s" } } An example of a request to remove a component from the catalog:
http://localhost:8500/v1/agent/service/deregister/[ServiceID]
An example of a request to put a service into maintenance mode:
http://localhost:8500/v1/agent/service/maintenanse/[ServiceID]?enable=true|false
Using the enable flag, we can switch the state and add an optional parameter, reason, which contains a text description of the reason for switching the component to maintenance mode.
If we need to register any external service and we do not have the opportunity to “teach” it to register with Consul on our own, then we can register it not as a service provider, but as an external service. After registration, we will be able to obtain external service data via DNS:
$ curl -X PUT -d '{"Datacenter": "dc1", "Node": "google", "Address": "www.google.com", "Service": {"Service": "search", "Port": 80}}' http://127.0.0.1:8500/v1/catalog/register In addition to the HTTP API, you can use agent configuration files with a description of services.
In the second part we will complete the story about the Consul service, namely, we will tell about its following functions:
- DNS interface.
- HTTP API
- Health Checks.
- K / V storage.
And of course, let's summarize the work with Consul.
Source: https://habr.com/ru/post/278085/
All Articles