Overview and Comparison of Ingress Controllers for Kubernetes

When you start a Kubernetes cluster for a specific application, you should understand what the requirements for this resource are for the application, business and developers. With this information, you can begin to take an architectural decision and, in particular, to select a particular Ingress-controller, which today is already a large number. To get a basic idea of the options available without having to study a lot of articles / documentation, etc., we prepared this review by including basic (production ready) Ingress controllers.

We hope that it will help colleagues in the choice of architectural solutions - at least, will be the starting point for more detailed information and practical experiments. Previously, we studied other similar materials on the network and, oddly enough, did not find any more or less complete, and most importantly - structured - review. So fill this gap!
')

Criteria

In order to make a comparison in principle and get some useful result, you need to understand not just the subject area, but also to have a specific list of criteria that will set the research vector. Without pretending to analyze all possible cases of Ingress / Kubernetes, we tried to highlight the most common requirements for controllers - be prepared that all of its specifics and particular in any case will have to be studied separately.

But let me begin with the characteristics that have become so familiar that they are implemented in all solutions and are not considered:

dynamic discovery of services (service discovery);
SSL termination;
work with websocket'ami.

Now - about the points of comparison:

Supported Protocols

One of the fundamental criteria for selection. Your software may not work using standard HTTP, or it may require working on multiple protocols at once. If your case is non-standard, be sure to take into account this factor, so that you do not have to reconfigure the cluster. All controllers have a list of supported protocols.

Software based

There are several application options on which the controller is based. Popular ones are nginx, traefik, haproxy, envoy. In the general case, it may not have a great influence on how traffic is received and transmitted, however, it is always useful to know the potential nuances and peculiarities of what is “under the hood”.

Traffic routing

Based on what can you decide on the direction of traffic to a particular service? Usually this is the host and path, but there are additional features.

Namespace within the cluster

Namespace (namespace) - the ability to logically split resources in Kubernetes (for example, on stage, production, etc.). There are Ingress-controllers, which must be placed separately in each namespace (and then it can direct traffic only to the pods of this space). And there are those (and their obvious majority) that work globally for the entire cluster - the traffic to them goes to any pod cluster, regardless of the namespace.

Upstream samples

How is traffic directed to healthy instances of the application, services? There are options with active and passive checks, retries (retries), circuit breakers (for more details, see, for example, in the article about Istio ) , custom implementations of custom health checks, etc. A very important parameter if you have high requirements for availability and timely withdrawal from the balancing of failed services.

Algorithms of balancing

There are a lot of options: from traditional round-robin to exotic ones like rdp-cookies , as well as individual features like sticky sessions .

Authentication

What authorization schemes does the controller support? Basic, digest, oauth, external-auth - I think that these options should be familiar. This is an important criterion if there are a lot of contours for developers (and / or just closed) that are accessed via Ingress.

Traffic distribution

Does the controller support such commonly used mechanisms for traffic distribution as canary, A / B testing, traffic mirroring (mirroring / shadowing)? This is a really sore subject for applications that require careful and accurate traffic management for productive testing, debugging product errors not in combat (or with minimal losses), traffic analysis, etc.

Paid subscription

Is there a paid version of the controller, with advanced functionality and / or technical support?

Graphical User Interface (Web UI)

Is there any graphical interface for managing the configuration of the controller? Mainly for “handyness” and / or for those who need to make some changes to the Ingress configuration, but working with “raw” templates is inconvenient. It can be useful if developers want to conduct any experiments with traffic on the fly.

JWT validation

The presence of built-in validation JSON web-tokens for authorization and validation of the user to the target application.

Possibilities for config customization

Extensibility of templates in the sense of having mechanisms that allow you to add your own directives, flags, etc. to standard configuration templates.

Basic DDOS Protection Mechanisms

Simple rate limit algorithms or more sophisticated traffic filtering options based on addresses, whitelists, countries, etc.

Query trace

Possibilities of monitoring, tracking and debugging requests from Ingress to specific services / pods, and ideally between services / pods too.

Waf

Application firewall support.

Ingress Controllers

The list of controllers was generated based on the official Kubernetes documentation and this table . Some of them we excluded from the review due to specificity or low prevalence (early stage of development). The rest are discussed below. Let's start with a general description of the solutions and continue the pivot table.

Ingress from Kubernetes

Website: github.com/kubernetes/ingress-nginx
License: Apache 2.0

This is the official controller for Kubernetes, which is developed by the community. Obviously from the name, it is based on nginx and is supplemented with a different set of Lua plug-ins used to implement additional features. Due to the popularity of nginx itself and minimal modifications over it when used as a controller, this option can be the simplest and most understandable in configuration average engineer (with web experience).

Ingress from NGINX Inc

Website: github.com/nginxinc/kubernetes-ingress
License: Apache 2.0

The official product of nginx developers. It has a paid version based on NGINX Plus . The main idea is a high level of stability, constant backward compatibility, the absence of any foreign modules and the declared increased speed (in comparison with the official controller), achieved due to the failure of Lua.

The free version is significantly curtailed, even when compared with the official controller (due to the lack of all the same Lua modules). Paid at the same time has a fairly wide additional functionality: metrics in real time, JWT validation, active health check and more. An important advantage over NGINX Ingress is full support for TCP / UDP traffic (and in the community version too!). Minus - the lack of features for traffic distribution, which, however, "has the maximum priority for developers," but takes time to implement.

Kong Ingress

Website: github.com/Kong/kubernetes-ingress-controller
License: Apache 2.0

Product developed by Kong Inc. in two versions: commercial and free. Based on nginx, the capabilities of which are expanded by a large number of modules on Lua.

It was initially focused on processing and routing API requests, i.e. as API Gateway, however at the moment it has become a full-fledged Ingress-controller. Main advantages: many additional modules (including those from third-party developers) that are easy to install and configure and with which a wide range of additional features are realized. However, the built-in functions already offer many features. Work configuration is performed using CRD resources.

An important feature of the product is work within one contour (instead of cross-namespaced) is a controversial topic: someone will seem a disadvantage (you have to create entities for each contour), and for someone - a feature (more isolation level, because If one controller is broken, then the problem is limited to only one loop).

Traefik

Website: github.com/containous/traefik
License: MIT

Proxy, which was originally created to work with request routing for microservices and their dynamic environment. From here, there are many useful features: configuration update without any reboots, support for a large number of balancing methods, web interface, forwarding of metrics, support for various protocols, REST API, canary releases and much more. A nice feature is also the support of Let's Encrypt certificates out of the box. The disadvantage is that in order to organize high availability (HA), the controller will need to install and connect its own KV storage.

Haproxy

Website: github.com/jcmoraisjr/haproxy-ingress
License: Apache 2.0

HAProxy has long been known as a proxy and traffic balancer. As part of the Kubernetes cluster, it offers a “soft” configuration update (without loss of traffic), DNS-based service discovery, dynamic configuration using the API. Full customization of the configs pattern can be attractive by replacing CM and also the possibility of using Sprig library functions in it. In general, the main focus of the decision is on high speed of work, its optimization and efficiency in consumed resources. The advantage of the controller - support a record number of different ways of balancing.

Voyager

Website: github.com/appscode/voyager
License: Apache 2.0

Based on HAproxy controller, which is positioned as a universal solution that supports wide opportunities on a large number of providers. Opportunity for balancing traffic on L7 and L4 is offered, and balancing TCP L4 traffic in general can be called one of the key features of the solution.

Contour

Website: github.com/heptio/contour
License: Apache 2.0

The basis of this solution is not only Envoy: it was developed together with the authors of this popular proxy. An important feature is the ability to split Ingress resource management using the IngressRoute CRD resources. For organizations with multiple development teams using one cluster, this helps to maximize the security of work with traffic in neighboring circuits and protect them from errors when changing Ingress resources.

It also offers an expanded set of balancing methods (there is a mirroring of requests, auto-repetition, a limit on the rate of requests and much more), detailed monitoring of the flow of traffic and failures. Perhaps for someone it will be a significant disadvantage of the lack of support for sticky sessions (although work is already underway ).

Istio ingress

Website: istio.io/docs/tasks/traffic-management/ingress
License: Apache 2.0

A comprehensive service mesh solution that is not only an Ingress controller that manages incoming traffic from outside, but also controls all traffic within the cluster. “Under the hood”, as a sidecar proxy for each service, is used by Envoy. In essence, this is a large combine that “can do everything”, and its main idea is maximum controllability, extensibility, security and transparency. With it, you can fine tune traffic routing, access authorization between services, balancing, monitoring, canary releases and much more. Read more about Istio in the “ Back to microservices with Istio ” article series.

Ambassador

Website: github.com/datawire/ambassador
License: Apache 2.0

Another Envoy based solution. It has a free and commercial version. Positioned as “completely native to Kubernetes,” which brings corresponding advantages (tight integration with the methods and entities of the K8s cluster).

comparison table

So, the culmination of the article is this huge table:

It is clickable for more detailed viewing, and is also available in the Google Sheets format.

Let's sum up

The purpose of the article is to provide a more complete understanding (though not at all exhaustive!) Of what choice to make in your particular case. As usual, each controller has its advantages and disadvantages ...

Classic Ingress from Kubernetes is good for its availability and verification, quite rich in features - in general, it should be "enough for the eyes." However, if there are increased requirements for stability, level of features and development, you should pay attention to Ingress with NGINX Plus and a paid subscription. Kong has a rich set of plug-ins (and, accordingly, the opportunities they provide), and even more in the paid version. He has ample opportunities to work as an API Gateway, dynamic configuration based on CRD-resources, as well as basic services Kubernetes.

With increased balancing requirements and authorization methods, look at Traefik and HAProxy. These are open source projects that have been proven over the years, are very stable and actively developing. Contour has been around for a couple of years, but it still looks young and has only the basic features added on top of Envoy. If there are requirements for the presence / embedding of WAF before the application, you should pay attention to the same Ingress from Kubernetes or HAProxy.

And the richest in function are products built on the basis of Envoy, especially Istio. It seems to be a complex solution, which “can do everything”, which, incidentally, means a much higher threshold of entry in configuration / launch / administration than in other solutions.

We have chosen Kubernetes's Ingress as the standard controller, which still covers 80–90% of needs. It is quite reliable, easily configurable, expanding. In general, in the absence of specific requirements, it should fit most clusters / applications. From the same universal and relatively simple products, you can recommend Traefik and HAProxy.