DPI Technology Overview - Deep Packet Inspection
For several years I have been actively engaged in the DPI theme, implementing pre-sale and direct implementation of these solutions. It prompted me to write this topic that the DPI topic on Habré is rather poorly developed, so I would like to tell a little about the devices that leading service providers and large corporate users use for intelligent traffic management in their networks, and also explain why need to.
The DPI system, as the name implies, performs an in-depth analysis of all packets passing through it. The term “deep” means analyzing the packet at the top levels of the OSI model, and not just by standard port numbers. In addition to studying packets by certain standard patterns, by which one can unambiguously determine whether a packet belongs to a particular application, say, by the header format, port numbers, etc., the DPI system also performs the so-called behavioral traffic analysis, which allows to recognize applications that do not use for data exchange pre-known headers and data structures. Bittorrent is a good example. To identify them, a sequence analysis of packets with the same characteristics, such as Source_IP: port - Destination_IP: port, packet size, opening frequency of new sessions per unit of time, etc., is carried out according to behavioral (heuristic) models corresponding to such applications. Naturally, as many manufacturers of such iron - so many interpretations of behavioral models of the relevant protocols, and hence the detection accuracy also varies. If we are talking about manufacturers, it is worth noting that the largest players and their products on the standalone DPI market are Allot Communications , Procera Networks , Cisco , Sandvine . DPI solutions integrated into routers are becoming more and more popular. So many do - Cisco, Juniper, Ericsson, etc. by the list. Such solutions, as a rule, are quite compromise, and cannot provide the whole range of services available to standalone solutions. However, for most tasks this is quite enough. Software products that run on servers (such as OpenDPI) I deliberately do not indicate, their market is very narrow and, as a rule, limited to corporate / campus networks, and this is not my profile. An important distinguishing feature of this DPI is the ability to analyze traffic by collecting various kinds of statistics broken down by applications, by tariff plans, by regions, by type of subscriber devices, etc. For this reason, the remarkable NBAR of the Cisco name, although it allows detection and monitoring of traffic across applications, is not a full-blown DPI solution, since It lacks a number of important components.
The DPI system is typically installed at the edge of the operator’s network into a break in existing uplinks that leave the border routers. Thus, all traffic that leaves or enters the network of the operator passes through the DPI, which makes it possible to monitor and control it. To solve specific tasks, you can install this system not at the network boundary, but lower it lower, closer to the end users, to the BRAS / CMTS / GGSN / ... level. This can be useful for operators who, for a number of reasons, also want to solve external channels. the task of controlling the internal. Naturally, here we are talking about rather large service providers with a large distributed network of country scales and with rather expensive channel capacities.
In the DPI market there are models for a very different wallet. The performance of the devices on the market ranges from hundreds of Mbit / s to 160 Gbit / s of FDX in a single box, which, as a rule, can be clustered. Accordingly, the cost floats very seriously - from a few thousand to millions of US dollars. In the case of corporate segments, the solutions assume low-speed connections over copper interfaces of 10/100/1000 types. Operator solutions are designed to connect multiple 1GE and 10GE links. As for completely adult solutions, so far the market for 100GE interfaces on network equipment is rather poor and expensive, but as soon as the first real business case appears, DPI vendors will offer appropriate solutions, because some of them already have workpieces.
')
The main problem of all existing DPI solutions is that in order to unambiguously determine whether a data stream belongs to one of the network applications, the device performing the traffic analysis should see both directions of the session. In other words, incoming and outgoing traffic within the same flow must pass through the same device. If the equipment understands that it sees only one direction within the session, it does not have the ability to relate this flow to any known traffic category with all the ensuing consequences. In this regard, when it comes to controlling uplinks, there is a very logical question about asymmetric traffic, which for more or less large operators is not exotic, but commonplace. Different vendors solve this problem in different ways:
Another important point that is critical for some customers is the frequency of updating signature files, based on which traffic analysis is performed. Some vendors do the update once a quarter, some - once a week. If necessary, a critical update (containing methods for detecting a new version of Skype, for example) may come out ahead of schedule. As a rule, all vendors adequately relate to the desires of customers to add some kind of new protocol to the list of supported and in every possible way help with this. It is no secret that in each local market there are specific applications that are practically absent in other countries. In Russia and the CIS countries, the most striking example is Mail.ru agent. Or, for example, a similar request may arise after the release of the next network game, which must be separated from the general data stream.
Now a logical question arises - so what now to do with all this? The operator has a sufficiently powerful tool, with the skillful use of which you can solve various problems of network operation and its development.
From the point of view of operation, the operator can control the utilization of channels connected via DPI at the application level. Previously, it solved the tasks of QoS (Quality of Service) implementation solely by means of building queues based on traffic marking with service bits in IP, 802.1q and MPLS headers, highlighting the most priority traffic (various kinds of VPNs, IPTV, SIP, etc.) , and guaranteeing him a certain bandwidth at any time. Best Effort type of traffic, which includes all home subscriber Internet traffic (HSI - High Speed ​​Internet), remained virtually unchecked, which made it possible for Bittorrent to also pick up the entire free band, which in turn led to the degradation of any other web -applications. Using DPI, the operator has the opportunity to distribute the channel between different applications. For example, at night to allow Bittorrent traffic to take more bandwidth than during the daytime, during rush hours, when a large amount of other web traffic goes on the network. Another popular measure among many mobile operators is blocking Skype-traffic, as well as any type of SIP-telephony. Instead of blocking, the operator can allow these protocols to work, but at a very low speed with a corresponding degradation of the quality of service provided by a particular application, to force the user to pay for traditional telephony services, or for a special service package allowing access to VoIP services.
The important point is that the rules, on the basis of which the shaping / blocking is performed, can be set by means of two basic bases - per-service or per-subscriber. In the first case, the simplest way is that the specific application is allowed to dispose of a certain strip. In the second, the application is linked to the band for each subscriber or group of subscribers independently of the others, which is done through the integration of DPI with the existing OSS / BSS systems of the operator. Those. You can configure the system in such a way that subscriber Vasya, who has downloaded torrents for 100 gigabytes in a week, will be limited in terms of the speed of downloading these same torrents by the end of the month to 70% of the purchased tariff. And the subscriber Petit, who bought an additional service called “Skype without problems”, will not block the traffic of the Skype application under any circumstances, but any other one is easy. You can bind to the User-Agent and allow browsing only with the help of recommended browsers; you can make clever redirects depending on the type of browser or OS. In other words, the flexibility of tariff plans and options is limited only by common sense. If we are talking about the traffic of mobile operators, then DPI allows you to control the download of each base station separately, fairly distributing the BS resources so that all users are satisfied with the quality of service. Of course, this problem can be solved by the mobile core, but it is not always budget. Since I mentioned mobile operators, I would like to note that every self-respecting manufacturer of the EPC (Evolved Packet Core) Packet Core for LTE integrates DPI functionality into its PDN-GW, which is designed for solving mobile operators.
This all sounds, of course, is not very optimistic, but for many operators for economic reasons it is much cheaper to put the DPI system to control the utilization of channels than to expand uplinks. Moreover, to do this without any significant loss of the subscriber base, since It has long been known that most of the traffic is generated by about 5% of the most active subscribers. And in this case, the operator is economically feasible to reduce the subscriber base, but pay less for aplinki, because the most active rockers will leave, because of which the operator has to pay a considerable amount each month for uplinks. This is a nightmare of any marketer, but in some cases losing customers is beneficial. The delicacy of the situation lies in the fact that sooner or later there will come a time when all operators will somehow shape something with the help of DPI. Those. if today one operator starts hacking torrents, the most active rockers will go to another at once. After that, the downloads of its channels will skip badly and customers will begin to complain that web browsing does not work well. The operator will think, calculate, and eventually buy DPI. And so on until all the players on the market acquire such a system. Of course, the DPI installation does not remove from the operator the task of periodically expanding uplinks and increasing access speed for subscribers. Just now, these extensions will not be uncontrolled. Those. the operator will always know what type of traffic and in what quantity will go through his channels, it will be predictable. Of course, when it comes to boxes worth $ 1M, it's not just uplinks, you need to understand that. My personal opinion in the first approximation, as a user of broadband Internet access services, is that something to cut and block is, of course, bad and completely wrong. But looking through the engineer’s eyes at how rapidly traffic volumes are growing, the use of DPI becomes a salvation for many operators, since torrents today are able to score tightly almost any uplink.
We smoothly moved on to the task of developing the network and its services. Looking at how subscribers use the band they bought, what applications they use, the operator can study the needs of each category of subscribers and offer them more flexible and sophisticated tariff plans. For example, based on the fact that Silver subscribers are actively using the services of third-party SIP telephony, you can offer them an additional package that allows you to use a similar service provided by the operator, but at a discount. Other subscribers, if they wish to use cheaper telephony, will be motivated to switch to a more expensive tariff, acquiring additional bonuses in the form of increased speed. You can come up with many cases, this is just one of them. Allot presented its vision of personalized services in its presentation, excerpts from which are mentioned in a material that was once published in Habré . The approach is very interesting and beneficial for both the user and the operator. The development trends of the telecommunications market are such that for operators to sell the pipe, as they are doing now, it will soon be simply unprofitable, there is a mass of studies confirming this. ARPU does not increase, competition is high, equipment needs to be upgraded more and more often, operator costs are rising, and the desire to make a profit does not go away. The objective of DPI in this context is to implement new models of providing services to the end user. Some world operators are already moving towards this idea in small steps. In Russia, obviously, this process will be long and painful, because to achieve the task, it is necessary to rebuild the subscribers' brains to a different frequency, which is very difficult, because Weaning a person not to download torrents, and buying legal content is not easy. I would not like to start a discussion on the topic “Where should I get legal content?”, This is a separate song, and I am very glad that it has moved from a dead point (using ivi, omlet, zabava, etc. together with increasing Smart TV sales). I hope these projects are not stalled. I don't dream about Netflix yet, but it would be great.
DPI perfectly knows how to work in conjunction with various VAS (Value Added Services) systems, such as antispam, antivirus, video optimizers, etc. The essence of the functional is to divert part of the traffic according to the criteria specified by the administrator, to third-party devices, for more in-depth analysis and processing.
It is quite easy to organize the provision of parental control services to users, which are becoming more and more relevant.
In the end, I would like to say a few words about what DPI is also purchased for, except for bullying subscribers. The DPI equipment, in connection with its ability to see everything and everything that happens on the network, is a very interesting device for comrades in uniform, without whom they are nowhere. With the help of DPI, special services can monitor the network activity of a particular user. You can block it VPN, HTTPS and other delights that make it impossible to analyze the content. Of course, it is possible to close users' access to sites that are undesirable to the authorities, which is very important in connection with the latest developments in legislative activity in Russia.
And finally, I would like to say a few words about the long-suffering network neutrality that exists in some countries. In short, operators, in the absence of overloads on uplinks, are now forbidden to block the traffic of legitimate / legal applications. Those. Start selective blocking of any traffic is now allowed only in the event of an overload. But, at the same time, there is still no clear wording on which applications are legitimate and which are not. Logically, only content can be illegal, not applications. For example, child pornography is clearly related to illegal content, but the HTTP and Bittorrent protocols, through which it can be transmitted, are completely legal. So there is still enough space for disputes, and the topic, in my opinion, is very interesting. So far, our network neutrality does not smell, therefore the operators have all the cards for controlling traffic using DPI.
I hope this article helped someone to structure their knowledge about DPI a little. I will think about staying at some points in more detail in the following opuses, if there is a demand for them, the topic is quite extensive. To avoid unnecessary controversy - personally, as a user of broadband access services, I am against anything being cut and blocked, which in no way prevents me from doing my job. I am happy to answer questions.
The basics
The DPI system, as the name implies, performs an in-depth analysis of all packets passing through it. The term “deep” means analyzing the packet at the top levels of the OSI model, and not just by standard port numbers. In addition to studying packets by certain standard patterns, by which one can unambiguously determine whether a packet belongs to a particular application, say, by the header format, port numbers, etc., the DPI system also performs the so-called behavioral traffic analysis, which allows to recognize applications that do not use for data exchange pre-known headers and data structures. Bittorrent is a good example. To identify them, a sequence analysis of packets with the same characteristics, such as Source_IP: port - Destination_IP: port, packet size, opening frequency of new sessions per unit of time, etc., is carried out according to behavioral (heuristic) models corresponding to such applications. Naturally, as many manufacturers of such iron - so many interpretations of behavioral models of the relevant protocols, and hence the detection accuracy also varies. If we are talking about manufacturers, it is worth noting that the largest players and their products on the standalone DPI market are Allot Communications , Procera Networks , Cisco , Sandvine . DPI solutions integrated into routers are becoming more and more popular. So many do - Cisco, Juniper, Ericsson, etc. by the list. Such solutions, as a rule, are quite compromise, and cannot provide the whole range of services available to standalone solutions. However, for most tasks this is quite enough. Software products that run on servers (such as OpenDPI) I deliberately do not indicate, their market is very narrow and, as a rule, limited to corporate / campus networks, and this is not my profile. An important distinguishing feature of this DPI is the ability to analyze traffic by collecting various kinds of statistics broken down by applications, by tariff plans, by regions, by type of subscriber devices, etc. For this reason, the remarkable NBAR of the Cisco name, although it allows detection and monitoring of traffic across applications, is not a full-blown DPI solution, since It lacks a number of important components.
The DPI system is typically installed at the edge of the operator’s network into a break in existing uplinks that leave the border routers. Thus, all traffic that leaves or enters the network of the operator passes through the DPI, which makes it possible to monitor and control it. To solve specific tasks, you can install this system not at the network boundary, but lower it lower, closer to the end users, to the BRAS / CMTS / GGSN / ... level. This can be useful for operators who, for a number of reasons, also want to solve external channels. the task of controlling the internal. Naturally, here we are talking about rather large service providers with a large distributed network of country scales and with rather expensive channel capacities.
In the DPI market there are models for a very different wallet. The performance of the devices on the market ranges from hundreds of Mbit / s to 160 Gbit / s of FDX in a single box, which, as a rule, can be clustered. Accordingly, the cost floats very seriously - from a few thousand to millions of US dollars. In the case of corporate segments, the solutions assume low-speed connections over copper interfaces of 10/100/1000 types. Operator solutions are designed to connect multiple 1GE and 10GE links. As for completely adult solutions, so far the market for 100GE interfaces on network equipment is rather poor and expensive, but as soon as the first real business case appears, DPI vendors will offer appropriate solutions, because some of them already have workpieces.
')
The main problem of all existing DPI solutions is that in order to unambiguously determine whether a data stream belongs to one of the network applications, the device performing the traffic analysis should see both directions of the session. In other words, incoming and outgoing traffic within the same flow must pass through the same device. If the equipment understands that it sees only one direction within the session, it does not have the ability to relate this flow to any known traffic category with all the ensuing consequences. In this regard, when it comes to controlling uplinks, there is a very logical question about asymmetric traffic, which for more or less large operators is not exotic, but commonplace. Different vendors solve this problem in different ways:
- Cisco is content with half the session and is trying to determine the type of network application using only this data. Obviously, with this technique, the accuracy of detection of applications suffers, especially those for which behavioral analysis models are required. Also in this implementation there are a number of restrictions imposed on the ability to control such traffic, each vendor has their own.
- Sandvine uses the following idea to solve the problem of asymmetric traffic - all traffic that is asymmetric is sent to all DPI devices in a single domain using encapsulation in broadcast frames. As a result of this transfer, devices that previously saw only one direction within the session will see the second, on the basis of which it will be possible to implement a full set of measures for analyzing and managing traffic. The disadvantage of this scheme is obvious - with large volumes of asymmetric traffic on the network, there are serious requirements for communication channels connecting DPI devices at different sites. In some cases, when it comes to the asymmetry of orders of several gigabits (or tens of gigabits) per second, this technique is not applicable due to the high overhead costs of organizing a channel between sites.
- The smarter ones are Procera and Allot. The idea is similar to the implementation of Sandvine with the difference that not asymmetric traffic is sent between sites, but metadata, clearly characterizing it. In general, we can assume that these are protocol headings, although in reality everything is a bit more complicated. Due to this optimization, the requirements for cross-site communication channels are much more humane, with respect to the implementation of Sandvine the gain can be up to 95%. Anticipating some comments, I will answer right away - yes, it works, confirmed in practice on production networks, implemented personally with my own hands.
Another important point that is critical for some customers is the frequency of updating signature files, based on which traffic analysis is performed. Some vendors do the update once a quarter, some - once a week. If necessary, a critical update (containing methods for detecting a new version of Skype, for example) may come out ahead of schedule. As a rule, all vendors adequately relate to the desires of customers to add some kind of new protocol to the list of supported and in every possible way help with this. It is no secret that in each local market there are specific applications that are practically absent in other countries. In Russia and the CIS countries, the most striking example is Mail.ru agent. Or, for example, a similar request may arise after the release of the next network game, which must be separated from the general data stream.
What's next?
Now a logical question arises - so what now to do with all this? The operator has a sufficiently powerful tool, with the skillful use of which you can solve various problems of network operation and its development.
QoS implementation
From the point of view of operation, the operator can control the utilization of channels connected via DPI at the application level. Previously, it solved the tasks of QoS (Quality of Service) implementation solely by means of building queues based on traffic marking with service bits in IP, 802.1q and MPLS headers, highlighting the most priority traffic (various kinds of VPNs, IPTV, SIP, etc.) , and guaranteeing him a certain bandwidth at any time. Best Effort type of traffic, which includes all home subscriber Internet traffic (HSI - High Speed ​​Internet), remained virtually unchecked, which made it possible for Bittorrent to also pick up the entire free band, which in turn led to the degradation of any other web -applications. Using DPI, the operator has the opportunity to distribute the channel between different applications. For example, at night to allow Bittorrent traffic to take more bandwidth than during the daytime, during rush hours, when a large amount of other web traffic goes on the network. Another popular measure among many mobile operators is blocking Skype-traffic, as well as any type of SIP-telephony. Instead of blocking, the operator can allow these protocols to work, but at a very low speed with a corresponding degradation of the quality of service provided by a particular application, to force the user to pay for traditional telephony services, or for a special service package allowing access to VoIP services.
Subscriber management
The important point is that the rules, on the basis of which the shaping / blocking is performed, can be set by means of two basic bases - per-service or per-subscriber. In the first case, the simplest way is that the specific application is allowed to dispose of a certain strip. In the second, the application is linked to the band for each subscriber or group of subscribers independently of the others, which is done through the integration of DPI with the existing OSS / BSS systems of the operator. Those. You can configure the system in such a way that subscriber Vasya, who has downloaded torrents for 100 gigabytes in a week, will be limited in terms of the speed of downloading these same torrents by the end of the month to 70% of the purchased tariff. And the subscriber Petit, who bought an additional service called “Skype without problems”, will not block the traffic of the Skype application under any circumstances, but any other one is easy. You can bind to the User-Agent and allow browsing only with the help of recommended browsers; you can make clever redirects depending on the type of browser or OS. In other words, the flexibility of tariff plans and options is limited only by common sense. If we are talking about the traffic of mobile operators, then DPI allows you to control the download of each base station separately, fairly distributing the BS resources so that all users are satisfied with the quality of service. Of course, this problem can be solved by the mobile core, but it is not always budget. Since I mentioned mobile operators, I would like to note that every self-respecting manufacturer of the EPC (Evolved Packet Core) Packet Core for LTE integrates DPI functionality into its PDN-GW, which is designed for solving mobile operators.
Why is all this necessary?
This all sounds, of course, is not very optimistic, but for many operators for economic reasons it is much cheaper to put the DPI system to control the utilization of channels than to expand uplinks. Moreover, to do this without any significant loss of the subscriber base, since It has long been known that most of the traffic is generated by about 5% of the most active subscribers. And in this case, the operator is economically feasible to reduce the subscriber base, but pay less for aplinki, because the most active rockers will leave, because of which the operator has to pay a considerable amount each month for uplinks. This is a nightmare of any marketer, but in some cases losing customers is beneficial. The delicacy of the situation lies in the fact that sooner or later there will come a time when all operators will somehow shape something with the help of DPI. Those. if today one operator starts hacking torrents, the most active rockers will go to another at once. After that, the downloads of its channels will skip badly and customers will begin to complain that web browsing does not work well. The operator will think, calculate, and eventually buy DPI. And so on until all the players on the market acquire such a system. Of course, the DPI installation does not remove from the operator the task of periodically expanding uplinks and increasing access speed for subscribers. Just now, these extensions will not be uncontrolled. Those. the operator will always know what type of traffic and in what quantity will go through his channels, it will be predictable. Of course, when it comes to boxes worth $ 1M, it's not just uplinks, you need to understand that. My personal opinion in the first approximation, as a user of broadband Internet access services, is that something to cut and block is, of course, bad and completely wrong. But looking through the engineer’s eyes at how rapidly traffic volumes are growing, the use of DPI becomes a salvation for many operators, since torrents today are able to score tightly almost any uplink.
New service model
We smoothly moved on to the task of developing the network and its services. Looking at how subscribers use the band they bought, what applications they use, the operator can study the needs of each category of subscribers and offer them more flexible and sophisticated tariff plans. For example, based on the fact that Silver subscribers are actively using the services of third-party SIP telephony, you can offer them an additional package that allows you to use a similar service provided by the operator, but at a discount. Other subscribers, if they wish to use cheaper telephony, will be motivated to switch to a more expensive tariff, acquiring additional bonuses in the form of increased speed. You can come up with many cases, this is just one of them. Allot presented its vision of personalized services in its presentation, excerpts from which are mentioned in a material that was once published in Habré . The approach is very interesting and beneficial for both the user and the operator. The development trends of the telecommunications market are such that for operators to sell the pipe, as they are doing now, it will soon be simply unprofitable, there is a mass of studies confirming this. ARPU does not increase, competition is high, equipment needs to be upgraded more and more often, operator costs are rising, and the desire to make a profit does not go away. The objective of DPI in this context is to implement new models of providing services to the end user. Some world operators are already moving towards this idea in small steps. In Russia, obviously, this process will be long and painful, because to achieve the task, it is necessary to rebuild the subscribers' brains to a different frequency, which is very difficult, because Weaning a person not to download torrents, and buying legal content is not easy. I would not like to start a discussion on the topic “Where should I get legal content?”, This is a separate song, and I am very glad that it has moved from a dead point (using ivi, omlet, zabava, etc. together with increasing Smart TV sales). I hope these projects are not stalled. I don't dream about Netflix yet, but it would be great.
DPI perfectly knows how to work in conjunction with various VAS (Value Added Services) systems, such as antispam, antivirus, video optimizers, etc. The essence of the functional is to divert part of the traffic according to the criteria specified by the administrator, to third-party devices, for more in-depth analysis and processing.
It is quite easy to organize the provision of parental control services to users, which are becoming more and more relevant.
Special services
In the end, I would like to say a few words about what DPI is also purchased for, except for bullying subscribers. The DPI equipment, in connection with its ability to see everything and everything that happens on the network, is a very interesting device for comrades in uniform, without whom they are nowhere. With the help of DPI, special services can monitor the network activity of a particular user. You can block it VPN, HTTPS and other delights that make it impossible to analyze the content. Of course, it is possible to close users' access to sites that are undesirable to the authorities, which is very important in connection with the latest developments in legislative activity in Russia.
Net neutrality
And finally, I would like to say a few words about the long-suffering network neutrality that exists in some countries. In short, operators, in the absence of overloads on uplinks, are now forbidden to block the traffic of legitimate / legal applications. Those. Start selective blocking of any traffic is now allowed only in the event of an overload. But, at the same time, there is still no clear wording on which applications are legitimate and which are not. Logically, only content can be illegal, not applications. For example, child pornography is clearly related to illegal content, but the HTTP and Bittorrent protocols, through which it can be transmitted, are completely legal. So there is still enough space for disputes, and the topic, in my opinion, is very interesting. So far, our network neutrality does not smell, therefore the operators have all the cards for controlling traffic using DPI.
Instead of conclusion
I hope this article helped someone to structure their knowledge about DPI a little. I will think about staying at some points in more detail in the following opuses, if there is a demand for them, the topic is quite extensive. To avoid unnecessary controversy - personally, as a user of broadband access services, I am against anything being cut and blocked, which in no way prevents me from doing my job. I am happy to answer questions.
Source: https://habr.com/ru/post/111054/
All Articles