Another article about caching web traffic

Introduction, or why another WCCP article?

Much has been written about organizing transparent caching of web traffic using the WCCP protocol, including a good article on Habré . Usually in these articles a scheme similar to that shown in the figure to the left is considered.



At first glance, the solution has solid advantages: the implementation is simple, caching is performed completely transparently to users, and if the proxy server fails, requests will automatically be redirected directly.
')
But does WCCP implementation always go smoothly? And if not, how to deal with emerging issues?

For example, in almost all articles it is mentioned that the caching server must be located in the same segment as the users, but the reasons for this are not specified. And what if the security policy requires all servers to be in the demilitarized zone and protected by a firewall (ME)?

Recently, we had to deal with a similar requirement when installing a caching server in a carrier network, a simplified diagram of which is shown in the title picture on the right.

If readers are interested in what problems they encounter when implementing such schemes, and how you can get around the limitations - welcome.

Theory - standards and implementation features

First, a little theory. WCCP is designed to redirect traffic (not just the web) in real time. Initially, the protocol was developed by Cisco, then became an open standard used by most vendors.

To date, version 2, which is in the status of Internet-Draft and described by the document draft-mclaggan-wccp-v2rev1-00, is relevant .

Let us dwell on several important points in the operation of this protocol (see figure).



All WCCP messages are UDP packets with a destination port number of 2048. The messaging order is as follows:

1. If the server is ready to handle traffic caching requests, it sends WCCP2_HERE_I_AM messages.
2. The router sends to the server a WCCP2_I_SEE_YOU message containing information about the settings, in particular, the “Receive ID” field.
3. The server in response sends another WCCP2_HERE_I_AM message, which contains the “Receive ID” field with the same value as in the previous step, thereby confirming that it is ready to work with the router.
4. The router, having received such a message, understands that from now on, user requests to websites should be redirected to the caching server.

The system is ready to go. The WCCP2_HERE_I_AM and WCCP2_I_SEE_YOU messaging process is repeated periodically (by default - once every 10 seconds), and if the router does not receive a response from the caching server, the latter is excluded from the process.
In reality, the protocol is somewhat more complicated, it provides for authentication, various redirection algorithms, etc., but we will deliberately omit details that are not important for further understanding. Interested readers can find them in the corresponding draft, the link to which is given above.

This implementation contributes to the fault tolerance of the solution — if the caching server fails and stops sending WCCP2_HERE_I_AM messages, the router stops trying to forward packets and starts sending them to the Internet directly. After the service is restored, the message exchange process of WCCP2_HERE_I_AM / WCCP2_I_SEE_YOU will repeat and the caching scheme will start working again.

For users, such a refusal is either completely imperceptible, or it may look like a one-time “Unable to connect” message, which will disappear after reloading the page in the browser.

In Wireshark, the WCCP messaging process looks like the following figure. Note the Time column. The traffic image is taken from a real-life system, therefore IP addresses are shown in a truncated form for security purposes.



Let's see what happens when a client tries to get data from a web server. For clarity, we will assign specific IP addresses to the hosts using the special ranges allocated for use in the examples , and for simplicity, we will exclude from consideration all the extra functionality (NAT, firewalling, etc.).

1. The user browser initiates a TCP session by sending a packet with SRC IP 198.51.100.150, DST IP 192.0.2.20, DST TCP port 80, with the TCP SYN flag.
2. The router, having received such a packet, does not send it further to the Internet, but packs it entirely into a GRE packet and sends it to the caching server. The GRE package has SRC IP 192.51.100.1 and DST IP 198.51.100.100, respectively. In Wireshark, it looks like the following figure.
   
3. The caching server, upon receiving such a packet, first of all decides whether it will process this packet. If not, the packet is sent back to the router for normal forwarding through the same GRE tunnel, and the algorithm ends. If so, the server proceeds to the next step.
4. The cache server on its own behalf establishes a connection with the web server, for which it sends a packet with SRC IP 198.51.100.100, DST IP 192.0.2.20, DST TCP port 80, with the TCP SYN flag.
5. In response, the web server sends a packet with SRC IP 192.0.2.20, SRC TCP port 80, DST IP 198.51.100.100, TCP SYN / ACK flags, i.e., so far everything goes according to the usual beginning of a TCP session using the three- way handshake.
6. The caching server, after receiving a response from the web server, does two things:
   
   • sends to the web server a packet with SRC IP 198.51.100.100, DST IP 192.0.2.20, DST TCP port 80, ACK flag, i.e., it continues normal TCP session, which looks to the web server as if it were normal client with IP address 198.51.100.100.
     
   • sends to the web client a packet with SRC IP 192.0.2.20, SRC TCP port 80, DST IP 198.51.100.150, TCP SYN / ACK flags, i.e. the situation for the client looks as if the web server responded directly to it. Remember this moment, it is key to further understanding.
   
   
7. So, we have two established TCP sessions, one between the client and the caching server, the other between the caching server and the web server. The caching server receives the content from the web server in the usual way, transmits it to the client, simultaneously storing it in memory or (and) on disk.
   Upon subsequent access to the same content, the caching server, under certain conditions, will not be able to download it again to the web server, but give it to the web client on its own.
   

The described algorithm is shown schematically in the figure.



Pay attention to several important points:

1. Packets within the GRE tunnel are sent primarily from the router to the cache server (except when the cache server cannot process the packet, and sends it back to the router for normal forwarding).
2. In the opposite direction, that is, from the caching server to the web client, the packets are sent directly, bypassing the router altogether.
3. The caching server sets for packages in the direction of the web client not its address, but the address of the website to which the request was made.

This implementation of the protocol significantly reduces the load on the router, since it only has to redirect traffic from the web client to the web server, which is usually small. Traffic from a web server, which is usually quite large, is not subjected to any complicated processing — it is simply routed.

But such an implementation creates asymmetric traffic, which, in turn, generates the difficulties discussed in the next section.

Practice - fighting routers and firewalls

Modifying the previous scheme - let's place the caching server behind the firewall:



We will assume that we use popular hardware — a Cisco router with Cisco IOS software version 12.3 and higher, a Cisco ASA firewall with software version 8.2 and higher, a Linux-based caching server (RHEL or CentOS distribution), and Squid caching software.
How in this case to set everything up? Suppose that the basic functionality is already configured, that is, the web client and the caching server are able to access resources on the Internet. We start by setting up WCCP on Cisco.

We will carry out the preparatory work, for which we will create two access lists:

ip access-list standard l_wccp_service permit 203.0.113.100 ip access-list extended l_wccp_redirect permit tcp host 198.51.100.150 any eq www 

The first determines which caching servers are allowed to receive WCCP2_HERE_I_AM messages.
The second determines what traffic needs to be routed to the caching server.

Configure WCCP and enable it on the interface, looking in the direction of internal users, ie, having the address 198.51.100.1. For definiteness, let it be FastEthernet0 / 0):

 ip wccp web-cache redirect-list l_wccp_redirect group-list l_wccp_service interface FastEthernet0/0 ip wccp web-cache redirect in 

On the firewall, we allow the exchange of WCCP and GRE packets between the router and the cache server.

 access-list l_wccp extended permit gre host 198.51.100.1 host 203.0.113.100 access-list l_wccp extended permit udp host 198.51.100.1 host 203.0.113.100 access-group l_wccp in interface outside 

Now configure the caching server. First, install and configure squid, for which, using your favorite text editor, open the /etc/squid/squid.conf file and make sure that it contains the following lines:

 # /etc/squid/squid.conf http_port 3128 transparent wccp2_router 198.51.100.1 wccp2_forwarding_method 1 wccp2_return_method 1 wccp2_assignment_method hash wccp2_service standard 0 

Let's create a tunnel interface, for which, again in our favorite editor, create the file / etc / sysconfig / network-scripts / ifcfg-tun0 with the following contents:

 # /etc/sysconfig/network-scripts/ifcfg-tun0 DEVICE=tun0 BOOTPROTO=none ONBOOT=yes TYPE=GRE PEER_OUTER_IPADDR=198.51.100.1 PEER_INNER_IPADDR=192.168.168.1 MY_INNER_IPADDR=192.168.168.2 

The PEER_INNER_IPADDR and MY_INNER_IPADDR IP addresses can be absolutely any — nothing will be routed through this tunnel in the normal way. Instead, all incoming TCP traffic from DST port 80 will be wrapped on squid using iptables. Assuming that squid is responding on port 3128, we will raise the tunnel interface and wrap the necessary traffic on squid:

 /etc/sysconfig/network-scripts/ifup tun0 iptables -t nat -A PREROUTING -i tun0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 203.0.113.100:3128 /etc/init.d/iptables save 

Check that the caching server is registered on the router:

 cisco# show ip wccp Global WCCP information: Router information: Router Identifier: 198.51.100.1 Protocol Version: 2.0 Service Identifier: web-cache Number of Service Group Clients: 1 Number of Service Group Routers: 1 Total Packets s/w Redirected: 175623 Process: 0 Fast: 0 CEF: 175623 Redirect access-list: l_wccp_redirect Total Packets Denied Redirect: 113892411 Total Packets Unassigned: 20590 Group access-list: l_wccp_service Total Messages Denied to Group: 26558 Total Authentication failures: 0 Total Bypassed Packets Received: 0 

Here we can expect an unpleasant ambush: the router usually has several interfaces with different IP addresses. And nothing prevents him from sending WCCP2_I_SEE_YOU packets from the SRC IP of one interface, and GRE packets from the SRC IP of another interface.
Some, but not all, firmware versions of Cisco IOS routers have a “ip wccp source-interface” command that allows you to hard-code an interface whose IP address will be used as the SRC IP for all packets related to the WCCP subsystem .

If your router supports this command, you're in luck. Run it:

 ip wccp source-interface FastEthernet 0/0 

If, in response to such a command, the router issues something like “Syntax error”, we proceed as follows - we run diagnostics on the ME, and on the caching server some kind of network analyzer (at least tcpdump) and find out which IP addresses come from WCCP packets, and with what — GRE packets.

Next, in the squid settings, we register the first IP address, in the settings of the tunnel interface and iptables - the second one. We modify accordingly the access lists on the ME.

To prevent the IP address from which WCCP packets come from between the interfaces during subsequent reconfiguration of the router, you can create a loopback interface on the last one. In this case, WCCP will use the largest IP address among all loopback interfaces to send its packets.

 interface lo0 ip address 198.51.100.20 255.255.255.255 

Check that the redirect works. First, make sure that the package counters in the access lists created earlier grow:

 cisco# show access-list l_wccp_redirect Extended IP access list l_wccp_redirect 10 permit tcp host 198.51.100.150 any eq www (2399 matches) 

Then open an arbitrary web page in the browser of the client machine. And surely we will not succeed. When trying to figure it out, we will surely find messages in the firewall logs of the following form:

 %ASA-4-313004: Denied ICMP type=0, from 192.0.2.20 on interface dmz to 198.51.100.150: no matching session 

If we try to google it, the first link will tell us something about asymmetric routing. We will understand what this means.
The Cisco ASA Firewall is a device running in Stateful Inspection mode, i.e., in order to pass a packet with TCP SYN / ACK flags from the caching server to the client, it is necessary that the corresponding packet with the TCP SYN flag from the client went to the website in the forward direction through the same ME.

In this case, the DOE will understand that the client has initiated a TCP session, will create appropriate internal structures and will begin to correctly monitor the status of this TCP session.

In our scheme, the initiating SYN packet passes through ME a) inside the GRE tunnel and b) “in the wrong direction”.
Accordingly, the ME does not start a TCP session in its table of connections and cannot understand that the session has started and its packets must be skipped.

What to do in this situation? If you can’t connect the caching server bypassing MEs, all that remains is to disable the check for an open TCP session for packets coming from the DMZ.

In Cisco ASA, the disable check feature is called TCP bypass . The function has limitations:

1. Does not work on Cisco ASA with software version younger than 8.2.
2. Unknown (at least, we were not able to find) a way to organize both the client zone and the DMZ on the same ME model of the Cisco ASA at the same time — predictably, IP address translation does not work.

So, turn on the TCP bypass function:

 access-list l_bypass extended permit tcp any eq www host 198.51.100.150 class-map c_bypass match access-list l_bypass policy-map p_bypass class c_bypass set connection advanced-options tcp-state-bypass service-policy p_bypass interface dmz 

You must put a range of client IP addresses in the l_bypass access list.

Now everything should work. At least it worked for us.

Conclusion

The article is based on the experience of introducing the function of caching web traffic in a small carrier network, and once again illustrates two old principles in the work of a network engineer:

• Do not neglect the standards and descriptions of the protocol;
• you do not understand what is happening - do not be lazy, connect the network analyzer.

Happy testing and implementation! And let now and always your channels transport as little extra traffic as possible.