How we pierced the Great Chinese Firewall (part 3)

Hello!
Any good stories end. And our story about how we came up with a solution to the rapid passage of the Chinese Firewall is not an exception. Therefore, I hasten to share with you the last, final part on this topic.

In the previous part, we talked about a lot of test benches invented by us, and what results they gave. And we settled on the fact that it would be nice to add a CDN! for viscosity in our scheme.

I will tell you how we tested the Alibaba Cloud CDN, Tencent Cloud CDN and Akamai, and what we ended up with. And of course, let's summarize.

Alibaba Cloud CDN

We are hosted by Alibaba Cloud, using IPSEC and CEN from them. It would be logical to try them first.

Alibaba Cloud has two types of product that may suit us: CDN and DCDN . The first option is a classic CDN for a specific domain (subdomain). The second variant is decoded as Dynamic Route for CDN (I call it dynamic CDN), it can be enabled in Full-site mode (for wildcard domains), it also caches static and accelerates dynamic content on itself, that is, page dynamics will also be loaded through fast network provider. This is important for us, because basically we have a dynamic site, it uses many subdomains, and it is more convenient to configure the CDN once for “asterisk” - * .semrushchina.cn.

We have already seen this product at the earlier stages of our Chinese project, but then it did not work yet, and the developers promised that the product would be available for all customers. And he became.

In DCDN you can:

• configure SSL termination with your certificate,
• enable dynamic content acceleration
• flexibly configure the caching of static files,
• do cache purge,
• prokidyvat web sockets,
• enable compression and even html beautifier.

In general, everything is like for adults and large CDN providers.

After the Origin (the place where the CDN edge servers go) is specified, it remains to create a CNAME for an asterisk, referring to all.semrushchina.cn.w.kunluncan.com (this CNAME was received on Alibaba Cloud console), and CDN will earn.

According to the test results, this CDN helped us a lot. Statistics below.

These are very good results, especially if you compare them with what the numbers were at the beginning. But we knew that the browser test of the American version of our site www.semrush.com works out from the United States for an average of 8.3s (a very approximate value). There is much to strive for. Moreover, there were still CDN-providers that were interesting to test.

So we smoothly move on to another giant in the Chinese market - Tencent .

Tencent cloud

Tencent is only developing its cloud - this is evident from a small number of products. During its use, we wanted to test not only their CDN, but also the network infrastructure in general:

• Do they have something like CEN?
• how does ipsec work for them? Is it fast, what uptime?
• do they have anycast?

Let us examine these questions separately.

Analog CEN

Tencent has a Cloud Connect Network ( CCN ) product that allows you to interconnect VPCs from different regions, including regions inside and outside of China. The product is now in the internal beta, and you need to create a ticket with a request to connect to it. From the support, we learned that global accounts (it’s not about Chinese citizens and non-legal entities) cannot participate in the beta testing program and generally connect the region inside China with the region outside. 1-0 in favor of Ali Cloud

IPSEC

Tencent's southernmost region is Guangzhou . We collected the tunnel and connected it to the Hong Kong region in the GCP (then this region was already available). Also simultaneously raised the second tunnel in Ali Cloud from Shenzhen to Hong Kong. It turned out that through the Tencent latency network to Hong Kong is generally better (10ms) than from Shenzhen to Hong Kong to Ali (120ms - what?). But this did not accelerate the work of the site aimed at working through Tencent and this tunnel, which in itself was an amazing fact and once again proved the following: latency - for China it is not an indicator that you really should pay attention to when developing a solution for passing the Chinese firewall.

Anycast Internet Acceleration

Another product that allows working through anycast IP is AIA . But it is also not available to global accounts, so I won’t tell you about it, but knowing that such a product exists can be useful.

But the CDN test showed quite interesting results. Tencent CDN cannot be enabled on the full-site, only on specific domains. We started domains and allowed traffic to them:

It turned out that this CDN has the following function: Cross Border Traffic Optimization . This feature should reduce costs when passing traffic through the Chinese firewall. As Origin , the IP address of Google GLB (GLB anycast) was specified. So we wanted to simplify the project architecture.

The results were very good - at the level of Ali Cloud CDN, and sometimes even better. This is surprising, because if the tests succeed, you can refuse a significant part of the infrastructure, tunnels, CEN, virtual machines and so on.

Were not happy for long, as the problem was revealed: the tests in Catchpoint faylilis for the Internet provider China Mobile. From any locations we received a timeout through Tencent's CDN. Correspondence with technical support did not lead to anything. About a day we tried to solve this problem, but nothing happened.

I was in China at the time, but I could not find public Wi-Fi on this provider’s network in order to verify the problem personally. Otherwise, everything looked fast and good.
However, due to the fact that the operator China Mobile is among the three largest operators, we were forced to return traffic to the Ali CDN.
But overall, this was a rather interesting solution that deserves more lengthy testing and troubleshooting of this problem.

Akamai

The last CDN provider we tested is Akamai . This is a huge provider that has its own network in China. Of course, we could not get past him.

From the very beginning, we agreed with Akamai about the test period so that we could switch the domain and see how it will work on their network. I will describe the result of the whole test in the form “What I liked” and “What I did not like”, and also give the results of the tests.

What you liked:

• The guys from Akamai helped a lot in all questions and accompanied us at all stages of testing. Constantly trying to improve something on their side. They gave good technical advice.
• Akamai works about 10-15% slower than our solution through Ali Cloud CDN. It is impressive that in Origin for Akamai we indicated the IP address of GLB, that is, the traffic did not go through our solution (potentially, it is possible to refuse part of the infrastructure). Still, the test results showed that this solution is worse than our current version (comparative results below).
• It was tested as Origin GLB, and Origin in China. Both options are about the same.
• There is Sure Route (automatic optimization of routing). You can host a test object on the Origin, and the Akamai Edge servers will try to pick it up (normal GET). For these requests, speed and other metrics are measured, on the basis of which the Akamai network optimizes routes so that traffic goes faster for our site and it was clear that the inclusion of this feature really had a strong effect on the speed of the site.
• Versioning the configuration in the web interface is awesome. You can make Compare for versions, see diff. View previous versions.
• You can roll out the new version at first only on the Staging network of Akamai - the same network as production, only this way does not affect real users. For this test, you need to spoof DNS records on a local machine.
• Very fast download speeds through their network of large static, and, apparently, any other files. The file from the “cold” cache gets several times faster than the same file from the “cold” cache Ali CDN. From the “hot” cache, the speed is already plus or minus the same.

Ali CDN test:

root@shenzhen1:~# curl -o /dev/null -w@curl_time https://en.semrushchina.cn/my_reports/build/scripts/simpleInit.js?v=1551879212 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5757k 0 5757k 0 0 513k 0 --:--:-- 0:00:11 --:--:-- 526k time_namelookup: 0.004286 time_connect: 0.030107 time_appconnect: 0.117525 time_pretransfer: 0.117606 time_redirect: 0.000000 time_starttransfer: 0.840348 ---------- time_total: 11.208119 ---------- size_download: 5895467 Bytes speed_download: 525999.000B/s 

Akamai test:

 root@shenzhen1:~# curl -o /dev/null -w@curl_time https://www.semrushchina.cn/my_reports/build/scripts/simpleInit.js?v=1551879212 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5757k 0 5757k 0 0 1824k 0 --:--:-- 0:00:03 --:--:-- 1825k time_namelookup: 0.509005 time_connect: 0.528261 time_appconnect: 0.577235 time_pretransfer: 0.577324 time_redirect: 0.000000 time_starttransfer: 1.327013 ---------- time_total: 3.154850 ---------- size_download: 5895467 Bytes speed_download: 1868699.000B/s 

We noticed that the situation from the example above depends on various factors. At the time of this writing, I conducted the test again. Results for both platforms were about the same. This tells us that the Internet in China, even for large operators and cloud providers, behaves differently from time to time.

I’ll add a big plus to Akamai to the previous point: if Ali has similar flashes of high performance and very low (this also applies to Ali CDN, and Ali CEN, and Ali IPSEC), then Akamai every time, no matter how I test their network, everything works stably.
Akamai really has a large coverage in China and works through many providers.

What did not like:

• Do not like the web interface and the scheme of work - such zero. But basically you get used to it (probably).
• Test results worse than our site.
• There are more errors during tests than on our site (uptime below).
• No their own DNS servers in China. Hence a lot of errors in the tests due to the DNS resolve timeout.
• Do not provide their IP ranges -> there is no possibility to register the correct set_real_ip_from on our servers.

Metrics (~ 3626 runs; all metrics except Uptime, in ms; statistics for one time interval):

Percentile distribution (in ms):

The conclusion is: the Akamai version is viable, but it does not provide the same stability and speed indicators as our own solution, along with Ali CDN.

Small notes

Some moments were not included in the narration, but I would like to write about them too.

Beijing + Tokyo and Hong Kong

As I said above, we tested the IPSEC tunnel to Hong Kong (HK). But we also tested CEN to HK. It costs a bit cheaper, and it was interesting how it will work between cities with a distance of ~ 100km. Interesting was the fact that latency between these cities is 100ms higher than in our original version (before Taiwan). Speed, stability was also better for Taiwan. As a result, we left the HK as a backup IPSEC region.

In addition, we tried to raise this installation:

• customer termination in Beijing,
• IPSEC and CEN to Tokyo,
• Ali CDN listed Beijing as the origin server.

This scheme was not so stable, although it was not inferior in speed to our solution. As for the tunnel, I saw periodic drops even for CEN, which was supposed to be stable. Therefore, we returned to the old scheme and dismantled this staging.

Below are statistics on latency between different regions through different channels. Maybe someone it will be interesting.

IPsec
Ali cn-beijing <-> GCP asia-northeast1 - 193ms
Ali cn-shenzhen <-> GCP asia-east2 - 91ms
Ali cn-shenzhen <-> GCP us-east4 - 200ms

CEN
Ali cn-beijing <---> Ali ap-northeast-1 - 54ms (!)
Ali cn-shenzhen <---> Ali cn-hongkong - 6ms (!)
Ali cn-shenzhen <---> Ali us-east1 - 216ms

General information about the Internet in China

As an addition to the problems with the Internet, described at the very beginning, in the first part of the article.

• The Internet in China works pretty fast inside.
  
  • The conclusion is made on the basis of testing public Wi-Fi networks in various locations where these networks are used by a large number of people.
  • The speed of downloads and uploads to servers inside China was about 20 Mb / s and 5-10 Mb / s, respectively.
  • The speed to the servers outside of China is just miserable, less than 1 Mb / s.
• The Internet in China is not very stable.
  
  • Sometimes sites can open quickly, sometimes slowly (at the same time of day on different days), provided that the configuration does not change. We observed this by the example of semrushchina.cn. This can be attributed to Ali CDN, which also works this way or that, depending on the time of day, the position of the stars, etc.
• Mobile Internet is almost everywhere 4G or 4G +. Catching in the subway, elevators - in short, everywhere.
• The fact that Chinese users only believe domains in the .cn zone is a myth. We learned this directly from users.
  
  • You can see how http://baidu.cn redirects at www.baidu.com (in mainland China as well).
• Many resources are really blocked. Primitive: google.com, Facebook, Twitter. But many Google resources work (of course, not on all Wi-Fi and VPN is not used at the same time (on the router side too, that's for sure).
• Many “technical” domains of blocked corporations also work. This means that one should not always foolishly cut out all Google and other seemingly blocked resources. We need to look for some list of denied domains.
• They have only three main Internet operators: China Unicom, China Telecom, China Mobile. There are still smaller, but their market share is insignificant

Bonus: the final solution

Total

A year has passed since the start of the project. We started with the fact that our site in general, in principle, refused to work normally from China, but simply GET curl took 5.5 seconds.

Then, with such indicators at the first decision (Cloudflare):

We finally got to these results (statistics for the last month):

As you can see, uptime in 100% has not yet been achieved, but we will think of something, and then we will tell you about the results in a new article :)

To the one who read to the end all three parts - Respect. I hope you all this was just as interesting as I was when I did it.

PS Previous parts

Part 1
Part 2