Developing a VPN Client for Android (Part 1)
Hello! The reason for writing this article was the awareness of the fact that when there are a large number of articles and reviews about VPN application clients for Android, there is not a single normal article describing development problems using the VpnService API. Moreover, in most cases you, as an application developer, cannot do anything with these problems.
Let's start from the beginning. Some time ago, on the basis of research conducted in the field of security of mobile devices, our company decided to release a small application ( WebGuard ) under Android to block all already annoying ads, as well as protection from surveillance and viruses when working from any browser. To implement this functionality, we needed to solve many problems, the most time consuming of which was the task of intercepting and processing application traffic. To intercept and filter browser connections, it was decided to use the VpnService API, which appeared in Android from version 4.0.3 and provides all the necessary functionality (though at times this functionality simply does not work for a bunch of different reasons, but it turned out a little later).
Here it is worthwhile to tell in more detail why this API was chosen and what methods exist at all for intercepting network traffic of applications by another application on a “non-routed” android device. Actually, there are only three methods (apart from various vulnerabilities) and it will be useful for application developers with various mechanisms of in-game purchases to know about them. Because many of them believe that the substitution of network traffic for an application is possible only on the “root” device and does not bother with the protection of transmitted data, although there is a possibility (and quite large) of the appearance of “cheat” applications that make changes to the transmitted data.
So, the first way is to install a local proxy server . Most antiviruses for Android use this method. An application for intercepting traffic just needs to implement the HTTP proxy support described in RFC 2068 and set the proxy server or APN in the settings of the mobile network in the WiFi network settings. In this case, the data transfer will occur as in the diagram below.
')
But this method has a lot of serious problems:
All these problems, except for the last, are successfully solved by the second method - by writing a VPN client application using the VpnService and VpnService.Builder classes. It is enough for an application to call a couple of functions (mandatory checks are omitted in the code):
, and all TCP / IP traffic of all applications (even those running as root) will be redirected to the TUN interface , which will create Android, and your application will have a read / write device file / dev / tun (or / dev / tun0, / dev / tun1, etc., from where you can read outgoing network packets, transfer them to a remote VPN server (usually via an encrypted connection) for processing, and then record incoming network packets. To ensure that the VPN client connections themselves are not “wrapped” in the TUN, the VpnService.protect method is used on TCP or UDP sockets created by the application.
The scheme presented above in this case will look as follows:
This method has two features:
There is a small additional problem in this method (apart from the “features” about which below) - you need a remote server to work.
The problem with the presence of the server is solved by the third method (somewhat modified method No. 2) - the VPN client application also contains the TCP / IP stack (you can take it ready or write it yourself due to the presence of shortcomings in the ready one) to analyze traffic from applications and handle connections as a proxy server. Then the application traffic processing scheme will change a little and will look as follows:
This is the way we use in WebGuard. Of the drawbacks, compared with the previous method, only one can be noted - it is the impossibility of normal processing of protocols other than TCP or UDP (or protocols "on top of them), because the application will need to create " raw "sockets for which root rights are usually needed. but. To make it clear what we are talking about, let's take a simple example: the user launches a shell via ADB and executes the “ping www.ya.ru ” command , which sends an ICMP echo request. Next, the VPN client application reads the IP packet from / dev / tun, parses it, and finds out that the packet contains an ICMP echo request to some server. And since the application cannot transfer the request further to the network, there are only two options: ignore the packet or emulate ping by trying to connect to the correct server and, if successful, write a fake ICMP echo reply to / dev / tun.
In the process of developing the application, testing and using the first versions by users, we are faced with a large number of errors or flaws associated with the VpnService API. Some of them managed to be fixed, since in fact these were flaws of our programmers (which we honestly wrote and got on bash.org.ru), and with the rest of it, it is difficult or impossible to do something:
So read this list and think - and maybe this is his VPN? Unfortunately, there is no other way to intercept all traffic on the "unruted" Android phone.
The list above didn’t get another problem, the consequences of which were quite unexpected for us (and not only for us). Therefore, we decided to talk about the consequences in more detail. It all started with the fact that at some point it was necessary to make the most important part of the application - a boring icon. No sooner said than done. A beautiful round icon is drawn and we happily test the pre-release version, when suddenly:
It turned out that on the part of the phone the application icon in the VPN notification can be very strangely displayed (wrong color, size or something else). After experiments and a brief but hot discussion.
, it wasordered to make an application icon in the nine-patch format, since the official documentation on this matter says nothing (that is, does not prohibit) and the egg-shaped problem is solved. But, after the release of the release, “victims” appeared very quickly. These were both applications for Android, and various online services working with apk-files, and not expecting to receive the application icon in the nine-patch format. We decided to place the most remarkable ones on a pedestal of three places:
3. Various launchers that fell when trying to display an icon (for example, LauncherPro).
2. Android application stores from Samsung and Yandex. When I tried to download the application in Yandex.Store, I got a quite clear description of the error: “I could not get the application icon from the APK”. C Samsung Apps was more fun. As a high-tech company, when adding an application, the verification results come in the appropriate form - a letter with a link to the video which records the process of testing the application and should be visible (in theory) an error. It turned out, however, as usual , a letter came with a link by which the video was not.
1. Well, the honorable first place, by right, is occupied by Sony with an Xperia phone. After some time after the release, the owners of Xperia L began to send messages that WebGuard "killed" their phone ( example ). It turned out that PackageManagerService crashes when trying to process an application icon during installation, after which the phone automatically reboots and there is an endless download:
As a result, from version 1.3, it was decided to use the icon without the nine-patch, especially since its use of all the problems with displaying the icon did not decide:
We did not want to make too much post, so it was decided to split the article into several parts. In the next part, we will explain why Android applications filtering traffic consume so much battery power (according to the android) and recall, using the example of a single performance test, that DalvikVM! = JavaVM.
We hope our article will help someone in writing an interesting application for Android. Successful development!
Let's start from the beginning. Some time ago, on the basis of research conducted in the field of security of mobile devices, our company decided to release a small application ( WebGuard ) under Android to block all already annoying ads, as well as protection from surveillance and viruses when working from any browser. To implement this functionality, we needed to solve many problems, the most time consuming of which was the task of intercepting and processing application traffic. To intercept and filter browser connections, it was decided to use the VpnService API, which appeared in Android from version 4.0.3 and provides all the necessary functionality (though at times this functionality simply does not work for a bunch of different reasons, but it turned out a little later).
A little about the technologies used
Here it is worthwhile to tell in more detail why this API was chosen and what methods exist at all for intercepting network traffic of applications by another application on a “non-routed” android device. Actually, there are only three methods (apart from various vulnerabilities) and it will be useful for application developers with various mechanisms of in-game purchases to know about them. Because many of them believe that the substitution of network traffic for an application is possible only on the “root” device and does not bother with the protection of transmitted data, although there is a possibility (and quite large) of the appearance of “cheat” applications that make changes to the transmitted data.
So, the first way is to install a local proxy server . Most antiviruses for Android use this method. An application for intercepting traffic just needs to implement the HTTP proxy support described in RFC 2068 and set the proxy server or APN in the settings of the mobile network in the WiFi network settings. In this case, the data transfer will occur as in the diagram below.
')
But this method has a lot of serious problems:
- You can programmatically install a WiFi proxy or APN for a mobile network only using a hidden API (for example, hidden methods in WifiManager);
- you need to monitor the emergence of new WiFi or mobile networks by setting up a proxy (APN) for them;
- This method works only for connections created using the Http (s) URLConnection class and, unless explicitly stated, do not use a proxy (for example, url.openConnection (Proxy.NO_PROXY)). There are also applications that check the presence of a proxy server and use it (or vice versa do not use it);
- if the user deletes or simply does not start the application with the proxy server, other applications will not be able to connect to any server, because the network settings will still have the proxy specified and the user will have to remove it from the settings ( for example );
- it is impossible to build a chain of several proxies, respectively, and there can be only one intercepting traffic application.
All these problems, except for the last, are successfully solved by the second method - by writing a VPN client application using the VpnService and VpnService.Builder classes. It is enough for an application to call a couple of functions (mandatory checks are omitted in the code):
// Activity Intent intent = VpnService.prepare(PromptActivity.this); startActivityForResult(intent, VPN_REQUEST_CODE); // // onActivityResult // requestCode == VPN_REQUEST_CODE && resultCode == RESULT_OK VpnService.Builder vpnBuilder = new VpnService.Builder(); vpnBuilder.addAddress(options.address, options.maskBits); vpnBuilder.addRoute("0.0.0.0", 0); ParcelFileDescriptor pfd = vpnBuilder.establish(); FileInputStream in = new FileInputStream(parcelFileDescriptor.getFileDescriptor()); FileOutputStream out = new FileOutputStream(parcelFileDescriptor.getFileDescriptor()); , and all TCP / IP traffic of all applications (even those running as root) will be redirected to the TUN interface , which will create Android, and your application will have a read / write device file / dev / tun (or / dev / tun0, / dev / tun1, etc., from where you can read outgoing network packets, transfer them to a remote VPN server (usually via an encrypted connection) for processing, and then record incoming network packets. To ensure that the VPN client connections themselves are not “wrapped” in the TUN, the VpnService.protect method is used on TCP or UDP sockets created by the application.
The scheme presented above in this case will look as follows:
This method has two features:
- The application must obtain the rights to use VpnService through a call to startActivityForResult, and the system will show the user the following dialog:
Moreover, Android remembers that it granted rights to the application, only before rebooting the device, so after rebooting, the rights must be requested again. As it turned out (we didn’t even expect it) there are users (“yes, there are hundreds of them” ©) who like to reboot their phone every 10 minutes and this window hinders them, for which they can rate the app for 1 point in the market; - "Pechalka" a little more than the previous one. Android after turning on VPN, without fail, displays the notification icon in the form of a key and the non-removable notification itself, by clicking on which you can see a small statistic:
Notification in the "curtain". Notification icon and status dialog
Here, user dissatisfaction was expected, since many current applications simply love to "hang up" notifications (sometimes several at once) and the status bar turns into a New Year's garland:
Also, if the user presses the “Disconnect” button, Android will disconnect the VPN and take away the rights from the application, after which, in order to activate the VPN, you will have to reapply the rights via the startActivityForResult call.
There is a small additional problem in this method (apart from the “features” about which below) - you need a remote server to work.
The problem with the presence of the server is solved by the third method (somewhat modified method No. 2) - the VPN client application also contains the TCP / IP stack (you can take it ready or write it yourself due to the presence of shortcomings in the ready one) to analyze traffic from applications and handle connections as a proxy server. Then the application traffic processing scheme will change a little and will look as follows:
This is the way we use in WebGuard. Of the drawbacks, compared with the previous method, only one can be noted - it is the impossibility of normal processing of protocols other than TCP or UDP (or protocols "on top of them), because the application will need to create " raw "sockets for which root rights are usually needed. but. To make it clear what we are talking about, let's take a simple example: the user launches a shell via ADB and executes the “ping www.ya.ru ” command , which sends an ICMP echo request. Next, the VPN client application reads the IP packet from / dev / tun, parses it, and finds out that the packet contains an ICMP echo request to some server. And since the application cannot transfer the request further to the network, there are only two options: ignore the packet or emulate ping by trying to connect to the correct server and, if successful, write a fake ICMP echo reply to / dev / tun.
"Features" VpnService API
In the process of developing the application, testing and using the first versions by users, we are faced with a large number of errors or flaws associated with the VpnService API. Some of them managed to be fixed, since in fact these were flaws of our programmers (which we honestly wrote and got on bash.org.ru), and with the rest of it, it is difficult or impossible to do something:
- There is no support for the TUN interface in the linux kernel , so VPN will not work. Basically, this problem is encountered on self-assembly firmware based on CyanogenMod, AOSP, and other projects. Assembly authors either remove support from the kernel altogether, or forget to put the tun.ko module in the firmware. It seems that the authors are guided by the principle “I don’t know what it is, so it’s not necessary”;
- There is no file VpnDialogs.apk which contains the dialogs of the request for rights and statistics, VPN again will not work. Strangely enough, this error most often happens on official assemblies from phone manufacturers ( including Google);
- GUI freeze on some Samsung phones with official firmware, if an unhandled exception occurs in the VPN client (and the shell via the ADB is quite working, but reconnects every 1-2 minutes). In general, manufacturers of phones with Android, sometimes collect “great” firmware for them, forgetting to check out some of the “rarely used” functionality. Our development team in this regard simply "adores" Samsung and believes that to test Android applications, it is imperative to have several phones from different manufacturers and several different phones from Samsung;
- Thanks to this change, the user will not be able to tick “I trust this application” in the VPN dialog if it is blocked by another activity that skips user clicks. This is done so that it would not be possible to draw on top of another dialogue and trick the user into putting a tick. The problem is that various utilities, launchers and some programs (for example, “Read! Free”) usually do this (and for a long time), creating a transparent activity. As a result, some users might think that they are being bullied;
- If the user turns on traffic restriction for background processes, then a working VPN client will be “distributed” (when sending data via a socket, there will be a SecurityException exception with Permission denied), respectively, there will be no Internet
kin; - Linux has such a wonderful mechanism as OOM Killer , and if the user suddenly launches an application that will “devour” a large amount of the device’s RAM (for experiments, you can take Firefox, that's why there is always little memory), OOM Killer will start to “dump” everything consecutive user processes except, as usual, the hero of the occasion (more precisely, he will be killed last). There are no exceptions for the necessary processes (launcher or VPN client), so you need to be ready to constantly restart your service;
- After an incorrect shutdown of the VPN client (even if it restarts afterwards), in rare cases, you may need to reboot the device, because Android will incorrectly reconfigure packet routes and packages from applications will go to / dev / null ;
- If the user turns on the distribution of the mobile Internet via WiFi simultaneously with the VPN client, then on most firmware, as in the case above, Android will incorrectly configure packet routing;
- Despite the decent list of features, Google developers may suddenly decide to
remake theimproved functionality in the new version, and Android developers usually do not warn about all changes, so you need to carefully monitor your Android committees and edit your code. The result of this approach is most often felt by the users of Nexus phones, who receive the update the fastest (read heartbreaking requests in the style of “PLIZZZZZZZZZ !!!!!!!!!!!! VPN !!!!!!!!!!!! !!! "," Please fix this issue "for more than 230 days here , here and here ). So, be careful and do not update your Nexus during a business trip.
So read this list and think - and maybe this is his VPN? Unfortunately, there is no other way to intercept all traffic on the "unruted" Android phone.
A little bit about NinePatch
The list above didn’t get another problem, the consequences of which were quite unexpected for us (and not only for us). Therefore, we decided to talk about the consequences in more detail. It all started with the fact that at some point it was necessary to make the most important part of the application - a boring icon. No sooner said than done. A beautiful round icon is drawn and we happily test the pre-release version, when suddenly:
It turned out that on the part of the phone the application icon in the VPN notification can be very strangely displayed (wrong color, size or something else). After experiments and a brief but hot discussion.
, it was
3. Various launchers that fell when trying to display an icon (for example, LauncherPro).
2. Android application stores from Samsung and Yandex. When I tried to download the application in Yandex.Store, I got a quite clear description of the error: “I could not get the application icon from the APK”. C Samsung Apps was more fun. As a high-tech company, when adding an application, the verification results come in the appropriate form - a letter with a link to the video which records the process of testing the application and should be visible (in theory) an error. It turned out, however, as usual , a letter came with a link by which the video was not.
1. Well, the honorable first place, by right, is occupied by Sony with an Xperia phone. After some time after the release, the owners of Xperia L began to send messages that WebGuard "killed" their phone ( example ). It turned out that PackageManagerService crashes when trying to process an application icon during installation, after which the phone automatically reboots and there is an endless download:
E / AndroidRuntime (790): *** FATAL EXCEPTION IN SYSTEM PROCESS: Thread-123 <br />
E / AndroidRuntime (790): java.lang.ClassCastException: android.graphics.drawable.NinePatchDrawable cannot be cast to android.graphics.drawable.BitmapDrawable <br />
E / AndroidRuntime (790): at com.android.server.pm.PackageManagerService $ SetIconCacheThread.run (PackageManagerService.java.3672) <br />
... <br />
E / AndroidRuntime (2981): FATAL EXCEPTION: ApplicationsProviderUpdater <br />
E / AndroidRuntime (2981): java.lang.RuntimeException: Package manager has died <br />
E / AndroidRuntime (2981): at android.app.ApplicationPackageManager.queryIntentActivitiesAsUser (ApplicationPackageManager.java:487) <br />
E / AndroidRuntime (2981): at android.app.ApplicationPackageManager.queryIntentActivities (ApplicationPackageManager.java:473) <br />
E / AndroidRuntime (2981): at com.android.providers.applications.ApplicationsProvider.updateApplicationsList (ApplicationsProvider.javaCl12) <br />
E / AndroidRuntime (2981): at com.android.providers.applications.ApplicationsProvider.access $ 300 (ApplicationsProvider.java:69) <br />
E / AndroidRuntime (2981): at com.android.providers.applications.ApplicationsProvider $ UpdateHandler.handleMessage (ApplicationsProvider.java:206) <br />
E / AndroidRuntime (2981): at android.os.Handler.dispatchMessage (Handler.java:99) <br />
E / AndroidRuntime (2981): at android.os.Looper.loop (Looper.java:137) <br />
E / AndroidRuntime (2981): at android.os.HandlerThread.run (HandlerThread.java:60) <br />
E / AndroidRuntime (2981): Caused by: android.os.DeadObjectException <br />
E / AndroidRuntime (2981): at android.os.BinderProxy.transact (Native Method) <br />
E / AndroidRuntime (2981): at android.content.pm.IPackageManager $ Stub $ Proxy.queryIntentActivities (IPackageManager.java:2027) <br />
E / AndroidRuntime (2981): at android.app.ApplicationPackageManager.queryIntentActivitiesAsUser (ApplicationPackageManager.java:481) <br />
E / AndroidRuntime (2981): ... 7 more <br />
As a result, from version 1.3, it was decided to use the icon without the nine-patch, especially since its use of all the problems with displaying the icon did not decide:
Notification of enabling VPN on Ainol Novo10 Hero
Conclusion
We did not want to make too much post, so it was decided to split the article into several parts. In the next part, we will explain why Android applications filtering traffic consume so much battery power (according to the android) and recall, using the example of a single performance test, that DalvikVM! = JavaVM.
We hope our article will help someone in writing an interesting application for Android. Successful development!
Source: https://habr.com/ru/post/231827/
All Articles