I present to the community a project that I have been working on recently: an ad blocker and trackers for Samsung smartphones on Android with open source.

Advertising is blocked at the device level, that is, not only in the standard browser, but in all other applications (for example, in Chrome and YouTube).

Prehistory

I use blockers on all of my computers, but increasingly I go to the Internet from a smartphone.
')
I like Mobile Chrome more than Mozilla or Opera, but Chrome does not support extensions, and the presence of advertising on a smartphone is more critical than on the desktop, as traffic is often limited and the display is several times smaller.

Plus, I sometimes like to watch videos on YouTube, but advertising that you shouldn’t miss misses the process.

Based on these prerequisites, I started looking for a solution to blocking advertising on a smartphone.


An example of the application on the Samsung Galaxy Tab 4 10.1. habr.ru

There are quite a few ready-made ad blocking solutions, but, unfortunately, they all have at least one major drawback:

1. Browsers with support for extensions to block (Mozilla, Opera).
2. Browsers with built-in ad blocking (from Ghostery, from AdBlockPlus).
3. Ad blockers for the standard Samsung browser.
4. VPN based blockers.
5. Semi-working solutions ( https://adblockplus.org/android-about ).
6. Root-based blockers.

The first three options block only in the browser. I tried to use them, but I like Chrome even more with ads. The fourth option is good, but not stable. The fifth does not filter mobile data.

The sixth option is the most ideal, if you have root or you are not averse to get it.

In each of the options have to sacrifice something, so I started thinking about whether I can do something myself. I did not find a solution at that time and it took a long time until I came across a platform from Samsung.

Samsung Knox Standard SDK

Knox is a solution developed by Samsung. There are many products under the Knox brand, but I’ll focus on one that I used.

The Knox Standard SDK is a free library that runs on almost all Samsung Android smartphones and has a rich API (about 1100 methods). If I were asked to describe the Knox Standard SDK in a nutshell, then I would say that it is almost root legal.

In general, a very useful thing that allows you to do a lot of interesting things. For example, fully block the device for one application, turn on the GPS silently and prevent it from shutting down, turn on Wi-Fi / Bluetooth silently, silently install / remove non-system applications, silently turn on / off any applications, prohibit changing settings, prevent device shutdown and many things other.

Here is a link to the official website , there are examples, guides and API Reference.

The unknown decision in wide circles, I explain for myself that it was created for Enterprise + in my opinion, insufficient marketing of Samsung.

Idea

Looking through the documentation, I realized that using the API, you can achieve ad blocking at the device level (as if there is root access in paragraph 6 above), and not at the application level. But the blocker is suitable only for owners of Samsung smartphones. I decided it was better to have a complete solution on one device than half to all. And he began to develop.

Getting the key

To start using the API, you need to register on the seap.samsung.com website and generate a free developer key with which the library can be activated on the device. When the project is finished, you need to get a production key.

The development key works for three months, at the end of the term you can generate a new one. Production key is perpetual, it must be tied to the package name and the key with which the application is signed. In this case, only your application can use the received key.

API usage

Next, you need to import three .jar that are downloaded after registration, they are not in Gradle, so you have to copy it manually.

I will not describe the code in detail here: you can see it on GitHub and everything is quite simple done there. I will dwell only on some features.

1. For the Knox Standard SDK to work, you need to ask the user to enter the application in administrator mode, then activate the received key and agree in the pop-up window.

2. It turned out that on different versions of Android from Samsung, different versions of the Knox Standard SDK are installed and they are updated only with the firmware update.

Therefore, it was necessary to support different versions, in which, in general, I succeeded and now the blocker works (I hope that) on all devices since version 2.0. The SDK version can be viewed in the device information in the phone settings.

I will be grateful if in case of problems with the application, you write in the comments or open the issue on GitHub.

And what to block?

In the first version, I visited popular sites from my list at Feedly and fished out advertising providers manually, in total, I got about 200 domains and everything worked, but often the advertising leaked out.

After I published the application on Google Play, I received an email with a list of advertising providers:

1. adaway.org/hosts.txt
2. pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
3. hosts-file.net/ad_servers.txt

I note that while in the project the first two links are used, which is about 2600 links in total, and this is enough not to see ads on 95% of the sites.

In the Knox Standard SDK version 5.6, all 2600 links are blocked normally. But on older 2600 links, the smartphone refuses to accept, only 1500 is stably blocked.

And now there is a problem, which 1500 out of 2600 links to choose? How to sort the list by popularity? I began to think and google.

I quickly found that Alexa Rank is giving the .csv file the 1 million most visited domains in the world over the past month in order of popularity. I assumed that if a domain is in this list and at the same time a domain (or its subdomain) is in the list of advertising providers, then the advertising provider is popular and often found on various sites.

With the help of this hypothesis it turned out to sort the list of 2700 links. Those domains that are not found in the list of 1M I added to the end.

As a result, those that meet the list turned out to be ~ 1300, which is quite suitable for our limitations.

In fact, I tried to crank the same thing not only with two, but also with three lists of advertising providers, which at the output gives more than 10 thousand links, and a quick scrolling of the list shows that there are many not relevant sites (for example, Indian or Chinese) . Therefore, while it was decided to leave the first two providers.

I would like in the future to make it possible to block advertising providers by language (as is done in the extension AdBlockPlus, for example).

First damn lump

Somewhere in March, I made a prototype in a week, almost manually collected advertising providers (at that time about 200). Just went to the sites that I usually go to and caught advertising links. I tested a little, put it on Google Play, wrote on w3bsit3-dns.com and started talking about it to everyone I know.

In two weeks there were about 300 downloads and 60 active (at that moment only Marshmallow was supported). At about the same time, they sent me links with advertising providers and I updated the application for joy.

In the morning, a letter came from Google that the application was blocked. Apparently there was no need to write in the update in capital letters: “The blocker of advertising and trackers. Blocks not only in the standard browser, but in all applications. ” To paraphrase with humor, a letter from Google said: “You are a good developer, but you put our entire business at risk, so we cut out your application.” And this is how the story began.

So that the project does not depend on Google Play and at the same time survive, I decided to make a website (where you can download the application directly), make the project open and write about it on Habr.

What is the result?

As a result, we received mobile Internet without ads, accelerated page load and traffic saving bonus.

I use two Galaxy S7 and OnePlus 2 smartphones and, returning to Chrome or YouTube on OnePlus, I wonder how it turns out to be in the way and advertising on a mobile display takes up a lot of space.

Here are some examples of working with and without blocker. The test was conducted on the same smartphones with the blocker turned off and on. I just drove the domain into the browser and waited for the page to load.


habr.ru


theverge.com


rbc.ru


adme.ru

Conclusion

The great advantage of the solution is simplicity and convenience for the user, but the disadvantage is platform dependency.

Thanks

For sharing links to advertising providers, I thank semax95 .

For the preparation of the site thank you very much yelnar .

GitHub project

»Created an organization , it contains three repositories.
" The application itself for Android
» Advertising link sorter by popularity
» Website

» Link to the project website
" Link to Google Play (until they block)

Update (January 19, 2017): Starting with version 1.2.0, you can select applications in which Adhell ​​will be disabled.

Update (April 21, 2017): Starting with version 1.2.6, it is possible to change the DNS servers on the device (only for devices on Nougat). In particular, this means that you can use DNS servers that block ads (or raise your DNS server, as dmitriylyalyuev did) and filter hundreds of thousands of links.
Or, for example, you want to change the DNS server of your mobile operator to Google 8.8.8.8.

Adhell ​​does not block ads in the YouTube app. If someone knows how ads are loaded in their application, please write in the comments.