I believe that all users of the Habr somehow found a way to get to the root-tracker, but sometimes it is too lazy to turn on your torus, proxy, VPN or something else. I now became lazy, and so I decided to write my little client. To bypass the blocking, I decided to use a google compression proxy. An interesting, good and useful thing - it is strange that there were no articles on it. Looking ahead, I will immediately say that everything turned out, and you can try the working version on your device. However, in the process there were many interesting sorts of nuances that are somewhat more curious than the application itself. So, let's begin!
Google Compression proxy
In order not to repeat Google manuals (you can find all links at the end of the article), I’ll just say that this proxy server allows your Google Chrome to significantly reduce the amount of perceived traffic due to its compression by Google servers. Proxy can work on HTTP and HTTPS. In the first case, the address is used by compress.googlezip.net, in the second - proxy.googlezip.net. Interestingly, the proxy requires its own header. In the official documentation it is not, but you can find the source from Google and a little dig into them. They look like
this (link to the most interesting file, I have already posted on the GiHab, since nothing can be seen in the
official repository on the google code).
From there we get such good:
var authHeader = function() { var authValue = 'ac4500dd3b7579186c1b0620614fdb1f7d61f944'; var timestamp = Date.now().toString().substring(0, 10); var chromeVersion = navigator.appVersion.match(/Chrome\/(\d+)\.(\d+)\.(\d+)\.(\d+)/); return { name: 'Chrome-Proxy', value: 'ps=' + timestamp + '-' + Math.floor(Math.random() * 1000000000) + '-' + Math.floor(Math.random() * 1000000000) + '-' + Math.floor(Math.random() * 1000000000) + ', sid=' + MD5(timestamp + authValue + timestamp) + ', b=' + chromeVersion[3] + ', p=' + chromeVersion[4] + ', c=win' }; };
It's pretty obvious, but
You can stop more.For each request there should be a Chrome-Proxy header.
It should have the following line:
ps=<timestamp>-<num1>-<num2>-<num3>, sid=<md5 string>, b=<build>, p=<patch>, c=<platform>
Where:
')
timestamp: time in linux timestamp
num1, num2, num3: some random numbers that can be set to 0
md5 string: md5 authorization string hash
auth string:
"<timestamp>" + "<auth key>" + "<timestamp>"
auth key: ac4500dd3b7579186c1b0620614fdb1f7d61f944 - just a certain key ... One for all, and all for one.
build: chrome build number - for example, 2214
patch: chrome patch number - for example, 115
platform: platform - for example, "win"
As a complete example, you can cite the following title:
Chrome-Proxy: ps = 1439961190-0-0-0, sid = 9fb96126616582c4be88ab7fe26ef593, b = 2214, p = 115, c = win
Strangely enough and not funny, you can use this very line for any number of requests without any changes ... For example, the extension for Firefox, which is engaged in recompression of traffic, is based on this. Apparently, protection from a lazy fool was simply made.
However, the honest option would be to rewrite it in Java like this:
public static String[] authHeader() { String[] result = new String[2]; result[0] = "Chrome-Proxy"; String authValue = "ac4500dd3b7579186c1b0620614fdb1f7d61f944"; String timestamp = Long.toString(System.currentTimeMillis()).substring(0, 10); String[] chromeVersion = {"49", "0", "2623", "87"}; String sid = (timestamp + authValue + timestamp); sid = Utils.md5(sid); result[1] = "ps=" + timestamp + "-" + Integer.toString((int) (Math.random() * 1000000000)) + "-" + Integer.toString((int) (Math.random() * 1000000000)) + "-" + Integer.toString((int) (Math.random() * 1000000000)) + ", sid=" + sid + ", b=" + chromeVersion[2] + ", p=" + chromeVersion[3] + ", c=win"; return result; }
Next you need to choose which option for proxying we choose. My provider is harsh and blocks requests if they go over HTTP via google proxy, so I had to go the right way over SSL.
WebView via SSL
In order not to go a long and sad way of writing a client from scratch, I decided to “just” show everything as it is through a standard WebView, since I had already written a simple client that does roughly the same thing, and works smartly even on a heavy web site. It seems - work for half an hour. How wrong I was ... If you look at the solutions for proxying WebView on the Internet, it becomes very sad - everyone does something like this:
public static void setKitKatWebViewProxy(Context appContext, String host, int port, String exclusionList) { Properties properties = System.getProperties(); properties.setProperty("http.proxyHost", host); properties.setProperty("http.proxyPort", port + ""); properties.setProperty("https.proxyHost", host); properties.setProperty("https.proxyPort", port + ""); properties.setProperty("http.nonProxyHosts", exclusionList); properties.setProperty("https.nonProxyHosts", exclusionList);
The rest of the deliberately lowered - there is still about a hundred lines with conditions for the version of the android and eerie shamanism. At the same time, many still don’t work, plus there are problems with switching the proxying mode - and it is “solved” by installing Thread.Sleep (1000) between operations. Although I am not a Java developer, I just sometimes indulge, I poplohelo. Common sense prompted me to intercept requests from Webview (for this, WebViewClient has a wonderful function shouldInterceptRequest), and then do the proxying myself. I even managed to do it:
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("compress.googlezip.net", 80)); HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection(proxy);
Everything is fine, everything works! Only one problem. As attentive readers have noticed, the 80th port is indicated in the function parameters. For a rather ridiculous reason. Namely - because HttpURLConnection does not know how to work with HTTPS proxies. Totally. No It took me a lot of time to understand that everything is so bad, and that it is impossible to make a HTTPS proxy through HttpURLConnection, or through the popular okHttp. I became a little thoughtful, then with a decisive gesture dismissed all the arguments of Google that the Apache libraries were not suitable for Android, shook the dust off the tested jars and strongly connected them to the project. And everything turned out! The fifth and sixth android cheers apprehended such a terrible offense. If someone knows how to solve the problem without using Apache libraries, tell us. Of course, one could do everything on sockets, but this is rather sad.
So, we were finally able to display the rutracker's main page. It would seem that victory is near. How wrong I was.
Advertising
Almost from the first attempt at debugging, I was faced with the fact that everything is monstrously slow. The reason is pretty obvious - an insane amount of advertisements of all kinds. I really did not want to do something with it - we all know the position of the rutracker, and the dudes clearly need a lot of gold to protect against DDOS and their native state - but with advertising the application was generally unrealistic to use. The correct solution would be to find it and cut it out from the body of the page, but the quicker approach to implement was simply to cut it into hosts:
pretty obvious code public static boolean is_adv(Uri url) { String[] adv_hosts = {"marketgid.com", "adriver.ru", "thisclick.network", "hghit.com", "onedmp.com", "acint.net", "yadro.ru", "tovarro.com", "marketgid.com", "rtb.com", "adx1.com", "directadvert.ru", "rambler.ru"}; String[] adv_paths = {"brand", "iframe"}; String host = url.getHost(); for (String item : adv_hosts) { if (StringUtils.containsIgnoreCase(host, item)) { return true; } } if (StringUtils.containsIgnoreCase(url.getHost(), "rutracker.org")) { String path = url.getPath(); for (String item : adv_paths) { { if (StringUtils.containsIgnoreCase(path, item)) { return true; } } } } return false; }
And after that we just chop off its receipt:
if (Utils.is_adv(url)) { Log.d("WebView", "Not fetching advertisment"); return new WebResourceResponse("text/javascript", "UTF-8", null); }
In hindsight, I just thought that it might be easier to make a list of allowed hosts ... But with this very mind, everyone is strong.
Now the application began to work not only with acceptable, but with a very vigorous and pleasant speed. If the owners of the rutracker are upset, then I will remove the blocking - but most likely with the application. With monstrous brakes it simply will not make sense.
Submitting forms
Leaning back on the chair, I found ... That authorization does not work. What was actually quite obvious was that, because in the request I intercepted and transmitted further, I did not send the data that should go to POST. It would seem a couple of minutes - and everything will be fine. How wrong I was ...
It turned out that there are no ways to intercept POST from WebView. Totally. No Best practices are to embed your javascript into a page and call special Java methods from it. Or transfer server to GET requests. From the first option, I am somewhat poplohelo, and the second is not available for obvious reasons. Yes, and would be incorrect. After scratching my head and trying to catch POST in several more places, I came to the conclusion that there was still no normal solution. As a result, made the decision funny. Namely, when a page is received, change the method of all forms from POST to GET. And after this, the next time you convert the parameters passed in the address bar to the body of the POST request. It sounds awful, but it's not so bad if you do not have an address bar in which you can disgrace, large variables or files to transfer. Although no, I lie of course. Everything is very bad, but I did not find another adequate way.
pretty obvious code public static UrlEncodedFormEntity get2post(Uri url) { Set<String> params = url.getQueryParameterNames(); if (params.isEmpty()) return null; List<NameValuePair> paramsArray = new ArrayList<>(); for (String name : params) { Log.d("Utils", "converting parameter " + name + " to post"); paramsArray.add(new BasicNameValuePair(name, url.getQueryParameter(name))); } try { return new UrlEncodedFormEntity(paramsArray, "UTF-8"); } catch (UnsupportedEncodingException e) { e.printStackTrace(); } return null; }
Where are my cookies, dude?
When I once again sat back in the chair after successfully passed the authorization and clicked the following link, I found that the authorization is dying on the next page. That, in general, is also quite logical, since nobody magically controls the cookies, and we send all the headers manually. But here, for once, I was not mistaken in that
Hidden text if (Utils.is_login_form(url)) { Header[] all = response.getAllHeaders(); for (Header header1 : all) { Log.d("WebView", "LOGIN HEADER: " + header1.getName() + " : " + header1.getValue()); } Header[] cookies = response.getHeaders("set-cookie"); if (cookies.length > 0) { String val = cookies[0].getValue(); val = val.substring(0, val.indexOf(";")); authCookie = val.trim(); CookieManager.put(MainContext, authCookie); Log.d("WebView", "=== Auth cookie: ==='" + val + "'"); //redirect does not work o_O String msgText = "<script>window.location = \"http://rutracker.org/forum/index.php\"</script>"; ByteArrayInputStream msgStream = new ByteArrayInputStream(msgText.getBytes("UTF-8")); return new WebResourceResponse("text/html", "UTF-8", msgStream); } }
True, an attentive reader will find that there is some kind of strange redirection javascript. Yes, that's it. The response should have been 302 authorization header, but for some reason it did not come anywhere. As a result, I found myself on the forum page at an incorrect address with the domain login.rutracker.org — everything worked, but since links are relative everywhere, the next click was a bummer. By the way, here you can also notice that we carefully save the user cookie so that you do not have to log in again later.
Now is that all ?!
It was with this question that I once again opened the page, already confident that I did not miss anything. How can I ... In general, you could freely search through a native search or through Google without authorization, you could watch any topic, you could not only one ... Download torrent. Since by clicking links nothing good happens. But this time it was also easy - the program menu with the Share button was added to the interface, which allows you to send a link to Magnet anywhere. It would be more convenient to throw a link to a torrent file, but you will obviously be blocked from it. Of course, it would be possible to download a torrent file and transfer it through sharing - but this is some other time.
Result
You can look at the screenshots - the
main page and
sharing magnet links from the topic .
It remains only to publish
Based on those strange android applications that I posted before, I got the feeling that they are missing everything. So, with a light heart, I sent the application for publication and sat down to write an article. Usually the application appeared on google play for a couple of hours, so there was just enough time. Unfortunately, this time was 8 hours. And the answer came
Here is a letter.After review, rutracker free - unofficial, ru.jehy.proxy_rutracker, has been suspended and removed from the play.
Next Steps- Read through the article.
- Make sure your app is in accordance with the Developer Policies . If you’re there, you’ll be able to get it.
- Sign up for your Developer Console and a new app name.
What if I have permission to use the content?Contact our support team to provide a justification for its use. If you want to have a google account, you can use it. If your account is terminated, you may be able to pay for it. may have been in error, please contact our
policy support team . You can get back to you within 2 business days.
It is not entirely clear who impersonated whom, but I had three options, what Google didn’t like:
- What I took from somewhere icons;
- What I mentioned in the comments is that the application works through a google compression proxy;
- That as banners, I used the options of the funny logos of the rutracker from the new contest;
Of course, in any case, it is rather unpleasant that you need to rename pakage for some reason and download it again. Yes, and you also warned that "I remember you, and once again do so - banned." Well, somehow gopnicheski. I did not expect it, it's a shame.
Of the options mentioned above, I decided that the second is most likely to blame - the mention of Google in vain. Well, okay, as you say - I did not write how the application works. Just "an application with proxying, which allows you to bypass the lock rutracker.org". Well, at the same time I put the curves drawn in five minutes icons and similar banners. And what do you think - my efforts were rewarded! Then came to me
next answer.After review, ru.rautracker_free, ru.rautracker_free.
Next Steps
Read through the webviews spam.
Make sure your app is compliant with the Spam Policies. Remember that you couldn’t have any apps.
It is possible to bring your app in compliance with your app.
In general, I was accused of not giving the application only referral links, or not doing anything other than a webview. There is nothing “left” in the application, and calling it “normal mapping” is also wrong - there is quite a lot of work on proxying. Well, and this especially surprised me in the light of the fact that I had already successfully downloaded applications on Google Play, which in fact only consist of web views to the site. There were two nuances - there was authorization in the application, and I was the owner of the site I was webcasting on. But I didn’t indicate this, and Google couldn’t find out.
In general, I answered both of these accusations with a request to sort it out - there is no answer day, maybe the answer will come another day. Although hope is somehow small. So we put so far the application from APK. If it does appear on Google Play, then it will be possible to upgrade from there.
ToDo
If you make a full-fledged pleasant application, then you could add a lot of good things. Including
- Styles with adaptation to viewing from the phone and tablet; UPD: done to some extent;
- Automatic update; UPD: done;
- Correct advertising cut;
- Transfer torrents torrent files, not magnet links;
- Exit from the authorized state on the rutracker (yes, now you will log in there forever);
- Any meaningful messages about probable errors;
- Compatibility with a large number of devices - now you can try to run on Android from 4.0 to 6 - but the result is unpredictable - you need to test a lot. I work on Nexus 5 with Android 6 and on Sony Xperia Z3 with Android 5;
- Convenient input of authorization and search without poking at the terrible little elements of the web form;
- Remove some copy-from-code from the code;
- Add encryption of a stored user cookie with a unique device key in case data is stolen from the phone;
- Implement the monetization of the application, its pop-up banners and links that would bring me tons of gold.
But, unfortunately, I don’t have time to do this seriously - I just wanted to make a working prototype - to show that web applications in Android applications work, work well and quickly. Along the way, however, the opinion was confirmed that the average quality of Android development at the hospital suffers quite a bit, standard libraries do not cover all cases, and there are a large number of strange tasks that have not been implemented by anyone.
I would be glad if someone takes this code to develop a more serious application or sends pull requests - I promise to watch them carefully and use them.
Q & A
- But it’s much easier to go to the root tracker through XXX (for example, simply turning on the traffic saving on the phone or in the browser).
- Yes.
- And what if Google blocks the root tracker?
- It may well be stated in their policy that they are blocking resources that are prohibited in your country. Well, that's okay - ways around the thousand.
- And if the root tracker blocks Google for a large number of visits from their proxy?
- To them, and so now go through it, just using chrome. So it is unlikely. And in general, banning everyone is a bad idea, so you can become an official.
- Your code is terrible!
- Yes, I have already mentioned that I am not a Java developer.
- It didn't work for me.
- Yes, this is a common Android problem - a full-fledged application should be tested 10 times longer than writing. Alas, I had no such opportunity. Send errors, pull requests - we will fix it.
- Is it possible to use Google Compression Proxy in general?
- While it is not clear - the application was banned before this potential issue.
- I do not trust you, for sure you steal all my passwords and download my own torrents.
- Please - build the application yourself from the source.
- Why are the icons so scary?
- Look under the heading “it remains only to publish” - everything is explained there.
- It's freedom for evil robots and spiders!
- No, do not flatter yourself. Obviously, there are limits and checks for robots too. There are much simpler ways for bots.
- The site is crookedly displayed.
- Yes, in some places there are problems. But rather related to poor-quality layout.
Hi rutrekeru
Separately - a few wishes for the rutraker administration in case they suddenly look in here.
Wishlist- Please consider alternative monetization options. Anyway, most users bypass these terrible banner garlands with adblokom.
- It would be good to introduce various authorization providers - google or facebook, for example.
- Update the layout and appearance to something more beautiful, comfortable and preferably with a mobile version.
Links
UPD :
post about how the update is implemented, and how to live without Google Play;
- For those who have not mastered the whole text - once again screenshots: the main page and sharing magnet links from the topic .
- Actual source code and github releases ;
- Here you can take me to work - yes, I'm looking for her;
- A fun article about how to use Google Compression Proxy + Squid. I started it with her;
- Here are the sources from the compression proxy from Google. Since the google code does not give them a look, you can look at my githaba . I also assorted it;
- What is a google compression proxy for the user and for the administrator ;
- Good answer and analysis of the google compression proxy standard , though only for HTTP traffic, not HTTPS. There is an example implementation for firefox. I translated it for the article;
- A java proxy implementation that proxies through the google data compression server. Made for some image board. Good code, there is something to see;
- The only option I found was for proxying HTTP traffic through an HTTPS proxy using apache libraries. I was finishing it.