CloudFlare ScrapeShield bypass in Java (Android)

At some point in time, I had to solve the problem described in the title. Long wondered whether to write the obvious, in the end decided that it might be useful to someone.

The reason is quite trivial - being the author of an Android client to a very niche site , I at the same time do not belong to either its administrators or co-founders. In this way, I am not aware of any decisions by the management of the site until their actual entry into force.
')
Not long ago, a DDoS attack began on this site, and the administration turned on DDoS protection against CloudFlare. Accordingly, the client application, which previously used standard authorization mechanisms through POST + Cookie, has stopped authorizing users. Communicating with the administration did not lead to anything - “what can we do is better without mobile clients than nothing at all”.

Naturally, all this began to affect the ratings and gave rise to very unflattering reviews.

The solution was to bypass the protection from CloudFlare by simulating browser behavior on their page. CloudFlare in this particular case uses a hash, a key and a random javascript code that the browser performs (calculating several arithmetic operations that look like obfuscated garbage) and later sends the resulting number along with the hash and key to the verification page. Our task, therefore, is to intercept the javascript task, solve it in any way and ask if our answer is correct. If yes, we get a bun (cookies cf_clearance). If not, we get 503.

Having rummaged in the search engine, exactly one link was found, leading to a project that does something very similar. Written in Python using node.js or another compatible provider for PyExecJS. With all due respect to Python, its use in a lightweight niche application was an unwarranted luxury, the integration of which would have to spend many hours. It was made a strategic decision to rewrite the code solver in Java.

Some remarks / non-obviousness when writing code:

• As a JS provider, Mozilla Rhino was chosen, providing, with the optimizations disabled, a Dalvik-bytecode-compatible interface.
• UserAgents inherent in automatic requests are rejected with Error 503. Any "Java / 1.5.0_08", "libcurl-agent / 1.0" and similar lines are instantly rejected. Before you try anything, disguise yourself as a modern browser UserAgent.
• The Apache implementation was used as the Http client. I trust her more than HttpURLConnection, which Android developers are promoting, but this is a matter of tastes. You can use any compatible implementation, for example, OkHttpClient
• Important: if you want to later display some data from the site in WebView, you need to consider two things:
  
  1. The http client must have exactly the same UserAgent as the WebView (use the settings.userAgentString of the WebView)
  2. After receiving the cf_clearance-cookie, you need to synchronize it with the WebView (sample code below)

The final version below. Jumbled in a hurry, but a basic idea of ​​how everything works, gives.

private final static Pattern OPERATION_PATTERN = Pattern.compile("setTimeout\\(function\\(\\)\\{\\s+(var t,r,a,f.+?\\r?\\n[\\s\\S]+?a\\.value =.+?)\\r?\\n"); private final static Pattern PASS_PATTERN = Pattern.compile("name=\"pass\" value=\"(.+?)\""); private final static Pattern CHALLENGE_PATTERN = Pattern.compile("name=\"jschl_vc\" value=\"(\\w+)\""); abstract public HttpResponse getPage(URI url, HashMap<String, String> headers) throws IOException; abstract public CookieStore getCookieStore(); public boolean cloudFlareSolve(String responseString) { //  Rhino Context rhino = Context.enter(); try { String domain = "www.example.com"; // CF      Thread.sleep(5000); //   Matcher operationSearch = OPERATION_PATTERN.matcher(responseString); Matcher challengeSearch = CHALLENGE_PATTERN.matcher(responseString); Matcher passSearch = PASS_PATTERN.matcher(responseString); if(!operationSearch.find() || !passSearch.find() || !challengeSearch.find()) return false; String rawOperation = operationSearch.group(1); //  String challengePass = passSearch.group(1); //  String challenge = challengeSearch.group(1); //  //    String operation = rawOperation .replaceAll("a\\.value =(.+?) \\+ .+?;", "$1") .replaceAll("\\s{3,}[az](?: = |\\.).+", ""); String js = operation.replace("\n", ""); rhino.setOptimizationLevel(-1); //    rhino    Android Scriptable scope = rhino.initStandardObjects(); //    // either do or die trying int result = ((Double) rhino.evaluateString(scope, js, "CloudFlare JS Challenge", 1, null)).intValue(); String answer = String.valueOf(result + domain.length()); //   javascript challenge final List<NameValuePair> params = new ArrayList<>(3); params.add(new BasicNameValuePair("jschl_vc", challenge)); params.add(new BasicNameValuePair("pass", challengePass)); params.add(new BasicNameValuePair("jschl_answer", answer)); HashMap<String, String> headers = new HashMap<>(1); headers.put("Referer", "http://" + domain + "/"); // url ,      String url = "http://" + domain + "/cdn-cgi/l/chk_jschl?" + URLEncodedUtils.format(params, "windows-1251"); HttpResponse response = getPage(URI.create(url), headers); if(response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) { //    ,   Referer response.getEntity().consumeContent(); //       return true; } } catch (Exception e) { return false; } finally { Context.exit(); //  Rhino } return false; } private void syncCookiesWithWebViews() { List<Cookie> cookies = getCookieStore().getCookies(); CookieManager cookieManager = CookieManager.getInstance(); // CookieManager    cookies  WebView for (Cookie cookie : cookies) { String cookieString = cookie.getName() + "=" + cookie.getValue() + "; domain=" + cookie.getDomain(); cookieManager.setCookie("diary.ru", cookieString); } } 

The client code is published under GPLv3, so, most likely, CloudFlare will soon become aware of it, which will lead to a change in the algorithm. Nevertheless, I am not committed to the principle of security by obscurity and the task of letting mobile users in before the DDoS downturn was solved.

Thanks for attention. Questions / comments in the comments.