Chronicle of hacking Pokemon Go

Introduction from the translator

I translated this story because it was considered interesting in my previous article about the relationship between Niantic and the developers of any additional software.
This cute story does not claim to be the greatest and most professional hacking of all times and peoples, but allows you to plunge into the cozy atmosphere of an open source community, engaged in a common cause. Perhaps someone will remember their past days ...
The author of the original admits that he is not very well versed in the design, but it is possible and for the better - the general atmosphere is well conveyed without unnecessary technical details. If you are interested in exactly the technical details, then you can follow the links in the text or just make out the resulting API. Yes, you may be confused that the text sometimes says something about creating an API, then about its hacking. This refers to two different APIs - the API of the game itself and the API that developers make to control the game from the outside. We'll have to distinguish them by context. Yes, it should also be borne in mind that the updates were written in a hurry and sometimes by a tired and sleepy person - some formulations will sound strange. I also apologize for the abundance of links, some of which are already inoperative. The format of the reddit is not very consistent with the format of the article, but it seemed to me that it would be interesting.

Prehistory

The API has stopped accepting requests from all sources except the updated official client. Now he needs an “unknown 6”, which was also in earlier versions of the API, but was not tested. At the moment, only an official client can create a correct "unknown 6". We are not completely sure that the case is in the “unknown 6”, but it’s very similar to that — this is actually the only big piece of data that we cannot recreate.

We cannot just take and compare the versions of the application before and after the update, because this variable was already in previous versions of the API, but the server did not check it.
')
The final value of the variable itself is unimportant, and a simple selection does not help - we need not a value, but a method of generating “unknown 6”.

We are trying to find exactly where it is generated, to recreate its version and restore the work of our API ...
This is pretty hard, because some code fragments are difficult to analyze.

Story

08/04/2016 - 00:49: Apparently, the logic and protocol has changed.
08/04/2016 - 01:37: The protocol has not changed, no new hashes have appeared. Perhaps now requests are signed by some cryptographic methods, but we are not sure yet.
08/04/2016 - 02:07: It becomes obvious that the change is nontrivial, and reverse engineering can take a lot of time.
08/04/2016 - 08:08: Everyone is working on the definition of “unknown 6”. All known data is collected here.
08/04/2016 - 15:06: We confirm that “unknown 6” is responsible for the changes that have occurred.
August 5, 2016, 14:00 - Breakthrough? The developers seem to have found where the “unknown 6” is generated. Now we need to recreate the process and hope that everything will work. And the fact that it was “unknown 6” that broke us all.
14:30 - Channel in the discord went into private mode, because our breakthrough is appropriated by other people. I was also thrown out of the discord, so there will probably be no more updates.
(16:20) I was allowed back. We are now confident that the case is in the “unknown 6”. This is nice, otherwise all the work of recreating it would be unnecessary. We are on the right track.
16:30 - A lot of difficult work and incomprehensible words. As I understand it, now we know where most of the “unknown 6” is created. Now we need to understand the input data for generation and how to further encrypt this unknown. There is still a lot of work to be done ...
18:00 - We have disassembled part of the encryption, and we can now partially understand the source data. As soon as we disassemble all encryption, we will be able to repeat its process, having driven through decompiled code.
19:30 - We are one step closer ...
20:30 - Breakthrough # 2: We tied together two pieces of code responsible for generating “unknown 6”. We found where the encryption function is called. As I said earlier, we now have a decompiled encryption function.
21:15 - We have two things left to do:
Bring the decompiled version of encryption into a human form. This is custom encryption, and the decompiled code would take about 200 pages of the text ... People are working on it - slowly but surely. Not the hardest part of the job, but time consuming.
Determine the remaining input data. This can be quite difficult ... There are 3-4 fields left, and we rejoice at every little breakthrough.
22:30 - No news, except that "they work," but I thought it would be good to write something about the last day.
It's incredibly cool to watch the work of programmers from the reddit and their joint efforts to crack the “unknown 6”. By the way, this is the same unknown, on which Ingres hackers “broke down”. But the Pokemon Go community of developers is much more. Some have been working on burglary for 20 hours out of the 24 that have occurred since the API was changed. / u / keyphact no longer slept for 40 hours (dude, seriously - go get some sleep). People work purposefully and tirelessly. I feel we can do it.

We found the core of the creation of “unknown 6” in a few hours. The encryption functions were decompiled and the places of their calls were found. The remaining 10% of the input data and the normal implementation of the encryption functions. But so close, but so far. Can we do it?

23:30 - We understood most of the encryption. However, they still do not understand how the input data is stored (protobuffer), it is incredibly difficult. This is required to get ahead of the remaining input data.
01:30 - We have fully working encryption! (although we did not fully realize it!) This can be considered a breakthrough # 3. It remains to cope with the protobuffer.

It is incredibly interesting and informative for me too. I hardly understand what the developers are doing. For example, what is the “protobuffer format”? I was told that it is somewhere between input and encryption. It takes the input data, forms some packet from it and sends it to encryption.

Now we have encryption, but we cannot determine the input data, because we do not know how the packet is formed from them. So we make our own package format (protobuffer). Gradually, step by step ... I hope that when we figure out the protobuffer, the input data will become clear.
03:30 - No special news. We are working and moving forward.

It should be mentioned that to ensure the efficiency of the community was extremely difficult. It was a nightmare without any logical decision, and the moderators had no “good” answer to it.
It all started easy and simple - an open channel on the discord, where everyone could work together on repairing the API. But a little later it became clear that everything is not so easy and simple ...

The number of people on the channel grew and grew. Which in turn led to a huge amount of spam, like “When will the API be ready?” Or “What happened to the API?”. It became impossible to work, and we had to restrict the right to post messages to the channel.

To make matters worse, some presented our work as their own and gave the community false promises as to when the new API would be ready. And the moderators were spammed with requests for the right of posting to the channel. We decided to completely hide the channel.

We tried to create a second channel in which to test suitability for general work. But there, in the end, the same problems began as in the first channel. And people were angry at the moderators for the fact that they no longer had access to the materials of the first channel, and they had to repeat the work that we have already completed long ago.
Now we are moving towards transparency again. We made the first channel available again for everyone to read. Hope no one abuses this again. We also made open to all githabs and wikis .

What other conclusions can we draw from this lesson? Yes, no, I guess. I just want to thank the moderators for the insane amount of work they have done to solve this unsolvable situation.

And now I will go to sleep. We opened the channel for API updates, since the messages in the main channel are somewhat complicated to understand with all sorts of technical terms. I'll come back to you tomorrow.

5 August 2016, 13:00 - Here is a detailed technical description of our current problem.
13:30 - No special news, we work work. There is some progress with the protobuffer, we have isolated several fields, but there are still quite a lot of incomprehensible data. We also parse the encryption functions to fully understand how it happens. Reverse engineering is hard and tedious work.
I also want to answer a few comments. Small FAQ:
Q: I believe that I can help, how to do it?
A: Unfortunately, the main channel is closed for new developers. But you can help in our public repository and do pull requests.
Q: Developers should try x.
A: I have no idea what you are talking about, but they probably already tried it. If you are sure that you have a really breakthrough idea - write it somewhere on the discord.
15:30 - No special news, still working work.

We have a new thread on reddit. There are more technical details that you can hardly figure out if you are not a programmer. If you do not see updates, this means that developers are actively working. We also raised Twitter , in which everything is somewhat more understandable - I will write there with each update as soon as they give me access there.

We also made invites on discord permanent, their lifespan should no longer expire, * crossed fingers *.
We want to keep you informed and avoid the appearance of fake Twitter accounts.
I have reached the limit on the length of the text. Further updates will be here .

5 August 2016, 16:00 - we defined another input field! Cool to see finally some progress. Do not rely on the decision right now, we have one more field left, but we are already working on it.
17:00 - We checked the field, which I mentioned earlier - yes, everything is fine! Everyone's mood is great, we are moving.
18:00 - We think that the last field is somehow related to the field that we just hacked. Hope this somehow helps.
18:30 - We want to repeat once again that the API hacking community does not support bots. We just break the API. So we want to confirm that Niantic can trace any MITM applications, for example, those that modify the poker ball throws so that they are always accurate. If you use such an application, Niantic most likely already knows about it.

We do not know if you will be banned for this, we just made sure that Niantic can (theoretically) determine this. But it does not concern us, we are engaged in hacking API.

20:00 - No special news on the development front. Still working on the remaining input data.
We adapt to the many channels through which we communicate with you. We have Discord, Twitter, Reddit, this post and githab repository. They just burst from the messages. However, our internal communication is much better. Setting up and supporting the interaction is sometimes time consuming and time consuming, but it is exciting. It's nice to know that developers can safely do their work by cracking the API.
23:30 - I am back and waiting for updates to translate them from programmer to human. We have already gone far, but to the final destination even further.
00:45 - Progress made in the last hours can be called Breakthrough # 4.
We opened 3 more input fields. One of the fields is an encrypted (or, more precisely, hashed) version of the authorization ticket, in combination with the GPS position, the second field is obtained. The third one is also somehow connected with the authorization ticket.
“Combination” is a strong simplification of what is actually there. The real complexity of the work of programmers is much higher than my level of understanding.
We are currently working on the remaining field. Or fields.
GMT 03:30 - We didn’t write anything special, as there is very little progress.
We have been trying to crack one of the fields for 12 hours already. We know more than when we started, but there has not been a breakthrough yet.
We know that the field is not combined with the authorization token, but depends on the session (perhaps not directly). We also know its length (16 bytes). We are working on clarifying information.
Right now, most developers still went to bed. I must say, this is a well-deserved rest. I'll go and I'll sleep.

6 August 2016, 13:00 - So, now it will be purely my comments. This is the only real source of information, the rest is fakes. Now I will speak I about myself and THEY about the developers. This decision was made to remove unnecessary pressure on developers.

While I was sleeping, nothing really happened, perhaps because the developers were also asleep. The field that brought us so many hemorrhoids deserves its own name, now it will be “unknown 22”. Additional complexity comes from the fact that it is tied to a session, and it is rather difficult to collect data. We parse this field and one more.
14:30 - No news, just wanted to answer the question “how has nothing happened so far? You said that there were 3-4 fields left, and you hacked much more! ”

In fact, these 3-4 fields turned out to be a combination of many other fields that also need to be hacked. You can not answer the question how many fields are left. We do not want to create false expectations. Yes, and do not know.
5:00 pm - Breakthrough # 5: The developers found out that the “unknown 22” is not needed. One of them responded to the expected "Yes, your mother!". Now developers are trying to build a demo application to test this hypothesis. They are very excited and pray.
It should be noted that even if the API call is successful, this is just a prototype, and not an API fix. Developers have scored on quite a few fields - for example, on a field that is transmitted only from Android devices. To get around it, developers emulate the use of iOS. Much more needs to be done so that it does not work so dumbly.
17:30 - The first versions of the prototype do not work.

18:00 - No news, I just wanted to tell why the “unknown 22” turned out to be such hemorrhoids - we have a working theory about this. “Unknown 22” is a random variable generated during the initialization of the application and then bound to the session.
The developers carefully looked for everything that could affect the “unknown 22”, until it was gradually realized that it had no input parameters. This is just a random variable. I will try to explain why it was so difficult to determine.

An example from the real world: Imagine that we want to determine the temperature in New York. There are a huge number of values ​​that correlate with it. For example, selling ice cream. When sales rise, the temperature rises. However, it is useless to calculate the temperature based on the sales of ice cream. Correlation does not mean causation. Keep this in mind when reading about the “unknown 22”.
At first, the developers tried to change the authorization token, and every time they did this, “Unknown 22” also changed! So they decided to check that the authorization affects the “Unknown 22”. For this they needed data.

The collection of this data took a lot of time, because they had to log in and go back. Also, the “unknown 22” could be affected by many other fields, including, for example, SessionID, Auth_token, Auth_ticket. They tried to understand the dependencies on each of them, until it came to someone: the “unknown 22” has no input parameters.

“Unknown 22” is generated randomly each time the application starts.
So, in no way can Niantic check what value “there should be” there - after all, it is random! So there you can just shove anything. This is still a theory, but all developers stick to it.
18:30 - Breakthrough # 6. It seems we have our first successful API call! Wait for updates, confirmation will be within an hour.
18:35 - Works! Believe me, it causes as much excitement as you have.
19:00 - Public chat in the discord is empty. Still waiting for updates from the developers. Do you also update the chat to see nothing as well?
20:00 - For quite some time there was no information. However, they say that they are working on implementation, so apparently there is nothing more to break. The next update should be great, stay with us ...
20:30 - They removed the public githab. Probably moving somewhere. Apparently, the work goes. Update: The repository is closed for all sorts of copyright reasons.
10:00 pm - I'm already starting to doubt, but I still believe that they could then make a successful API call. Although apparently, they think when, how and what to publish. Hard work, given that they closed the githab.
From the very beginning, everyone knew that this work had 2 stages. First reverse engineering, then creating a new API. A successful API test call is important because it means that the work is somewhere in the middle.

However, this does not mean that reverse engineering is complete. Some bypassed fields were not necessary, but they may still want to hack them.

I will look like an idiot if it turns out that they did not get an API call, but I’m ready to put up with this opportunity.
I reached the limit of the number of characters, I will continue here .

Here I will also write my own wash, as in the previous comment. I am not an advanced programmer, but I was with the “unknown 6” group from the very beginning.
6 august 2016, 23:00 - Small update. They are looking for how to bypass copyright so that their materials are not demolished again. They also write that they are working on implementation, which means that they have made a successful API request and are already working on a working version of the API.
00:00 - They say that the “last spurt” is left, let's hope that it means something good.
Their work is hampered by wild spam requesting rights and updates. Please just let them code! Faster, they will not work, and believe me, you can live another day without an API.

Some also blame the developers for doing this for their own benefit. It is not true. I know many of these guys, and they do it just for fun, for the sake of Challenge. They are not going to sell anyone the API.
They also say “The fact that a certain paid service says that it has a fix API, does not mean that we sold it to them.”
00:30 - Just wanted to say - I hate bots.
00:45 - They just confirmed that the API is working (NOT COMPLETELY). But they generally wrote not about this:

For all those who spread rumors that we first launched our bot.
Excited by our success, the team member used our not yet completely ready API in his bot and posted a screenshot. And some other team members also added API to projects not related to bots (for example, pgoapi and RocketAPI).

In the end, as soon as we finish, everyone will have access to the completed version at the same time.

1:15 - Done, API released!
Victory. The developers broke the API in 3 days and 5 hours. A worthy achievement.
1:30 AM - This API is easy to track, and Niaintic can determine that you are not using an official client. Developers are quite satisfied, and you should not roll a barrel on them.
For example, the height above sea level is not taken into account. Also, all the requests allegedly come from the IOS device, which is extremely suspicious, since it's obviously Android ... There is still a lot of work, but we have a working API and with this our participation in the project is completed.

1:45 - I'm going to sleep. Last night I slept very little. I want to say a huge thank you to all the developers, moderators and everyone else who participated. And those who patiently waited for a solution.
Your support was amazing. A week ago, I could not even imagine that I would be a full-fledged manager of the hacker community Pokemon Go.
Thanks to all,
/ u / DutchDefender

From translator

As you know, the story is not over, the story is just beginning. Will Niantic listen to requests for improvements to the application? Or try to defend again? Does the community have enough strength, desire and skills to crack new ways of protection? We learn soon.