Google invented distributed AI for a billion smartphones

Just think, what is the total computing power of all smartphones in the world? This is a huge computing resource that can fully emulate even the work of the human brain. It is impossible that such a resource would stand idle, stupidly burning energy kilowatts on chatiki and social network tapes. If you give these computing resources to a single distributed world AI, and even provide it with data from custom smartphones - for training - then such a system can make a qualitative leap in this area.

Standard machine learning methods require that the data set for learning the model (“primary”) be assembled in one place — on one computer, server, or in the same data center or cloud. From here, it is picked up by a model that is trained on this data. In the case of a cluster of computers in a data center, the stochastic gradient method (Stochastic Gradient Descent, SGD) is used - an optimization algorithm that constantly passes through a set of data that is homogeneously distributed across servers in the cloud.

Google, Apple, Facebook, Microsoft and other AI players have long been doing just that: they collect data — sometimes confidential — from users' computers and smartphones into a single (supposedly) secure storage where they train their neural networks.

Now scientists from Google Research have offered an interesting addition to this standard method of machine learning. They proposed an innovative approach called Federated Learning. It allows all devices that are involved in machine learning to share a single model for forecasting for all, but not to share primary data for training a model !
')
Such an unusual approach, perhaps, reduces the effectiveness of machine learning (although this is not a fact), but it significantly reduces Google’s costs for maintaining data centers. Why would a company invest huge sums in its equipment if it has billions of Android devices all over the world that can share the load among themselves? Users can be happy with such a load, because they thereby help to make better services that they themselves use. They also protect their confidential data without sending them to the data center.

Google emphasizes that in this case it is not just that the already-trained model runs directly on the user's device, as it happens in the Mobile Vision API and On-Device Smart Reply services. No, it is the training model is carried out on the end devices.

The system of federated learning works according to the standard principle of distributed computing, such as SETI @ Home, when millions of computers solve one big complex problem. In the case of SETI @ Home, this was a search for anomalies in the radio signal from space across the entire width of the spectrum. And in the case of Google’s federated machine learning, this is the improvement of a single common model (so far) for weak AI. In practice, the learning cycle is implemented as follows:

1. smartphone downloads the current model;
2. using a mini-version, TensorFlow performs a learning cycle on unique data of a specific user;
3. improves the model;
4. computes the difference between the improved source models, makes a patch using the Secure Aggregation crypto protocol, which only allows decryption if there are hundreds or thousands of patches from other users;
5. sends the patch to the central server;
6. the adopted patch is immediately averaged with thousands of patches received from other participants in the experiment, according to the algorithm of federated averaging;
7. rolls out a new version of the model;
8. An improved model is sent to the participants of the experiment.

Federated averaging is very similar to the aforementioned stochastic gradient method, only here the initial calculations do not occur on servers in the cloud, but on millions of remote smartphones. The main achievement of federated averaging is 10-100 times less traffic with customers than traffic with servers using the stochastic gradient method. Optimization has been achieved due to high-quality compression of updates that are sent from smartphones to server. Well, plus it uses the cryptographic protocol Secure Aggregation.


Google promises that the smartphone will perform calculations for a distributed world AI system only at times of inactivity, so this will not affect performance in any way. Moreover, you can set the operating time only at the time when the smartphone is connected to the mains. Thus, these calculations will not even affect the battery life. At the moment, federated machine learning is already being tested on contextual prompts in the Google keyboard - Gboard on Android .

The algorithm of federated data averaging (Federated Averaging) is described in more detail in the scientific work Communication-Efficient Learning of Deep Networks from Decentralized Data , which was published on February 17, 2016 at arXiv.org (arXiv: 1602.05629).