What is Federated Learning?
It is a distributed Machine Learning technique. It basically enables machine learning engineers and data scientists to work productively with decentralized data with privacy by default.
The data is present on the end nodes only. The models are trained on them and only the updated parameters are sent to the central server. These updates are then aggregated on the main server to produce the required Machine Learning model.
Federated Learning
Federated Averaging algorithm can be used to train the main model. the steps are as follow:
- Select k clients from the pool
- Send the initial parameters θt to these clients
- A client receives this θt from the server
- Run some iterations of SGD (Stochastic Gradient Descent) to produce updated parameter θ’
- Return θ’ – θt to the server
- θt+1 = θt + data-weighted average of the client updates
Cons of Federated Learning
As we already know, to get a properly trained model, thousands of iterations of training is required. Now each of these iterations will require sending the original parameters to thousands, or maybe millions of edge nodes, and then allowing them to train. Then after the local training, updates are sent from these millions of devices to the central server.
Now to get a good model, this process has to be repeated multiple times. All this causes a huge communication overhead. Even though the size of the communication message might be small, but the frequency is high. This communication overhead prevents it from being more viable. So a need for a better approach, that can reduce the number of communications between the server and the nodes, is strongly felt.