Projected Federated Averaging with Heterogeneous Differential Privacy

Download PDF

Abstract

Cross-silo Federated Learning (FL) emerges as a promising framework for multiple institutions to collaboratively learn a joint model without directly sharing the data. In addition to high utility of the joint model, rigorous protection of the sensitive data and communication efficiency are among the key design desiderata of a successful FL algorithm. Many existing efforts achieve rigorous privacy by ensuring differential privacy for the intermediate model parameters, however, they typically assume a uniform privacy parameter for all the sites. In practice, different institutions may have different privacy requirements due to varying privacy policies or preferences of the data subjects. In this paper, we focus on explicitly modeling and leveraging the heterogeneous privacy requirements of different institutions. We formalize it as the heterogeneous differentially private federated learning problem and study how to optimize utility for the joint model while minimizing communication cost. As differentially private perturbations inevitably affect the model utility, a natural idea is to make better use of information submitted by the institutions with higher privacy budgets (referred to as “public” clients, and the opposite are “private” clients). The challenge is how to use such information without biasing the global model. To this end, we propose the Projected Federated Averaging with heterogeneous differential privacy, named as PFA, which extracts the top singular subspace of the model updates submitted by “public” clients and then utilizes them to project the model updates of “private” clients before aggregating them. We further propose the communication-efficient PFA+, which allows “private” clients to upload projected parameters instead of original parameters using the projection space learned from the previous round. Our experiments on both statistical learning and deep learning verify the utility boost of both algorithms compared to the baseline methods, whereby PFA+ achieves over 99% uplink communication re-duction for “private” clients. Our implementation is publicly available.