PiPar: Pipeline parallelism for collaborative machine Learning

Zihan Zhang*, Philip Rodgers, Peter Kilpatrick, Ivor Spence, Blesson Varghese

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Collaborative machine learning (CML) techniques, such as federated learning, have been proposed to train deep learning models across multiple mobile devices and a server. CML techniques are privacy-preserving as a local model that is trained on each device instead of the raw data from the device is shared with the server. However, CML training is inefficient due to low resource utilization. We identify idling resources on the server and devices due to sequential computation and communication as the principal cause of low resource utilization. A novel framework PiPar that leverages pipeline parallelism for CML techniques is developed to substantially improve resource utilization. A new training pipeline is designed to parallelize the computations on different hardware resources and communication on different bandwidth resources, thereby accelerating the training process in CML. A low overhead automated parameter selection method is proposed to optimize the pipeline, maximizing the utilization of available resources. The experimental results confirm the validity of the underlying approach of PiPar and highlight that when compared to federated learning: (i) the idle time of the server can be reduced by up to 64.1×, and (ii) the overall training time can be accelerated by up to 34.6× under varying network conditions for a collection of six small and large popular deep neural networks and four datasets without sacrificing accuracy. It is also experimentally demonstrated that PiPar achieves performance benefits when incorporating differential privacy methods and operating in environments with heterogeneous devices and changing bandwidths.
Original languageEnglish
Article number104947
Number of pages17
JournalJournal of Parallel and Distributed Computing
Volume193
Early online date2 Jul 2024
DOIs
Publication statusE-pub ahead of print - 2 Jul 2024

Keywords

  • Collaborative machine learning
  • Resource utilization
  • Pipeline parallelism
  • Edge computing

Fingerprint

Dive into the research topics of 'PiPar: Pipeline parallelism for collaborative machine Learning'. Together they form a unique fingerprint.

Cite this