【摘 要】
:
The proliferation of massive datasets has led to significant interests in distributed algorithms for solving large-scale machine learning problems.However,the communication overhead is a major bottleneck that hampers the scalability of distributed machine
【机 构】
:
National Laboratory for Parallel and Distributed Processing (PDL),College of Computer,National Unive
论文部分内容阅读
The proliferation of massive datasets has led to significant interests in distributed algorithms for solving large-scale machine learning problems.However,the communication overhead is a major bottleneck that hampers the scalability of distributed machine learning systems.In this paper,we design two communication-efficient algorithms for distributed learning tasks.The first one is named EF-SIGNGD,in which we use the 1-bit (sign-based) gradient quantization method to save the communication bits.Moreover,the error feedback technique,i.e.,incorporating the error made by the compression operator into the next step,is employed for the convergence guarantee.The second algorithm is called LE-SIGNGD,in which we introduce a well-designed lazy gradient aggregation rule to EF-SIGNGD that can detect the gradients with small changes and reuse the outdated information.LE-SlGNGD saves communication costs both in transmitted bits and communication rounds.Furthermore,we show that LE-SIGNGD is convergent under some mild assumptions.The effectiveness of the two proposed algorithms is demonstrated through experiments on both real and synthetic data.
其他文献
This research discussed a deep learning method based on an improved generative adversarial network to segment the hippocampus.Different convolutional configurations were proposed to capture information obtained by a segmentation network.In addition,a gene
Event temporal relation extraction is an important part of natural language processing.Many models are being used in this task with the development of deep learning.However,most of the existing methods cannot accurately obtain the degree of association be
Road pricing is an urban traffic management mechanism to reduce traffic congestion.Currently,most of the road pricing systems based on predefined charging tolls fail to consider the dynamics of urban traffic flows and travelers\' demands on the arrival
Integer overflow is a common vulnerability in Ethereum Smart Contracts (ESCs) and often causes huge economic losses.Smart contracts cannot be changed once it is deployed on the blockchain and thus demand further testing.Mutation testing is a fault-based t
Lesion detection in Computed Tomography (CT) images is a challenging task in the field of computer-aided diagnosis.An important issue is to locate the area of lesion accurately.As a branch of Convolutional Neural Networks (CNNs),3D Context-Enhanced (3DCE)
In distributed training,increasing batch size can improve parallelism,but it can also bring many difficulties to the training process and cause training errors.In this work,we investigate the occurrence of training errors in theory and train ResNet-50 on
Identifying the association between metabolites and diseases will help us understand the pathogenesis of diseases,which has great significance in diagnosing and treating diseases.However,traditional biometric methods are time consuming and expensive.Accor
N400 is an objective electrophysiological index in semantic processing for brain.This study focuses on the sensitivity of N400 effect during speech comprehension under the uni-and bi-modality conditions.Varying the Signal-to-Noise Ratio (SNR) of speech si
The prevalence of missing values in the data streams collected in real environments makes them impossible to ignore in the privacy preservation of data streams.However,the development of most privacy preservation methods does not consider missing values.A
Online advertising click-through rate (CTR) prediction is aimed at predicting the probability of a user clicking an ad,and it has undergone considerable development in recent years.One of the hot topics in this area is the construction of feature interact