Deep networks have been widely used in many domains in recentyears.However,the pre-training of deep networks is time consuming with greedy layer-wise algorithm,and the scalability of this algorithm is greatly re-stricted by its inherently sequential nature where only one hidden layer can be trained at one time.In order to speed up the training of deep networks,this pa-per mainly focuses on pre-training phase and proposes a pipelined pre-training algorithm because it uses distributed cluster,which can significantly reduce the pre-training time at no loss of recognition accuracy.Its more efficient than greedy layer-wise pre-training algorithm by using the computational cluster.The contrastive experiments between greedy layer-wise and pipelined layer-wise algorithm are conducted finally,so we have carried out a comparative ex-periment on the greedy layer-wise algorithm and pipelined pre-training algo-rithms on the TIMIT corpus,result shows that the pipelined pre-training algo-rithm is an efficient algorithm to utilize distributed GPU cluster.We achieve a 2.84 and 5.9 speed-up with no loss of recognition accuracy when we use 4 slaves and 8 slaves.Parallelization efficiency is close to 0.73.