This paper studies the 3-D turbulent channel flows over rough walls with stochastic roughness height distributions by using the large eddy simulation and the immersed boundary method. The obtained mean and fluctuating velocity profiles for the smooth and rough channel flows agree well with the available numerical and experimental results. The stochastic surface roughness is found to have a more significant influence than the uniform surface roughness on the turbulent velocity statistics and the coherent structures, with the same average roughness height. With a greater variation in the roughness height, the mean velocity and the streamwise fluctuating velocity is decreased and the spanwise velocity and the wall-normal fluctuating velocity are increased. In addition, one observes larger and more profuse quasi-streamwise vortices, hairpin vortices and elongated structures above the crest plane of the roughness array in cases of highly stochastic rough walls. However, the low-speed streaky structures are broken up locally and the ejection and sweep events are depressed by the stochastic roughness below the average roughness height. The results of this study support Townsend\'s wall-similarity hypothesis for both stochastic and uniform rough wall turbulences, demonstrating that in both cases the effects of the surface roughness on the turbulent flow are limited to the rough sub-layer.