A numerical approach was performed to predict the propagation and transformation of nonlinear water waves. A numerical wave flume was developed based on the non-periodic high-order spectral (HOS) method. The flume was applied to analyze the effect of wave steepness and wavelength on the propagation of nonlinear waves. The results show that for waves of low steepness, the wave profile and the wave energy spectrum are stable, and that the propagation can be predicted by the linear wave theory. For waves of moderate steepness and steep waves, the effects associated with the interactions between waves in a wave group become significant and a train of initially sinusoidal waves may drastically change its form within a short distance from its original position.