In order to investigate the impact of channel estimation error on channel capacity of multiple input multiple output (MIMO) system, a novel method is proposed to explore the channel capacity in correlated Rayleigh fading environment. A system model is constructed based on the channel estimation error at receiver side. Using the properties of Wishart distribution, the lower bound of the channel capacity is derived when the MIMO channel is of full rank. Then a method is proposed to select the optimum set of transmit antennas based on the lower bound of the mean channel capacity. The novel method can be easily implemented with low computational complexity. The simulation results show that the channel capacity of MIMO system is sensitive to channel estimation error, and is maximized when the signal-to-noise ratio increases to a certain point. Proper selection of transmit antennas can increase the channel capacity of MIMO system by about 1 bit/s in a flat fading environment with deficient rank of channel matrix.