An experimental study of the dependence of SiO2 waveguide side wall roughness on the etch conditions and etch masks in CHF3/O2 based reactive ion etching plasma was reported. When working under standard low-pressure (20mtorr) etching conditions, a novel etch roughening phenomenon has been observed in the plasma, that is, the roughness of the etched front surface increases with the amount of material etched, independent of etch rate, RF power, and gas composition. Besides, the etched underlying side wall will be tapered as the upper SU-8 resist pattern degradation transfers downward. A process using double-layered mask, consisting of SU-8 resist and thin Chromium film, was developed for improving the side wall smoothness. Based on the studies, SiO2/Si channel waveguides with the propagation loss less than 0.07dB/cm were fabricated at last.