Ta2O5 films are deposited on fused silica substrates by conventional electron beam evaporation method. By annealing at different temperatures, Ta2O5 films of amorphous, hexagonal and orthorhombic phases are obtained and confirmed by x-ray diffractometer (XRD) results. X-ray photoelectron spectroscopy (XPS) analysis shows that chemical composition of all the films is stoichiometry. It is found that the amorphous Ta2O5 film achieves the highest laser induced damage threshold (LIDT) either at 355 or 1064nm, followed by hexagonal phase and finally orthorhombic phase. The damage morphologies at 355 and 1064nm are different as the former shows a uniform/used area while the latter is centred on one or more defect points, which is induced by different damage mechanisms. The decrease of the LIDT at 1064 nm is attributed to the increasing structural defect, while at 355nm is due to the combination effect of the increasing structural defect and decreasing band gap energy.