We propose and experimentally demonstrate an optimized setup to implement quantum controlled-NOT operation using polarization and orbital angular momentum qubits.This device is more adaptive to inputs with various polarizations,and can work both in classical and quantum single-photon regime.The logic operations performed by such a setup not only possess high stability and polarization-free character,they can also be easily extended to deal with multi-qubit input states.As an example,the experimental implementation of generalized three-qubit Toffoli gate has been presented.