We demonstrate a polarization-insensitive silicon $4\times 4$ optical switch based on Mach–Zehnder interferometer (MZI) switch elements. On-chip polarization controllers are integrated before the switch fabric to automatically adjust an arbitrary input polarization to the transverse electric mode. The $4\times 4$ switch fabric is based on a dilated double-layer network architecture to completely cancel the first-order crosstalk. Thermo-optic phase shifters are integrated in the MZI switch elements and the polarization controllers for adjustment of the switching state and polarization, respectively. We develop a polarization control algorithm based on a gradient descent method for automated polarization control. The polarization recovery time is less than 4 ms, and the measured polarization-dependent loss is $\sim 2\mathrm{dB}$. The scheme provides a new solution for realizing polarization-insensitive silicon optical switches.