The Hofstadter model, well known for its fractal butterfly spectrum, describes two-dimensional electrons under a perpendicular magnetic field, which gives rise to the integer quantum Hall effect. Inspired by the real-space building blocks of non-Abelian gauge fields from a recent experiment, we introduce and theoretically study two non-Abelian generalizations of the Hofstadter model. Each model describes two pairs of Hofstadter butterflies that are spin–orbit coupled. In contrast to the original Hofstadter model that can be equivalently studied in the Landau and symmetric gauges, the corresponding non-Abelian generalizations exhibit distinct spectra due to the non-commutativity of the gauge fields. We derive the genuine (necessary and sufficient) non-Abelian condition for the two models from the commutativity of their arbitrary loop operators. At zero energy, the models are gapless and host Weyl and Dirac points protected by internal and crystalline symmetries. Double (8-fold), triple (12-fold), and quadrupole (16-fold) Dirac points also emerge, especially under equal hopping phases of the non-Abelian potentials. At other fillings, the gapped phases of the models give rise to topological insulators. We conclude by discussing possible schemes for experimental realization of the models on photonic platforms.