Subsea pipelines passing through the shallow area are physically protected against the environmental, accidental, and operational loads by trenching and backfilling. Depending on construction methodology, environmental loads, and seabed soil properties, the stiffness of backfilling material may become largely different from the native ground (softer than native ground in most of the cases). The different stiffness between the backfill and native ground affects the soil failure mechanisms and lateral soil resistance against large pipeline displacements that may happen due to ground movement, landslides, ice gouging, and drag embedment anchors. This important aspect is not considered by current design codes. In this paper, the effect of trench-backfill stiffness difference on lateral pipeline-backfill-trench interaction was investigated by performing centrifuge tests. The soil deformations and failure mechanisms were obtained by particle image velocimetry (PIV) analysis. Three experiments were conducted by using three different backfills including loose sand, slurry, and chunky clay that represent the purchased, natural in-fill, and pre-excavated materials, respectively. The study shows that the current design codes underestimate the lateral soil resistance for small to moderate pipe displacements inside the trench and overestimate it for large lateral displacement, where the pipeline is penetrating into the trench wall.