Objective There is strong evidence that the body composition can affect the progression-free survival (PFS) and overall survival (OS) in patients with a variety of cancers. The main objective of this study was to investigate the effect of body composition on the prognosis of patients with advanced gastrointestinal and colorectal cancers who received first-line palliative chemotherapy. Methods Patients who were newly-diagnosed with advanced gastrointestinal or colorectal cancer and received standard first-line palliative chemotherapy from January 2017 to December 2018 were included in this retrospective study. An analysis of computed tomography images was performed to determine the skeletal muscle index (SMI), which reflects the skeletal muscle mass and skeletal muscle density (SMD) related to muscle strength. A Kaplan-Meier survival analysis and log-rank test were used to compare the survival relationships among groups stratified by the SMI, and a Cox proportional hazard model was used for a multivariate analysis. Results A total of 108 patients met the inclusion criteria, including 41 cases of gastric cancer, 46 cases of left colorectal cancer, and 21 cases of right colon cancer. In patients with gastric cancer, the OS of women was significantly shorter than that of men. The OS of patients with a low SMI, low SMD, and low phase angle (PA) was significantly shorter than that of patients with high values (P ≤ 0.05). In the multivariate analysis, the SMD was significantly associated with the patients' long-term survival [Hazard Ratio (HR) = 0.904, 95％ CI: 0.840～0.974, P = 0.008]. For patients with a low SMI and PA, the PFS was significantly shorter than that of patients with high values (P ≤ 0.05). In patients with left colon cancer, the PA and SMD were both independent risk factors for a poorer long-term prognosis (HR = 0.375, 95％ CI:= 0.167～0.840, P = 0.017; HR = 0.887, 95％ CI: 0.824～0.954, P = 0.001). Among right colon cancer patients, the PFS and OS of those with a low SMD were significantly lower than those for patients with high values (P ≤ 0.05). Conclusion The PA is an independent risk factor for the OS of left colon cancer patients; the SMD is an independent risk factor for the survival of patients with gastric cancer, left colon cancer, and right colon cancer.