Abstract Purpose To compare late genitourinary (GU) and gastrointestinal (GI) toxicity following different prostate cancer treatment modalities. Materials and methods This study included 1084 consecutive prostate cancer patients treated with conventional radiotherapy, intensity-modulated radiotherapy (IMRT), permanent iodine-125 implantation (PI) alone, and PI combined with external beam radiotherapy (PI + EBRT). The effects of treatment- and patient-related factors on late grade ⩾ 2 (G2+) GU/GI toxicity risk were assessed. Results The median follow-up was 43 months (range, 12–97 months). Compared to the PI + EBRT, there was significantly less G2+ GU toxicity in the conventional radiotherapy (hazard ratio [HR] = 0.39; 95% CI, 0.20–0.77) and the IMRT (HR = 0.45, 95% CI, 0.27–0.73). Compared to the PI + EBRT, there was significantly more G2+ GI toxicity in the IMRT (HR = 2.38; 95% CI, 1.16–4.87). In PI-related groups, prostate equivalent dose in 2 Gy fractions was a significant predictor of G2+ GU toxicity (p = 0.001), and the rectal volume receiving more than 100% of the prescribed dose was a significant predictor of G2+ GI toxicity (p = 0.001). Conclusion The differences in the late G2+ GU/GI risk cannot be explained by the differences in treatment modalities themselves, but by the total radiation dose to the GU/GI tract, which had a causal role in the development of late G2+ GU/GI toxicity across all treatment modality groups.