My doctor actually doesn't recommend colonoscopy until age 50. But starting at age 40 they have you do the "poop in a box" test instead, and then only have you come in if that shows anything.
The complication rate for colonoscopy is about 3 in 1000, and that is skewed towards people who have polyps, which in and of themselves could be dangerous if not removed.
So it's always a risk tradeoff. You can skip the procedure and risk the effects of the disease it's supposed to detect instead. But if you do the math, you're statistically better off doing the procedure.