There are, I think, three big hurdles for freshmen in college:

First, college can be a very abrupt shift from "someone is watching over you" to "you, and only you, are responsible for yourself." If you slack off in high school, someone--your parents, your teachers, your school administration--is likely to get on your ass to put you back on track, and this is more likely to be the case for people who get into tougher colleges ("helicopter parents" are very real things). But in college, if you sleep in and miss all of your classes... no one is going to care. Well, your parents might, but your parents aren't with you in college to do anything about it. I've seen quite a few people who went from straight A's in high school to failing most of their college courses because of this.

Second, a lot of those accepted are going to be the smartest person in their class, who are now going to a college where they might be average--or even below average.

Finally, college courses are generally more difficult than a high school course, even an AP course, teaching the same material; there's just a lot more that's expected as students. Freshmen courses generally mitigate this somewhat to help ease in students, but by the time you get to a senior-level course, reading multiple research papers a day for per class is considered pretty normal. Of course, if you're a kid with a truckload of AP courses who let you skip your first year of college, now you're dropped into courses that expect you to have already adjusted to college life first.