I think it's a bit judgemental to declare what "the fun part" is. As a kid, I loved both -- I'd build sets as defined and keep them as-is forever. I'd play with them, too, but I liked the model aspect.

I also had tons of extra bricks that I'd use for free-form play. I loved both aspects.

As an adult, I don't really have the time or interest in "playing space explorers" or "driving" little cars and trains through lego "city" towns. But I still love building the prettier models and having them on a shelf.