I came to a similar realization recently. I used to care so much about how pristine my books looked and then I realized that some of my favorite books I have found in used bookstores are ones which are written all over and have a well-worn feel to them. I now annotate all my books with pencil and also write my name in them. I love seeing a history of who's owned a given book when I open the front cover and see a name and phone number.
I once read that the only way you can actually damage a book, is by not reading it.