You have an error there (bad copy-paste?).

Should be "Stały 4 książki na półce".

Anyway, to make sense of the second one: treat "5 książek" as "5 of books", or "a 5-set of books"

— Four books were sitting on a shelf ("books" are plural, feminine)

— A five-set of books was sitting on a shelf ("five-set" is singular, neutral)

Now, why 5 becomes a set and 4 doesn't is not something I have a clue about.

But hope it helps grok how it affects the form of the noun being enumerated :)