That's my take. I break the DRM off books I've bought. I own those copies. I'll format shift them for my own convenience. Bought on Kindle but want to read on my Kobo? It's impossible to make me feel guilt about that.
But I don't read books I haven't legally acquired, whether through a paid bookstore, or temporarily borrowed via Libby, or Standard Ebooks or whatever. I won't yell at other people for doing that, but I don't do it myself. In a nutshell, I follow the same rules as with physical books I own (or temporarily possess).