This is exactly what I've done. I don't enjoy Rails very much but I love working with Ruby and Sinatra. The community has done a great job to keep Rails separated from Ruby when they release tools, so that they don't depend on or require Rails.

A lot of the aversion towards ruby I've gotten from people that have worked with it before mainly stems from complaints I share with the philosophy of Rails (making big changes to it when needed is like driving a freight ship instead of a small boat) and the complexity that can go with that, or with horror stories around poorly implemented metaprogramming, which are both valid points to me because I honestly share them. But Rails is not Ruby, and the community in general has been a lot more responsible with using metaprogramming carefully than it was in the early days.

Ruby becoming successful through Rails was a boon for it in the short term but I think also detracted from it's future growth at the same time as it became hard to untether developers' bad experiences with huge Rails apps with their experience using Ruby itself.

Personally, I love using Ruby so much for my own needs that if it ever was to fall into obscurity to the point of being unusable for modern tasks, I have decided that will be the point that I probably retire from programming and move on to other things in my life. It's not that I can't learn a new language for my daily driving, it's that I... don't really want to at this point.