Having written three books, what I found was that you either self publish or you write books that publishers want. Some of that is choosing publishers that specialize in certain types of books. Some publishers want “Learn AI in 21 Minutes with Python,” while others want “Deep Dark Secrets of Java Class Loaders.” O’Reilly is the best for niche technical stuff. Most of the rest of the industry wants beginner stuff because that’s where the volume is. Fortunately, self-publishing is easier than ever and various sites make it easy to sell copies online, so you don’t necessarily have to just give it away for free. But yea, there’s no magic formula here. If you really want to write something niche, don’t expect that a publisher will be interested in. Expect that you’ll have to self publish and promote it yourself.
That’s my experience too. My most recent book (my fifth) was closer to one publisher’s stated mission than 95% of its catalog, but they all but admitted that they had given up on it. After that rejection I didn’t have the energy to approach anyone else and did it myself instead. LeanPub first, then Amazon (print and kindle), Apple, Kobo, and Google.