> therefore most likely overpriced at ~6€/month

> something where security is high and support is 24/7 available

You are not going to get great 24/7 support inside anything like €6/month (though many may promise it!), so that service may not be as overpriced as you think.

Another thing to consider is that many VPS providers have “dirty” IP ranges so you will have trouble with getting your mail delivered. A common solution to this is to use a mail delivery service (like mailgun, mxroute, and many others) so your mail comes from a managed “clean” source and has less chance of summarily being declared spam just from the source address alone (these services also reduce the need to think about spf/dkim/etc and keep an eye out on changes that may happen in that realm). Essentially by self-hosting a service like this, you are taking back the admin of managing all that so you have to ask yourself if it is worth saving a few € per month, and consider what you might lose if something goes wrong and all your mail starts bouncing.

> switch to some VPS/root or anything where I can SSH and install, compile my own services

Be wary of most really inexpensive options here. In most cases you will experience performance degradation (at random times, or just almost always) and other issues due to noisy neighbours on oversold infrastructure.

> I really want to try this, but I'm afraid my DNS will be blacklisted if I do.

The best (but unfortunately time-consuming) way to get comfortable with this is probably to register a sacrificial domain and set it all up for that domain on hosts not associated with your domains that normally send mail. Play with it, break it, fix it, break it again, etc., until you feel confident you aren't going to screw yourself by trusting your own admin of all this on a domain that actually matters.