The performance impact depends substantially on whether you've configured it to fsync the binlog on every group commit.

Also, it's important to consider that replication and backups serve different purposes. Backups alone are insufficient for high availability, change data capture, point-in-time recovery / undoing a bad change, etc.

Turning it off cause you're not using it seems reasonable, but I'm surprised it has a big effect on performance. Sequential appends to a file are pretty easy as long as you're not doing so many writes per second that there's contention on the write.

I've used bin log for almost decades and never experienced a big performance impact. This even holds for write heavy MySQL instances in ancient times where servers had spinning disks.