If you don't need to switch versions at runtime (ABR), you don't even need to chunk it manully. Your server has to support range requests and then the browser does the reasonable thing automatically.
The simplest option is to use some basic object storage service and it'll usually work well out of the box (I use DO Spaces with built-in CDN, that's basically it).
Yes, serving an MP4 file directly into a <video> tag is the simplest possible thing you can do that works. With one important caveat: you need to move the "MOOV" metadata to the front of the file. There are various utilities for doing that.