Actually, I was going to completely edit this comment with a different suggestion, but decided to leave that to show the thought process.

Thinking more about the hours, I like the circle progressing round, but also maybe you could do it as a box with 1-4 sides present enclosing a polygon of 0-5 points. It's easy to mentally convert that to both 12 and 24 hour clock format.

For the minutes, I'd be tempted to do something like the current hours, done as a fraction of the circle arc with an additional polygon of 0-5 points for the minute. It's relatively easy to eyeball the size of the 5 minute segment, and the polygon just refines that a bit. You could locate the polygon so it touches the arc at the 1/12 of the circle corresponding to the base, which would help with estimating the arc length too.

Either way, I wouldn't make the encoding rely on colours, especially colours that look very similar.