I thought about a similar concept for fun - each hex digit was replaced by 4x4 pixel matrix, where amount of pixels roughly corresponded to the value. So dot for 0, two dots for 1, checkerboard for 8 etc.

Then byte was represented as 16x16 matrix where each 4x4 area had the lower digit pattern, and these were arranged in the shape of the higher digit.

But at the end of the day, it wasn't really more readable.