I wonder if NNs could be trained well enough with 1 bit weights (i.e. 0 and 1) with some layers doing addition, while others substraction (i.e. weight sign would be "hardcoded" into the network architecture). Or with zero-less weights (e.g. -1, -0.5, 0.5, 1).