Typically one defines relations before functions anyway (unless you're doing type theory/programming, in which case types matter), and relations also offer a fix (the empty relation from the empty set is left inverted by the empty relation from the target to the empty set).

Most algebra books I've read are either explicitly or at least implicitly setting up structural analogies to introduce categories, where your A and B are indeed fixed/"typed".