Routing tables don't work here, because the routing table looks something like:

  fe80::/64 dev eth0 proto kernel metric 256 pref medium
  fe80::/64 dev eth0.11 proto kernel metric 256 pref medium
  fe80::/64 dev eth0.13 proto kernel metric 256 pref medium
  fe80::/64 dev eth0.14 proto kernel metric 256 pref medium
  fe80::/64 dev eth0.15 proto kernel metric 256 pref medium
  fe80::/64 dev eth0.16 proto kernel metric 256 pref medium

Which interface's fe80::4 are you talking about? They all have an fe80::4.