Not necessarily but you will have some explaining to do when you apply for a new E-3 visa again.