This is not sufficient to stop XSS, because an attacker can redirect users to file.svg URL as the top-level document, bypassing restrictions of <img>.

If you're going that route, add CSP headers on HTTP level to disable scripting, and/or host the SVG on a separate domain that has nothing valuable, or use data: URLs.