A spherical balloon 20cm in radius is displacing 41g of air. Even ignoring compression (which I don’t know enough to quantify the effects of, except that it will make the numbers more unfavourable), nitrogen’s 3.3%-lighter gives you a budget of only 1.35g for the balloon. I believe balloons hare heavier than this, so the balloon will still sink (a little more slowly than an air-filled one, but I’m not sure how noticeable the difference will be).