We know what something with about 2 solar masses looks like at about 5 light years, because there is such a thing a bit closer: the Alpha Centauri system.

So (and again, this is very rough approximation) I mean AU: consider the high speed of interstellar encounters so it won't be there for long, and that it takes the full pull of Sol 6 months to turn us around 180°.

Also, yes, in general this would cause a lot of the Oort cloud to destabilise and a long period of bombardment of all the planets.

At 5 AU, gravitational force would be 1/25 or 4% the strength of the sun at 1 AU. If the black hole was four solar masses, it would exert a pull about 16% as strong as the sun. Probably not enough to rip us out of orbit, but enough to seriously perturb it. I think it's at least an end-of-civilization event.

5 AU is the perfect distance to really wreck Jupiter, though.