If controller isn't being rotated in the moment, then camera should be stationary regardless of controller orientation, yea.

I guess you've used some strange gyro to stick emulation (never heard of such thing, but sounds like it).