So I experimented a bit more. Turns out, it was bad calibration all along! I 3D printed a new calibration shape to calibrate camera-to-robot and everything is fine now.
As a side effect, it doesn’t really matter if you use the “corrected” DH parameters at all, which opens up the possibility of using Inverse Kinematics formulas, which are a lot faster then the Jacobian method. But if you need “Polyscope”-accuracy, eg your calculations should match the PolyScope display, you should use the robot calibration correction values, including the smoothing on the rotation part!