I suddenly don't have this issue anymore at all. I don't know what changed. 

Looks identical to this problem: https://shaderforge.userecho.com/topic/368408-screen-position-node-output/#
Does it still happen at runtime or is it only in the #scene work view?

@Joachim You will see the jitter if you view the shader in a Unity viewport and rotate the camera. It only happens in point filtering and when the wrap mode is on repeat. The problem is much more obvious if you use a small 2x2 or 4x4 texture and divide it over the width of the screen.

edit: Just found out that it doesn't actually happen at runtime. It's just the #scene work viewport that shows the jitter.