Hall of mirrors bug when using refraction.

I had tried plugging both vector 2 values and an append node bringing in alpha channels to do refraction. It seemed that once doing so I got the refraction effect fine, but sometimes when turning my player's first person view a certain angle away from the object using it, I'd see my view freeze, and then if I hit a key to detach my camera from the player I'd see a hall of mirrors effect as I moved my mouse and the player hand\weapon moved around to leave trails of itself.

If I removed refraction from my shader it would work fine. This was using a deferred rendering camera. I tried changing my camera to forward rendering but it still occurred.