I solved it by setting the sort order for objects in front of the refracting object higher (e.g. to Transparent(3000))