Okay, gave all those settings a try. Same results. For reference, was always using CBR and just regular x265, no smart codec.
I have an unmounted camera that is identical in focal length, I will swap it out later today and see if it has the same issue.
Here is another example, see the ghost...