My guess is that because almost 1/2 of the image that your camera sees is your siding, ... when the IR hits the siding at night, it makes the siding "bright". Your camera automatically adjusts for this brightness, which makes the rest of your yard too dark.
If you take the dome off of the camera (while its still mounted to your soffit), the lens itself should be able to be pointed up/down, as well as being rotated left/right 360'. You should be able to push the lens away from the house, to where there's no vinyl in the image. If the image is crooked after that, you should be able to "twist" the lens to straighten it back out.