Ok, we are now just over 1 hour in and the delay has not returned. It looks like the default buffer size was causing the issue. I want to thank everyone who gave me some areas to look at.
On another note, @wittaj you mentioned that this camera had issues with the processor not being scaled...