Make sure you're using go2rtc, it's included in frigate.
If you just use the built-in detect stream, it gets decoded by frigate for motion, object detection and so there will be additional latency.
Go2rtc will let you restream the RTSP feed via webrtc, you should see much lower delays as the...