(Very) Slow RTSP Using pfsense + VLAN on Dahua

[ScottOregon]

Hi all,

We have a situation where an RTSP stream viewed with VLC on the VLAN is quick to start (1 second), but when it goes through pfsense and a different VLAN, it's very slow (12 seconds). This is reproducible and has been tested with multiple machines on both Windows 10 and MacOS.

The specific setup is as follows:

• Camera: Dahua DH-IPC-HFW4613H-ZSA (as one example, but also an Amcrest 4k which is Dahua hardware)
• Client hardware: Dell with Intel i9 or MacBook (same results)
• Switches: all Gigabit D-Link Web Smart Switch (a couple of models betweeen the PoE feeding the cameras and the computer, but all D-Link Web Smart)
• Firewall: pfsense on a Netgate SG-3100
• Client software: Win10 or MacOS; VLC 3.0.8 Vetinari

We set substream 1 to a low resolution for testing:
H.264H, 704x576(D1), 20FPS, VBR, Quality 6

We start VLC with:
vlc -vvv "rtsp:/admin:xxxxxxxxxxx@192.168.200.123/cam/realmonitor?channel=1&subtype=1"


If the machine running VLC is on VLAN200, it consistently starts in almost exactly 1 second.
If the machine is switched to VLAN1, it consistently takes 12 seconds.

To prove there's nothing else in the mix (e.g. VLC settings or OS firewall stuff), we have performed the following test. Take a Mac notebook with DHCP enabled and wifi disabled. Plug its Ethernet into a switch set to untagged VLAN1.

• Refesh address via DHCP
• get an address on 192.168.34 (which is for legacy reasons, currently VLAN1)
• Issue the command: vlc -vvv "rtsp:/admin:xxxxxxxxxxx@192.168.200.123/cam/realmonitor?channel=1&subtype=1"
• Wait 12 seconds, and it appears.
• Switch the port to untagged VLAN200 (removed from 1, add to 200, actually)
• Issue the exact same command again: (vlc -vvv "rtsp:/admin:xxxxxxxxxxx@192.168.200.123/cam/realmonitor?channel=1&subtype=1")
• Wait 1 second
• it appears

I am new to pfsense, but I can't imagine it's this slow... We must be doing something wrong. The specific hardware (SG-3100) comes highly recommended and we didn't take any chances with setting up our own box (however tempting that was!)

I've heard mixed reviews about whether camera streaming should all be UDP but didn't get that working after a medium amount of effort. I'm interested in best practices there and have lots of questions, but separately, I'd like to get to the bottom of why this is painfully slow because there are lots of other IoT things we were hoping to use the pfsense+VLANs to secure.

This is our first test and it was really exciting seeing everything come together until this. Does anyone have advice?

Note also that both the Mac notebook and new Windows 10 machines behave the same: ~1 second on VLAN1, ~12 seconds on VLAN200 with pfsense doing L3 routing. So... different OS and hardware, but same results based only on switching from VLAN200 to VLAN1+pfsense.

Finally, and perhaps very significant--or maybe not:

We have a middle-aged iPhone 8 on wifi(obviously?) on the 192.168.34 (i.e. VLAN1) which is, as per above, the super slow access situation. On it, we have a "Live Cams Pro" app configured with the exact same RTSP URL. It loads in maybe 2 seconds and it is also controlled by pfsense... same network as the Mac notebook and the Win10 PCs. So... the best we can come up with is that the problem is somehow tied to the interaction between VLC and the pfsense and neither one individually, since VLC can be very fast with everything the same but bypassing pfsense and being in the same VLAN... but pfsense routing the stream can be pretty fast going to the iPhone.

We'd very much appreciate any ideas!

 

[crw030]

not a VLAN expert but jumping in to watch this conversation unfold as I am "interested" in learning more about VLANs, and happen to have pfSense firewall/router.

Can you post the models of the D-Link Switches and firmware versions? I read online some Websmart switches might not support certain VLAN standards or might not on specific (older) firmwares.

Also can you roughly map out the config of which ports are tagged & untagged ports to each respective VLAN and which ports are trunks running between switches to illustrate the network layout a little better than just mentioning the equipment. I wouldn't think internal network diagram would be super-secret. Please also let us know what the gateway IP addresses are configured for each network.

 

[ScottOregon]

Sure @crw030; I'll do that shortly--hopefully within a couple of hours after some work stuff. Also, FWIW, I cross-posted at Slow Dahua RTSP stream with VLC when going through pfSense and replied to a post there with a packet capture.

 

[crw030]

Got it, read your post over on the netgate forums. I think you are right to assume the "fast link" where the devices are on the same VLAN probably doesn't have exactly the same path (through pfSense), because my (limited) understanding of VLANs is devices on the same VLAN can talk directly to each other via MAC Address, the IP becomes less important.

You've definitely isolated the problem, I'm sure that 10 seconds pause is the issue now just to root cause. Normally those TCP 0 requests are responded within a few thousandths of a second by the device, but in your 10-second trace that feels almost like a timeout occurred or something. Can you tryin implicitly setting VLC --network-caching option to something small (say 1000) and see if that reduces the lag? Also what is your FPS and i-frame settings on the camera you are trying to watch? (just trying to figure out if it is VLAN negotiation or something video related).

 

[ScottOregon]

Hey @crw030...

Yes, I'll try the VLC stuff after this and let you know.

For now, here's the info you requested on the switches... we were actually going to go through and update all the firmware but haven't yet. We could do that shortly. By the way, we've had some of these switches for years and only recently discovered that they lose config on power loss unless you go through a "Save", "Save Configuration", "Apply" cycle! That turns out to be pretty important with VLAN config where you're actually putting in some effort, and relying on them to isolate cams! (and not wanting the default password)

Anyway, these are the switches involved; there are a couple more but I don't think they're in the picture as far as this route.

DGS-1100-08P Gigabit Ethernet Switch [GaragePoE]
Boot PROM Version Ver 1.0.1
Firmware Version Ver 1.00.B035
Hardware Version B1
IP 192.168.34.12

DGS-1210-10 [LivingRoomAboveTV]
Boot Version 1.00.016
Firmware Version 4.10.004
Hardware Version C1
IP 192.168.34.11

DGS-1210-28 [LivingRoomMediaCabinet]
Boot Version 1.00.016
Firmware Version 4.00.064
Hardware Version C1
IP 192.168.34.9

DGS-1210-28 [Office1]
Boot Version 1.00.005
Firmware Version 4.10.B051
Hardware Version C1
IP 192.168.34.8

... have another work call; after that, I'll see if I can get a network diagram up and try the VLC stuff you suggested.

 

[ScottOregon]

Only had a second, but your (@crw030) idea about timeouts produced a really interesting first hit from google... vlc rtsp session timeout - Google Search => RTSP timeout with VLC as client - The VideoLAN Forums which says (among other things)

the timeout is 10 seconds when no data is received and you cannot change it without changing the source code. There is no option for this at the moment.

Since it's almost exactly 10 seconds without any traffic, that looks relevant!

Of course, the post is pretty old... April 2012. ... but I think you're onto something with the timeout idea. I might try another client on Windows... have never really used anything but vlc and hope I don't have to change.

Also, I tried:

vlc -vvv "rtsp:/admin:xxxxxxxxxx@192.168.200.123/cam/realmonitor?channel=1&subtype=1"  --network-caching=1000

and got ~the same results.

 

[catcamstar]

You mention an (old) iPhone with app that didn't show this 10 second delay, what happens if you play the video streams from your browser (I assume you might already have the no-plugin firmware on board)? Try from the same IPC Vlan and your "other" Vlan. If there is a stream hickup, you can be sure it is the network. Otherwise, you might run into that vlc timeout. I put 2c$ on the latter

Good luck!
CC

 

[TonyR]

Any change if you use the main stream (subtype=0) ?

 

[ScottOregon]

@TonyR ... Thanks for the suggestion; using the main stream adds an extra 1.5-2.0 seconds and looks better (it's full resolution).

@catcamstar I'm not sure what you mean about the no-plugin firmware. Could you tell me a little more about that? One of the cameras we got is

https://www.amazon.com/gp/product/B07TQM83XJ

and I didn't think firmware was upgradable (I might be getting confused though; we ordered a few to test and some definitely said they were not upgradable). For example, another we got was

https://www.amazon.com/gp/product/B0827SM54V

which says "Note: The international stable firmware cannot be updated".

I was actually going to look for a non-VLC client to try to further isolate. We have now tried VLC on Mac, Windows 10, and iPhone. It has the same 11-12 second delay every time from VLAN1/subnet34.

@crw030 Based on your comment, we're going to try upgrading all the switch firmware and rebooting them all, then will post more on the network layout as soon as we can... along with the updated firmware versions. It seems like a good idea anyway, so we'll try to take "old firmware" out of the mix and in the off chance that a reboot changes something, I actually haven't tried that either.

Also, the behavior is identical for both of the above cameras, both of which are on VLAN 200.

 

[catcamstar]

@catcamstar I'm not sure what you mean about the no-plugin firmware. Could you tell me a little more about that? One of the cameras we got is

https://www.amazon.com/gp/product/B07TQM83XJ

and I didn't think firmware was upgradable (I might be getting confused though; we ordered a few to test and some definitely said they were not upgradable). For example, another we got was

https://www.amazon.com/gp/product/B0827SM54V

which says "Note: The international stable firmware cannot be updated".

What I mean is: connect to the webservice of your camera (eg

http://ip-in-vlan-200-of-your-camera

). If you are on a (sub)recent no-plugin version, you can actually view live footage from the camera. If that is also "buffering" for 10 seconds, you know it is a network latency issue. If you do not have the latest no-plugin firmware version, you can try palemoon browser or something similar to try this way. Only works on that Dahua cam off course, as I don't have experience with your other camera.

Good luck!
CC

 

[ScottOregon]

Hi all; I got caught up with other things and revisited this a few days ago. I'm happy to announce that this has been resolved. Thanks so much for the comments and help @crw030, @catcamstar, and @TonyR

The issue apparently was that VLC was trying to use UDP instead of TCP for the video streams and then timing out and reverting to TCP. I was able to fix this three different, independent ways. That is, any of these these works individually--you don't need all there.

1. open UDP ports to the cameras
2. VLC with command line option --rtsp-tcp
3. change preferences, Live555 stream transport to RTP over RTSP (TCP); this is in Input/Codecs, toward the bottom

I've been favoring few firewall rules, and don't have a compelling reason (at least so far) to use UDP, so options 2 and 3 are what I'm using for now, but I might revisit later.

Should I somehow mark this issue as "resolved"?

 

[crw030]

In your pfSense firewall rule you can enable both TCP & UDP by selecting the TCP/UDP option for whichever rule was blocking the access. The would potentially be a 4th fix. I would probably enable UDP because it has lower overhead, doesn't typically require as solid a connection (so it requires less work than TCP establishing conn/sending data/terminating connection) and is typically used for broadcast traffic anyhow. If you are on a wired internal network probably not a huge impact, but if you were on wireless or mobile data, UDP might tolerate it better.

 

[ScottOregon]

Thanks @crw030; I might try that--I'm so relieved that the problem wasn't just poor performance from the new pfsense. I actually didn't care quite as much about rule count (even though that's what I actually said) vs. just trying to have minimal ports open for security. But having said that, I don't know why allowing all UDP to the cameras, with connections originating in the secure network would pose any risk... I just wanted to start minimal and go from there.

BTW, if you ever want info on our VLAN config per your earlier comment, I feel pretty good about how that went, and it was ultimately not difficult to set up. The more I read about dahua cams, the more happy I am that we did that.

The three next projects for us are (1) fully automating video conversions off the cam (dav => something VLC plays) and (2) seeing if we actually can upgrade the firmware, though they are clearly marked as "not upgradable" by the vendor. You might be seeing some new threads from me on that before long, haha. Especially on the latter issue. Oh, and (3) is configuring the cams via the HTTP API vs the GUI... we have a few steps done on that, but a lot left.