CPU Power Options in Windows Server 2016

[spock]

I might be causing my own problem here by insisting on running Windows Server 2016 as my OS for Blue Iris, but alas that is what I have set up. I have been dealing with poor BI performance on my old hardware which was a very old i7-3770K (Ivy Bridge), and I assumed that was the problem. So this week I finally gave it "new" hardware, an i7-7700 (non-K) and 12GB of DDR4. Well, the performance is actually far worse now! BI records video (direct-to-disk with substreams enabled) just fine, but playback of a saved clip is laggy and averages 1fps. (If I download the clip and play it offline, it's properly 30fps so the camera source and recording is fine.) Background: I'm running six 2MP cameras in H.265, all with H.264 substreams. I have done all the other optimization I can find (FPS = keyframe interval, etc.).

I debugged some and found that the Power Options have a HUGE effect. Clicking "High Performance" instantly boosts playback to 30fps through UI3. Click "Balanced" and it drops to 1fps again. I can see in Task Manager while just recording (no viewing/playback) the CPU utilization is 6% and the CPU idles at 0.76GHz. Seems ok. Opening UI3 and viewing live cameras or the "all camera" thumbnail works fine. The trouble is, when I play a saved clip in UI3, the CPU usage (task manager) rises only to 9-10% and the CPU frequency only comes up to 0.86GHz, and the playback is 1-2fps as mentioned. Setting the High Performance power plan causes the CPU to permanently run at the full speed of 4.0GHz and playback is perfectly fine at 30fps.

So, that's the "fix" -- leave it in High Performance mode. Or that's a workaround at least. However, I dug further as I didn't like that. Some experiments:

I tried setting "EPP" to 0 and it didn't help. Commands, for reference:
powercfg -setacvalueindex scheme_current sub_processor PERFEPP 0
powercfg -setactive scheme_current

I wondered if the CPU or GPU frequency was the limiting factor. In Balanced mode the GPU runs at 350MHz and in High Performance it's pegged at 1150MHz. So I modified the Balanced power mode by setting Minimum processor frequency to 50/75% (I tried both). This modifies ONLY the CPU frequency and GPU stays at 350MHz. It resolves the 1fps issue, proving that the CPU frequency is the problem not GPU.

I'm still stumped though, why is the CPU usage so low in Task Manager if it is bottlenecking the video playback so badly? Why won't the CPU hardware and/or Windows kernel increase the CPU frequency on demand as needed? I feel like trying Blue Iris on Windows 10/11 on the same hardware to see if that fixes it. Windows Server 2016 is pretty old now and is showing its age. But that's a big annoying process just as an experiment, so I'm looking for other things to try first.

 

[bp2008]

In the advanced settings for the Balanced profile, you might find that the max CPU frequency was not set to 100%?

Ideally you would want the CPU to be able to reduce its clockspeed when max speed is not needed, to save energy.

 

[spock]

Great idea and I had the same thought so I had checked it already -- the max CPU frequency was already 100%. In fact I did the "restore defaults for this plan" to make sure there wasn't anything corrupted, and it didn't make any difference.

I agree, I would like the CPU to reduce clock speed when not necessary. The heat and stress might be ok now that it's winter but it'll only get worse as the weather warms up...

 

[JustAnotherCameraGuy]

Are you running the server OS on bare metal or as a VM? Microsoft states that if running on bare metal ECC memory is required. I’m assuming since you are running a core i7 you’re not running ECC RAM. With that said, this might not be related to the issue you are seeing but is an example of perhaps why you might not want to run a server OS if your hardware doesn’t meet the recommendations for what is supported for a reliable system. I’ve ran server OS’s on all sorts of devices without ECC but would never do it on something this important if my hardware didn’t follow the requirements 100%.

Maybe check BIOS settings as well for any CPU / power savings variables.

Also, some of the BI experts will be chiming in on why you don’t necessarily need 30fps.

 

[spock]

OS is running on bare metal. And you're right, there's no ECC memory in this system. I could run a couple days of memtest86 to rule out bad memory I suppose. But the failure pattern is so distinct and connected to the CPU frequency governor that I doubt I can blame the RAM. I did reset BIOS defaults but haven't played with the settings too much, so I'll check it out.

And I agree that 30fps is probably overkill but I do need more than ONE fps! It's seriously laggy and chunky in Balanced mode and silky smooth in High Performance. Can't be a coincidence. Honestly the right answer is probably to switch to win10/win11, but I don't really like that answer either. If it came to that, I'd probably just leave it in High Performance mode. OR maybe upgrade to Windows Server 2022??

 

[IAmATeaf]

What other features of Server 2016 do you leverage?

 

[ludshed]

You didn’t mention anything about hard drive.

 

[spock]

You didn’t mention anything about hard drive.

Ah, yes I omitted that for two reasons. First, it was my initial suspicion and I so did a deep investigation. IO latency, queue depth, split IO, etc... all the file/disk metrics are not only healthy but trivially low. I'm running a mirrored SSD for New and Alert storage and mirrored HDDs for the rest of it. The disks are downright bored and functioning properly. Secondly, when I found that the CPU power governor setting had such a dramatic impact on the performance, I don't see how that knob is directly connected to IO performance. There's a tertiary link through IRQ handling of IO transactions but I would also expect that to appear in the performance logs if anything were lagging, and it's all-clear.

My mistake for not including those details, but I believe I have ruled it out as a cause.

 

[spock]

What other features of Server 2016 do you leverage?

Honestly, just the fact that I own the license key. I don't really use the server features I just thought "hey, camera server is a /server/ so why not run the server OS I happen to have a key for?" And I may not even have a spare win10 license to put there instead!

I did play around with reFS for a while. Professionally I'm experienced with Solaris and Linux but not so much Windows. I'm a die hard ZFS advocate and when I went looking for an equivalent in the Windows ecosystem I found reFS and Storage Spaces. I'm not impressed... it's very nearly abandonware and has a LONG way to go to catch up with ZFS from a decade ago. Anyway, yeah, I don't really NEED the server OS, it's just a pain to swap!