It sounds like you are also running the cameras through the router, so when you put that many on it, it cannot handle and process it. Unlike streaming services like NetFlix, these cameras do not buffer, so it is trying to pass a large load 24/7 and is bottlenecked.
That plus no reason to max these cameras and units out. Your system may have been able to keep up with 2 cameras, but when you doubled it, you magnified the issues.
Shutter speed is much more important than FPS. Most of us use 10-15 FPS.
Bitrate also impacts the picture quality - too low and it is a pixel mess. Too high and you are wasting storage space with no real appreciable return.
For a 4MP cam, I start at 8192 and CBR and then adjust up and down until my eyes do not see a difference. Some will run 20,000 and others can run 4192. YMMV based on field of view.
Keep in mind that these type of cameras and NVRs, although are spec'd and capable of these various parameters, real world testing by many of us shows if you try to run these units at higher fps and higher bitrates than needed that you will max out the CPU in the unit and then it bugs out just long enough that you miss something or video is choppy. My car is rated for 6,000RPM redline, but I am not gonna run it in 3rd gear on the highway at 6,000RPM...same with these types of units - gotta keep them under rated capacity. Some may do better than others, but trying to use the rated "spec" of every option available is usually not going to work well, either with a car or a camera or NVR.
I have a cheapo camera I use for overview purposes, but one of the cool things that camera has though in the gui is it shows the CPU usage. If I max out the FPS, bitrate, use it's motion detection and set it to middle sensitivity, the CPU maxes out 100% quite often. If I run it at 15 FPS with an appropriate bitrate and motion detection at a reasonable level, the CPU sits around 40%. I suspect even the more expensive cameras function close to this.