New build 40 cameras 4K , what would you do ?

Cire3

n3wb
Joined
Mar 7, 2020
Messages
14
Reaction score
13
Location
US
I have a build I need to do with 40 cameras @ 4K recording on motion. BI is the flavor for PCNVR.

I need 36 for sure, but possibly 40 max. I have built a few PCNVR's but nothing on this scale.

Client wants 30 days storage. I figure 12 hours active and 12 off, 7 days a well.

Is this even possible with one PC / Server ?

What kind of hardware will run it ? Is anyone running anything this large ? What do I need to watch out for ?

I tried searching, but didn't really see anything this large with 4K cameras.

Thanks in advance !
 

The Automation Guy

Known around here
Joined
Feb 7, 2019
Messages
1,374
Reaction score
2,734
Location
USA
I'll simply say that it doesn't have to be one computer running everything. You can easily split the system into two parts and have two computers running BI.

There are obviously some pros and cons to this.

Cons:
Requires another computer and BI license so it might be more expensive (although two mid range computers is likely less expensive than one really fast computer)
Adds to the administrative work behind running the system - pulling footage from two machines, etc

Pros:
Don't have to use really fast computers (which are expensive) to run a large number of cameras
Has a limited amount of redundancy. If one machine goes down for some reason, the other machine and it's cameras might still be working

That being said, if you are recording directly to disk and try to utilize ONVIF triggers instead of the BI motion detection, you might be surprised at how many cameras you can run without issues. Obviously the new main stream/sub stream utilization will also help cut down on CPU usage even if you do need to do some motion detection with BI.

If you want to utilize AI, you can offload that to another computer as well to help lower CPU usage.
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
If you are using 4K cameras, if they are all inside with good light. Most 4K cameras are nearly useless at night. Show the customer a good 4MP camera with a 1/1.8 sensor. something like a IPC-T5442TM-AS-LED or a IPC-T5442TM-AS.

What is the application, business inside? Motion may not be the best way to go. A storage warehouse motion may work.
What is your expected frame rate .
How many monitors, independent view station?

Storage at 15 FPS continuous will be around 50TB for 12 hours per day for 40 cameras at 8MP(4K) with h.265 compression. If
 

Cire3

n3wb
Joined
Mar 7, 2020
Messages
14
Reaction score
13
Location
US
Many thanks for the quick reply !

3 viewers max at any one given time. One more than likely inside. Retail environment.

My understanding is indoor building with 10-12 foot ceilings. Great lighting. The cameras are all ready up to my dismay, and as far as traffic, it will be busy.

I'm thinking 6 X 10 Tb drives JBOD with windows storage spaces for recording ? I hate the thought of raid as a rebuild will not allow it to record on build, or I wouldn't think so anyway.

i9-10900k with 16Gb memory ? Being the 10900k has quick sync. Or will I need a card as well ?

Also I see some use 2 NVMe for boot , 2 SATA SSD for images and then large storage. What is the term images drive being used for ? Also are the NVMe RAID ?

This big of build also raises questions on my end. He may not get 4K on all cameras, and it's a reality he may have to accept.
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
I am a hard core KISS person, keep it as simple as possible.

I would use 6 disks, Each completely independent of each other, Assign 7 cameras to each drive. If a drive fails you loose 7 cameras. In a JBOD a disk failure takes out the complete system. In a retail environment, i would record continuous, there is too much movement, it will cut down on your processor needs.

Quicksync will work. I do not see a need for a separate video card to process video. You can use 16 GB of memory, but I would go for 32 GB with 40 cameras. More memory is better.

I am not an expert on multiple displays, but You can run multiple work station using UI3. So you can use a a Small NUC, keyboard, mouse and a good 4K monitor.

I do not think you need the SSD for video storage, write direct to the separate disks. Moving video files around is wasting CPU. A single NVMe will work for the windows 10 pro and the BI system.

Keep the system and cameras off the internet, NO internet access after install.
 

biggen

Known around here
Joined
May 6, 2018
Messages
2,539
Reaction score
2,765
Many thanks for the quick reply !

3 viewers max at any one given time. One more than likely inside. Retail environment.

My understanding is indoor building with 10-12 foot ceilings. Great lighting. The cameras are all ready up to my dismay, and as far as traffic, it will be busy.

I'm thinking 6 X 10 Tb drives JBOD with windows storage spaces for recording ? I hate the thought of raid as a rebuild will not allow it to record on build, or I wouldn't think so anyway.

i9-10900k with 16Gb memory ? Being the 10900k has quick sync. Or will I need a card as well ?

Also I see some use 2 NVMe for boot , 2 SATA SSD for images and then large storage. What is the term images drive being used for ? Also are the NVMe RAID ?

This big of build also raises questions on my end. He may not get 4K on all cameras, and it's a reality he may have to accept.
JBOD is fine and what I use for my installations. Its not RAID 0 so losing a single disk in JBOD just means you lose footage of whatever cameras you have assigned to write to the failed drive. The rest of the drives operate independently. Splitting the cameras up to have equal amounts write to different drives like @SouthernYankee suggested is a good idea.

It really depends on how redundant do you need the system though. Do you need fault tolerance? How big of a deal is it if a drive fails and you loose day/weeks of cameras footage for those cameras? If its mission critical then you have to use RAID and for an array this size you basically have to use RAID 10. RAID 5 is a no-no for an array this size and RAID 6 will take FOREVER to rebuild at 60TB.

Putting the OS on an SSD is standard operating procedure nowadays. Yes, I'd use NVMe. There is no reason to use standard SATA SSDs any longer if your motherboard has a NVMe M.2 interface. If not, then a regular SATA SSD is also acceptable. I wouldn't RAID up the OS. Since the video footage will be kept on the mechanical spinners, its safe from the OS SSD failing. Upon a Windows and Blue Iris reinstall, you just point BI back to your mechanical spinners so it can rebuild the DB and all your clips re-appear like magic.
 
Joined
May 1, 2019
Messages
2,215
Reaction score
3,504
Location
Reno, NV
I have a build I need to do with 40 cameras @ 4K recording on motion. BI is the flavor for PCNVR.

I need 36 for sure, but possibly 40 max. I have built a few PCNVR's but nothing on this scale.

Client wants 30 days storage. I figure 12 hours active and 12 off, 7 days a well.

Is this even possible with one PC / Server ?

What kind of hardware will run it ? Is anyone running anything this large ? What do I need to watch out for ?

I tried searching, but didn't really see anything this large with 4K cameras.

Thanks in advance !
what is the customer's expectation max price tag?
 

biggen

Known around here
Joined
May 6, 2018
Messages
2,539
Reaction score
2,765
You can get a tower case, add a nice enclosure, and stuff it with 18-20TB drives so getting to 64TB is pretty easy. Its just gonna be a high pricetag.
 

sebastiantombs

Known around here
Joined
Dec 28, 2019
Messages
11,511
Reaction score
27,690
Location
New Jersey
A full tower Obsidian case has eight drive bays for 3.5" drives, four drive bays for SSD drives and, if the mobo supports it, an M2 drive or two as well. Then, a SATA card, PCIe, can be added for another eight drives. Way more than would be needed for storage.
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
You must leave room to grow.

Make sure you have at least eight SATA connectors on the motherboard. Get a large case that supports eight 3.5 inch drives. Something like a
Thermaltake Core V71. Add additional fans to the case. I had a friends large PC custom built and tested, by an online PC shop. Work great but was a little expensive.

If you do not have motion detection on all cameras and use continuous record you should be able to handle the load. But it will be tight and require detailed BI configuration.

 

Cire3

n3wb
Joined
Mar 7, 2020
Messages
14
Reaction score
13
Location
US
Many thanks for the ideas ! Funny I was actually looking at the Corsair Obsidian Series 750D . I'll take a look @ the Thermaltake Core V71 as well.

Having an issue finding a motherboard with 8 SATA that the m.2 doesn't kill. In a LGA 1200 Chipset socket for the 10900K. I wont go any lower for this project.

He knows its going to be expensive. As far as who installed the cameras, it's his buddy that does all his TV's and what not in his new buildings. But doesn't know anything about NVR or PCNVR's. I was was to do the whole job at first. Any camera issues aren't my issue. I hope he did some research, as I wasn't sharing.

They are Hikvision from my understanding. And I'll check to make sure they are real Hikvision :) So I have the build / setup of the server, switches, PCNVR, and terminations.

Lots of great info, thanks !!

I'll be sure to post my results to help the group for sure !
 

bp2008

Staff member
Joined
Mar 10, 2014
Messages
12,666
Reaction score
14,005
Location
USA
RAID 5 is a no-no for an array this size and RAID 6 will take FOREVER to rebuild at 60TB
RAID 5 definitely a no-no. Raid 6 however is not necessarily bad. I have a RAID-Z3 (ZFS) with 11x 6TB disks. 42 TB usable. 86% full. A disk swap happened last month and the resilvering took 18 hours 18 minutes . Granted this isn't relying on some cheap RAID card; it was running on a haswell i5-equivalent Xeon and was basically idle the whole time besides resilvering.
 
Joined
Apr 26, 2016
Messages
1,090
Reaction score
852
Location
Colorado
Sure, this ought to work.

Finally Completed - New EPYC build & Limit Test - (Extremely Unnecessary Builds)

I estimate (40 * 8.3MP cameras @ 20FPS) using substreams if I did the math would be about 6500 MP/s effective full-rez recorded and another 600MP/s in substreams. With this configuration in testing, I got up to 52 cameras and 7200 MP/s effective running dual-GTX1650 video cards and drawing 200+ watts constantly from the wall.

Of course you could just build multiple little desktop systems for a lot less.
 

biggen

Known around here
Joined
May 6, 2018
Messages
2,539
Reaction score
2,765
RAID 5 definitely a no-no. Raid 6 however is not necessarily bad. I have a RAID-Z3 (ZFS) with 11x 6TB disks. 42 TB usable. 86% full. A disk swap happened last month and the resilvering took 18 hours 18 minutes . Granted this isn't relying on some cheap RAID card; it was running on a haswell i5-equivalent Xeon and was basically idle the whole time besides resilvering.
Idle makes a big deal. If you’re still writing to the array (which one would assume if using it for camera recording) then resilvering would take longer. Another issue is during a resilver the entire array is being hit hard by the parity calculation. The longer that goes on the higher the chance of a second drive failure and then you’re getting into trouble.

Im a RAID 10 guy with mechanicals just because of this. Resilvers are super fast since it just a straight copy. YMMV though.
 

Cire3

n3wb
Joined
Mar 7, 2020
Messages
14
Reaction score
13
Location
US
Again, the client knows it's going to be expensive. Lets say he has the funds to do whatever he wants to be honest. One of those, he wants what he wants.

Is there somewhere I can see the limit of BI ?

Windows 10 , this is what I get from Microsoft Docs :

NTFS can support volumes as large as 8 petabytes on Windows Server 2019 and newer and Windows 10, version 1709 and newer (older versions support up to 256 TB). Supported volume sizes are affected by the cluster size and the number of clusters. With (232 – 1) clusters (the maximum number of clusters that NTFS supports), the following volume and file sizes are supported.

Cluster sizeLargest volume and file
4 KB (default size)16 TB
8 KB32 TB
16 KB64 TB
32 KB128 TB
64 KB (earlier max)256 TB
128 KB512 TB
256 KB1 PB
512 KB2 PB
1024 KB4 PB
2048 KB (max size)8 PB
 

Cire3

n3wb
Joined
Mar 7, 2020
Messages
14
Reaction score
13
Location
US
Idle makes a big deal. If you’re still writing to the array (which one would assume if using it for camera recording) then resilvering would take longer. Another issue is during a resilver the entire array is being hit hard by the parity calculation. The longer that goes on the higher the chance of a second drive failure and then you’re getting into trouble.

Im a RAID 10 guy with mechanicals just because of this. Resilvers are super fast since it just a straight copy. YMMV though.
All my rigs are RAID10 as well. One 14 X 3Tb image server. For computer images, not cameras. I seen many pop the second drive trying to rebuild for days with other raids. However 64Tb is a crazy rebuild.. I would ask if the data was that important at that point (If the footage wasn't in question for any reason)
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
Another solution to using a raid. Is to use local drives for primary storage. Then use a network drive and have clone cameras write to the network drives. This may cause a problem with 40 cameras as BI has a maximum support of 64 cameras. I do not know if clone cameras count against the 64 max. Using a network drives you can have different storage time duration than the primary storage.

I use a network drive as a backup for my primary drive.

Advanced storage:
If you are using a complete disk for large video file storage (BVR) continuous recording, I recommend formatting the disk, with a windows cluster size of 1024K (1 Megabyte). This is a increase from the 4K default. This will reduce the physical number of disk write, decrease the disk fragmentation, speed up access.
 
Top