smb/cifs partitions used by Hikcams seem to require reformatting on a monthly basis

[PRC_Save_Me]

I've been experiencing a strange problem. Every month or so, a 32 GB NTFS partition mounted through samba on my router for the DS-2CD2332-I seems to fail, the status changes to uninitialized and the only solution to this issue seems to be to completely reformat the partition in a partition editor, and then have Hikvision format that partition.

Has anyone ever encountered this? The medium being used does not matter, it seems. I have encounter the same problem whenever I used an SD card, an HDD, or an SSD.


Any help would be appreciated

 

[nayr]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

will nothing else mount it either? what about locally on your router? seems odd to me your using NTFS through Samba, why not a Unix filesystem? or is this something the Hikcams require?

If your using NTFS in Linux isint that using FUSE? Ive had all sorts of problems with long term stability w/FUSE... but that was a long time ago.

 

[PRC_Save_Me]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

hi nayr, really appreciate the response

In order: no, nothing else will mount it either. Either locally on my router or locally on my workstation. The other partitions on the drive work fine. I've tried using EXT2-EXT4 but my hikvision cameras were not able to format the drive.

If your using NTFS in Linux isint that using FUSE? Ive had all sorts of problems with long term stability w/FUSE... but that was a long time ago.

Good point. I don't know what to do. I can recreate the bug when the drives/SD cards are formatted as FAT32 as well.

 

[nayr]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

what kinda scripting environment do you have on your router? can you setup python on it by any chance?

I think the NFS implementations are horribly convoluted, you might try using plain old FTP if you dont mind playing back the files externally as most wont playback off FTP.. personally thats not a problem as I find the web UI very slow and would rather just transfer the files over and play them back at 60x with a real app.

I use this script: https://github.com/jakejohns/py-direg to keep the cameras from filling up the HDD's and giving them each individual disk quotas, run in a cron a few times an hour and erases the old files... then a simple ftpd just churns em all out, the load is extremely low with 4 cameras on a low powered arm server.. I could probably handle 32 cams this way if I added a sata expander... my real NAS could handle hundreds of cams via FTP.

 

[PRC_Save_Me]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

I see a lot of "HDD 9 Error" messages in the log, but I've always seen those. I have no idea what it means.

 

[PRC_Save_Me]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

I'm currently running a Linksys WRT1200ac with stock firmware, but I have to option of flashing it with OpenWRT. I will have to look into that.

keep the cameras from filling up the HDD's and giving them each individual disk quotas, run in a cron a few times an hour and erases the old files.

Doesn't the hikvision firmware do this automatically? My disk quotas are set to the defaults. I figure this is the culprit. The 32 GB partition only allows for 3-4 days worth of recordings+JPEG snapshots, afterwards, it's continuously erasing old files and writing new ones.

 

[PRC_Save_Me]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

nayr, did you have to resort to using that py-direg script out of necessity because you were experiencing similar issues?

 

[nayr]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

might be performance issues? have you tried to manually transfer data constantly via smb/cifs via this router of yours? if its choking out for a few moments it might be throwing errors and eventfully resulting in corruption.

how many cameras are you recording with? I presume your using USB for disk access? Do you have any other computers you could setup some shares on and see if the problem disappears on different hardware?

 

[nayr]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

no my Dahua's dont include support for samba/cifs storage.. it was either this or save the RSTP streams to disk directly.. KISS lead me to use FTP and its pretty damn reliable.. I dont know why more people dont use it, but FTP does not report back used/free space like samba/cifs.. so the cameras cannot manage the drive, it will just keep pushing data until it can take no more.

FTP is also incapable of streaming video stored on it, so the cameras cant playback recorded video through there UI with it either.. but I share the ftp path out via read-only samba/cifs and remotely play back the raw files directly off the storage.

Samba/Cifs has alot more overhead than basic FTP, so when your trying to reliably store videos on a very low power machine its really the best choice.

 

[PRC_Save_Me]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

might be performance issues? have you tried to manually transfer data constantly via smb/cifs via this router of yours? if its choking out for a few moments it might be throwing errors and eventfully resulting in corruption.

Not constantly, only very occasionally. But I was transferring some large files today, in fact - but only for half an hour. I don't want to believe something like that would overload my WRT1200AC (it runs a dual-core ARM processor, the Marvell Armada 385, which you will find in consumer NASes these days). Its NAS read/write performance is comparable to that of Synology's more cheaper NASes.

On the other hand, the router's FTP server has been 100% reliable for me, so I may have to go with what that you did

how many cameras are you recording with? I presume your using USB for disk access? Do you have any other computers you could setup some shares on and see if the problem disappears on different hardware?

I'll have to experiment with this. I suspected for a long time the issue might be with the router. It hasn't been the most reliable, even with the beefy hardware. The quality of Linksys (well, now Belkin) products aren't what they used to be.

thanks for your help

 

[PRC_Save_Me]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

if its choking out for a few moments it might be throwing errors and eventfully resulting in corruption.

how does this happen? may you expound a little bit on the technical details?

 

[nayr]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

I am on a CuBox i4 Pro, thats like a QC 700mhz armv7 with 2gb ram and an eSata port w/WD Purple.. I get about 60MB/s disk performance and use more power on the HDD than the CPU.. iirc ~2-3W total.

system load while direg.py is doing its thing, top procs: (21% for direg, runs occasionally.. ~4.5% for each camera)

[B]17325 root      20   0    9520   4912   1736 R  21.7  0.2   0:28.88 direg.py                                                                                                                                                                 
[/B]14894 ftp       20   0    4256    884    444 S   5.6  0.0   0:33.37 vsftpd                                                                                                                                                                    
17042 ftp       20   0    4256    680    324 S   4.9  0.0   0:07.07 vsftpd                                                                                                                                                                    
14881 ftp       20   0    4256    876    444 S   4.3  0.0   0:26.55 vsftpd                                                                                                                                                                    
14884 ftp       20   0    4256    876    444 S   3.6  0.0   0:26.20 vsftpd

With 2 1080p and 2 3MP cameras recording at 15fps full quality im seeing about 4-5MB/s average coming in.. been running for 2 years now without as much as a hiccup.

also runs all my home automation crap, and voip services, and a few other things.. load average: 0.39, 0.55, 0.56

all these guys running giant i7's with full cpu load just to record nothing crack me up..

 

[nayr]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

how does this happen? may you expound a little bit on the technical details?

The camera is basically laying down its own filesystem ONTOP of your already provided system, so if I understand the implementation they are basically making it a virtural disk and to the end program it will look and function like a local disk.. ever try to run a program off the network and have it hang unexpectedly because the network had a hiccup? well now all the nice stuff samba/cifs does to handle that wont be known by the end software so it will treat it as a disk lockup and this can result in that virtual disk of yours not getting writes and going sideways as far as integrity.. like unplugging a computer suddenly while writing and corrupting the filesystem.

I suspect this implementation is not very fault tolerant, having the camera format a network share just oozes bad mojo.. someone at Hik does not understand what a network filesystem is when it thinks it has to put another on top..

 

[alastairstevenson]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

The camera is basically laying down its own filesystem ONTOP of your already provided system, so if I understand the implementation they are basically making it a virtural disk and to the end program it will look and function like a local disk..

I suppose it depends on your definition of a file system.

I suspect this implementation is not very fault tolerant, having the camera format a network share just oozes bad mojo.. someone at Hik does not understand what a network filesystem is when it thinks it has to put another on top..

'Format' is a misleading term for what is done by the camera. It's just a fixed framework of placeholder files and folders with the metadata held in some index files, using the remote file system the share has been offered under.
My money would be on the remote server file system implementation.

 

[nayr]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

then why is it so hell bent on using Fat32/NTFS? that should entirely transparent to the camera.. it should have no freaking idea or care what the underlying filesystem is.

something is very wonkey with there implementation.. this is evidenced by the number of people Ive seen having issues with the Hik's NFS solution.. which is alot of em.

 

[alastairstevenson]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

then why is it so hell bent on using Fat32/NTFS? that should entirely transparent to the camera.. it should have no freaking idea or care what the underlying filesystem is.

It isn't, and doesn't. What gives you that idea?

something is very wonkey with there implementation.. this is evidenced by the number of people Ive seen having issues with the Hik's NFS solution.. which is alot of em.

Yes - the issue with few exceptions is that when the volume size on the share is larger than a ceiling size, around 200GB on the older ~ 5.2.0 firmware, the 'format' operation can't cope as the arithmetic breaks down.
There's no doubt the design of how the files are stored and organised is flawed.

 

[nayr]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

It isn't, and doesn't. What gives you that idea?

This:

I've tried using EXT2-EXT4 but my hikvision cameras were not able to format the drive.

NTFS Support in Linux sux.. always has, I am not really surprised his FS goes sideways if he's using it heavily in linux.

 

[alastairstevenson]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

What was the volume size?

*edit*
This works OK both SMB/CIFS and NFS with Hik cameras and NVRs. With the user quota or volume size limited to what they can handle:

/dev/mapper/cachedev2 on /share/CACHEDEV2_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,delalloc,noacl)

 

[nayr]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

32gb, and how is it that a network client can corrupt the filesystem of a network share so its entirely unreadable? he claims he had same problem with FAT32.. this should not be possible with normal usage.

so how does that happen?

 

[alastairstevenson]

Re: smb/cifs partitions used by Hikcams seem to require reformatting on a monthly bas

32gb, and how is it that a network client can corrupt the filesystem of a network share so its entirely unreadable?

Well, one likely possibility is by the file server not being able to reliably and safely handle the requests of the client, and corrupting the system. Whatever bad things the client may do, it should not be able to corrupt the server volume.