Upgraded firmware failed at 88% experience and recovery

[klasipca]

I'll share my experience while bricking several cams during firmware upgrade. Two of them were repaired by Jack, but 1 I recovered with the help from @alastairstevenson. Thank you!
First a little background on the recent cam

• It was purchased 7 month ago
• Upgraded to IPNC_S2l2.0.1_build201603031830 firmware 5 month ago
• Hasn't been touched in terms of configuration since then
• Recently I attempted initially simply to make changes to the settings and discovered that camera would not apply any changes
• I've tried resetting the camera to default without much luck
• The camera was working fine, but it seemed to be stuck in the "read only" state
• It did not have SD card
• So finally I decided to reload the same EXACT firmware using Batch Control/Update tool
• It failed at 88% (Exactly same way as the previous two cams as well)
• I did not reboot the camera after the failed upgrade

Note: If your cam was connect to NVR which means it probably using 192.168.254.x subnet, so you will need to make some changes to access it. What I did is simply changed my network configuration on PC to:

IP 192.168.254.2
Mask 255.255.255.0
Gateway 192.168.254.1

and then simply accessed 192.168.254.3 via browser which is what my camera used. (you will lose network access, but you can toggle back forth)


So I knew the firmware was good and worked before on the same exact camera, but my previous experience taught me to not reboot the camera after failed upgrade. Since I already knew the telnet was working I started gathering information what I needed to do. Here are steps that I ended performing:

Capture Serial and MAC address of your bricked camera before performing any steps. You can do this via batch tool, web ui or telnet. You will need those to update the configuration file later, otherwise it will be duplicate and will cause problems when using in Hikvision NVR.

1. Create a share on your window pc. Lets say c:\tmp
2. Login to telnet on working camera (port 3232)
3. Mount the share command: mount -t cifs -o username=<share user>,password=<share password> //yourpcip/tmp /mnt
4. Navigate to dev/ and copy mtdblock4 using command 'cp mtdblock4 /mnt/mtdblock4'
5. Login to telnet on a bricked camera
6. Mount the share on a bricked camera command: mount -t cifs -o username=<share user>,password=<share password> //yourpcip/tmp /mnt
7. Navigate to share cd /mnt
8. Copy mtdblock4 to a tmp folder command: cp mtdblock4 /tmp/mtdblock4
9. Run command: flashcp -v mtdblock4 /dev/mtd4
10 Reboot cam
11. After the camera booted login to telnet again
12. Perform the same steps to copy file /home/config/config_hw.ini to your windows share
13. Download Geany program if you are using Windows to edit linux text files mini-IDE/Editor, from here : https://www.geany.org/Download/Releases
14. Open config_hw.ini
15. Change Edit > Preferences > Files > End-of-line set to default to LF
16. Replace previously captured serial number and MAC address
17. Save and transfer the file back in to /home/config/config_hw.ini
18. Chmod 644 /home/config/config_hw.ini to set the same permission
19. Reboot the camera

Other optional steps:
20. Access SDAP program to reset IP
21. Access web ui and reset the camera for a good measure
22. Upload firmware via web ui to test it out

Note: Clone mtdblock4 was from a newer camera with most recent firmware.


I am pretty sure things steps could be condensed a bit and I did end up doing certain things incorrectly which I fixed with the help, so I am not listing that.

So this is actually simple enough because I did not reboot the camera and had normal telnet access, if you happened to reboot it then it will be much more headache. Please refer to Unbricking Mini v2 thread for details.

 

[alastairstevenson]

I think you did pretty well there, and probably learned a few things along the way.
One comment if I may -

In this case the mtdblock4 grabbed from another camera is quite big at about 15MB, and I think you were quite luck you didn't run into a space problem.
So if you do need to copy it over (say using tftp to do the transfer) it is necessary to take a look to see where it would fit.
The df command (or df -h if it supports it) and the mount command can help choose what mount points are in RAM and are therefore usually able to provide the needed space.
/tmp would normally be a safe choice, if it's not in use for anything else.

If you have the benefit of being able to mount a CIFS/SMB or NFS share, you can avoid having to copy over the file, as it's already available at the mount point in the file system tree.
Your command 8 here could be omitted, for example, removing the need for any additional space.

7. Navigate to share cd /mnt
8. Copy mtdblock4 to a tmp folder command: cp mtdblock4 /tmp/mtdblock4

I have to say I'm impressed with what you've done here, and what you've added to the forum knowledge base.
There is no doubt that having telnet access to a root shell opens up lots of possibilities, and risks too.

 

[klasipca]

I couldn't have done without all the groundwork done by you already. So one step remaining to update serial and mac address, once I get my hands on this tonight. Otherwise both cams don't want to work at the same time on NVR

 

[klasipca]

BTW, what is your guess why camera was a) not resettable b) bricked using the same firmware
Is this some bad programming in the early firmware that could have caused this or perhaps hardware issue?

 

[klasipca]

All done and all good. Thanks again! I've updated steps with additional info. I think I am starting to like all this hacking... but not really

 

[alastairstevenson]

Excellent!
So messing with the hardware definitions file didn't trigger an integrity tripwire .. interesting.
I wonder what the checkcode is for.

By the way - just in case any adventurous person is going to follow in your footsteps a small correction offered:
Step 9 in your post #1 is
flascp mtdblock4 /dev/mtd4
should read
flashcp mtdblock4 /dev/mtd4

BTW, what is your guess why camera was a) not resettable b) bricked using the same firmware
Is this some bad programming in the early firmware that could have caused this or perhaps hardware issue?

To be honest - I haven't a clue, sorry.

 

[klasipca]

Excellent!
So messing with the hardware definitions file didn't trigger an integrity tripwire .. interesting.
I wonder what the checkcode is for.

By the way - just in case any adventurous person is going to follow in your footsteps a small correction offered:
Step 9 in your post #1 is
flascp mtdblock4 /dev/mtd4
should read
flashcp mtdblock4 /dev/mtd4

thanks and I'll correct your code as well, need a parameter -v

To be honest - I haven't a clue, sorry.

Not even a wild guess? Knowing what we know, is there a chance that camera for 5 month was writing something to somewhere which caused it lock up or perhaps admin user lost admin privelgases (I did check though in the web gui it did say admin)

 

[alastairstevenson]

Well, they do write some log stuff and some command history to flash on-going, but it's quite small.
On admin privileges, there is no real segregation.
Everything runs with root access.

thanks and I'll correct your code as well, need a parameter -v

If you want to have more to worry about while it's doing its thing, -v for Verbose would certainly help.

On the bricked Mini PTZ V2 that @pozzello had, prior to changing anything I dumped the entire existing flash area using the Amboot bootloader and converted it back into binary and the original partitions and mounted them to inspect the contents. I'd not seen this camera before, so I wanted a look around before trying anything.
What was evident was the complete lack of any app-related files and programs, leaving just the basic Linux kernel and a skeleton structure that's probably just the initramfs (initial RAM file system - just enough to get going and start the main file system and apps).
So you could imagine as part of the firmware update process a preparatory wipe of any app and rootfs related areas with an intended reload that for unknown reasons didn't happen.
Then the appearance of a bricked camera on the next reboot.