Reducing Gain through isapi/cgi ?

[WestOzGuy]

Hello,
Sorry for smashing the board with another post. I think it is in addition to my previous post re: firmware 5.5.53 and it being a right balls up of a firmware.

Previous post here
I didn't think this question belonged with that post.

Basically I rolled back to 5.5.0 but I just don't like being that far behind in firmware. I have read a few things on these forums and other places about a pretty nasty hole in these earlier firmwares for port forwarded services. Well, I happen to like port forwarding my NVR so I can check in while I work away (remote).

So the latest firmware version I have found that actually allows me to connect to the cameras via pale moon was version 5.5.5 build 180111.

The problem is, it has a missing gain slider in the exposure settings. It appears to be set to 100% gain by default with no way to adjust this. My night time image is god awful!
I really do miss that nice smooth picture I had by reducing the gain to about 40 at my place.

So I have spent all night having a read and a fiddle. I have accessed the cameras via the command line using curl. I really would like to adjust the gain behind the scenes but the closest I've come is:
curl http://username: password@192.168.0.200:10001/Video/inputs/channels >dump.xml

This produces a dump.xml
<?xml version="1.0" encoding="UTF-8"?>
<VideoInputChannelList version="1.0" xmlns="http://www.hikvision.com/ver10/XMLSchema">
<VideoInputChannel version="1.0" xmlns="http://www.hikvision.com/ver10/XMLSchema">
<id>1</id>
<powerLineFrequencyMode>50hz</powerLineFrequencyMode>
<whiteBalanceMode>auto1</whiteBalanceMode>
<gainLevel>100</gainLevel>
<brightnessLevel>50</brightnessLevel>
<contrastLevel>50</contrastLevel>
<saturationLevel>50</saturationLevel>
<DayNightFilter>
<dayNightFilterType>auto</dayNightFilterType>
</DayNightFilter>
</VideoInputChannel>
</VideoInputChannelList>

I have tried changing the <gainLevel> tag to 40, and then uploading it with 'curl -t' but it just doesn't want to go back in the camera. I have poked around a little to see if I can find anywhere else to set it but have no luck so far. It appears that the gain is set for the source. Not sure how reducing gain will affect daytime recordings but I suspect not much. I can find a a happy medium between the two anyway, or script my raspberry pie to cron some magic

I was wondering if anyone of you guys have done something like this before? is it doable? I really don't want to roll back to version 5.5.0 but the difference with 100% gain is like stepping backwards 15 years in CCTV technology. Or maybe I am just wasting daylight hours over here

 

[SBBcarl]

I wrote this recently which runs as a cronjob on a raspberry pi I have on my network.
Every 15 minutes, it checks to see if the current time is within 10 minutes of the sunset/sunrise time for my location.
If so, it will open up my XML file for either Day or Night and send that configuration to the camera.

This way, each camera can have its own Day and Night settings since the physical location of them could play a factor in what needs to be changed.
While this automated part might not be what you need, you should be able to get an idea of how I am sending the XML settings over.

What I did was login to each camera directly and go to the image settings.
From there, I opened my developer tools on chrome and looked at the response.
I made all the settings I needed to for Day Time and clicked save. It then showed ALL of the settings in XML being sent to the API.
I saved this XML response and called it "day.xml".

That evening, I did the same thing except I tweaked all of the settings to make it look how I wanted at night and clicked save to generate the "night" XML response.

I now have two XML files with all of the settings I use for the day time and the night time.

From here, it's just a matter of getting the XML to the camera.
I used CURL with a PUT request to send the XML file to the API endpoint.

Its a little confusing but it's working.
Each day around sunset/sunrise the script will run and each camera will update its configuration image settings to what I defined in its XML file.

I attached the script to this post. Here is the main code:

#!/usr/bin/php
<?php

define('USERNAME', 'script');
define('PASSWORD', 'secret');
define('LAT', '33.0581');
define('LONG', '-112.0476');
define('SUNSETOFFSET', 10);
define('SUNRISEOFFSET', 10);
date_default_timezone_set('America/Phoenix');

/**
 * Update our camera feeds based on time sunrise / sunset
 */
function updateCamera($camera)
{
    try {

        $ch  = curl_init();
        curl_setopt($ch, CURLOPT_URL, "http://" . $camera['ip'] . "/ISAPI/Image/channels/1");
        curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_DIGEST);
        curl_setopt($ch, CURLOPT_USERPWD, USERNAME . ":" . PASSWORD);
        curl_setopt($ch, CURLOPT_TIMEOUT, 30);
        curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT");
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
        curl_setopt($ch, CURLOPT_POSTFIELDS, file_get_contents(__DIR__ . "/". $camera['ip'] . "_".dayOrNight() . ".xml"));
        $resp = curl_exec($ch);

        // validate CURL status
        if (curl_errno($ch)) {
            doLog(curl_error($ch));
            throw new Exception(curl_error($ch), 500);
        }

        // Validate HTTP status code
        $status_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        if ($status_code != 200) {
            $msg = "Response with Status Code [" . $status_code . "].";
            doLog($msg);
            throw new Exception($e, 500);
        }

        // Log Data
        $xml = new SimpleXMLElement($resp);
        $msg = "[".$camera['ip']."] - " . $camera['name'] . " | D/N: ".dayOrNight()." | Sunrise: ".date("H:i:s", sunInfo()['sunrise'])." | Sunset: ".date("H:i:s", sunInfo()['sunset'])." | Status: ". $xml->statusCode ." (". $xml->statusString.")";
        doLog($msg);

    }
    catch (Exception $ex) {
        if ($ch != null) {
            curl_close($ch);
            $msg = "Unable to Connect. - " . $ex;
            doLog($msg);
            throw new Exception($e, 500, $ex);
        }
    }
    if ($ch != null) {
        curl_close($ch);
    }
}

/**
 * Get the sunrise / sunset times based on the geo coordinates
 */
function sunInfo(){
    return date_sun_info(strtotime("now"), LAT, LONG);
}

/**
 * Based on the current time.
 * Check if we are prior to sunset or sunrise.
 */
function dayOrNight(){

    // Sunrise
    $sunrise = date("H:i:s", sunInfo()['sunrise']);
    $sunR = strtotime($sunrise);
    $sunR = $sunR - (SUNRISEOFFSET * 60);
    $sunrise = date("H:i:s", $sunR);

    // Sunset
    $sunset = date("H:i:s", sunInfo()['sunset']);
    $sunS = strtotime($sunset);
    $sunS = $sunS - (SUNSETOFFSET * 60);
    $sunset = date("H:i:s", $sunS);

    // If the current time is greater than or equal to sunrise-offset, and not past sunset-offset minutes use day time.
    if(strtotime(date("H:i:s", time())) >= strtotime($sunrise) && strtotime(date("H:i:s", time())) < strtotime($sunset)) {
        return "day";
    } else {
        return "night";
    }
}

/**
 * Log information for each run
 */
function doLog($message){
    $t=time();
    $fp = fopen(__DIR__ . "/log.txt", "a") or die("Unable to open log!");
    $data = date("m/d/Y @ g:i:s A", $t);
    fwrite($fp, $data. " " . $message . "\n");
    fclose($fp);
}



/*************************************************************************************************/



// Vars
$json = json_decode(file_get_contents(__DIR__  .  "/config.json"),TRUE);

// Do we need to update the cameras?
if($json['currentTimeOfDay'] != dayOrNight()){

    // Do we have cameras?
    if(count($json['cameras']) > 0 && !$json === FALSE){

        // Prepend to log
        doLog("---------------------");

        // Loop over our cameras
        for ($i = 0; $i < count($json['cameras']); $i++) {
            updateCamera($json['cameras'][$i]);
        }

        // Append to log
        doLog("---------------------");

    }else{
        doLog("Unable to open JSON Config.");
    }
    // Update JSON
    $json['currentTimeOfDay'] = dayOrNight();
    file_put_contents(__DIR__  .  "/config.json", json_encode($json));
}else{
    doLog("Current Time of Day: " . dayOrNight() . " | Last Logged Time of Day: " . $json['currentTimeOfDay'] . " - No updated needed.");
}

 

[WestOzGuy]

SBBcarl that's awesome thank you for sharing all of this with me.
I've got a pi on my network too. Once I figured the proper way to reduce gain I was going to script up a cron as well.
I didn't think of sniffing the protocol, I was just blindly poking around, reading. Thanks for your advice. I've got some IPC's on the network with the firmware that has the gain slider so I'll give that a bash and see what it comes up with thank you.

 

[WestOzGuy]

For anyone who is following:
I managed to use SBBcarl's guidance on this and have found where to adjust the night time settings for gain through sniffing the headers to the camera with firmware version 5.5.0. I can now manipulate the settings in the later firmware that have the gain slider missing from the image settings.

The process was quite simple:
Step1:
curl http://username:password@IPaddress:Port/ISAPI/Image/channels/1/gain/night >temp.xml
Contents of temp.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Gain version="2.0" xmlns="http://www.hikvision.com/ver20/XMLSchema">
<GainLevel>100</GainLevel>
</Gain>
​

Step2:
Just change the 100 to 60 (I wanted a gain of 60 but you can make it what ever you like)

Step3:
curl -T temp.xml http://IPaddress:Port/ISAPI/Image/channels/1/gain/night
Which returned the following response

<?xml version="1.0" encoding="UTF-8"?>
<ResponseStatus version="2.0" xmlns="http://www.hikvision.com/ver20/XMLSchema">
<requestURL>/ISAPI/Image/channels/1/gain/night</requestURL>
<statusCode>1</statusCode>
<statusString>OK</statusString>
<subStatusCode>ok</subStatusCode>
</ResponseStatus>​

Optional step:
You can then run the first command again to check that the setting has changed, without sending it to temp.xml so it just displays it on the screen
curl http://username:password@IPaddress:Port/ISAPI/Image/channels/1/gain/night
Which will return the same data to the screen and you can confirm your change was successfull

<?xml version="1.0" encoding="UTF-8"?>
<Gain version="2.0" xmlns="http://www.hikvision.com/ver20/XMLSchema">
<GainLevel>60</GainLevel>
</Gain>​

Keep in mind, to use these commands you will need to install the original cURL. Not Microsofts barstardisation of 'curl'. Seeing as this goes into the /night section I presume it sticks in the camera so probably wont need to be scripted. Just modified when you are setting up your cameras.

Thanks all.

 

[WestOzGuy]

Oh and just found something else:
Looks like that is the settings for "auto-switch" not "scheduled-switch". Which is funny because the camera with the earlier firmware was on scheduled-switch. I might just leave it on auto and fiddle with the settings to get it right. Previously I just used to set schedule-switch.

 

[WestOzGuy]

Well, I dont know what to say...
haha this is very odd.
Now that I have manually fiddled with the gain, changing it from the default 100 to 60, then switched between image settings "auto-switch" and "scheduled-switch" the gain slider as just appeared out of nowhere!
Its there now on both auto and switch. It's a little bit buggy this hikvission stuff, but it is fun

 

[WestOzGuy]

I know no one cares... but I just told my wife all this and her answer was 'meh'.
So I am gonna type it all out here because it absolutely has done my head in.

So I have 3 cameras out the front which were all on 5.5.0 firmware. Upgrading to 5.5.53 seen me unable to access the cameras via pale moon.
So I rolled back to 5.5.5 and it works. I can live with that, but like I say, the gain slider was missing.

So I spent a significant amount of time trying to figure out how to adjust it in the xml and send it to the camera, suspecting that it was still there in the back ground somewhere!

Solution appeared to be above. However, upon further investigation it was not actually adjusting the gain in the "scheduled switching" picture in the image setting or the live view picture.
However, changing to "Auto-switching" showed me an image with significant gain reduction. Perhaps I was changing that, even though the xml location was sniffed changing the slider on a 5.5.0 camera inside the image settings with scheduled-switching selected.

Trying not to ramble on too much here. But at some stage while mucking around with changing the xml and uploading the camera and switching between live-view, and configuration/image/auto and scheduled my gain sliders appeared back again.

So I thought I had a solution. install the 5.5.5 firmware, and squeeze a new gain level into it and the sliders come back. With that in mind I upgraded the second camera and went to check it out... OMG the gain sliders are there.
Maybe there is something going on with the cache in the browser so I cleared everything. Both cameras have the gain slider.. interesting.

Update the 3rd camera. No gain sliders. Push a new gain level onto it. No gain sliders. Flick back and forth into different settings while pushing xml at it. no gain sliders.
Reboot camera, no gain sliders.
Reboot other cameras, they have the sliders.

So I decide to pull the configuration off the 2nd camera and push it onto the 3rd. Maybe there is manual setting that is causing issues and if I just push all the settings from one to the other I could jag it.

Oh look you cant just export a config file now.
You can import, but the closest thing I found was the attached picture. Click Device Parameters and its asking for some encryption strings. So I make one up.
I go to import the file into the new camera, which fails obviously because it never asked for a decryption string.




Flick back to image... there is my gain sliders.

Seriously, very buggy system but its been a fun ride. I just wish I knew how these gain sliders all came back so I could let others know.
All 3 cameras were the same models, bought at the same time, from the same supplier, and probably the same batch from the factory.
Setting the bastards up how I like and not touching them for a very long time!

 

[alastairstevenson]

I just wish I knew how these gain sliders all came back so I could let others know.

Maybe not what happened here - and you probably can't test this long-shot theory now, but worth bearing in mind for any future similar situations -

Sometimes on a firmware update, where the underlying web layout is different from the earlier firmware, some elements can be missing or not rendered correctly.
This can be due to the page history / caching incorrectly affecting the updated page.
In those cases, clearing the browser history fixes up the rendering problem.

 

[WestOzGuy]

Yeah good point Alastair, but I did try that.
When I put the new firmware onto the 2nd camera and seen the sliders were there without me having to push xml onto the camera I thought maybe its cached the layout from the first camera. After all, I am using the nvr vhost so as far as the browser is concerned it is the same host, just a different port. I thought perhaps it had used some of the layout from the cache. I cleared cache a few times along the process

I was a bit cheesed at that point because I was set up to record my PC to show me looking at the fresh install of firmware with no slider, the retrieval/modify/upload of XML and then the resulting slider. I thought it could help others. I have spotted a few people in different places complaining that the gain slider was missing in the new firmware (which is old now)