Running CPAI on OrangePi/RK3588--RKNN modules are "not available"

[PeteJ]

Newbie here, but I've googled for hours but have not found much success.

I currently run BI and CPAI on the same server, and it works ok. The server does not have a GPU and it takes 300-700ms for AI to run, and this is on only a few cameras. The server is a 11th gen Intel i5, 16G of ram. For comparison, I run Frigate on Linux with a Coral TPU and inference times are sub 10ms, on the same server, so I'm looking for a way to speed things up w/o a GPU.

I want to split the CPAI workload off to an OrangePI 5 Max to see if performance would improve.. but I can't figure out how to get the RKNN modules to install since they are currently listed as "Not Available" on the CPAI server dashboard.

The Orange Pi runs the official Ubuntu distribution from Orange Pi (22.04.5 LTS, 6.1.43 kernel), I am running CPAI in Docker. CPAI is 2.9.7. I am running the container in privileged mode, with /dev/bus/usb passed thru as devices. Within the container, I can see those devices, and Docker confirms the container is privileged. I am running the amr64 image.

my docker compose file:

services:
CodeProjectAI:
image: codeproject/ai-server:arm64
container_name: codeproject-ai-server-arm64
privileged: True
hostname: codeproject-ai-server
restart: unless-stopped
ports:
- "5000:32168"
environment:
- TZ=America/Los_Angeles
volumes:
- /dev/bus/usb:/dev/bus/usb
- codeproject_ai_data:/etc/codeproject/ai
- codeproject_ai_modules:/app/modules
devices:
- /dev/bus/usb:/dev/bus/usb
volumes:
codeproject_ai_data:
codeproject_ai_modules:


If you have CPAI running on RK3588, would love some advise on how to get this going.

Thanks!

 

[wittaj]

Before you go to all that effort, unless you like to tinker, what is the benefit?

Does improving sub 1-second process times result in any additional improvements to your response to a situation? 300ms is faster than the reaction time of most people.

If it were 15,000ms, then yeah giving a perp a 15 second head start on you is bad, but sub 1 second, whether it is 300ms to 5ms, that is human reaction time differences that you might be chasing something that doesn't add any value.

 

[PeteJ]

Yes, sort of.

I do not currently have plans to act on the data, just collecting it is sufficient. I only have 1 camera doing LPR currently and it's point out to a very quiet street. I'd be surprised if I would have more than 30 reads per day. However, where I plan to install this is going to be toward a main street where I'd be in the 1000 reads per day range, in a single direction (that's just a guess, but it is a busy street).

I don't think BI and CPAI running on the same server can keep up, so that's why I am interesting in splitting it up. But, I am not really clear on how BI pipelines the LPR to CPAI--does it just queue it up in a FIFO queue and will process each one as it waits for CPAI to return the results, or does it overflow or start blocking if the requests are not returned quick enough?

From what I have seen, after about 8-10 secs, it looks like BI stops checking? The *.dat data is all grayed out.

And yes, I do like to tinker.

 

[PeteJ]

Okay, looks like the problem was the official Ubuntu 22.04.5 LTS image from Orange Pi's site. Under platform, it comes up as Linux/Arm64--which is accurate, but looks like CPAI actually looks for OrangePi as the platform:



I was able to get this to work by installing Joshua Reik's Ubuntu image: GitHub - Joshua-Riek/ubuntu-rockchip: Ubuntu for Rockchip RK35XX Devices

Once installed, the platform was detected properly and the option to install the RNKK modules was available.

Thanks very much to @MikeLud1 for pointing me in the right direction. Hopefully this info would be useful to someone else in the future.

 

[PeteJ]

In case anyone is considering running CPAI on the RK3588, this is what I am seeing in terms of perf on the RK3588



Below is on CPU.



CPAI avg is slightly under 40ms running average.



I only have a 15w power supply on it, so it's probably running at half of that right now.

 

[AlwaysSomething]

I'm interested in giving this a try but not really familiar with using Docker (I understand the concept just never used it). Did you try without Docker? Reason you used Docker?

Your numbers seem really good. Any negatives or complaints?

 

[PeteJ]

I'm interested in giving this a try but not really familiar with using Docker (I understand the concept just never used it). Did you try without Docker? Reason you used Docker?

Your numbers seem really good. Any negatives or complaints?

No complaints, it's running very well.

Don't let Docker scare you, it's not that hard to learn, and you can install something like Portainer to manage it via a web interface.

 

[AlwaysSomething]

Which of Joshua's image did you use? I see he has 24.04 as the latest but I thought when I read his notes it sounded like there were still errors with that version. He lists 22.04 as deprecated.

Also curious, did you install the desktop version or just server?

I appreciate the help. SBC is all new to me and I'm still a Linux newb. Just realized I bought the Ultra and not Max. Only difference is one HDMI port is IN and not OUT so hopefully I can still use the Max image.

 

[PeteJ]

Which of Joshua's image did you use? I see he has 24.04 as the latest but I thought when I read his notes it sounded like there were still errors with that version. He lists 22.04 as deprecated.

Also curious, did you install the desktop version or just server?

I appreciate the help. SBC is all new to me and I'm still a Linux newb. Just realized I bought the Ultra and not Max. Only difference is one HDMI port is IN and not OUT so hopefully I can still use the Max image.

I'm running 24.04, yeah, it should work fine with the Ultra. I'm running the desktop version, I think the only difference is that the server doesnt come with some apps and X.

 

[AlwaysSomething]

Thanks for info.

So I ended up trying both desktop versions and neither worked for me. They would load and I got the background images but that was it. I noticed if I moved my mouse all the way up, down, or right, I hit the edge of the screen. If I moved the mouse all the way to the left, it was running off the screen (probably where the icons/login was). I'm guessing there is a config different between the MAX and the ULTRA. Bummer

I just ordered a MAX with 16 GB. Pretty crazy this thing idles at 3 watts!!

EDIT: I did try right clicking to see if I could get the display properties but nothing happened.

 

[PeteJ]

Thanks for info.

So I ended up trying both desktop versions and neither worked for me. They would load and I got the background images but that was it. I noticed if I moved my mouse all the way up, down, or right, I hit the edge of the screen. If I moved the mouse all the way to the left, it was running off the screen (probably where the icons/login was). I'm guessing there is a config different between the MAX and the ULTRA. Bummer

I just ordered a MAX with 16 GB. Pretty crazy this thing idles at 3 watts!!

EDIT: I did try right clicking to see if I could get the display properties but nothing happened.

Yeah. that's a little weird, I didn't have any issues installing the image on the MAX. Maybe try a different mouse or monitor to see if it makes a difference? There is a few youtube videos that go over the install, depending if you are booting from microSD card or nvme.

Update:

I found my notes:

To install it on a nvme drive:

sudo fdisk -l
sudo ubuntu-rockchip-install /dev/nvme0n1
sudo u-boot-install-mtd

The Orange Pis seem to be sensitive to power supply issues, when I got my first one, it would crash from time to time. Nothing in the logs, just hard lockup.

I install a few tools to help debug:

sudo apt install htop
sudo apt install sensors
sudo apt install stress

watch sudo stress --cpu 4 --io 3 --vm 2 --vm-bytes 256M --timeout 600s

This caused it to crash. Swapped the USB-C PSU to a better one, and the problem went away

I increased the timeout to 2h once it was stable and used that as a sort of a burn in.

sensors will give you the board temps to make sure it doesn't get too warm.

Hope this helps.

 

[AlwaysSomething]

So I got my Orange Pi 5 Max today with their official power supply (to hopefully avoid PS issues).

Installed 24.04 Desktop version from Joshua's git account. Tried installing without Docker but there is a new version 2.9.7 for ARM and it wasn't working for me. Uninstalled that.

Then I took the plunge and went the Docker route with Portainer as you mentioned. You are correct, it was pretty easy. Thank you recommending it. Just wish the font color for Portainer was different because I am color blind and it was hard to read (too close to the white background for me).

I was getting excited but then...

CPAI thinks I'm on 22.04 and also did NOT see it as Orange Pi.



Glad I installed the desktop version as it was nice and easy to compare. LOL

When you installed the OS did you have internet access setup and let Ubuntu run updates before installing everything else? Originally I did and it would keep crashing (locking up) when I opened Chromium. I started over with installing Ubuntu and removed the Ethernet cable so no internet access until the installation was complete. Doing that allowed me to open Chromium without it crashing. I did NOT do any updates at that point. I then installed the Apps (Docker, Portainer, and CPAI).

Very strange.

I need to take a break and then I will try from scratch again.

One last question, besides configuring the port did you have to configure anything else? Any devices like the NPU?

I really appreciate your help. Thank you.

FYI - Joshua also made a comment that he may no longer be supporting these images so If that is the case we need to make sure we back these up very well or never update anything (include CPAI) again.

 

[MikeLud1]

This is for the ALPR
You can try deleting lines 25 and 26 in ALPR_adapter.py and this should make it work

 

[MikeLud1]

For the Object Detection you can try deleting lines 34 and 35 in objectdetection_fd_rknn_adapter.py and this should make it work

 

[AlwaysSomething]

For the Object Detection you can try deleting lines 34 and 35 in objectdetection_fd_rknn_adapter.py and this should make it work

It's not even letting me install the RKNN modules (neither LPR or Object Detection).



Unless you mean manually pull the modules from GitHub and then edit the files as you recommended? Then finally run the install.sh?

 

[AlwaysSomething]

Actually, I'm looking at the docker hub site

https://hub.docker.com/r/codeproject/ai-server/tags?page=&page_size=&ordering=&name=2.9.5

and wondering if I pulled the wrong image file.

I pulled:

docker pull codeproject/ai-server:rpi64-2.9.5

I'm going to pull:

docker pull codeproject/ai-server:arm64-2.9.5

and try it.

I just saw "linux/arm64" in the first option and figured that was it. Scrolling to the bottom I see the second option.

FYI - Viewing the docker hub in Chromium is horrible (unless there is dark mode in Chromium without logging in). I went to my mac to browse (which I'm using Brave and dark mode without logging in).

 

[PeteJ]

I had this issue too when I used the Linux ISO from Orange PI, it would list the platform as Linux-Arm64. When it is listed as Linux-Arm64, CPAI will not let you install the RNKK modules.

I was able to get it working when I switched to Joshua's ISO, and in CPAI, it will show the platform as Orange PI:

 

[PeteJ]

I'm using this image: ubuntu-24.04-preinstalled-desktop-arm64-orangepi-5-max.img.xz.sha256


sha256sum ubuntu-24.04-preinstalled-desktop-arm64-orangepi-5-max.img.xz.sha256
319d52b60398dfbc046b9761876e705f9545d0889ece4f0074c6290568111ca4

 

[AlwaysSomething]

I thought I used Joshua's image but I tried so many things that my eyes may have mixed up the files when burning them (could also be lack of sleep LOL).

I'm going to start from scratch. Did you see my lost post about the Docker image? Which Docker image did you use?

Thank you both Pete and Mike. I'm probably making user errors but appreciate your help.

 

[AlwaysSomething]

Oh and are you using any cases/fans for your board? I bought a case with heatsinks and a fan. This tiny fan is the loudest thing in my office!! Just curious if you're not using a fan if getting any overheating/throttling.