Skip to content

Commit 193b6be

Browse files
LJ-HaoLJ-Hao
committed
Run_Clip_Application_with_Hailo_NPU
1 parent fb356c4 commit 193b6be

1 file changed

Lines changed: 115 additions & 0 deletions

File tree

‎articles/Chapter_3-Computer_Vision_Projects_and_Practical_Applications/Run_Clip_Application_with_Hailo_NPU.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
---
2+
sidebar_position: 3
3+
---
4+
5+
# Deploying Custom AI Models Across Applications with Hailo NPU
6+
7+
## Overview
8+
9+
[CLIP](https://github.com/openai/CLIP)(Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. We found CLIP matches the performance of the original ResNet50 on ImageNet “zero-shot” without using any of the original 1.28M labeled examples, overcoming several major challenges in computer vision.
10+
11+
This wiki will teach you how to deploy the clip application on a [Raspberry Pi5](https://www.seeedstudio.com/Raspberry-Pi-5-8GB-p-5810.html) or [Recomputer r1000](https://www.seeedstudio.com/reComputer-R1000-Series-Optional-Accessories.html), clip will inference on [AI kit](https://www.seeedstudio.com/Raspberry-Pi-AI-Kit-p-5900.html).
12+
13+
14+
## Install Hailo Software & Verify Installation
15+
16+
### update the system:
17+
18+
```
19+
sudo apt update
20+
sudo apt full-upgrade
21+
```
22+
23+
:::note
24+
Sometimes you may encounter the following issues during updates.
25+
```
26+
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
27+
Get:2 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
28+
Get:3 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
29+
Get:4 http://archive.raspberrypi.com/debian bookworm InRelease [39.0 kB]
30+
Reading package lists... Done
31+
E: Release file for http://deb.debian.org/debian/dists/bookworm/InRelease is not valid yet (invalid for another 58d 8h 26min 35s). Updates for this repository will not be applied.
32+
E: Release file for http://deb.debian.org/debian-security/dists/bookworm-security/InRelease is not valid yet (invalid for another 84d 18h 23min 59s). Updates for this repository will not be applied.
33+
E: Release file for http://archive.raspberrypi.com/debian/dists/bookworm/InRelease is not valid yet (invalid for another 84d 13h 13min 5s). Updates for this repository will not be applied.
34+
E: Release file for http://deb.debian.org/debian/dists/bookworm-updates/InRelease is not valid yet (invalid for another 85d 0h 52min 29s). Updates for this repository will not be applied.
35+
```
36+
This is because the time on the Raspberry Pi is set incorrectly, and you need to manually set the time on the Raspberry Pi with command below:
37+
```
38+
# This command only you can connect google.com
39+
sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"
40+
```
41+
After set your raspberry time, you can update your raspberry.
42+
:::
43+
44+
### Set pcie to gen2/gen3(gen3 is faster than gen2):
45+
46+
Add following text to ```/boot/firmware/config.txt```
47+
48+
```
49+
#Enable the PCIe external connector
50+
51+
dtparam=pciex1
52+
53+
#Force Gen 3.0 speeds
54+
55+
dtparam=pciex1_gen=3
56+
57+
```
58+
:::note
59+
If you want to use gen2, please comment dtparam=pciex1_gen=3
60+
:::
61+
62+
### Install hailo-all and reboot:
63+
64+
Open terminal on the Raspberry Pi5, and input command as follows to install Hailo software.
65+
66+
```
67+
sudo apt install hailo-all
68+
sudo apt-get -y install libblas-dev nlohmann-json3-dev
69+
sudo reboot
70+
```
71+
### Check Software and Hardware:
72+
73+
Open terminal on the Raspberry Pi5, and input command as follows to check if hailo-all have been installed.
74+
75+
```
76+
hailortcli fw-control identify
77+
```
78+
79+
The right result show as bellow:
80+
<p style={{textAlign: 'center'}}><img src="https://files.seeedstudio.com/wiki/reComputer-R1000/YOLOV8/check_software.png" alt="pir" width={1000} height="auto"/></p>
81+
82+
Open terminal on the Raspberry Pi5, and input command as follows to check if hailo-8L have been connected.
83+
84+
```
85+
lspci | grep Hailo
86+
```
87+
88+
The right result show as bellow:
89+
<p style={{textAlign: 'center'}}><img src="https://files.seeedstudio.com/wiki/reComputer-R1000/YOLOV8/check_hardware.png" alt="pir" width={1000} height="auto"/></p>
90+
91+
## Run Project
92+
93+
### Install Project
94+
95+
```
96+
git clone https://github.com/hailo-ai/hailo-CLIP.git
97+
cd hailo-CLIP
98+
python3 -m pip install -v -e .
99+
```
100+
101+
### Run the project
102+
Input the command below you will see a clip demo:
103+
```
104+
clip_app --input demo
105+
```
106+
And if you want to use your camera, you should input command below after you make sure raspberry connect your own camera:
107+
```
108+
clip_app --input /dev/video0
109+
```
110+
111+
## Result
112+
113+
In the video shown below, you can see that when I input "banana," the CLIP model recognizes a banana, and when I input "apple," the model recognizes an apple. You only need to input different words, and the CLIP model will recognize different objects.
114+
115+
<iframe width="800" height="400" src="https://www.youtube.com/embed/JMHtqSmAGCA" title="CLIP Zero Shot Classification on Raspberry Pi 5 with Hailo AI Accelerator" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

0 commit comments

Comments
 (0)