Easy-to-use realtime CV/AI pipelines for Nvidia Jetson Platform. This project:

• Builds a typical multi-camera pipeline, i.e. N×(capture)->preprocess->batch->DNN-> <<your application logic here>> ->encode->file I/O + display. Uses gstreamer and deepstream under-the-hood.
• Gives programatic acces to configure the pipeline in python via jetmulticam package.
• Utilizes Nvidia HW accleration for minimal CPU usage. For example, you can perform object detection in real-time on 6 camera streams using as little as 16.5% CPU. See benchmarks below for details.

You can easily build your custom logic in python by accessing image data (via np.array), as well object detection results. See examples of person following below:

We have 3 intependent cameras with ~270° field of view. Red Boxes correspond to DashCamNet detections, green ones to PeopleNet. The PeopleNet detections are used to perform person following logic.

Robot is operated in manual mode.

Robot is operated in manual mode.

All demos are performed in real-time onboard Nvidia Jetson Xavier NX.

Install:

git clone https://github.com/NVIDIA-AI-IOT/jetson-multicamera-pipelines.git
cd jetson-multicamera-pipelines
bash scripts/install_dependencies.sh
pip3 install Cython
pip3 install .

Run example with your cameras:

source scripts/env_vars.sh 
cd examples
python3 example.py

import time
from jetmulticam import CameraPipelineDNN
from jetmulticam.models import PeopleNet, DashCamNet

if __name__ == "__main__":

    pipeline = CameraPipelineDNN(
        cameras=[2, 5, 8],
        models=[
            PeopleNet.DLA1,
            DashCamNet.DLA0,
            # PeopleNet.GPU
        ],
        save_video=True,
        save_video_folder="/home/nx/logs/videos",
        display=True,
    )

    while pipeline.running():
        arr = pipeline.images[0] # np.array with shape (1080, 1920, 3), i.e. (1080p RGB image)
        dets = pipeline.detections[0] # Detections from the DNNs
        time.sleep(1/30)

Notes:

• All figures are in 15W 6 core mode. To reproduce do: sudo nvpmodel -m 2; sudo jetson_clocks;
• Test platform: Jetson Xavier NX and XNX Box running JetPack v4.5.1
• The residual GPU usage in DLA-accelerated nets is caused by Sigmoid activations being computed with CUDA backend. Remaining layers are computed on DLA.
• CPU usage will vary depending on factors such as camera resolution, framerate, available video formats and driver implementation.

pipeline = CameraPipelineDNN(
    cam_ids = [0, 1, 2]
    models=[
        models.PeopleNet.DLA0,
        models.PeopleNet.DLA1,
        models.PeopleNet.GPU,
        models.DashCamNet.DLA0,
        models.DashCamNet.DLA1,
        models.DashCamNet.GPU
        ]
    # ...
)