Hi everyone,
I’m working on a project using the rpicam-apps (1.7.0) suite and have successfully implemented a custom post-processing stage (TemplateMatchStage) that performs real-time template matching on the main camera stream using OpenCV. The typical use case is to match a template image (from a second camera or stream) against the main video stream and visualize the result as an overlay.
Because the telephoto camera captures only a small, zoomed-in region of the wide-angle image, the template image must be scaled down before matching.
For example, with a 70 mm setting on a 70–300 mm telephoto lens and a wide-angle lens of 6–8 mm, the scaling factor is typically around 0.1.
This means the template from the telephoto camera is reduced to about 10% of its original size to match the scale of the corresponding region in the wide-angle image.
In the code, this is implemented as follows:This scaling is crucial for successful template matching, as the size difference between the telephoto and wide-angle images can be significant.
My specific scenario:
I have two Raspberry Pi cameras attached: one with a wide-angle lens and one with a telephoto (zoom) lens. My goal is to find the current view of the telephoto camera (the "zoomed-in" image) within the wide-angle camera’s image in real time. The template for matching is taken from the telephoto camera, and the search image is the wide-angle camera stream.
Planned extension:
This functionality is intended as the basis for an automated pan-tilt-zoom (PTZ) camera control system for camera object tracking. The idea is to use the result of the template matching to automatically steer the telephoto camera (via pan, tilt, and zoom) so that it follows a selected object or region of interest detected in the wide-angle view.
In the final system, object detection will be performed using Hailo’s YOLO inference on the wide-angle stream. Detected objects will be tracked, and the telephoto camera will be automatically moved (with maximum possible zoom) to keep the selected object centered and in focus.
Currently, I’m using GStreamer to pipe the second camera’s video stream (as RTSP/H264) into the post-processing stage, where it is decoded and used as the template. While this works, it is quite CPU-intensive, especially on a Raspberry Pi, due to the overhead of decoding and handling the stream with GStreamer.
Here is the relevant code I’m currently using for the GStreamer approach:
// In the TemplateMatchStage constructor:
// In the Process() method:
My question:
Is there a more CPU-efficient way to provide a second video stream (from a second Raspberry Pi camera) as a template source to a post-processing stage in rpicam-apps, without using GStreamer?
Ideally, I’d like to access both camera streams directly in the post-processing stage, or use a lightweight mechanism to share frames between two running camera pipelines, so that template matching can be performed in real time with minimal CPU usage.
Details:
Both cameras (Raspberry Pi HQ-Cameras) are connected to the same Raspberry Pi 5 /8GB (but I’m open to solutions for two Pis as well).
The main camera stream is processed as usual in rpicam-vid.
The template should be updated regularly from the second camera (ideally at a lower frame rate).
The goal is to avoid the CPU overhead of GStreamer and H264 decoding for the template stream.
I’m open to using shared memory, direct buffer access, or any other efficient method supported by the rpicam-apps framework.
What I’ve tried:
Using GStreamer with udpsrc and appsink in OpenCV’s VideoCapture for the template stream (works, but high CPU usage, about 60-70%).
Reading from a file or static image (not suitable, as the template needs to be live).
// gstreamer start parameter//main rpicam-vid with template_matching start parameters
Questions for the developers:
Is there a recommended way to access two camera streams simultaneously within a post-processing stage in rpicam-apps?
Can rpicam-apps natively support dual camera input for such use cases, or is there an efficient workaround?
Are there examples or best practices for sharing frames between two camera pipelines or post-processing stages?
Would using shared memory or another IPC mechanism be feasible within the current framework?
Any advice, code examples, or pointers to documentation would be greatly appreciated!
Thank you very much in advance for your help!
Best regards,
Tom
I’m working on a project using the rpicam-apps (1.7.0) suite and have successfully implemented a custom post-processing stage (TemplateMatchStage) that performs real-time template matching on the main camera stream using OpenCV. The typical use case is to match a template image (from a second camera or stream) against the main video stream and visualize the result as an overlay.
For example, with a 70 mm setting on a 70–300 mm telephoto lens and a wide-angle lens of 6–8 mm, the scaling factor is typically around 0.1.
This means the template from the telephoto camera is reduced to about 10% of its original size to match the scale of the corresponding region in the wide-angle image.
In the code, this is implemented as follows:
Code:
cv::resize(img1, scaled_img1, cv::Size(), fixed_scale_, fixed_scale_, cv::INTER_AREA);// Example: fixed_scale_ = 0.1 for a 70mm telephoto vs. wide-angle setupMy specific scenario:
I have two Raspberry Pi cameras attached: one with a wide-angle lens and one with a telephoto (zoom) lens. My goal is to find the current view of the telephoto camera (the "zoomed-in" image) within the wide-angle camera’s image in real time. The template for matching is taken from the telephoto camera, and the search image is the wide-angle camera stream.
Planned extension:
This functionality is intended as the basis for an automated pan-tilt-zoom (PTZ) camera control system for camera object tracking. The idea is to use the result of the template matching to automatically steer the telephoto camera (via pan, tilt, and zoom) so that it follows a selected object or region of interest detected in the wide-angle view.
In the final system, object detection will be performed using Hailo’s YOLO inference on the wide-angle stream. Detected objects will be tracked, and the telephoto camera will be automatically moved (with maximum possible zoom) to keep the selected object centered and in focus.
Currently, I’m using GStreamer to pipe the second camera’s video stream (as RTSP/H264) into the post-processing stage, where it is decoded and used as the template. While this works, it is quite CPU-intensive, especially on a Raspberry Pi, due to the overhead of decoding and handling the stream with GStreamer.
Here is the relevant code I’m currently using for the GStreamer approach:
// In the TemplateMatchStage constructor:
Code:
TemplateMatchStage(RPiCamApp *app) : PostProcessingStage(app), active_methods_{true, true, true}, cap_("udpsrc port=8554 caps=\"application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96\" " "! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! appsink", cv::CAP_GSTREAMER){}Code:
cv::Mat cam1_frame, img1;if (!cap_.isOpened() || !cap_.read(cam1_frame) || cam1_frame.empty()) { std::cerr << "RTSP-Stream not available or no frame received!" << std::endl; return false;}if (cam1_frame.channels() == 3) cvtColor(cam1_frame, img1, cv::COLOR_BGR2GRAY);else img1 = cam1_frame;My question:
Is there a more CPU-efficient way to provide a second video stream (from a second Raspberry Pi camera) as a template source to a post-processing stage in rpicam-apps, without using GStreamer?
Ideally, I’d like to access both camera streams directly in the post-processing stage, or use a lightweight mechanism to share frames between two running camera pipelines, so that template matching can be performed in real time with minimal CPU usage.
Details:
Both cameras (Raspberry Pi HQ-Cameras) are connected to the same Raspberry Pi 5 /8GB (but I’m open to solutions for two Pis as well).
The main camera stream is processed as usual in rpicam-vid.
The template should be updated regularly from the second camera (ideally at a lower frame rate).
The goal is to avoid the CPU overhead of GStreamer and H264 decoding for the template stream.
I’m open to using shared memory, direct buffer access, or any other efficient method supported by the rpicam-apps framework.
What I’ve tried:
Using GStreamer with udpsrc and appsink in OpenCV’s VideoCapture for the template stream (works, but high CPU usage, about 60-70%).
Reading from a file or static image (not suitable, as the template needs to be live).
// gstreamer start parameter
Code:
rpicam-vid --camera 1 -t 0 --qt-preview --info-text %focus --width 640 --height 480 --lores-height 300 --lores-width 400 --lores-par 1 --codec libav --libav-format mpegts --inline -o - | gst-launch-1.0 fdsrc fd=0 ! tsdemux ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=192.168.0.30 port=8554Code:
rpicam-vid --info-text FoM:%focus,-FPS:%fps,-Frame:%frame --qt-preview --height 990 --width 1330 --camera 0 -t 0 --preview 500,50,600,400 --lores-width 320 --lores-height 240 --lores-par 1 --post-process-file assets/template_match.json --awb cloudy --ev +1.5Questions for the developers:
Is there a recommended way to access two camera streams simultaneously within a post-processing stage in rpicam-apps?
Can rpicam-apps natively support dual camera input for such use cases, or is there an efficient workaround?
Are there examples or best practices for sharing frames between two camera pipelines or post-processing stages?
Would using shared memory or another IPC mechanism be feasible within the current framework?
Any advice, code examples, or pointers to documentation would be greatly appreciated!
Thank you very much in advance for your help!
Best regards,
Tom
Statistics: Posted by kletternaut — Wed May 28, 2025 8:25 pm — Replies 0 — Views 52