Articles on Lluvia

Color mapping for data visualization

Fri, 07 Jul 2023 00:00:00 +0000

Jupyter notebook:

A Jupyter notebook with the code in this article is available in Google Colab. Check it out!

Introduction

Data visualization is an important tool for presenting the results of new algorithms. For 2D data in particular, it is possible to visualize the data as colored images so that our brains can interpret the data in a visual way. This article presents new color mapping nodes available in Lluvia to transform 2D data into color images. These nodes use several color maps available in the Matplotlib project to accelerate data to color conversion using the GPU.

Raspberry Pi 4 build

Mon, 28 Nov 2022 00:00:00 +0000

Introduction

The Raspberry Pi 4 project announced back in November 2020 that the Vulkan 1.0 conformance tests successfully passed for its GPU driver. More recently in August 2022, Vulkan 1.2 conformance testing has been completed.

The conformance tests are a large set of tests run against a driver implementation to see if it conforms with the Vulkan specification. This is essential to maintain the Vulkan API portable across platforms and GPU vendors.

Android integration using mediapipe

Tue, 08 Nov 2022 00:00:00 +0000

Introduction

The previous post on mediapipe integration explored how to integrate Lluvia with Mediapipe to create complex Computer Vision pipelines leveraging mediapipe’s integration with other frameworks such as OpenCV and Tensorflow. This post expands this integration to run Lluvia on Android systems.

The video below shows the Optical Flow Filter algorithm running on a Samsung Galaxy S22+ phone.

Graph for mobile applications

The figure below illustrates the pipeline used on Android for running the LluviaCalculator. First, a FlowLimiter calculator receives the images from the input_stream. This calculator controls the rate at which packets are sent downstream; it receives an additional input from the last calculator, ImageFrameToGpuBuffer, that indicates processing has been completed and that a new packet can be received.

Mediapipe integration

Sat, 08 Oct 2022 00:00:00 +0000

Introduction

Mediapipe is a cross-platform framework to create complex Computer Vision pipelines both for offline and real-time applications. It leverages popular frameworks such as OpenCV and Tensorflow to process audio, video, and run deep learning models. By integrating Lluvia into mediapipe, it is possible to speed up some of those computations by creating a GPU compute pipeline.

Difference 1: project scope

Mediapipe is a more general framework than Lluvia. Mediapipe, at its core, is a compute graph scheduler, where each node can contain any arbitrary processing logic. The integration of third-party frameworks (e.g. OpenCV, Tensorflow, Lluvia) gives the framework its power for developing complex Computer Vision pipelines.

Camera undistort

Sun, 18 Sep 2022 00:00:00 +0000

Jupyter notebook:

A Jupyter notebook with the code in this article is available in Google Colab. Check it out!

Background

Camera undistort is the process by which distortions generated by the optics used in the camera during the capture process are corrected in software. The process requires a mathematical model of the distortion, and a calibration procedure to estimate the parameters of such model given actual images.

An overview of the camera modeling is pressented in the Computer Vision book of Szeliski and the Multiple View Geometry book of Hartley and Zisserman, as well as the articles of Zhang, Wei and Ma.

Implementing the Horn and Schunck optical flow algorithm

Sun, 07 Aug 2022 00:00:00 +0000

Jupyter notebook:

A Jupyter notebook with the code in this article is available in Google Colab. Check it out!

Background

The Horn and Schunck variational method for computing optical flow is one of the seminal works in the field. It introduces the idea of using a global smoothness constrain on the estimated optical flow. This constrain helps the numerical solution to find a good flow estimate even in image regions with poor texture.

Working with floating point precision

Sat, 12 Feb 2022 00:00:00 +0000

Jupyter notebook:

A Jupyter notebook with the code in this article is available in Google Colab. Check it out!

GPU devices support several floating point number precisions, where precision refers to the number of bits used for representing a given number. Typical representations are:

FP16: or half precision. Numbers are represented in 16 bits.
FP32: or single precision. It uses 32 bits for representing a number.
FP64: or doble precision. 64 bits are used for represeting a number.

FP64 is used when numerical precision is required, while FP16 is suitable for fast, less exact calculations, and FP32 sits in the middle. The IEEE 754 standard defines the specification of floating point numbers used in modern computers. It defines the rules for interpreting the bit fields that form a number, as well as the arithmetic rules to process them.