Documentation

• 1: About
• 2: Getting started
• 2.1: Installation
• 2.1.1: Ubuntu
• 2.1.2: Linux using docker
• 2.1.3: Raspberry Pi 4
• 2.1.4: Windows 10
• 2.2: Mediapipe integration
• 3: Reference
• 3.1: Session
• 3.2: Memory
• 3.3: Objects
• 3.3.1: Buffer
• 3.3.2: Image
• 3.3.3: ImageView
• 3.4: Node system
• 3.4.1: Compute nodes
• 3.4.2: Container nodes
• 3.4.3: Running
• 4: Examples
• 4.1: Real-time optical flow

1 - About

Designing and developing real-time Computer Vision algorithms is a difficult task. On one hand, one needs to formulate the algorithm mathematically in such a way that enables fast calculations on modern compute hardware. On the other hand, such mathematical formulation has to be transformed into machine code, optimized and tested to satisfy system requirements such as hardware usage and frame rate. Accomplishing both tasks successfully is the job of research and engineering teams with highly specialized knowledge.

Lluvia has been designed around the idea of reducing the effort for designing and implementing Computer Vision algorithms for real-time applications. The engine is built on top the Vulkan graphics and compute API. By using Vulkan, it is possible to run the algorithms on any modern GPU that supports the API. The core libraries are coded in C++ and can be compiled for several operating systems, currently Linux and Android. Wrappers for high-level languages such as Python are maintained as well.

Workflow

Lluvia uses a compute graph to organize and schedule computations on the GPU. The development workflow circles around coding and debugging nodes in such a graph until the whole algorithm is built:

1. The node’s inputs, outputs and parameters are described in a Lua script . This description will later be used to instantiate nodes in the graph.

2. The node’s computation in the GPU is coded as a compute shader in Open GL Shading Language (GLSL). Shaders are compiled into SPIR-V intemediate representation for later load into the GPU.

3. The node’s description and compute shader are loaded into Lluvia’s runtime. After this, nodes can be instantiated to build the compute graph and be dispatched to the GPU.

From a user perspective, one needs to only care about describing nodes (inputs, outputs, compute shader, etc.) and connecting nodes to form a graph. Lluvia takes care of the low-level details of dispatching the graph for execution onto the GPU. This workflow allows porting the compute graph from one platform to another with ease.

Check the Getting Started guides for examples on how to describe computations in Lluvia.

Alternatives

There are many other alternatives to use for coding and deploying Computer Vision algorithms. The list below is by no means an exhaustive review. Please contact me if you want other frameworks to be included.

• OpenCV The go-to alternative for fast prototyping and deployment of CV algorithms. OpenCV is a mature project that can be used on many platforms (Linux, OSX, Windows, Android). It contains a bast library of algorithms, some of them with GPU implementations.

• Halide is a programming language for coding high-performance image processing algorithms. The language is embedded in C++ and can dispatch execution of the algorithms to CPUs and GPUs depending on the available hardware.

• Mediapipe A framework for developing complex Computer Vision pipelines combining several frameworks such as OpenCV, TensorFlow, TFLite.

2 - Getting started

2.1 - Installation

2.1.1 - Ubuntu

Dependencies

• Install the following packages in your system if they not available yet:

  1 2 3

  sudo apt install \ build-essential \ clang

• Bazel:

  1 2 3 4

  curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add - echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list sudo apt update sudo apt install bazel

• LunarG Vulkan SDK:

  1 2 3 4

  wget -qO - https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo apt-key add - sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-1.3.211-focal.list https://packages.lunarg.com/vulkan/1.3.211/lunarg-vulkan-1.3.211-focal.list sudo apt update sudo apt install vulkan-sdk

  Verify that the SDK was successfully installed by running:

  1	vulkaninfo

• Python3 dependencies

  1	sudo apt install python3-pip

Build C++ Libraries

Clone and compile Lluvia’s C++ libraries:

1
2
3
4
5
6
7

git clone https://github.com/jadarve/lluvia.git
cd lluvia

# install requirements listed in lluvia's root folder
sudo python3 -m pip install -r requirements.txt

bazel build //lluvia/cpp/...

Run the tests to verify that your compilation runs properly:

1

bazel test //lluvia/cpp/...

Python3 package

To build the Python3 package, execute the commands below from the repository’s top-level directory. You can create a virtual environment to isolate the installation:

1
2

bazel build //lluvia/python:lluvia_wheel
pip3 install bazel-bin/lluvia/python/lluvia-0.0.1-py3-none-any.whl

Open a Python3 interpreter and import lluvia:

1
2
3

import lluvia as ll

session = ll.createSession()

If the import completes successfully, lluvia is ready to use.

2.1.2 - Linux using docker

Install docker following the official documentation and the post installation guide.

Clone lluvia and build the container

1
2
3
4

git clone https://github.com/jadarve/lluvia.git
cd lluvia

docker build ci/ --tag="lluvia:local"

Run the container, mounting lluvia’s repository at /lluvia:

1
2
3

docker run \
    --mount type=bind,source="$(pwd)",target=/lluvia \
    -it --rm "lluvia:local" /bin/bash

Inside the container, build and test all of lluvia package:

1
2
3

cd lluvia
bazel build //...
bazel test --test_output=errors //...

2.1.3 - Raspberry Pi 4

Raspberry Pi config

OS Installation

On the desktop machine, download the Raspberry Pi Imager and install a fresh version of the operating system in a micro SD card.

1

sudo apt install rpi-imager

Go to extra settings and enable SSH access, configure the WiFi, if needed. Click in write and wait for the process to complete.

Insert the micro SD card in the RPi and next, update and upgrade the operating system:

1
2
3
4

sudo apt update
sudo apt upgrade

sudo reboot

Expand storage

1

sudo raspi-config

1

sudo reboot

Increase swap memory

Some of the instructions bellow require more memory than the one available in the RPi4. For this, it is recommended to increase the SWAP memory available to the system so that compilation does not fail unexpectedly. Open the /sbin/dphys-swapfile and /etc/dphys-swapfile, and edit the line CONF_MAXSWAP=<some_value> to 4096. After it, reboot the RPi.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# unmount the swap
sudo dphys-swapfile swapoff

# edit the swap configuration to:
# CONF_SWAPSIZE=2048 
# CONF_MAXSWAP=4096
sudo nano /sbin/dphys-swapfile
sudo nano /etc/dphys-swapfile

# configure with new values
sudo dphys-swapfile setup

# start swap
sudo dphys-swapfile swapon

Camera module

If you have a camera module available, you can enable it by following the official guide

1

sudo raspi-config

1

sudo reboot

Vulkan SDK

Dependencies

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

sudo apt install -y \
  bison \
  cmake \
  g++ \
  gcc \
  git \
  libglm-dev \
  liblz4-dev \
  libmirclient-dev \
  libpciaccess0 \
  libpng-dev \
  libwayland-dev \
  libx11-dev \
  libx11-xcb-dev \
  libxcb-dri3-0 \
  libxcb-dri3-dev \
  libxcb-ewmh-dev \
  libxcb-keysyms1-dev \
  libxcb-present0 \
  libxcb-randr0-dev \
  libxml2-dev \
  libxrandr-dev \
  libzstd-dev \
  mesa-vulkan-drivers \
  ninja-build \
  ocaml-core \
  pkg-config \
  python \
  python3 \
  python3-distutils \
  qt5-qmake \
  qtbase5-dev \
  qtbase5-dev-tools \
  qtcreator \
  vulkan-tools
  wayland-protocols

Build and install

Download and build the Vulkan SDK

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

mkdir -p ~/local/vulkan
cd ~/local/vulkan

wget -O vulkansdk.tar.gz \
  https://sdk.lunarg.com/sdk/download/1.3.231.2/linux/vulkansdk-linux-x86_64-1.3.231.2.tar.gz

tar -xvf vulkansdk.tar.gz

# remove archive to save space
rm vulkansdk.tar.gz

cd 1.3.231.2

# remove prebuilt x86_64 libraries are not needed in the RPi
rm -rf x86_64

# build the SDK

./vulkansdk -j 4 glslang
./vulkansdk -j 4 vulkan-headers
./vulkansdk -j 4 vulkan-loader
./vulkansdk -j 2 vulkan-validationlayers  # this one consumes all the RAM and uses SWAP memory
./vulkansdk -j 4 vulkan-tools
./vulkansdk -j 4 vulkantools
./vulkansdk -j 4 shaderc
./vulkansdk -j 4 spirv-headers
./vulkansdk -j 4 spirv-tools
./vulkansdk -j 4 spirv-cross
./vulkansdk -j 4 gfxreconstruct
./vulkansdk -j 4 spirv-reflect
./vulkansdk -j 4 vulkan-extensionlayer
./vulkansdk -j 4 vulkan-profiles
./vulkansdk -j 4 DirectXShaderCompiler
./vulkansdk -j 4 volk
./vulkansdk -j 4 VulkanMemoryAllocator
./vulkansdk -j 4 VulkanCapsViewer

Configure the VULKAN_SDK environment variable permanently in the system:

1
2
3
4
5
6

# edit .bashrc or .zshrc accordingly
nano .bashrc

# include the following environment variables at the end of the file
export VULKAN_SDK=~/local/vulkan/1.3.231.2/aarch64                                          
export PATH=$VULKAN_SDK/bin:$PATH

Reload the profile:

1
2

cd
source .bashrc

Install the SDK in the system:

1
2
3
4
5
6
7

sudo cp -r $VULKAN_SDK/include/vulkan/ /usr/local/include/
sudo cp -P $VULKAN_SDK/lib/libvulkan.so* /usr/local/lib/
sudo cp $VULKAN_SDK/lib/libVkLayer_*.so /usr/local/lib/
sudo mkdir -p /usr/local/share/vulkan/explicit_layer.d
sudo cp $VULKAN_SDK/etc/vulkan/explicit_layer.d/VkLayer_*.json /usr/local/share/vulkan/explicit_layer.d

sudo ldconfig

OpenCV

Dependencies

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

sudo apt install -y \
    build-essential \
    cmake \
    gfortran \
    gstreamer1.0-gl \
    gstreamer1.0-gtk3 \
    libatlas-base-dev \
    libavcodec-dev \
    libavformat-dev \
    libblas-dev \
    libcanberra-gtk* \
    libdc1394-22-dev \
    libgif-dev
    libgstreamer-plugins-base1.0-dev \
    libgstreamer1.0-dev \
    libgtk-3-dev \
    libgtk2.0-dev \
    libhdf5-dev \
    libjasper-dev \
    libjpeg-dev \
    liblapack-dev \
    libopenblas-dev \
    libpng-dev \
    libswscale-dev \
    libtbb-dev \
    libtbb2 \
    libtiff-dev \
    libv4l-dev \
    libx264-dev \
    libxvidcore-dev \
    pkg-config \
    protobuf-compiler \
    python3-dev \
    python3-numpy \
    unzip

Build

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

mkdir -p ~/local/opencv && cd ~/local/opencv

# Download and unpack sources
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.5.2.zip
wget -O opencv.zip https://github.com/opencv/opencv/archive/4.5.2.zip

unzip opencv.zip
unzip opencv_contrib.zip

mv opencv-4.5.2 opencv
mv opencv_contrib-4.5.2 opencv_contrib

# remove archives
rm opencv.zip
rm opencv_contrib.zip

# Create build directory and switch into it
mkdir -p opencv/build && cd opencv/build

# Configure
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D OPENCV_EXTRA_MODULES_PATH=~/local/opencv/opencv_contrib/modules \
-D ENABLE_NEON=ON \
-D WITH_OPENMP=ON \
-D WITH_OPENCL=OFF \
-D BUILD_ZLIB=ON \
-D BUILD_TIFF=ON \
-D WITH_FFMPEG=ON \
-D WITH_TBB=ON \
-D BUILD_TBB=ON \
-D BUILD_TESTS=OFF \
-D WITH_EIGEN=OFF \
-D WITH_GSTREAMER=ON \
-D WITH_V4L=ON \
-D WITH_LIBV4L=ON \
-D WITH_VTK=OFF \
-D WITH_QT=OFF \
-D OPENCV_ENABLE_NONFREE=ON \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=OFF \
-D BUILD_opencv_python3=TRUE \
-D OPENCV_GENERATE_PKGCONFIG=ON \
-D BUILD_EXAMPLES=OFF ..

# Build
make -j 4

# Install
sudo make install

Lluvia

Bazel

1
2
3
4

wget -O bazel https://github.com/bazelbuild/bazel/releases/download/5.3.2/bazel-5.3.2-linux-arm64

chmod +x bazel
sudo mv bazel /usr/local/bin

Build and install C++ and Python packages

Install clang compiler

1

sudo apt install -y clang

Clone and build Lluvia

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

git clone https://github.com/jadarve/lluvia.git
cd lluvia

# install requirements listed in lluvia's root folder
sudo python3 -m pip install -r requirements.txt

bazel build //lluvia/cpp/...

# Run the tests to verify runtime is configured propertly
bazel test //lluvia/cpp/...

# Build and install the python wheel
bazel build //lluvia/python:lluvia_wheel
python3 -m pip install bazel-bin/lluvia/python/lluvia-0.0.1-py3-none-any.whl

Examples

With the Raspberry Pi camera module installed, it is possible to run the webcam demo located at samples/webcam/webcam.py. The command below configures the camera to output images at 320x240 resolution and fed to the webcam/HornSchunck container node, defined in the horn_schunck.lua script:

1
2

./samples/webcam/webcam.py --width=320 --height=240 \
  ./samples/webcam/scripts/horn_schunck.lua webcam/HornSchunck

The resulting color-encoded optical flow is displayed in a second window.

2.1.4 - Windows 10

Dependencies

• Python2. This is needed for Bazel to be able to run Python binaries

  1 2 3

  choco install python2 python -m pip install jinja2

• Python3 dependencies

  1	pip3 install cython numpy pytest jinja2 markupsafe

• Vulkan SDK: follow the official installation instructions from LunarG.

• Bazel: follow the official installation guide from bazel.build.

Clone and configure the repository

Clone the Lluvia repository from Github:

1
2

git clone https://github.com/jadarve/lluvia.git
cd lluvia

Open platform/values.bzl and change the paths to Python2 and Python3 according to your installation. Initially the file looks like this:

1
2
3
4
5
6
7
8
9

# Linux
python2_path_linux = "/usr/bin/python2"
python3_path_linux = "/usr/bin/python3"

# Windows
python2_path_windows = "C:/Python27/python.exe"

# get this value by running where.exe python3
python3_path_windows = "C:/hostedtoolcache/windows/Python/3.7.9/x64/python3.exe"

Build the C++ libraries

1

bazel build //lluvia/cpp/...

Run the tests to verify that your compilation runs properly:

1

bazel test //lluvia/cpp/...

Python3 package

To build the Python3 package, execute the commands below from the repository’s top-level directory.

1
2

bazel build //lluvia/python:lluvia_wheel
pip3 install bazel-bin/lluvia/python/lluvia-0.0.1-py3-none-any.whl

Open a Python3 interpreter and import lluvia package

1
2
3

import lluvia as ll

session = ll.createSession()

If the import completes successfully, lluvia is ready to use.

2.2 - Mediapipe integration

Mediapipe is a cross-platform framework for creating complex Computer Vision and Deep Learning pipelines both for offline and streaming applications. It includes support to OpenCV and TensorFlow. By integrating Lluvia in mediapipe, it is possible to leverage its runtime capabilities as well as the interfacing with other popular frameworks.

Setup

Follow the linux instructions to install the basic dependencies to build lluvia in your host machine.

Mediapipe

Clone the mediapipe repository in the same folder as lluvia.

1
2

git clone https://github.com/google/mediapipe.git
cd mediapipe

Setup Clang as default C++ compiler for mediapipe. Add the following line to mediapipe’s .bazelrc file

build:linux --action_env=CC=clang

Follow the installations instructions to configure OpenCV according to your installation. Also, enable GPU support. Once completed, run the hello_world application to check the build process:

1
2
3
4

export GLOG_logtostderr=1

bazel run --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
    //mediapipe/examples/desktop/hello_world:hello_world

the output should look like:

I20221006 15:04:52.196460 12142 hello_world.cc:57] Hello World!
I20221006 15:04:52.196496 12142 hello_world.cc:57] Hello World!
I20221006 15:04:52.196501 12142 hello_world.cc:57] Hello World!
I20221006 15:04:52.196537 12142 hello_world.cc:57] Hello World!
I20221006 15:04:52.196563 12142 hello_world.cc:57] Hello World!
I20221006 15:04:52.196588 12142 hello_world.cc:57] Hello World!
I20221006 15:04:52.196615 12142 hello_world.cc:57] Hello World!
I20221006 15:04:52.196640 12142 hello_world.cc:57] Hello World!
I20221006 15:04:52.196666 12142 hello_world.cc:57] Hello World!
I20221006 15:04:52.196691 12142 hello_world.cc:57] Hello World!

Modifications to embed Lluvia as mediapipe’s dependency

The next step is to include lluvia as a dependency of mediapipe. Append the configuration below to mediapipe’s WORKSPACE file to configure lluvia as a local_repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

###########################################################
# LLUVIA
###########################################################
local_repository(
    name = "lluvia",
    path = "../lluvia" # assuming lluvia was cloned in the same folder as mediapipe
)

load("@lluvia//lluvia/bazel:workspace.bzl", "lluvia_workspace")
lluvia_workspace()

# Python configuration
register_toolchains("@lluvia//platform:python_toolchain")

load("@rules_python//python:pip.bzl", "pip_repositories")
pip_repositories()

# Platform configuration
# Linux
load("@lluvia//platform/linux:python.bzl", "python_linux", "numpy_linux")
python_linux(name = "python_linux")
numpy_linux(name = "numpy_linux")

# Windows
load("@lluvia//platform/windows:python.bzl", "python_windows", "numpy_windows")
python_windows(name = "python_windows")
numpy_windows(name = "numpy_windows")

# Packaging rules
load("@ll_rules_pkg//:deps.bzl", "rules_pkg_dependencies")
rules_pkg_dependencies()

# Vulkan rules
load("@rules_vulkan//vulkan:repositories.bzl", "vulkan_repositories")
vulkan_repositories(
    android_use_host_vulkan_sdk = True
)

# Lua rules
load("@rules_lua//toolchains:repositories.bzl", "lua_repositories")
lua_repositories()

Rerun mediapipe’s hello_world binary again to confirm the new workspace configuration works:

1
2
3
4

export GLOG_logtostderr=1

bazel run --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
    //mediapipe/examples/desktop/hello_world:hello_world

Extra configuration for Android builds

Install Android Studio, SDK 33 and SDK 30, and the NDK 21 (r21e). Configure the ANDROID_HOME and ANDROID_NDK_HOME environment variables in your .bashrc or .zshrc file, for instance:

1
2

export ANDROID_HOME=~/local/Android/Sdk
export ANDROID_NDK_HOME=${ANDROID_HOME}/ndk/android-ndk-r21e

Install the default JDK and the ADB in the system:

sudo apt install default-jdk adb

WORKSPACE configuration

In addition to the WORKSPACE configuration mentioned above, add the following lines for Bazel to configure the Android SDK and NDK.

1
2
3
4
5

###########################################################
# ANDROID
###########################################################
android_sdk_repository(name = "androidsdk", api_level = 33, build_tools_version = "30.0.3")
android_ndk_repository(name = "androidndk", api_level=21)

lluvia-mediapipe repository

The lluvia-mediapipe project is an auxiliary repository containing the Calculators to interface with mediapipe. This repository needs to be cloned within mediapipe in order to consume its dependencies.

1
2
3
4
5

# clone lluvia-mediapipe inside mediapipe

# assuming you are in the in the root folder were mediapipe was cloned
cd mediapipe/mediapipe
git clone https://github.com/jadarve/lluvia-mediapipe.git

The directory structure for all repositories should look like:

lluvia                          <-- lluvia repository
mediapipe                       <-- mediapipe repository
├── BUILD.bazel
├── docs
├── LICENSE
├── MANIFEST.in
├── mediapipe                   <--
│   ├── BUILD
│   ├── calculators
│   ├── examples
│   ├── framework
│   ├── gpu
│   ├── ...
│   ├── lluvia-mediapipe        <-- lluvia-mediapipe repository
├── ...
├── .bazelrc
└── WORKSPACE

Next, run the lluvia_calculator_test target to verify the build and the runtime is correctly configured:

1
2

bazel test --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
    //mediapipe/lluvia-mediapipe/calculators:lluvia_calculator_test --test_output=all

LluviaCalculator

The lluvia-mediapipe repository declares a new LluviaCalculator. This calculator is in charge of initializing Lluvia, binding input and output streams from mediapipe to lluvia ports, and running a given compute pipeline.

The figure below illustrates a basic mediapipe graph utilizing lluvia, while the code below shows the graph description using protobuffer text syntax

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

input_stream: "input_stream"
output_stream: "output_stream"

node: {
  calculator: "LluviaCalculator"
  input_stream: "IN_0:input_stream"
  output_stream: "OUT_0:output_stream"
  node_options {
      [type.googleapis.com/lluvia.LluviaCalculatorOptions]: {
          enable_debug: true
          library_path: "path to .zip node library file"
          script_path: "path to .lua script defining the main container node"
          container_node: "mediapipe/examples/Passthrough"
          
          input_port_binding:  {
              mediapipe_tag: "IN_0"
              lluvia_port: "in_image"
              packet_type: IMAGE_FRAME
          }

          output_port_binding:  {
              mediapipe_tag: "OUT_0"
              lluvia_port: "out_image"
              packet_type: IMAGE_FRAME
          }
      }
  }
}

where:

1. The enable_debug flag tells whether or not the Vulkan debug extensions used by Lluvia should be loaded during session creation. This flag might be set to false in production applications to improve runtime performance.
2. The library_path declare paths to node libraries (a .zip file) containing Lluvia nodes (Container and Compute). This attribute can be repeated several times.
3. The script_path is the path to a lua script declaring a ContainerNode that Lluvia will instantiate as the “main” node to run inside the calculator.
4. input_port_binding, maps mediapipe input tags to the main ContainerNode port. In the example above, mediapipe’s input tag IN_0 is mapped to lluvia’s in_image port.
5. output_port_binding does the same for outputs of the ContainerNode. Both input and output bindings have a packet_type attribute indicating the expected type of the binding. Possible values are: IMAGE_FRAME and GPU_BUFFER.

Android Archive build

The lluvia-mediapipe repo contains an example Android Archive build target that contains the LluviaCalculator as well as some mediapipe graph examples. To build the archive, run:

1
2
3
4

bazel build -c opt \
    --host_crosstool_top=@bazel_tools//tools/cpp:toolchain \
    --fat_apk_cpu=arm64-v8a \
    //mediapipe/lluvia-mediapipe/java/ai/lluvia:lluvia_aar

The generated AAR file will be located at bazel-bin/mediapipe/lluvia-mediapipe/java/ai/lluvia/lluvia_aar.aar and can be exported into an Android project.

Examples

There are two example applications packed with lluvia-mediapipe

single_image

This app receives as command line arguments the path to an image file, a mediapipe graph definition and a lua script describing the ContainerNode to run inside of the LluviaCalculator.

The commands below execute the single_image app loading sample graphs packaged in the repository. The command assumes both lluvia and mediapipe repositories are cloned in the ${HOME}/git folder.

Passthrough

This graph simply copies the input image to the output, without any processing:

1
2
3
4
5

bazel run --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
    //mediapipe/lluvia-mediapipe/examples/desktop/single_image:single_image -- \
    --input_image=${HOME}/git/lluvia/lluvia/resources/mouse.jpg \
    --script_file=${HOME}/git/mediapipe/mediapipe/lluvia-mediapipe/examples/desktop/graphs/passthrough/script.lua \
    --graph_file=${HOME}/git/mediapipe/mediapipe/lluvia-mediapipe/examples/desktop/graphs/passthrough/graph.pbtxt

BGRA to Gray

This graph runs the lluvia/color/BGRA2Gray compute node to convert from the BGRA input to gray scale:

1
2
3
4
5

bazel run --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
    //mediapipe/lluvia-mediapipe/examples/desktop/single_image:single_image -- \
    --input_image=${HOME}/git/lluvia/lluvia/resources/mouse.jpg \
    --script_file=${HOME}/git/mediapipe/mediapipe/lluvia-mediapipe/examples/desktop/graphs/BGRA2Gray/script.lua \
    --graph_file=${HOME}/git/mediapipe/mediapipe/lluvia-mediapipe/examples/desktop/graphs/BGRA2Gray/graph.pbtxt

webcam

This application tries to open the default camera capture device in the host system using OpenCV VideoCapture class. Then it feeds the mediapipe graph the captured frames to be processed by the LluviaCalculator.

BGRA to Gray

1
2
3
4

bazel run --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
    //mediapipe/lluvia-mediapipe/examples/desktop/webcam:webcam -- \
    --script_file=${HOME}/git/mediapipe/mediapipe/lluvia-mediapipe/examples/desktop/graphs/BGRA2Gray/script.lua \
    --graph_file=${HOME}/git/mediapipe/mediapipe/lluvia-mediapipe/examples/desktop/graphs/BGRA2Gray/graph.pbtxt

Horn and Schunck optical flow

This is a more elaborate graph running inside the LluviaCalculator. First, the input image is converted from BGRA to gray scale, then is passes through the Horn and Schunck optical flow algorithm, and finally, the estimated optical flow is converted to RGBA (and then to BGRA) for visualization:

@startuml
skinparam linetype ortho

state LluviaCalculator as "LluviaCalculator" {

    state input_stream as "IN_0:input_stream" <<inputPin>>
    state output_stream as "OUT_0:output_stream" <<outputPin>>

    state ContainerNode as "mediapipe/examples/HornSchunck" {
        
        state in_image <<inputPin>>

        state BGRA2Gray
        state HS as "HornSchunck"
        state Flow2RGBA
        state RGBA2BGRA

        input_stream -down-> in_image

        in_image -down-> BGRA2Gray
        BGRA2Gray -down-> HS: in_gray
        HS -down-> Flow2RGBA: in_flow
        Flow2RGBA -down-> RGBA2BGRA: in_rgba

        RGBA2BGRA -down-> out_image <<outputPin>>
    }
    
  
  out_image -down-> output_stream <<outputPin>>
}

@enduml

The command to run the example is:

1
2
3
4

bazel run --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
    //mediapipe/lluvia-mediapipe/examples/desktop/webcam:webcam -- \
    --script_file=${HOME}/git/mediapipe/mediapipe/lluvia-mediapipe/examples/desktop/graphs/HornSchunck/script.lua \
    --graph_file=${HOME}/git/mediapipe/mediapipe/lluvia-mediapipe/examples/desktop/graphs/HornSchunck/graph.pbtxt

References

• Mediapipe
• lluvia-mediapipe

3 - Reference

The diagram below illustrates the suggested order for reading the documentation:

graph
    Session
    Memory

    Session --> Memory
    Memory --> Objects
    Objects --> Buffer
    Objects --> Image
    Image --> ImageView

    Session --> NodeSystem
    NodeSystem --> ComputeNode
    ComputeNode --> ContainerNode

    %% Interaction
    click Session "/docs/reference/session" "Session"
    click Memory "/docs/reference/memory" "Memory"
    click Objects "/docs/reference/objects" "Objects"
    click Buffer "/docs/reference/objects/buffer" "Buffer"
    click Image "/docs/reference/image" "Image"
    click ImageView "/docs/reference/image_view" "ImageView"
    click NodeSystem "/docs/reference/node_system" "NodeSystem"
    click ComputeNode "/docs/reference/node_system/compute_node" "ComputeNode"
    click ContainerNode "/docs/reference/node_system/container_node" "ContainerNode"

3.1 - Session

A Session is the main object in a Lluvia application. It holds the references to the underlying device used for computation. To see the available devices, run:

• Python
• C++

1
2
3
4

import lluvia as ll

for device in ll.getAvailableDevices():
    print(device)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

#include <lluvia/core.h>

#include <iostream>

int main() {

    const auto availableDevices = ll::Session::getAvailableDevices();

    for(auto deviceDesc : availableDevices) {

        std::cout << "ID: " << deviceDesc.id
                  << " type: " << ll::deviceTypeToString(std::forward<ll::DeviceType>(deviceDesc.deviceType))
                  << " name: " << deviceDesc.name << std::endl;
    }

    return 0;
}

and the output can look like:

id: 7040 type: DiscreteGPU   name: GeForce GTX 1080
id: 0    type: CPU           name: llvmpipe (LLVM 12.0.0, 256 bits)
id: 1042 type: IntegratedGPU name: Intel(R) HD Graphics 4600 (HSW GT2)

To create a session:

• Python
• C++

1
2
3
4
5
6
7
8

import lluvia as ll

devices = ll.getAvailableDevices()

# ... select the device appropriate to your needs
selectedDevice = devices[0]

session = ll.createSession(enableDebug=True, device=selectedDevice)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

#include <lluvia/core.h>

#include <memory>

int main() {

    const auto availableDevices = ll::Session::getAvailableDevices();

    // ... select the device appropriate to your needs
    auto selectedDevice = availableDevices[0];

    auto desc = ll::SessionDescriptor()
                    .enableDebug(true)
                    .setDeviceDescriptor(selectedDevice);    

    std::shared_ptr<ll::Session> session = ll::Session::create(desc);

    return 0;
}

The enableDebug flag enables the Vulkan validation layers for receiving messages about bad usage of the API. This can be useful while building your compute pipelines, but should be disabled in Production for reducing the communication overhead with the GPU.

Several object types are creating from a session, among the most important are:

graph
    Session --> Memory
    Session --> Program
    Session --> CommandBuffer
    Session --> Duration
    Session --> ComputeNode
    Session --> ContainerNode

What’s next

Check the Memory page to know about the different memory types in Lluvia.

3.2 - Memory

Memory objects represent regions of memory that can be used to allocate objects. Lluvia uses the memory types defined by the Vulkan API. You may also refer to this article by Adam Sawicki on how memory is offered by different GPU vendors.

Memory types

Memory objects are created from a Lluvia Session. The code block below enumerate the available memory options:

• Python
• C++

1
2
3
4
5
6
7
8

import lluvia as ll

session = ll.createSession(enableDebug=True)

for n, memflags in enumerate(session.getSupportedMemoryPropertyFlags()):
    print('Memory index:', n)
    print('    supported flags:', [p.name for p in memflags])
    print()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

#include <iostream>
#include "lluvia/core.h"

#include <vulkan/vulkan.hpp>

int main() {
    
    ll::SessionDescriptor desc = ll::SessionDescriptor().enableDebug(true);

    std::shared_ptr<ll::Session> session = ll::Session::create(desc);

    std::vector<vk::MemoryPropertyFlags> flagsVector = session->getSupportedMemoryFlags();

    for(int i = 0; i < flagsVector.size(); ++i) {
        const vk::MemoryPropertyFlags &flags = flagsVector[i];
        
        std::cout << "Memory index: " << i << std::endl;
        std::cout << "    Supported flags: " << vk::to_string(flags) << std::endl;
    }

    return 0;
}

Memory index: 0
    supported flags: ['DeviceLocal']

Memory index: 1
    supported flags: ['HostCoherent', 'HostVisible']

Memory index: 2
    supported flags: ['HostCached', 'HostCoherent', 'HostVisible']

Memory index: 3
    supported flags: ['DeviceLocal', 'HostCoherent', 'HostVisible']

The possible MemoryPropertyFlags values are:

For Lluvia, the two most important memory flag tuples are:

Creation

The code block below shows how to create memory objects:

• Python
• C++

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import lluvia as ll

session = ll.createSession(enableDebug=True)

deviceMemory = session.createMemory(ll.MemoryPropertyFlagBits.DeviceLocal, pageSize=32*1024*1024, exactFlagsMatch=False)

# use default page size
hostMemory = session.createMemory([ll.MemoryPropertyFlagBits.DeviceLocal,
                                   ll.MemoryPropertyFlagBits.HostVisible,
                                   ll.MemoryPropertyFlagBits.HostCoherent])

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

#include "lluvia/core.h"
#include <vulkan/vulkan.hpp>
#include <iostream>
#include <memory>

int main() {

    auto session = ll::Session::create(ll::SessionDescriptor().enableDebug(true));
    
    const vk::MemoryPropertyFlags deviceFlags = vk::MemoryPropertyFlagBits::eDeviceLocal;
    const vk::MemoryPropertyFlags hostFlags = vk::MemoryPropertyFlagBits::eDeviceLocal |
                                              vk::MemoryPropertyFlagBits::eHostVisible |
                                              vk::MemoryPropertyFlagBits::eHostCoherent;

    std::shared_ptr<ll::Memory> deviceMemory = session->createMemory(deviceFlags, 32*1024*1024, false);

    std::shared_ptr<ll::Memory> hostMemory = session->createMemory(hostFlags, 32*1024*1024, false);

    return 0;
}

Memories are created by passing the set of flags the memory should have. It is possible to also pass the size of a page, which defaults to 32MB, and an extra parameter to indicate if the flags should match perfectly with any of those listed by session.getSupportedMemoryPropertyFlags().

Internally, a Memory manages regions of memory as pages. On each page, there can be several objects allocated, such as Buffer or Image.

stateDiagram-v2

    Memory --> Page_0
    Memory --> Page_1
    Memory --> Page_2

    state Page_0 {
        Buffer_0
        Buffer_1
    }

    state Page_1 {
        Image_1
        Buffer_2
    }

    state Page_2 {
        Image_2
    }       

It is possible to query the memory attributes as:

• Python
• C++

1
2
3
4

print('flags      :', [p.name for p in hostMemory.memoryFlags])
print('isMappable :', hostMemory.isMappable)
print('pageCount  :', hostMemory.pageCount)
print('pageSize   :', hostMemory.pageSize)

1
2
3
4

std::cout << "flags      : " << vk::to_string(hostMemory->getMemoryPropertyFlags()) << std::endl;
std::cout << "isMappable : " << hostMemory->isMappable() << std::endl;
std::cout << "pageCount  : " << hostMemory->getPageCount() << std::endl;
std::cout << "pageSize   : " << hostMemory->getPageSize() << std::endl;

which prints:

flags      : ['DeviceLocal', 'HostCoherent', 'HostVisible']
isMappable : True
pageCount  : 0
pageSize   : 33554432

In particular, the isMappable flag tells whether or not the memory space can be mapped to the host memory space. At the moment of creation, there are no actual pages allocated, and hence, pageCount equals 0.

Object allocation

There are two types of objects that can be allocated from a Memory:

graph
    Memory --> Buffer
    Memory --> Image

    click Buffer "/docs/reference/objects/buffer" "Buffer"
    click Image "/docs/reference/image" "Image"

The code block below shows how to allocate a buffer and an image object. Each allocated object has an allocationInfo to see the allocation values.

• Python
• C++

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

import lluvia as ll

session = ll.createSession(enableDebug=True)

deviceMemory = session.createMemory(ll.MemoryPropertyFlagBits.DeviceLocal, pageSize=32*1024*1024)

# A 1024 byte size buffer
buffer = deviceMemory.createBuffer(1024)

# A 32x32 pixels image where each pixel is of type Uint8
image = deviceMemory.createImage((32, 32), ll.ChannelType.Uint8)

print('buffer:')
print('  page         :', buffer.allocationInfo.page)
print('  offset       :', buffer.allocationInfo.offset)
print('  left padding :', buffer.allocationInfo.leftPadding)
print('  size         :', buffer.allocationInfo.size)

print()
print('image:')
print('  page         :', image.allocationInfo.page)
print('  offset       :', image.allocationInfo.offset)
print('  left padding :', image.allocationInfo.leftPadding)
print('  size         :', image.allocationInfo.size)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#include "lluvia/core.h"
#include <vulkan/vulkan.hpp>
#include <iostream>
#include <memory>

int main() {
    auto session = ll::Session::create(ll::SessionDescriptor().enableDebug(true));
    
    const vk::MemoryPropertyFlags deviceFlags = vk::MemoryPropertyFlagBits::eDeviceLocal;

    std::shared_ptr<ll::Memory> deviceMemory = session->createMemory(deviceFlags, 32*1024*1024, false);
    
    std::shared_ptr<ll::Buffer> buffer = deviceMemory->createBuffer(1024);
    
    // 32x32 image with one uint8 color channel per pixel
    ll::ImageDescriptor desc = ll::ImageDescriptor(1, 32, 32, ll::ChannelCount::C1, ll::ChannelType::Uint8);
    std::shared_ptr<ll::Image> image = deviceMemory->createImage(desc);
    
    std::cout << "buffer:" << std::endl;
    std::cout << "  page         : " << buffer->getAllocationInfo().page << std::endl;
    std::cout << "  offset       : " << buffer->getAllocationInfo().offset << std::endl;
    std::cout << "  left padding : " << buffer->getAllocationInfo().leftPadding << std::endl;
    std::cout << "  size         : " << buffer->getAllocationInfo().size << std::endl;

    std::cout << std::endl;
    std::cout << "image:" << std::endl;
    std::cout << "  page         : " << image->getAllocationInfo().page << std::endl;
    std::cout << "  offset       : " << image->getAllocationInfo().offset << std::endl;
    std::cout << "  left padding : " << image->getAllocationInfo().leftPadding << std::endl;
    std::cout << "  size         : " << image->getAllocationInfo().size << std::endl;
}

The amount of memory reserved for a given object can be higher than the actual needed. This is because Vulkan imposes certain requirements on the allocation such as alignment.

buffer:
  page         : 0
  offset       : 2048
  left padding : 0
  size         : 1024

image:
  page         : 0
  offset       : 0
  left padding : 0
  size         : 2048

What’s next

Check the Objects page for an overview of the objects available in Lluvia.

3.3 - Objects

3.3.1 - Buffer

Buffers are unstructured regions of contiguous memory. Buffers are created from Memory objects:

• Python
• C++

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

import lluvia as ll

session = ll.createSession()

hostMemory = session.createMemory([ll.MemoryPropertyFlagBits.DeviceLocal,
                                   ll.MemoryPropertyFlagBits.HostVisible,
                                   ll.MemoryPropertyFlagBits.HostCoherent])

aBuffer = hostMemory.createBuffer(1024, usageFlags=[ll.BufferUsageFlagBits.TransferDst,
                                                    ll.BufferUsageFlagBits.TransferSrc,
                                                    ll.BufferUsageFlagBits.StorageBuffer])

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

#include <iostream>
#include <vulkan/vulkan.hpp>
#include "lluvia/core.h"

#include <vulkan/vulkan.hpp>

int main() {
    
    ll::SessionDescriptor desc = ll::SessionDescriptor().enableDebug(true);

    std::shared_ptr<ll::Session> session = ll::Session::create(desc);

    hostMemory = session.createMemory([ll.MemoryPropertyFlagBits.DeviceLocal,
                                   ll.MemoryPropertyFlagBits.HostVisible,
                                   ll.MemoryPropertyFlagBits.HostCoherent])

    const auto usageFlags = vk::BufferUsageFlags { vk::BufferUsageFlagBits::eStorageBuffer
                                                 | vk::BufferUsageFlagBits::eTransferSrc
                                                 | vk::BufferUsageFlagBits::eTransferDst};

    auto aBuffer = hostMemory->createBuffer(1024, usageFlags);
}

The first parameter is the requested size in bytes. The usageFlags indicated the intended usage of this buffer; the values are taken directly from the Vulkan BufferUsageFlagBits. The most used values are:

3.3.2 - Image

3.3.3 - ImageView

3.4 - Node system

%%{init: {'theme': 'neutral', 'themeVariables': {'fontSize': '32px', 'primaryColor': '#FF0000'}}}%%

classDiagram

    class Session
    class Memory
    class Program
    
    class Buffer
    class Image
    class ImageView

    class Node
    class ComputeNode
    class ContainerNode

    class CommandBuffer

    Session "1" --> "*" Memory
    Session "1" --> "*" Node
    Session "1" --> "*" CommandBuffer

    Memory "1" --> "*" Buffer: allocates
    Memory "1" --> "*" Image: allocates
    Image "1" --> "*" ImageView: creates

    Node <|-- ComputeNode
    Node <|-- ContainerNode

    ComputeNode "1" --> "1" Program

    ContainerNode "1" --> "*" ComputeNode: contains
    ContainerNode "1" --> "*" ContainerNode: contains

3.4.1 - Compute nodes

3.4.2 - Container nodes

3.4.3 - Running

4 - Examples

4.1 - Real-time optical flow