Mali GPU with LiteRT-LM

Key points

Armbian has published a step-by-step guide to running Google's Gemma 4 E2B model with LiteRT-LM on a board with a Mali-G610 GPU, using the mainline panthor kernel driver. Both the CPU and GPU paths work; the GPU path needs a one-line local patch to Mesa's panvk driver. The guide was verified on an Orange Pi 5 Max (RK3588, 8 GB RAM) with Armbian on Debian 13 (trixie), kernel 7.2.4-edge-rockchip64, Mesa 26.1.2 and LiteRT-LM v0.17.1.

Requirements

• An aarch64 system with a Mali GPU, at least 8 GB RAM and a minimal or console Armbian image; the guide says not to use a desktop image.
• The panthor kernel driver active (panfrost for older Mali), and a CPU with ARMv8.2-A dotprod, otherwise the LiteRT binaries crash with SIGILL.

ls /dev/dri/renderD128
lsmod | grep panthor
lscpu | grep -w asimddp

Install

• Mesa and Vulkan from backports, then the LiteRT-LM command-line tool in user space:

sudo apt install -y -t trixie-backports mesa-vulkan-drivers libvulkan1 vulkan-tools
curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install litert-lm

• vulkaninfo --summary should report the device Mali-G610 MC4 with the driver panvk.
• The model file gemma-4-E2B-it.litertlm (int4, 2.58 GB) comes from the litert-community repository on Hugging Face.

The panvk patch

• LiteRT's GPU accelerator uses Dawn (WebGPU), which requires maxImageDimension3D of at least 2048. Stock panvk reports 512 on this GPU and then crashes.
• The fix raises that limit to 2048 in src/panfrost/vulkan/panvk_vX_physical_device.c and builds a panvk-only Mesa 26.1.2 in the user's home (10 to 25 minutes on the board).
• The patched driver is selected per process with VK_ICD_FILENAMES and LD_LIBRARY_PATH. The system Mesa is untouched, and unsetting the two variables reverts it.
• The WebGPU prebuilt libraries (libLiteRtTopKWebGpuSampler.so, libwebgpu_dawn.so) are not bundled with the CLI and must be fetched from the litert-lm repository at the matching tag.

Results for Gemma 4 E2B

• CPU (XNNPACK, 8 threads): 123.6 tokens/s prefill, 12.3 tokens/s decode, 8.4 s to first token.
• GPU (patched panvk): 374.0 tokens/s prefill, 8.7 tokens/s decode, 2.9 s to first token.
• The GPU is about 3 times faster at prefill and time to first token, but slower at decode.
• With an image, the GPU cuts vision-encoder time by about 2.7 to 3 s (around 30%). Text on CPU plus vision on GPU gave 6.2 to 6.6 s to first token, against 8.9 to 9.4 s all on CPU.
• --cache disk stores compiled shaders and a weight cache of about 0.8 GB next to the model, so only the first GPU run is slow.
• Leave the GPU devfreq governor at its default; forcing it was counterproductive.

Gemma 4 E4B

• The stock E4B file (3.66 GB) cannot run its text path on the GPU on this 8 GB board: it exceeds panthor's fixed 1 s job timeout and runs out of memory.
• The text-only -web variant (2.97 GB) works on the GPU: 75.1 tokens/s prefill and 4.54 decode, against 52.6 and 5.58 on CPU.
• The stock E4B vision encoder does run on the GPU with text on CPU: 9.6 s to first token instead of 12.3 to 12.5 s.

Using it

litert-lm run ~/litert-lm-bench/gemma-4-E2B-it.litertlm --cache disk --prompt "What is the capital of France?"
litert-lm serve ~/litert-lm-bench/gemma-4-E2B-it.litertlm

• --attachment adds images or audio; --vision-backend and --audio-backend choose the encoder lane separately from --backend.
• litert-lm serve exposes an OpenAI-compatible API on port 9379 (/v1/models, /v1/chat/completions, with tool calls bridged). Settings live in ~/.litert-lm/config.json.
• Memory: max_num_tokens of 4096 uses about 3.1 GB and is recommended; 8192 uses about 7 GB.

Which lane to choose

• Chat and streaming: CPU. Long prompts, RAG and document Q&A: GPU. Images: GPU for vision, CPU for text when replies are short.

What does not work

• Rockchip's proprietary libmali needs the vendor kernel driver and cannot attach to mainline panthor.
• OpenCL: Mesa's rusticl exposes no device on panthor.

Summary by DistroNews, based on the original post.