Development of a computing backend for Llama.cpp based on Rockchip NPU with support for low-bit computations

Abstract

The article is devoted to solving the problem of energy-efficient inference of large language models (LLMs) on low-power devices. The paper examines the problem of high computational costs during text generation and presents an overview of existing hardware acceleration methods. Particular attention is paid to using the Neural Processing Unit (NPU) of the Rockchip platform to optimize resource-intensive tensor operations without significant quality loss.

Within the scope of the research, a specialized computational backend for the Llama.cpp framework was developed, integrated into the modular GGML architecture. The proposed software delegates matrix multiplication operations to the NPU, supporting computations in FP16, INT8, and INT4 formats. Mechanisms for weight preprocessing with adaptation to the hardware format, algorithms for smoothing outliers in activations using orthogonal Hadamard transformations, as well as an execution pipeline with computation parallelization were implemented. Experimental testing and comparative analysis of the developed solution against traditional execution on the Central Processing Unit (CPU) were performed. Experimental results on models ranging from 350 million to 8 billion parameters confirm the superiority of the proposed approach: a speedup of input context processing by more than 3 times compared to the CPU backend was recorded. Offloading computations to the specialized accelerator enabled reducing system energy consumption by more than 2 times with minimal accuracy degradation for models exceeding 1 billion parameters. The developed backend demonstrates high efficiency and makes the deployment of modern LLMs on devices with passive cooling and limited power capacity more realistic.