librav1e AV1 Encoder Assembly Optimizations

This article explores the specific assembly optimizations integrated into librav1e, the prominent AV1 video encoder written in Rust. We will examine how the project leverages hardware-specific assembly instructions to accelerate performance-critical operations, focusing on the targeted CPU architectures, key algorithms optimized, and how these low-level implementations integrate into the Rust-based codebase.

Supported CPU Architectures and Instruction Sets

While librav1e is built in Rust to ensure memory safety, pure Rust code is often insufficient for the extreme computational demands of real-time video encoding. To achieve competitive speeds, librav1e integrates hand-written assembly language tailored for specific processor architectures:

• x86_64 (Intel and AMD): The encoder features extensive optimizations for SSE2, SSSE3, and AVX2 instruction sets. Work is also ongoing to implement AVX-512 vector instructions for high-end server processors.
• AArch64 (ARM64): For mobile devices, Apple Silicon, and ARM-based servers, librav1e incorporates Neon SIMD (Single Instruction, Multiple Data) assembly.

These assembly routines allow the encoder to perform mathematical operations on multiple data points simultaneously, bypassing the overhead of standard compiler-generated machine code.

Key Encoder Operations Optimized in Assembly

The assembly optimizations in librav1e target the most computationally expensive bottlenecks in the AV1 encoding pipeline.

1. Motion Estimation (SAD and SATD)

Motion estimation is the process of finding matching blocks between video frames to reduce temporal redundancy. To speed this up, librav1e uses hand-written assembly for: * SAD (Sum of Absolute Differences): Calculates the absolute difference between pixel blocks. AVX2 and Neon assembly process multiple pixels per instruction cycle. * SATD (Sum of Absolute Transformed Differences): A more accurate metric that applies a Hadamard transform to the pixel differences. Assembly optimizations significantly speed up the execution of these transform matrices.

2. Forward and Inverse Transforms

AV1 utilizes various transform sizes ranging from 4x4 up to 64x64, using Discrete Cosine Transforms (DCT) and Asymmetric Discrete Sine Transforms (ADST). Librav1e includes specialized assembly for: * Multi-dimension matrix multiplications required for forward transforms (compressing spatial data into frequency data). * Inverse transforms (reconstructing pixels for the encoder’s internal reference frame).

3. In-Loop Filtering (CDEF and Deblocking)

AV1 relies heavily on in-loop filters to reduce compression artifacts. * CDEF (Constrained Directional Enhancement Filter): This filter requires identifying the direction of edges in a block and applying a directional smoothing filter. Both the direction search and the filtering execution are highly optimized using AVX2 and Neon instructions to prevent CPU stalls during post-processing. * Loop Restoration Filter (LR): Wiener and Self-Guided restoration filters are optimized to handle pixel smoothing operations across large frame areas efficiently.

4. Intra Prediction

For spatial redundancy reduction within a single frame, the encoder predicts block values based on neighboring pixels. Assembly routines optimize the directional prediction formulas, enabling the encoder to evaluate multiple intra-prediction modes rapidly.

Integration of Assembly in the Rust Ecosystem

To maintain its safety guarantees while utilizing raw assembly, librav1e utilizes a structured integration pipeline:

• NASM (Netwide Assembler): The x86 assembly is written in NASM syntax and compiled during the cargo build process using the nasm-rs build helper.
• Foreign Function Interface (FFI): Safe Rust wrappers bind to the compiled assembly functions. The Rust code handles high-level logic, scheduling, and threading, while passing raw pointers of pixel buffers to the assembly functions for heavy lifting.
• Checkasm: To prevent crashes and rendering bugs, librav1e utilizes a testing tool called checkasm. This tool compares the output of the assembly optimizations against the standard Rust fallback implementation to ensure absolute mathematical equivalence and prevent buffer overflows.