ffmpeg: ffmpeg_dec: core module framework and audio decoders (FLAC, MP3, AAC, Opus) by lgirdwood · Pull Request #11308 · thesofproject/sof

lgirdwood · 2026-10-14T14:14:35Z

Overview

This is PR 1 in the series splitting monolithic PR #10990 into modular, reviewable PRs rebased on main.

This PR introduces the Core Engine & Audio Decoders (ffmpeg_dec) module, enabling playback of compressed audio elementary streams (FLAC, MP3, AAC-LC, and Opus) on SOF hardware platforms (including PTL, LNL, MTL, NVL, TGL, and WCL).

The module sources libavcodec and libavutil directly from the official SOF fork thesofproject/FFmpeg via the West manifest (west.yml).

Subsequent PRs in the series will build on this core:

PR 2: WebRTC audio processing modules (AEC, NS, VAD, RNNoise)
PR 3: FFmpeg audio encoders (ffmpeg_enc: MP3, AAC)
PR 4: Full topology integration for compressed playback & capture
PR 5: Documentation & tests
PR 6: FFmpeg audio filters (afftdn, alimiter)

Changes in this PR

1. West Manifest Integration (`west.yml`)

Adds ffmpeg pointing to official fork thesofproject/FFmpeg pinned to the sof-hifi branch revision (08ab4baa3400d55be75be1280f6119e210ce354e).
Placed under modules/audio/ffmpeg.

2. Audio Processing Module (`src/audio/ffmpeg_dec`)

Core module framework: implements SOF processing module interface supporting both built-in (=y) and LLEXT (=m) modular builds, plus a stub backend (CONFIG_COMP_FFMPEG_DEC_STUB) for CI validation without external dependencies.
Audio Decoders:
- FLAC: lossless integer decoder (fixed-point arithmetic, minimal footprint).
- MP3: MPEG-1/2 Layer III decoder via mpegaudio parser and float/fixed synthesis.
- AAC-LC: Low Complexity AAC decoder.
- Opus: SILK+CELT decoder.
Private libc and newlib double math:
- ffmpeg_dec-builtin-libc.c: provides isolated libc symbols required by FFmpeg without conflicting with Zephyr minimal libc.
- Vendored newlib math archive (libnewlib_m.a) compiled with -mlongcalls -mtext-section-literals to satisfy double-precision math without calling into unsupported host libc.
DSP Heap Allocator:
- ffmpeg_dec-builtin-alloc.c: wraps malloc/free/realloc/calloc onto the unshared DSP heap (.heap_mem), allowing large decoder working sets (e.g. AAC-LC MDCT tables) without overflowing the common-libc arena.
Xtensa Compiler & Longcalls Compliance:
- Passes -mlongcalls -mtext-section-literals to prevent call8 out-of-range relocation errors across segmented DSP memory.
- Soft-float scalar math options for platforms with scalar FPU unconfigured.
IPC & Base Firmware Capabilities:
- Adds SND_AUDIOCODEC_OPUS_RAW (0x11) definition to src/include/ipc/compress_params.h.
- Advertises FLAC, MP3, AAC, and Opus capabilities in src/audio/base_fw.c when configured.
- Rimage configuration registration across PTL, LNL, MTL, NVL, TGL, TGL-H, and WCL.

3. Pipeline & Topology Definitions

include/pipelines/cavs/compr-playback-ffmpeg.conf: compress playback pipeline definition routing through ffmpeg_dec.
platform/intel/compr-ffmpeg.conf: platform instance with FLAC STREAMINFO setup control, gated on COMPR_FFMPEG.
Development topology build targets in development/tplg-targets.cmake:
- sof-hda-generic-ffmpeg-compr (FLAC)
- sof-hda-generic-ffmpeg-mp3-compr (MP3)
- sof-hda-generic-ffmpeg-aac-compr (AAC-LC)

Verification & Build Results

Baseline PTL Build:
- xtensa-build-zephyr.py -z -p ptl: PASS (code 0)
Stub Backend Build:
- xtensa-build-zephyr.py -z -p -o ffmpeg_dec_stub.conf ptl: PASS (code 0)
Real Decoders (PTL / ACE 3.0):
- ffmpeg_dec_flac_y.conf: PASS (code 0)
- ffmpeg_dec_mp3_y.conf: PASS (code 0)
- ffmpeg_dec_aac_y.conf: PASS (code 0)
- ffmpeg_dec_opus_y.conf: PASS (code 0)
TGL (CAVS 2.5):
- xtensa-build-zephyr.py -z -p -o ffmpeg_dec_stub.conf tgl: PASS (code 0)
Topology Compilation:
- Built targets: topology2_dev_sof-hda-generic-ffmpeg-compr, topology2_dev_sof-hda-generic-ffmpeg-mp3-compr, topology2_dev_sof-hda-generic-ffmpeg-aac-compr: PASS (code 0)

Copilot

🟡 Changes recommended

Codec selection, memory sizing, allocator concurrency, stream draining, metadata handling, and modular math linkage contain correctness and reliability defects.

15 open findings

Make allocator binding instance-safe · New
Prevent unsynchronized image-wide malloc interception · New
Link real libm for modular FFmpeg decoder · New
Zero-initialize attribute buffer before dispatch · New
Increase modular heap for AAC decoder working set · New
Advertise only codecs enabled by Kconfig · New
Recreate parser during decoder reset · New
Support maximum FLAC and Opus frame sizes · New
Parse per-instance codec ID from IPC4 setup payload · New
Incrementally discard oversized ID3v2 tags · New
Parse oversized FLAC metadata incrementally · New
Flush parser and decoder before signaling EOS · New
Clamp final playback copy to drain boundary · New
Avoid hard-coded FLAC STREAMINFO in topology · New
Remove ineffective extra EDF stack reservation · New

What changed in this PR

Adds SOF’s FFmpeg-backed compressed-audio decoder framework for FLAC, MP3, AAC-LC, and Opus, including firmware, IPC, topology, build, and platform integration.

Changes:

Adds built-in/LLEXT decoder implementations, allocators, conversion, and private math/libc support.
Adds compressed-playback topology targets and codec capability advertisement.
Extends IPC drain handling, cross-core operations, manifests, and platform module packaging.

File	Description
`west.yml`	Adds the pinned SOF FFmpeg fork.
`uuid-registry.txt`	Registers the decoder UUID.
`zephyr/Kconfig`	Adjusts EDF stack configuration.
`zephyr/include/rtos/idc.h`	Adds no-timeout IDC mode and payload alignment.
`src/audio/CMakeLists.txt`	Includes the decoder build.
`src/audio/Kconfig`	Exposes decoder configuration.
`src/audio/base_fw.c`	Advertises compressed codec capabilities.
`src/audio/host-zephyr.c`	Adds compressed-stream drain tracking.
`src/audio/ffmpeg_dec/*`	Implements decoder framework, backends, allocators, conversion, build integration, and metadata.
`src/audio/ffmpeg_dec/libm/*`	Vendors private newlib math sources.
`src/idc/idc.c`	Handles cross-core attributes and reset operations.
`src/idc/zephyr_idc.c`	Implements indefinite blocking IDC waits.
`src/include/ipc/compress_params.h`	Adds the raw Opus codec identifier.
`tools/topology/topology2/sof-hda-generic.conf`	Includes FFmpeg compressed playback.
`tools/topology/topology2/cavs-mixin-mixout-hda.conf`	Adds FFmpeg codec selection and inclusion.
`tools/topology/topology2/development/tplg-targets.cmake`	Adds FLAC, MP3, and AAC development targets.
`tools/topology/topology2/include/components/ffmpeg_dec*`	Defines decoder widgets and FLAC setup data.
`tools/topology/topology2/include/pipelines/cavs/compr-playback-ffmpeg.conf`	Defines the compressed playback pipeline.
`tools/topology/topology2/platform/intel/compr-default.conf`	Adds decoder memory defaults.
`tools/topology/topology2/platform/intel/compr-ffmpeg.conf`	Instantiates the FFmpeg compressed pipeline.
`tools/rimage/config/{lnl,mtl,nvl,ptl,tgl,tgl-h,wcl}.toml.h`	Registers the module across Intel platforms.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

intel-sofci · 2026-10-14T18:29:34Z

PR 11308: test results

Run date: 2026-10-15 09:41 UTC

Tested commit: 76ef42158b4da11238cf08f9bea30290d012253a

struct idc_payload lives in coherent (SHARED_DATA) memory and is shared across cores, but it is only a byte array with no explicit alignment, so the linker was free to place it (and every per-core slot) on an odd address. Handlers that read the payload as a wider type -- e.g. the cross-core get_attribute path, which reads a uint32 type field -- then take an EXCCAUSE 9 (load/store alignment) fault when a component runs on a secondary core (ptl/ace30). Align the payload to the cache line. That is the granularity the payload is already built around (IDC_MAX_PAYLOAD_SIZE == DCACHE_LINE_SIZE * 2) and it guarantees every per-core slot starts word-aligned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The remote (cross-core) get_attribute handler passed the caller coherent (uncached) IDC buffer straight to comp_get_attribute(), which memcpy()s a whole ipc4_base_module_cfg into it. On Xtensa the compiler lowers that struct copy to wide accesses (AE_L32X2 / 64-bit) that the uncached window does not support, faulting EXCCAUSE 9 -- so binding a component that lives on a secondary core (e.g. ffmpeg_dec offloaded to core 1 on ptl/ace30) crashed. Have the handler fill a local cacheable copy and publish it to the coherent buffer with aligned 32-bit word stores (ipc4_coherent_copy32); do the same for the read-back on the initiator side. Only COMP_ATTR_BASE_CONFIG is supported remotely, so the fixed-size local copy is always correct. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A remote component prepare() can be expensive: a DP decoder one-time codec open builds large trig tables -- measured ~580 ms for mp3 on a secondary core. Waiting for it stalls the initiator core IPC/EDF thread for that whole time, and with the default CONFIG_IDC_TIMEOUT_US the blocking k_p4wq_wait() instead times out mid-prepare (-EAGAIN) and races pipeline teardown into a crash. Send the prepare IDC non-blocking. Ordering is preserved without the wait: each core drains its IDC work through a single in-order p4wq worker, so the prepare completes before the cross-core trigger (START) queued behind it. For a compress stream the START IPC also arrives only after the app first write, giving the remote prepare time to finish off the critical path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

comp_reset_remote() sent the cross-core IDC_MSG_RESET with IDC_BLOCKING, which only waits CONFIG_IDC_TIMEOUT_US (15 ms) before returning -EAGAIN. A data-processing (DP) module reset runs schedule_task_cancel() -> k_thread_join() on the target core and cannot return until the DP thread finishes its in-flight process() call. That call is bounded but can be long during a codec cold-start (a deferred avcodec_open2 table build, up to ~1.4 s for aac, or the first decode after it), far exceeding the IDC timeout. When it timed out the component was left un-reset (not COMP_STATE_READY), so pipeline_reset() failed and the host-side free was then refused with an invalid-state error and the remote core could not be disabled - every teardown wedged until reboot. The caller (pipeline_reset -> host) depends on the reset actually being finished before it frees the component and powers down the remote core, so the reset must stay synchronous. Add IDC_BLOCKING_NO_TIMEOUT, which is IDC_BLOCKING but waits K_FOREVER for the handler to complete, and use it for comp_reset_remote(). A normal end-of-stream teardown joins an idle DP thread and returns immediately, so this only waits when it must. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

lgirdwood · 2026-10-15T14:51:02Z

Review Feedback Addressed & CI Build Resolution

All 15 review findings from GitHub Copilot have been addressed and folded into the corresponding commits of the PR series, rebased onto origin/main, and force-pushed. In addition, the CI build failure affecting POSIX environments (Unit tests, testbench, and User space tools) has been resolved.

1. POSIX RTOS / CI Failure Resolution

posix/include/rtos/idc.h: Added #define IDC_BLOCKING_NO_TIMEOUT 4 matching zephyr/include/rtos/idc.h. Fixes the undeclared symbol build error in src/include/sof/audio/component_ext.h during non-Zephyr native builds.
CI Workflows Passed:
- Unit tests/cmocka_utests: PASS (all 32 cmocka tests passing 100%)
- User space tools/ directory: PASS (both Ubuntu-latest and Ubuntu-24.04)
- testbench/build-and-test: PASS

2. Review Findings Addressed (15 Items)

Allocator Instance Safety (ffmpeg_dec-alloc.c): Embedded mod and size in struct ffmpeg_dec_block_hdr at 16-byte alignment boundary. Core binding is scoped with arch_proc_id() (arch_inlines.h); free() retrieves mod directly from the block header, guaranteeing instance safety across multi-core/DP threads.
Heap Synchronization (ffmpeg_dec-builtin-alloc.c): Protected all sys_heap operations and lazy initialization under a dedicated struct k_spinlock ffmpeg_dec_heap_lock.
Modular LLEXT Double Math (llext/CMakeLists.txt, ffmpeg_dec-shims.c): Linked real newlib_m (libnewlib_m.a) into ffmpeg_dec_libs and added build dependencies; removed conflicting stubs and provided int *__errno(void).
Coherent Attribute Zero-Init (src/idc/idc.c): Zero-initialized struct ipc4_base_module_cfg local_cfg = { 0 }; before dispatch in idc_get_attribute().
AAC Working Set Heap Budget (tplg-targets.cmake, compr-default.conf): Configured COMPR_FFMPEG_DEC_HEAP=901120 (880 KiB) for the AAC target in tplg-targets.cmake and documented worst-case working sets in compr-default.conf.
Codec Capability Advertisement (src/audio/base_fw.c): Removed spurious FLAC and MP3 advertisements from #ifdef CONFIG_FFMPEG_DEC_OPUS.
Decoder Reset Recreation (ffmpeg_dec-ffmpeg.c): Recreated ff->parser via av_parser_close() and av_parser_init() on reset to prevent stale buffered frame carryover.
Frame Bounds & Allocation (ffmpeg_dec.c): Updated ffmpeg_dec_max_frame_samples() (FLAC 8192 / STREAMINFO max_bs, Opus 5760 @ 48 kHz), dynamically sized pcm_buf, and guarded direct sink writes with contiguous >= max_frame.
Dynamic Per-Instance Codec ID (ffmpeg_dec.c): Dynamically parse struct snd_codec from cfg->ext_data->module_data to map SND_AUDIOCODEC_FLAC, SND_AUDIOCODEC_MP3, SND_AUDIOCODEC_AAC, SND_AUDIOCODEC_OPUS_RAW, and SND_AUDIOCODEC_BESPOKE.
Incremental ID3v2 Tag Discard (ffmpeg_dec.c): Discard arbitrary-sized ID3v2 tags incrementally across input chunks using cd->skip_bytes.
Incremental FLAC Metadata Parsing (ffmpeg_dec.c): Added incremental block tracking (cd->flac_parsing_meta, cd->flac_last_block) to handle oversized FLAC metadata blocks (PICTURE, PADDING) across chunk boundaries.
EOS Flush Sequence (ffmpeg_dec.c, ffmpeg_dec-ffmpeg.c): Implemented drain state sending flush packets (decode(NULL, 0)) through the parser and decoder to drain all buffered samples before signaling EOS.
Committed Drain Byte Clamping (src/audio/host-zephyr.c): Clamped playback copy bytes to remaining committed drain bytes (drain_bytes - total_data_processed) on EOS.
Dynamic FLAC STREAMINFO (ffmpeg_dec.c, compr-ffmpeg.conf): Removed hard-coded flac_streaminfo.conf topology widget; dynamically extract 34-byte STREAMINFO from bytes 8..41 of incoming FLAC stream and apply via ffmpeg_dec_store_extradata().
EDF Stack Clean-Up (zephyr/Kconfig): Dropped the CONFIG_STACK_SIZE_EDF override commit cleanly; stack sizing remains governed by DP thread topology requirements.

Copilot

🟡 Changes recommended

Drain-counter truncation, DMA-tail hangs, and insufficient default decoder memory must be fixed before approval.

1 open finding

64-bit remainder truncation stalls copying at 4 GiB · New

15 resolved since last review

🧠 Review effort: Balanced

…count The HDA compress gateway free-runs its write pointer circularly and re-presents already-played data, so host_handle_eos() never sees dma_stat.pending_length reach 0 and the source buffer is never flagged END_OF_STREAM. A compress drain therefore never completes. On DRAIN the kernel now hands the host copier the total committed gateway byte count via a new SET_LARGE_CONFIG param (IPC4_COPIER_MODULE_CFG_PARAM_DRAIN_BYTES = 100), stored in host_data.drain_bytes. Once the copier has delivered that many bytes (total_data_processed >= drain_bytes), host_handle_eos() sets the source buffer to END_OF_STREAM and stops the copy, giving the pipeline a correct end of input. drain_bytes is cleared on reset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add the FFmpeg project to west.yml using the official thesofproject fork repository (thesofproject/FFmpeg) pinned to the sof-hifi branch revision (08ab4baa3400d55be75be1280f6119e210ce354e). This supplies libavcodec and libavutil sources for the ffmpeg_dec audio decoder module. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Define SND_AUDIOCODEC_OPUS_RAW (0x11) and update SND_AUDIOCODEC_MAX so that compressed Opus playback streams can be advertised and instantiated across SOF IPC interfaces. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Advertise supported compressed audio playback formats (FLAC, MP3, AAC-LC, and Opus) to the host in get_codec_info() when their respective CONFIG_FFMPEG_DEC_* symbols are enabled. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

… AAC, Opus) Add the ffmpeg_dec audio processing module providing compressed audio decoding for SOF pipelines using FFmpeg libavcodec. Key capabilities: - Module framework supporting both built-in (=y) and LLEXT (=m) loadable module builds, as well as a dependency-free stub backend for CI. - Support for FLAC (lossless fixed-point), MP3 (MPEG-1/2 Layer III), AAC-LC (Low Complexity), and Opus audio elementary streams. - Built-in private libc and newlib double-math archive compiled with -mlongcalls to satisfy libavcodec mathematical functions without polluting the minimal libc namespace. - DSP heap wrap allocator (ffmpeg_dec-builtin-alloc.c) to handle large decoder working sets without exceeding common-libc arena limits. - End-of-stream (EOS) signaling and compress drain handling. - Rimage module manifest toml and configuration registration for Intel platforms (PTL, LNL, MTL, NVL, TGL, TGL-H, WCL). Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Add topology definitions for driving compressed audio playback through the ffmpeg_dec widget instead of the legacy Cadence decoder: - New pipeline: include/pipelines/cavs/compr-playback-ffmpeg.conf - Platform instances: platform/intel/compr-ffmpeg.conf gated on COMPR_FFMPEG - Decoder component definition: include/components/ffmpeg_dec.conf - Configurable memory budgets (COMPR_FFMPEG_DEC_STACK / HEAP) in compr-default.conf - Added development topology targets: - sof-hda-generic-ffmpeg-compr (FLAC) - sof-hda-generic-ffmpeg-mp3-compr (MP3) - sof-hda-generic-ffmpeg-aac-compr (AAC-LC) Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

lgirdwood requested review from abonislawski, dbaluta, iuliana-prodan, jsarha, kv2019i, lbetlej, lyakh, mmaka1, pblaszko, plbossart, ranj063 and tmleman as code owners October 14, 2026 14:14

Copilot AI balanced review requested due to automatic review settings October 14, 2026 14:14

lgirdwood mentioned this pull request Oct 14, 2026

WIP: Audio Processing & Codec Suite: FFmpeg dec/enc, WebRTC APM, and DSP offload filters #10990

Draft

Copilot started reviewing on behalf of lgirdwood October 14, 2026 14:15

View session

Copilot AI reviewed Oct 14, 2026

View reviewed changes

lrgirdwo and others added 4 commits October 14, 2026 20:28

lgirdwood force-pushed the topic/ffmpeg-decoders branch from 618c239 to 76ef421 Compare October 14, 2026 19:32

lgirdwood requested a balanced review from Copilot October 15, 2026 15:14

Copilot started reviewing on behalf of lgirdwood October 15, 2026 15:15

View session

Copilot AI reviewed Oct 15, 2026

View reviewed changes

Comment thread src/audio/host-zephyr.c Outdated

lrgirdwo and others added 4 commits October 15, 2026 16:36

lrgirdwo added 2 commits October 15, 2026 16:36

lgirdwood force-pushed the topic/ffmpeg-decoders branch from 76ef421 to e7f67ec Compare October 15, 2026 15:37

Navigation Menu

Repository navigation

ffmpeg: ffmpeg_dec: core module framework and audio decoders (FLAC, MP3, AAC, Opus) - #11308

ffmpeg: ffmpeg_dec: core module framework and audio decoders (FLAC, MP3, AAC, Opus)#11308
lgirdwood wants to merge 10 commits into
thesofproject:mainfrom
lgirdwood:topic/ffmpeg-decoders

lgirdwood commented Oct 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

intel-sofci commented Oct 14, 2026 •

edited

Loading

Uh oh!

lgirdwood commented Oct 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

lgirdwood commented Oct 14, 2026

Overview

Changes in this PR

1. West Manifest Integration (west.yml)

2. Audio Processing Module (src/audio/ffmpeg_dec)

3. Pipeline & Topology Definitions

Verification & Build Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

intel-sofci commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR 11308: test results

Uh oh!

lgirdwood commented Oct 15, 2026

Review Feedback Addressed & CI Build Resolution

1. POSIX RTOS / CI Failure Resolution

2. Review Findings Addressed (15 Items)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

1. West Manifest Integration (`west.yml`)

2. Audio Processing Module (`src/audio/ffmpeg_dec`)

intel-sofci commented Oct 14, 2026 •

edited

Loading