Audio Effect

This document mainly introduces the relevant content of Audio Effect, including its architecture, operating principle with the Application layer, and implementation.

As shown in the figure below, the Audio Effect is a high-level function module of the Audio Subsystem in parallel with the Stream, Notification, Signal and Volume modules. The Audio Effect module includes EQ, NREC, WDRC, Sidetone, Beamforming and Vendor Specific Effect (VSE). The Effect modules act on the Stream modules or the Signal modules, generating actual effects through the algorithm engine inside the DSP.

../../../_images/audio_subsystem_architecture.png — Audio Subsystem Architecture

Architecture Overview

The following diagram provides a simple description of the Audio Effect architecture. The Application invokes API from Audio Effect submodules to enable, disable, or flush specific effects. On the other hand, the Application invokes API from Stream submodules to start, stop or restart specific data stream. If the Application wishes to apply a certain effect to a data stream, it should actively invoke the API from the Stream submodules to attach it. The underlying AudioPath module will deliver the specific effect information attached to the data stream to the DSP at an appropriate time.

../../../_images/audio_effect_architecture.png — Audio Effect Architecture

Operating Principle

This section mainly describes the effect related interaction flow between Application and Audio Subsystem, and Audio Subsystem internal principles. The Audio Subsystem supports two types of effects: built-in effects and vendor specific effects.

Built-in Effect

The built-in effects include EQ, NREC, WDRC, Sidetone and Beamforming. The interaction process with Application is similar for these effects, and EQ will be used as an example for explanation.

As shown in the figure below, the Application calls eq_create() to create an EQ and calls eq_enable() to enable it. Then, the Application calls audio_track_effect_attach() to attach the EQ to the target playback data stream. In addition, during the runtime of the playback data stream, the Application can invoke eq_set() to flush the EQ coefficients, invoke eq_disable() to disable the EQ effect temporarily, or invoke eq_release() to release the EQ effect.

Note

For more detailed information about EQ, such as EQ types and configuration steps, please refer to EQ.

../../../_images/eq_interaction_with_app.png — EQ Interaction with APP

The figure below illustrates the internal interaction for EQ. When the API provided by the EQ module is called, it triggers a series of internal events within the Audio Effect and Audio Track. These events are ultimately translated into commands for the DSP.

../../../_images/eq_interaction_internal.png — EQ Internal Interaction

Vendor Specific Effect

The interaction between vendor specific effects and Application differs slightly from that of built-in effects. When integrating vendor specific effects, the Audio Subsystem primarily functions as a transmission layer, transparently passing information between the Application and the vendor custom algorithm library. The specific format of this information is defined by the algorithm vendor.

As illustrated in the diagram below, the Audio Subsystem implements three modes of interaction:

Request-Response: The Application sends a command to the DSP, expecting a response in return.
Indicate-Confirm: The DSP provides feedback to the Application and expects a response from the Application.
Notify Only: The DSP sends a notification to the Application without expecting any response.

In all these interaction modes, interactions initiated by the DSP are ultimately converted into events in the VSE module and then passed to the Application.

../../../_images/vse_interaction_with_app.png — Vendor Specific Effect Interaction with APP

Implementation

This section introduces the inheritance relationships derived from the abstract module Audio Effect and the internal details of binding an Audio Effect to a data stream.

Effect Inheritance Hierarchy

The diagram below illustrates the inheritance hierarchy of the effect-related software modules in the Audio Subsystem. As shown in the diagram, the Audio Effect serves as the base module, providing a set of general interfaces. The derived modules, such as EQ, NREC, …, and Audio_VSE, extend the general interfaces provided by the base module according to their respective needs. For example, eq_enable() is an extension of audio_effect_enable, indicating a desire to activate the EQ; eq_set() is an extension of audio_effect_update, representing an update to the EQ parameters. Specifically, both audio_vse_apply and audio_vse_confirm are extensions of audio_effect_update, suitable for different interaction modes mentioned in the Vendor Specific Effect.

../../../_images/inheritance_hierarchy.png — Effect Inheritance Hierachy

Binding to Data Streams

The Audio Subsystem supports dynamic binding of effects and data streams. As shown in the figure below, any specific effect can be bound to any kind of data stream. Furthermore, the binding and unbinding of data streams and effects can be dynamically managed at runtime, allowing Applications to control effects more flexibly. This design uses an abstract Audio Effect as a mediator to decouple the data stream model from specific effects, facilitating the independent expansion of the Audio Subsystem along the dimensions of data streams and effects.

../../../_images/binding_to_data_stream.png — Effect Binding to Data Streams

Taking Audio Track as an example, each Audio Track instance can aggregate one or more Audio Effect instances, thereby binding to specific Audio Effect instances. The Application explicitly calls the following set of APIs to bind and unbind Audio Effects to an Audio Track instance.

bool audio_track_effect_attach(T_AUDIO_TRACK_HANDLE    handle,
                               T_AUDIO_EFFECT_INSTANCE instance);

bool audio_track_effect_detach(T_AUDIO_TRACK_HANDLE    handle,
                               T_AUDIO_EFFECT_INSTANCE instance);

Note

A data stream can be attached to multiple Audio Effect instances simultaneously, but an Audio Effect instance can only be bound to one data stream at a time. This is because each Audio Effect instance is uniquely associated with a lower-level Audio Path instance. In practice, as shown in the figure below, an Audio Effect instance internally records the specific Audio Path instance it is truly bound to.

../../../_images/unique_effect_owner.png — Unique Effect Owner at A Time