Audio and video development and learning

We first assume that someone has zero foundation in audio and video, and has never learned any digital signal processing related knowledge. The mathematical foundation is basically high school level, but he is familiar with C/C++ development, at least familiar with the compilation and debugging IDE under a certain platform.

Focus on the two open source projects ffmpeg and webRTC, mainly looking at the video part of ffmpeg and the audio part of webRTC.

First of all, start with video decoding. This is the simplest one. The basic code for calling ffmpeg video decoding is only about 100 lines. Video decoding only needs to understand h.264, 80%+ scenes are in this encoding format, understand basic H.264 sps pps NAL and other file format concepts, understand the principle of YUV image format, and understand the conversion of YUV to RGB , So you can do image rendering, you can achieve a simple video player. (Basically in the first stage)

Then, to learn about video coding, it is recommended to start with x264, a simple call to x264 encoding code is less than 100 lines. Encoding is more complicated than decoding in that decoding does not require any parameters to be configured, and encoding has a large number of parameters that need to be configured. At the beginning, just understand a few basic concepts, such as frame rate, common flow control methods ABR CRF, GOP, I/B What does /P Frame mean respectively. (Familiar with X264 encoding again)

The codec of the audio part is similar to that of the video. In the initial stage, you only need to understand the two encoders EAAC+ and Silk, understand the concepts of sound channel, sampling rate, etc., and understand the Wave file format. Can be a simple audio player. (Make an audio player)

The complex part of audio is not the codec, but the sound effect. Take a look at the Audio Processing module in webRTC and understand the following concepts: denoising NS, echo cancellation AEC, mute detection VAD, automatic gain control AGC, and the built-in webRTC. Although the algorithm is not the best, it can solve 90%+ of the problems, and it is worth learning. (Learn webrtc audio processing)

After learning this, I need to think about a question, whether I want to take an engineering route or an algorithmic route. If it is an algorithmic route, then I must learn mathematics. Behind digital signal processing is a large number of basic mathematical theories. If it is an engineering route, then you have to learn cross-platform development, learn the collection, playback and processing of audio and video under multiple platforms, and learn assembly language optimization under multiple platforms. (Platform related)

Audio and video are a very deep field, and you will never finish learning in a lifetime, but this field has the advantage that the speed of knowledge update is relatively slow, the things learned are not easy to be eliminated, and the accumulation of experience is very important.v