"Rating guide
With constantly updated iteration recent broadcast technology, high-quality, low-bandwidth, low-cost broadcast industry has become an important goal to pursue. This article introduces the R & D of the H.265 player based on WEBASSEMBLY.
This article is from the national public number of the pepper.
Background introduction
With constantly updated iteration recent broadcast technology, high-quality, low-bandwidth, low-cost broadcast industry has become an important goal to pursue, in this context, H.264 standard has become the mainstream industry, and a new generation of HEVC (H.265) standard is also being broadcast field is more and more widely used. Pepper has been live for HEVC (H.265) research, application and continuous improvement.
Technology research
HEVC (H.265)
High-efficiency video coding (High Efficiency Video Coding, referred to as the HEVC), also known as H.265 and MPEG-H Part 2, is a video compression standard, is regarded as ITU-T H.264 / MPEG-4 AVC standard successor. HEVC is considered not only improves image quality, but also to achieve a compression ratio H.264 / MPEG-4 AVC twice as (equivalent to a bit rate reduced to 50% at the same picture quality). Hereinafter referred to as H.265.
Some major improvements to H.264 relative to H.264 include:
1. The more flexible block-sized image
H.265 divide the image into a more flexible, "" Tree coding units (Coding Tree Unit, CTU) "", rather than H.264 macroblock is divided into 4 × (Micro Block) 4 ~ 16 × 16 in. CTU using quadtree structure, can be (recursively) is divided into 64 × 64,32 × 32,16 × 16,8 × 8 sub-region size. As the video resolution is from 720p, 1080p to 2K, 4K continuously, the H.264 relatively small macroblock division generates a large amount of redundant parts, while H.265 provides a more flexible dynamic area division. Meanwhile, H.265 used by the encoding unit (Coding Unit, CU), a prediction unit (Predict Unit, PU) and the conversion unit (Transform Unit, TU) consisting of a new encoding scheme can effectively reduce the bit rate.
2. The more perfect prediction unit
Intra prediction: with respect to the nine kinds of intra prediction modes provided by H.264, to provide 35 as H.265 intra prediction modes (Planar-Mode, DC mode, 33 kinds of angle mode), and a better motion compensation and vector prediction method.
Inter prediction: refers to the current block to be coded picture predicted from neighboring blocks during a reference image, for removing temporal redundancy of the video signal. There are eight H.265 inter prediction mode, comprising four kinds of ways and four symmetrical division of asymmetric division manner.
3. Better quality and lower bit rate
After Deblock H.265 adds the new "" adaptive compensation samples (Sample Adaptive Offset) "" filter, comprising an edge compensation (EO, Edge Offset), compensating strip (BO, Band Offset), a fusion mode parameters (the Merge), for reducing a distortion between the source image and the reconstructed image, and bit-rate reduction. Test data indicate that although the use of such codecs SAO will increase complexity of about 2%, but can be reduced by 2% to 6% of the stream.
Can be seen, in H.265 offers higher compression ratio, while a lower bit rate, the better the quality, but also increases the complexity of the codec, H.265 decoding Statistics show that the amount of computation have been several times in H.264.
Since this is a live stream of practice for Web playback end, it focuses on decoding of this article.
Hardware solutions and software solutions
Decoding is usually divided into hard and soft decoding decoding. Hard decoding refers to decoding, such as GPU, DSP, FPGA, ASIC chip through dedicated hardware instead of CPU decoding; soft decoding means decodes the decoding software run by the CPU. Strictly speaking there is no pure hardware decoding, because the hard decoding process still need software to control.
Although hardware decoding can get better performance, but because of licensing fees as well as the device supports hardware decoding is also not popular (only part of the GPU supports H.265 hardware decoding on the market). With the rapid increase while the computer CPU performance, H.265 software decoder has begun to be widely used.
Web soft decoding end
Native Support for all major browsers currently on the H.265 playback is not ideal, Web end all major browsers do not support native H.265 playback, Web terminal H.265 player needs to be done through software decoding.
Web soft decoding in the end will first think of using JavaScript. Libde265.js is the JavaScript version of the open source H.265 codec libde265 (exactly the libde265 ASM.JS version, will be explained later). Not an audio and video playback has been tested using libde265.js improve the program, there is a low frame rate and audio and video out of sync issues. In addition, JavaScript acts as an interpreted scripting language, which is not an ideal choice for H.265 decoding this severe CPU-intensive computing task, so continues to explore better programs.
Chrome native audio / video players principle
Find Chromium Projects document, we can see the whole process roughly as follows:
video tag creates a DOM object, instantiating a request for the multimedia data WebMediaPlayerplayer driven Buffer FFmpeg decapsulating the audio and video decoding and the decoded data passed to the corresponding renderer object to render the display tag or card draw video playback
Decoding video object is decompressed to restore the original pixel data into the video, it is to restore the audio decoding mp3 / aac format into the original PCM format. Ffmpeg is an old, cross-platform audio and video processing tool, has a long history, outstanding performance, and has a large FFMPEG-based codec and player. Chrome can also be used to see it as one of its decoder. According to the principle of native Audio / Video player, we can use ffmpeg to implement the playback of H.265.
FFmpeg from the earlier version 2.1 has begun to support for H.265 video decoding, but pepper live is based on the HTTP-FLV H.265 video streams, but so far does not support the official FFmpeg "" HEVC over FLV (and thus RTMP ) "", of course, this is certainly not because of what FFmpeg problems on the technical side, but also because so far no official Adobe supports H.265 FLV to encapsulate data.
HTTP-FLV extension
Belonging to one of the three-FLV HTTP Live Protocol (RTMP and the other two are HLS), the name suggests, the audio and video data is encapsulated FLV format and then transmitted via HTTP protocol. HTTP-FLV low delay, scheduling and load balancing based on the data port 80 can penetrate the firewall stream protocol, and supports HTTP 302.
We mentioned above, do not support the official FFmpeg codec to H.265 FLV format to encapsulate data, but unofficial solutions already exist, such as domestic manufacturers Kingsoft video cloud to do for FFmpeg extended, added support for the FFmpeg H.265 codec functions FLV data package. Thus, the extended FFMPEG can support H.265 data in the FLV format that decodes the HTTP-FLV live stream.
But we know, FFmpeg is developed in C language, how FFmpeg runs on a Web browser, and input to a live stream of data to be decoded it? Using WebAssembly can solve our problem.
WEBASSEMBLY
Webassembly is a new coding method that can be run in a modern web browser - it is a low-level class assembly language with a compact binary format, providing a compilation target for other languages so that they can be on the web. run. It is also designed to coexist with JavaScript, allowing both to work together. In recent years, it has been widely supported by mainstream browsers:
Before understanding the characteristics and advantages of Wasm, first look at the JavaScript code in the V8 engines are running and how they are resolved, which can be divided into the following steps exist between (different versions of the JS engine or engines Some differences):
1.JavaScript Code Parser is converted by the abstract syntax tree AST
According to AST 2.Ignition generate bytecode (V8 engine Without this step before v8.5.9, but directly compiled into machine code, after v8.5.9 Ignition byte code interpreter will be the default startup)
3.TurboFan (JIT) optimizing compiler generates native machine code bytecode
Wherein step (1) AST, more JS code, the longer the time-consuming, but also the whole process a relatively slow link. Wasm itself is already bytecode, no need this part, so the overall running speed is faster.
In step 3, since Wasm data type is already established, and therefore does not require data types JIT assumptions based on information gathered at run time, there is no duplication of optimization period. Further, since Wasm bytecode, to achieve the same functionality than the JavaScript code (even after compression) will be much smaller volume.
Thus, to achieve equivalent functionality Wasm either download speed or running speed will be better than JavaScript. The ASM.js mentioned earlier is essentially JavaScript, and several steps are also subject to the above steps in the JS engine.
So far, there have been many high-level language support has compiled Wasm, from the earliest C / C ++, Rust to later TypeScript, Kotlin, Scala, Golang, or even Java, C # server-side language such veteran. Development of language level support Wasm situation is so flourishing, also shows the development prospects WebAssembly technology to look forward to from the side.
Speaking in front of us, WebAssembly FFmpeg technology can help us to run in the browser, in fact, through our on-demand tool Emscripten, FFmpeg cropped compiled into Wasm files loaded into the page, interact with the JavaScript code.
Practices
Overall architecture / flow diagram
Technical stack
WebAssembly, FFmpeg, Web Worker, WebGL, Web Audio API
Key Description
WASM is used to receive HTTP-FLV live stream data from JavaScript, decoding these data, and then transmits the decoded YUV video data and PCM audio data back to JavaScript by callback mode, and ultimately draw video screens on Canvas through WebGL while playing audio through the Web audio API.
Web Worker
Web Worker Web content in a background thread running scripts provide an easy way. Thread can perform tasks without interfering with the user interface. In addition, they can use XMLHttpRequest to perform I / O. Once created, a worker can send a message to the JavaScript code that created it, the event handler specified by the message posted to the code.
The main thread initializes two Web Worker, Downloader and Decoder, respectively, and a pull-stream decoding, and wherein Decoder Wasm data interaction between the three threads through postMessage communication using the Transferable object when transmitting the data stream is transmitted only references , Not copy data, improve performance.
Downloader uses the Streams API to pull the live stream. Fetch pulling stream data and returns a ReadableStreamDefaultReader object (the default), which can be used to read a stream from one Chunk them. The read method returns a Promise target object can be continuously obtain a set {done, value} value of the object by the Promise, wherein done indicating whether the current stream has ended, if not ended, then, value.buffer i.e. is pulled to the The binary data segment, this data is sent to the Decoder via PostMessage.
Decoder responsible for compiling the FFmpeg Wasm generated by sending the original data to be decoded, and the decoded reception data. Sending raw data to WASM, each data segment into a Uint8Array array, with Module._ma buffer memory alloc allocated the same length, then this data is written to the buffer Module.HEAPU8.set point memory space, and finally passed to the buffer start address Wasm along the length in memory and this data . When receiving data from the decoded WASM, callback data received through the video and audio data as defined in Decoder two Callback callback method, to be followed by the main thread to postMessage transfer.
AudioQueue audio decoding completion queue which will put the main thread, the video will be decoded into the main queue inside VideoQueue thread, the main thread is waiting to read. Role is to ensure smooth playback experience, but also audio and video synchronization.
FFmpeg
FFmpeg is mainly composed of several lib directory consists of:
libavcodec: providing encoding and decoding functions
libavformat: Package (MUX) decapsulation (DEMUX)
libswscale: image scaling and pixel format conversion
First, use libavformat's API to unpack the container to obtain information such as the location of the audio video in this file, and then use libavcodec to obtain image and audio data.
YUV video data presentation
YUV sampling mainly YUV4: 4: 4, YUV4: 2: 2, YUV4: 2: 0 three kinds, each of a Y component respectively represent a group corresponding to UV components, each of two Y components share a UV component, every four Y-components share a UV component, YUV4: 2: 0 yards required minimum stream.
The arrangement of YUV data includes two formats of Planar and Packed. The Planar YUV format successively storing pixel point Y, U, V data; Y of each pixel stored alternately Packed YUV format, U, V data.
Here we decoded video data is YUV420P format, but can not be directly Canvas rendering data YUV format, and can only receive data RGBA format. Converting YUV data to RGBA data, consumes a range of performance. We then rendered to the WebGL treatment by Canvas YUV data, so you can omit the cost of lost data conversion, utilizing the GPU's hardware acceleration to improve performance.
Memory ring / ring buffer (Circular-Buffer)
Live stream is a data source that continuously transmits, unknown total length, and the data taken is temporarily stored before being read by the Decoder Worker. After reading, it will be timely clear or override, otherwise the client will be occupied. more memory and disk resources.
One possible way is to write each data segment pulled into an annular space in the memory, a memory starting address to read data from a Head pointer to each decoding Decoder required, then a pointer to a subsequent Tail stream data segment written to the memory address, and with decoding, continuously moving two rearward position pointer, which would allow the stream data is written to the memory in the ring, is decoded, are covered, so that the overall memory usage control, the client will not take too much resources in the live process.
FFMPEG Custom Data IO
FFMPEG allows developers to customize data IO sources, such as file systems or memory. In our scenario memory to transmit data to be decoded FFmpeg, i.e. by creating a AVIOContext avio_alloc_context, AVIOContext structure is defined as follows:
buffer pointer is pointing to a memory buffer customized;
buffer_size this is the length of the buffer;
write_flag write data that identifies (1, using the encoding) or otherwise, such as read data (0, using decoding) from memory to memory;
pointing the pointer contains a set of opaque custom data source is an optional parameter;
read_packet and write_packet are two callback functions, one for reading and writing to the custom data source from a custom data source, note that these two methods of data to be processed is not empty cycle call;
seek for a custom data source specified in the byte positions. FFmpeg processing by reading data from the decoding IO defined as follows:
Optimization of the volume of Wasm
FFmpeg provides a package for large media format / decompression supports encapsulation, encoding / decoding support, and a variety of protocols, color space, filter, hardware acceleration or the like, may be used ffmpeg command to view the details of the current version of FFmpeg specific information.
Since we mainly practice the H.265 decoding, you can customize FFMPEG only support only the necessary decoders when compiling. Unlike conventional ./configure used when compiling FFmpeg, FFmpeg when compiling to Wasm call requires the use of emconfigure ./configure Emscripten provided:
After such a custom compiled version of FFmpeg, file merge with the decoder C compiler generated Wasm size is 1.2M, 1.4M down to 15 percent from the previous optimization, enhance the loading speed.
Practice results
Web pepper implement end H.265 decoding and playing live streaming. Tested, on the MacBook Pro 2.2GHz Intel Core i7 / 16g memory notebook, using the Chrome browser for a long time, the memory usage is stable between 270m to 320m, and the CPU usage is between 40% and 50%.
Main reference materials or websites
1.FFmpeg official website (http://ffmpeg.org/)
2. About FFMPEG does not support HTTP-FLV / RTMP discussion (http://trac.ffmpeg.org/ticket/6389)
3.WebAssembly official website (https://webassembly.org/)
4. Google V8 engine (https://v8.dev/)
5.Emscripten official website (https://emscripten.org/)
Pepper front-end recruitment
Front-end development intern:
Familiar with Vue, NodeJS, HTML5, CSS3, familiar with the development of excellent learning skills and team communication skills
Voice video engineer / intern:
Familiar h265, h264 video coding standard familiar hls, flv video package and related agreements, re-opened with a player development experience to understand webrtc, webassembly
Resume delivery:
[email protected]
latest events
In the mobile terminal, driven by the cost and efficiency of cross-platform development of one-stop slowly becoming a trend. Flutter is Google's development of a new cross-platform, open-source UI framework, including support for iOS, Android, and other application development platform, the past two years with its multi-platform compatibility, the majority of developers attention and favor.
The event we have invited a number of experts in the industry principle, Flutter use and practical application of technology and other aspects of exploration Flutter from internal technical details Flutter.
Fanger Wei code or click to read the text under the identification, registration immediately
Click on "read the original" Register now
Give me a small flower on the right. "
Our other product:
