"Since April 2020, AV1 real-time coding in the independent webrtc and millicast platforms has been available, as described in our previous article. However, for those who want to use it alone in web applications, you must recompile chrome. Although we provided precompiled binaries for the community and a few brave people tested them early, this is a single-layer implementation and does not support SVC.
With the development of the use of codec from closed circuit and dedicated lines to more and more use on the public Internet, codec itself is also developing, and uses some functions to improve the media experience on the public Internet. As the latest appendix of h264 (Appendix g), SVC has developed into a necessary function for any modern codec. By default, AV1 is the first codec to support SVC. For those interested in more details about how SVC works, Dr. Alex E. wrote a good explanatory blog in 2016. Written about VP9, most points are valid for AV1.
Throughout the past year, aomedia's real-time group (part of the code group) has worked hard to complete the RTP payload specification, which allows RTP endpoints to take advantage of all codec SVC functions, but can also be used for intermediate sfus to become better, stronger and faster. Cosmo software took the lead in implementing all test and reference sfus. Now, the payload specification of av1rtp can almost be considered as the final version, which has been tested up to 95% +.
Now is a good time to review why AV1 is more important for real-time media than just improving efficiency. At the same time, we will also provide details about performance expectations.
1 innovation does not occur in an isolated vacuum
In our fast-paced world, it's too easy to focus on small things and ignore the overall situation. However, no innovation occurs in a vacuum like isolation, and the analysis of trends and their trajectories is even more fascinating.
Dr. Alex Eleftheriadis (also known as Dr. Alex) perfectly recorded the development of the whole communication system in his recent article. It is well written and well documented by a person who not only experienced evolution from within, but also had to teach college students. He founded one of the most technologically creative companies in this field: vidyo. I strongly recommend that you read it.
In less than two weeks, the following three major technologies have become standard or can be used in chrome:
On Wednesday, January 20, all IETF rtcweb drafts eventually became standards (or references) and received an RFC number. This represents more than a decade of work, and more than 100 of the smartest people in the world are studying the protocol used as the basis of webrtc. Dozens of new standards generated by hard work have been made public and can be obtained through the web platform!
On January 26, the W3C announced the use of webrtc 1.0 as a standard, which consolidated the standard so that people can safely join and start implementing it. On January 21, Google finally enabled AV1 SVC real-time coding in chrome. In a few hours or days, this feature will be available in Canary version.
On January 21, Google finally enabled AV1 SVC real-time coding in chrome. A few hours or days later, this feature will be available in Canary version.
This is no accident. Real time media needs to integrate multiple elements to work normally. All these elements work and develop in parallel:
Codec with SVC (scalable video coding)
Media engine (codec, coupling of media and network transmission)
Sfus (selective forwarding unit) replaces MCU (multipoint conference unit)
This is a typical example of the whole, which shows that the whole is greater than the sum of the parts. Using these technologies in parallel also has amazing advantages:
Better network resilience
Faster adaptability
Back end no media processing
Support for next-generation new features, such as end-to-end encryption.
2rtc Innovation Track
Bernard aboba, chief architect of Microsoft, once wrote an article about AV1 (the link we added):
"Steve jobs once said," I've always been attracted by more revolutionary changes. I don't know why? Because they are more difficult. "
AV1 aims to integrate with the next wave of webrtc video Innovation: end-to-end encryption, SVC and codec independent forwarding. Therefore, this has nothing to do with the video codec, but with the next generation architecture.
1. With webrtc now merging E2E encryption through pluggable streams (and sframe), and NSA now recommends E2E security, since the payload may be opaque, the conference system needs RTP header extension to forward packets. Therefore, if browsers and codecs do not support pluggable streams or forwarding header extensions integrated with next-generation codecs, NSA requirements will not be met, and conference providers will not be able to provide complete functionality.
2. SVC support is important for meetings. AV1 has built-in SVC; In hevc, it is an extension. The dependency descriptor (defined in the AV1 RTP payload specification) is superior to the framemarking RTP header extension for spatial scalability patterns. If browsers (and next-generation codecs) do not support SVC with forwarding header extensions, it is not competitive.
3. AV1 contains screen coding tools as basic functions, rather than extensions like hevc. This is the main competitive advantage of the conference. "
A. Screen sharing
For text content and ultra-high dynamic content, AV1 is very efficient when encoding screen content. In fact, AV1 real-time performance is so superior that, as Cisco did in WebEx, AV1 real-time may only be deployed in a single use case.
When sharing screens or applications, if "optimize motion and video" is selected, and your machine has at least four cores, AV1 transmission is supported. Any machine with at least two cores supports receiving AV1. As long as all participants in the conference support AV1, AV1 will automatically be used to share such screen content, otherwise it will automatically revert to H.264.
Interestingly, the constraints of 4 and 2 kernels are mentioned here. Cisco also expressed the same view during the live demonstration at the big apple conference in June 2019.
We will continue to discuss performance in the next section, but in order to provide relevant background information, MacBook Air has been using Intel Core-2 chip with 2 cores since 2008, and Intel i7 or higher with 4 or more cores in MacBook Pro since 2011. For laptops and desktops, it is expected that having four cores is not a big problem.
B. End to end encryption
E2ee is the next issue of concern. Maybe it's because this is one of the original promises made by webrtc. Or maybe it has become an overused marketing term, and zoom has become black and blue. Maybe it's because most people still claim to have it, but they don't.
On e2ee, one of the best responses to this question is Emil ivov's speech.
Although many people think that e2ee encryption is only a video conference or chat application function, it is used in the form of the abbreviation "DRM" (Digital Rights Management) throughout the media industry. However, the first mock exam of traditional DRM in browser and media player is not real end to end, but only covers the delivery module. When people upload their content to a platform, they still need to trust the platform (and anyone who can access the platform legally or illegally) with their original content.
True e2ee requires that the media be encrypted at the source when it is encoded and decrypted only when it is played. It allows content providers not to trust the platform.
Webrtc can be widely used by inserting stream API scheme, because it can be used in many aspects. It is an API that enables you to access media and is a necessary step to enable e2ee. However, it has no encryption function or encryption key management function.
The closest to webrtc compatible e2ee media encryption is the proposed IETF sframe standard. It still needs an external system to provide secure external key management. So far, apple reported that at the monthly webrtc interim meeting held on January 18, they added the first version of sframe security implementation to safari. This has received good feedback from Firefox. The Firefox team usually attaches great importance to security features and protecting Internet users. Progress is also being made on web platforms.
The subtlety here is that the design of sframe is forward-looking. When its predecessor perc forces users to enter the old RTP media transmission and is limited to the video conference use case, the sframe is designed as follows:
Do not distinguish between use cases (i.e. for streaming media)
Not related to the agreement (RTP today, quic tomorrow)
Use less bandwidth overhead (than SRTP or PERC)
SVC codec compatible.
C. SVC with sfus and modern media infrastructure
Most people pay attention to the coding efficiency of the new codec:
Using a new codec results in reduced bandwidth usage
Using the same input, you can produce the same quality on the viewer side.
SVC used in the next generation media architecture provides more than these functions.
ABR is not required
SVC provides the ability to generate multi-level resolution from a single encoder in a single bit stream. In other words, SVC makes server-side transcoding and ABR obsolete (although there are still other reasons to transcode VOD server-side).
Since the low resolution content is encoded only once, encoding a layer of bitstream is also more effective than the current layer 3, 5 or 7 independent bitstreams such as simulcast or ABR. In the modern media communication system based on adaptation, it has a great impact on the bottom line.
Better network resilience
For those unfamiliar with the concepts of media transmission and partial reliability, we recommend reading our previous articles on this topic.
There are three main methods to deal with network faults: retransmission, redundancy and forward error correction. Each is a relative compromise:
1. Retransmission assumes that you have time to send packets again and that you reserve a packet cache for each ongoing stream. The advantage is that it's actually easy to implement.
2. Redundancy assumes that you have the ability to use more bandwidth. It won't help if your packet loss is caused by congestion (quantity problem) rather than quality problem.
3. FEC can reduce bandwidth overhead without waiting for retransmission. However, this will increase the complexity of the sender and receiver.
In the layered codec, only the basic layer is very important to the call, and the loss of other layers will only reduce the resolution of a single frame at the receiver.
Therefore, you do not have to protect the entire flow, but only the underlying. This makes FEC more interesting because the complexity is automatically reduced. If red or FEC is used on all packets, the relative bandwidth overhead is only a small part of it.
Moreover, the time frequency of base layer packets is a small part of the stream time frequency, which means you have more time to deal with lost packets. This also makes RTX more attractive.
No matter which network resilience method you use, SVC will only make it more efficient.
Faster (than light) adaptation
Similarly, the function of SFU is relatively simple: to obtain the incoming packet, you need to check which packet should be proxied to a given destination, and then push it to that destination. To decide which package should be proxied to a specific destination, you first need to decide which resolution / layer to proxy, and then make the change.
This decision is usually made according to some enlightenment methods, which are partially based on viewer bandwidth capacity, screen size and device hardware performing resolution / layer changes.
If simulcast is used, the resolution of the stream can be determined according to the source ID (SSRC) of the stream. After providing the resolution mapping of streamid =, SFU decides to stop sending a given stream and send another stream with the same content at a different resolution. In order for the viewer decoder to track changes without artifacts, it needs to wait an entire frame before switching.
SVC codec has a special structure called scalability structure, which defines the dependencies between different layers. This is a codec and bitstream function. In the past few years, a very wise progress has been to replicate and expand this scalable structure at the media transport level.
The ultimate goal is to make decipherability decisions on the decoder in real time!
Due to these additional structures, the SFU can decide whether to discard any packet when receiving it given the target decoding resolution. This is a very powerful function:
By not sending the NACK of non critical packets, the bandwidth usage feedback with the sender is reduced
The use of forward media bandwidth is reduced by not sending redundant packets that will eventually be discarded by the decoder
Change the resolution from one packet to the next (in milliseconds) instead of waiting for the full frame
To be honest, this is the most interesting feature brought by SVC codec so far. This presentation by Emil ivov (again) and his team demonstrates a very intuitive way to understand the advantages it provides in the user experience.
This is a highly technical new function. I won't repeat the technical details here. You can refer to Sergio, the technical director of our media server, for technical details.
3 adoption, performance and expectations
A. Adoption
AV1 as a codec is not a very new thing in technology. It was announced in April 2018. This decoder has been available on desktops and laptops since then.
Netflix and Youtube have adopted this technology and even enabled AV1 playback on their mobile clients in early 2020.
LG and Samsung announced in 2020 that they support decoders in their smart TVs, while Sony just announced that it supports AV1 in 2021 TV models.
From March 2021, all Android TV devices must support AV1 by default.
Our other product:
