In order to make users with slow online speed to talk to HD calls, Ying Weida can be said to race their brains. The new AI algorithm developed, can compress the traffic of the video call by more than 90%.
Compared with other videos, the call scene is relatively single, essentially only the head of the person is moving. Therefore, the flow rate can be greatly saved as long as the avatar data is largely compressed.
Yingda's new algorithm Face Vid2vid is from this point. As long as a picture, you can realize the picture of all kinds of head posture.
The bandwidth required for H.264 video is 2 to 12 times the new algorithm. It can also be seen from the previous demonstration, if the two use the same bit rate, H.264 video is almost unworthy.
Turn the face without distortion
Ying Weida offers a trial DEMO, can rotate in three directions in Pitch, YAW (yaw), ROLL (roll angle).
Enter a face, up to the maximum rotation of 30 degrees in each direction. The following is a picture that rotates to a maximum angle in three directions.
With the previous method, the technology of Ying Weida even when the surface rotation is large, the face does not distort the deformed.
However, the picture is not moving, and it is necessary to put the generated face in a movie video.
Synthetic facial video
We use the uploaded clear photos as a source image, from which the appearance is obtained. Then, a frame screen in the video is used as a basis for reconstructing video, and information such as facial expression and head posture is extracted.
The two data can be encoded by key points, so that human identity information and motion information are separated. Spend a traffic information when transmitting video, thereby saving traffic.
From the source image S, we get two groups of data: key coordinate X and Jacobby matrix J. These two sets of parameters are independent of the specific features of the face, only the geometric characteristics of the person.
Among them, the Elegant matrix represents how to convert local patches around the key to another image by affine transformation. If it is a constant and elevation, the patch will be copied and pasted into the new location.
The following figure shows the process of calculating the top 5 key points. Give a normative key point for the source image and model prediction.
The rotation and displacement from the motion video estimation is applied to the key point, and the changes in the head posture are driven. You can then sense the deformation of the expression to adjust the key point as the target expression.
Next start synthesis video. The key points of the source and motion are estimated to estimate the stream WK from the Jacobby matrix, and the synthesis flow field W can be generated from the generation stream into a mask M.
The output image Y can then be generated by entering the human face feature f.
This approach not only can be used for video calls, but also other "new gameplay".
For example, it feels a little sour, you can manually enter the corrected data to turn the face.
Alternatively, the facial features of a person and the Jacobby matrix are used for another person, and the migration of facial video operations.
Team Profile
The first author of this article is a senior researcher Ting-Chun Wang from Yingda.
The paper author of the article is the famous researcher in Yingda Liu Wei.
If you pay long-term attention to the CV field, you must be very familiar with these two authors. They have made a lot of work in image style migration, GaN.
△ gaugan
There have been many cooperation many times before the two. for example. There is no monitoring image migration network (NIPS 2017), and Gaugan (CVPR 2019) from the graffiti generated photo is from these two hands.
Editor in charge: Psy, Read more
Our other product:
