"In the front, the audio codes are touched, and learn to make hard coding through MediaCodec. Compact the PCM audio data acquired by AudiRecord to the audio data in the AAC format, then decode to play with the PCM via Audiotrack. Reference Demo Link
We can understand the audio data in front of us, and then how to code audio data and operate these data. The codec for the data of the video also wants to have that image understanding.
In the previous article, we use advanced API MediaRecord to perform video recording including video and audio data. It shields the implementation details of how we have completed the completion of the underlying. But we have to understand the implementation details of the underlying as developers.
This article implements features:
Collecting NV21 data via Camera, encoding a H.264 video file and saves YV12 data encoding the H.264 video file and saves the H.264 video file with the SurfaceView Decoding Display Camera Encoded H.264 Video File Displaying Camera Code Saved by TextureView .264 video file
Video codec foundation
This has already learned in front of it, and then listened to the code of the video. We carefully learn the codec of the video format. The format of the package video has the mainstream of the H.26X series and the MPEG series. When we use Camera to record video, you set up the codec of the audio video of MediaArecord. Code Review:
MMEDiaRecorder.setVideoEncoder (MediaRecorder.videoEncoder.mpeg_4_sp);
mmediaarecorder.setaudiOencoder (MediaRecorder.AudioEncoder.AAC); 12
We have set video encoded by MediaRecorder to encoding the MPEG series, and the recorded video takes out to see if the details are MPEG encoding.
You can see the encoding method of the audio and video we set.
This is the Android's advanced API, and the underlying implementation details are blocked. We don't just know how these, you will know how simple PCM data is known as the data format we want.
Video coding
We also also encoded a similar process in WAV with the original data PCM data of the audio. For video data, the video data data collected by the camera in front is YUV or RGB format. This data format is great because of the original data, which we will compress the size of the video data through various coding methods. And do not use MediaRecorder to do, but by the MediaCodecode of the underlying data from the encoded video data.
After the function of the function we want to implement, after the CAMERA collected the YUV raw data recorded video, the encoded YUV raw data saves the video file of the H.264 encoding format in the H.264 video format.
Code YUV is H.26X encoded video format
We use hard-knitted methods to complete the function, mainly familiar with the use of the API. Just started to encode the camera preview data with the camera preview data with the Camera2, which has experienced a lot of twists and turns, and it is difficult to switch on the format of the preview data. We have to know well for several basic data formats.
YUV data
YUV represents the color space by Y, U, V, y represents brightness, UV represents chromaticity. RGB color spaces have separate RGB three color components per pixel point. YUV is different: YUV is divided into YUV444, YUV422, YUV420, etc. according to the number of UV sampling.
YUV420
Indicates that each pixel point has a separate brightness representation, ie Y; chroma UV component is shared by each four pixel points. For example, a 4x4 picture is in the YUV420 format, there are 16 Y, UV four. YUV420 is divided into different formats based on the storage order of UV chroma. It is divided into two Class of YUV420P and YUV420SP, YUV420P UV order storage, YUV420SP UV interlaced storage. Take 4x4 picture format as an example part of the format as follows:
Name Data Storage Orientation Big Class I420YYYYYYYYYYYYYYYUUUUUUUVVV220PYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYUVUVUVUVUV420SP
Coding main code
Initialization MediaCodec
Public Avcencoder (int Width, int Height, int framete, file outfile, boolean iscamera) {
THIS.MISCAMERA = ISCAMERA;
MWIDTH = Width;
MeiHEight = Height;
Mframerate = Framerate;
Moutfile = OUTFILE;
Mediaformat MediaFormat = MediaFormat.createvideoFormat ("VIDEO / AVC", Width, Height)
MediaFormat.setinteger (MediaFormat.Key_Color_Format,
Mediacodecinfo.codeccapabilities.color_formatyuv420semiplanar);
Mediaformat.setinteger (MediaFormat.key_bit_Rate, Width * Height * 5);
MediaFormat.setinteger (MediaFormat.key_frame_rate, 30);
MediaFormat.setinteger (MediaFormat.key_i_frame_interval, 1);
Try {
MMEDIACODEC = Mediacodec.createEncoderbyType ("VIDEO / AVC");
} catch (ioexception e) {
E.PrintStackTrace ();
}
MMEDIACODEC.CONFIGURE (MediaFormat, null, null, mediacodec.configure_flag_encode);
MMEDIACODEC.START ();
Createfile ();
} 12345678910111213141516171819202122
Start encoding:
Public void startencoderthread () {
Thread EncodeRThread = New Thread (New Runnable () {
@Suppresslint ("NewAPi")
@Override
Public void run () {
Isorunning = true;
BYTE [] INPUT = NULL;
Long PTS = 0;
Long generateIndex = 0;
While (isrunning) {
IF (myuvqueue.size ()> 0) {
INPUT = Myuvqueue.Poll ();
IF (Miscamera) {
// NV21 data The space required is as follows, so the buffer is established.
Byte [] yuv420sp = new byte [mWidth * MHEIGHT * 3/2];
NV21TONV12 (INPUT, YUV420SP, MWIDTH, MWIGHT);
INPUT = yuv420sp;
} else {
Byte [] yuv420sp = new byte [mWidth * MHEIGHT * 3/2];
YV12TONV12 (INPUT, YUV420SP, MWIDTH, MWIDTH, MWIDTH, MHEIGHT);
INPUT = yuv420sp;
}
}
IF (Input! = null) {
Try {
Bytebuffer [] InputBuffers = mmediacodec.getinputbuffrs ();
INT INPUTBUFFERINDEX = mmediacodec.dequeueInputBuffer (-1);
IF (InputBufferIndex> = 0) {
PTS = ComputePresentationTime (GenerateIndex);
Bytebuffer InputBuffer = INPUTBUFFERS [InputBufferIndex];
InputBuffer.clear ();
InputBuffer.Put (Input);
MMEDIAcodec.queueInputBuffer (InputBufferIndex, 0, Input.length, PTS, 0);
GenerateIndex + = 1;
}
Bytebuffer [] outputbuffers = mmediacodec.getoutputbuffrs ();
Mediacodec.bufferinfo bufferinfo = new mediacodec.bufferinfo ();
Int outputBufferIndex = mmediacodec.dequeueoutputbuffer (bufferInfo, timeout_usec);
While (OutputBufferIndex> = 0) {
// LO G.i ("Avcencoder", "" Get H264 Buffer Success! Flag = "+ BufferInfo.flags +" ", PTS =" + BufferInfo.PresentationTIMEUS + "" ""
BYTEBUFFER OUTPUTBUFFER = OUTPUTBUFFERS [OUTPUTBUFFERINDEX];
Byte [] Outdata = new byte [bufferinfo.size];
OutputBuffer.get (Outdata);
IF (bufferinfo.flags == 2) {
Mconfigbyte = new byte [bufferinfo.size];
Mconfigbyte = Outdata;
} else if (bufferinfo.flags == 1) {
Byte [] KeyFrame = new byte [bufferinfo.size + mconfigbyte.length];
System.Arraycopy (Mconfigbyte, 0, KeyFrame, 0, Mconfigbyte.length);
System.ArrayCopy (Outdata, 0, KeyFrame, MconfigbyTe.length); Outdata.length
OutputStream.write (KeyFrame, 0, KeyFrame.Length);
} else {
OutputStream.write (Outdata, 0, Outdata.length);
}
MMEDIACODEC.RELEASEOUTPUTBUFFER (OutputBufferIndex, False);
OutputBufferIndex = mmediacodec.dequeueoutputbuffer (bufferinfo, timeout_usec);
}
} catch (throwable t) {
T.PrintStackTrace ();
}
} else {
Try {
Thread.sleep (500);
} catch (interruptedexception e) {
E.PrintStackTrace ();
}
}
}
}
});
Encoderthread.start ();
}
Stop coding
Public void stopthread () {
IF (! isrunning) return;
Isorunning = false;
Try {
STOPENCODER ();
IF (OutputStream! = null) {
OutputStream.flush ();
OutputStream.Close ();
OutputStream = NULL;
}
} catch (ioexception e) {
// Todo Auto-Generated Catch Block
E.PrintStackTrace ();
}
} 123456789101112131415
This code encoding this use of MediaCodec is hereby
I have made a small distinction here:
When using Camera, set the preview of the data format to NV21 when using Camera2, you cannot support preview data as NV21, I set the preview data format YV12
Why did you do the following code processing after setting?
IF (myuvqueue.size ()> 0) {
INPUT = Myuvqueue.Poll ();
IF (Miscamera) {
// NV12 data The space required is as follows, so the buffer is established.
// y = w * h; u = w * h / 4; v = w * h / 4, so total add is w * h * 3/2
Byte [] yuv420sp = new byte [mWidth * MHEIGHT * 3/2];
NV21TONV12 (INPUT, YUV420SP, MWIDTH, MWIGHT);
INPUT = yuv420sp;
} else {
Byte [] yuv420sp = new byte [mWidth * MHEIGHT * 3/2];
YV12TONV12 (INPUT, YUV420SP, MWIDTH, MWIDTH, MWIDTH, MHEIGHT);
INPUT = yuv420sp;
}
} 1234567891011121314
It can be seen that I convert the Camera's NV21 preview data and Camera2's YV12 preview data into data in NV12 format. And when I encode the picture for H.264 video, I didn't have a little green on the above situation. So I got this conclusion:
H.264 Encoding must use NV12, so we get the preview data to do the format to convert this conclusion, the conclusion of some articles I refer, but my practice code is my conclusion code successful output format. H.264 file, and think that my conclusion is correct.
OK finally finished this thing, as for the code for obtaining preview data, the previous implementation of Camera is very simple, and Camera2 is obtained through ImageRender. The generated video is very happy through the VLC player, very happy. Complete DEMO is posted at the bottom.
Video decoding
The problem that the video decoding I feel is the process of formatting the video. For the MediaCodec decoding of the video I feel that the problem I encountered is to blurred the long conceptual problem of the video displayed.
Decoding the main code
Initialization MediaCodec
Public voidinitcodec () {
File f = new file (mfilepath);
IF (null == f || f.exists () || f.Length () == 0) {
TOAST.MAKETEXT (MCONTEXT, "" Specify the file does not exist "", toast.length_long) .show ();
Return;
}
Try {
// Get file input stream
MINPUTSTREAM = New DataInputStream (New FileInputStream (New File (MfilePath)));
} catch (filenotfoundexception e) {
E.PrintStackTrace ();
}
Try {
// Create an available decoder through multimedia format name
McOdec = Mediacodec.createdEcoderbyType ("VIDEO / AVC");
} catch (ioexception e) {
E.PrintStackTrace ();
}
// Initialization encoder
Final Mediaformat MediaFormat = MediaFormat.createvideoFormat ("VIDEO / AVC", MvideoWidth, MVIDEOHT);
/ / Get PPS and SPS data in H264
IF (iSUSEppsandsps) {
Byte [] header_sps = {0, 0, 0, 1, 103, 66, 0, 42, (byte) 149, (byte) 168, 30, 0, (byte) 137, (byte) 249, 102, (Byte 224, 32, 32, 32, 64};
Byte [] header_pps = {0, 0, 0, 1, 104, (byte) 206, 60, (byte) 128, 0, 0, 0, 1, 6, (byte) 229, 1, (Byte) 151, (BYTE) 128};
MediaFormat.setBytebuffer ("CSD-0", ByteBuffer.wrap (Header_SPS));
MediaFormat.seTbytebuffer ("" "CSD-1", Bytebuffer.wrap (Header_PPS));
}
// Set the frame rate
Mediaformat.setinteger (MediaFormat.key_Frame_Rate, MFramerate);
McOdec.configure (MediaFormat, Msurface, NULL, 0);
} 12345678910111213141516171819202122232425262728293031
Open decoding thread
Public class decodeh264thread implements runnable {
@Override
Public void run () {
Try {
Decodeloop ();
} catch (exception e) {
}
}(Files! = null && files.length> 0) {
Mfile = files [0];Tencoderthread ();
TOAST.MAKETEXT (MCONTEXT, "" Start recording video success "", toast.length_short.show ();
}
MAVCENCODER.PUTYUVDATA (DATA);
Break;
}
} 1234567891011121314151617181920212223242526272829303333334353637
Reflecting the above code, I set up the Camera's mcapera.setdisplayorientation method to rotate 90 degrees, but to note that the data in the onpreviewFrame has not selected 90 degrees, and we set up MediaCodec, it determines our code. The final width of the video of the video file. This explains our video to the VLC of the PC side. Playback is to play 90 degrees in the form of a left.
It turns out that the data of each frame of my encoded source video file is data with Camera Sensor under the vertical direction. It is necessary to pass clockwise 90 times is the effect of our physics world, through code. Select, and set the width and height of the encoding, OK. Perfectly solve the problem of decoding playing video under our vertical screen.
One of the best widths we have obtained is higher than the height. This is obtained through the Camera's API, and its measurement is seen through the left side of Camera Sensor. So we get and set the best display width, such as such a 1920x1080.
Note that the source file of the video I decoded is the decoding process of the source file generated by the preview of Camera2, and I am using the decoding process of the source file generated by Camera2 preview, and I am using the millet 5 Android 7.0 system. Get a best display width size through the CAMERA-related API is 1920x1080, and get the best width of 1440x1080 through the CAMERA2 related API. When decoding the video file, I fixedly write about MediaCodec's width height is 1920x1080.
Complete Demo: View Reference Link: True God's summary and differences for audio and video encoding and differences for YUV format H.264 video decoding reference
}
IF (mfile == NULL) {
Toast.maketext (decodeh264tosurfaceviewActivity.this, "" Video file does not exist, first "", toast.length_short.show ();
Return;
}
MavcDecoder = New avcdecodertosurface (Mhandler,
Decodeh264tosurfaceviewActivity.this, mfile.getabsolutepath (),
Holder.getsurface (), 1080, 1920, 30);
mavcdecoder.initcodec ();
}
}
@Override
Tencoderthread ();
TOAST.MAKETEXT (MCONTEXT, "" Start recording video success "", toast.length_short.show ();
}
MAVCENCODER.PUTYUVDATA (DATA);
Break;
}
} 1234567891011121314151617181920212223242526272829303333334353637
Reflecting the above code, I set up the Camera's mcapera.setdisplayorientation method to rotate 90 degrees, but to note that the data in the onpreviewFrame has not selected 90 degrees, and we set up MediaCodec, it determines our code. The final width of the video of the video file. This explains our video to the VLC of the PC side. Playback is to play 90 degrees in the form of a left.
It turns out that the data of each frame of my encoded source video file is data with Camera Sensor under the vertical direction. It is necessary to pass clockwise 90 times is the effect of our physics world, through code. Select, and set the width and height of the encoding, OK. Perfectly solve the problem of decoding playing video under our vertical screen.
One of the best widths we have obtained is higher than the height. This is obtained through the Camera's API, and its measurement is seen through the left side of Camera Sensor. So we get and set the best display width, such as such a 1920x1080.
Note that the source file of the video I decoded is the decoding process of the source file generated by the preview of Camera2, and I am using the decoding process of the source file generated by Camera2 preview, and I am using the millet 5 Android 7.0 system. Get a best display width size through the CAMERA-related API is 1920x1080, and get the best width of 1440x1080 through the CAMERA2 related API. When decoding the video file, I fixedly write about MediaCodec's width height is 1920x1080.
Complete Demo: View Reference Link: True God's summary and differences for audio and video encoding and differences for YUV format H.264 video decoding reference Public Void SurfaceChanged (Surfaceholder Holder, int format, int width, int {
Log.e (tag, "" SurfaceChanged: "");
}
@Override
Public void SurfaceDestroyed (Surfaceholder Holder) {
Log.e (tag, "" SurfaceDestroyed: "");
}
});
Mbutton = findviewbyid (r.id.button);
12345678910111213141516171819202122232425262728293033383334414444344454647484950
Problem Description
I use SurfaceView to display the decoded data to be displayed, but the video is a display mode for rotating 90 degrees left on the vertical screen? Why is this? And my video is sent to the computer to display the way to select 90 on the left side of the vertical screen?
This problem I first thought about it. The SurfaceView clockwise 90 degrees should be, so using TextureView to display the decoded data, the key code is as follows:
Private void initTextureView () {
MTEXTUREVIEW.SETSURFACETEXTURELISTENER (New TextureView.SurfaceTextureListener () {
@Override
Public void OnsurfaceTextureavailable (SurfaceTexture Surface, Int Width, INT Height) {
// matrix matrix = new matrix ();
// Matrix.postrotate (90, mtextureview.getwidth () / 2, mtextureview.getHEight () / 2);
// mTextureView.SetTransform (Matrix);
IF (mavcdecoder == null) {
File file = getExternalFileSDir (Environment.directory_movies);
FILE [] files = null;
IF (file.exiss ()) {
FILES = file.listfiles (new filefilter () {
@Override
Public Boolean Accept (File Pathname) {
Return Pathname.GetabsolutePath (). Endswith ("" "" "" "" ");
}
});
}
IF (files! = null && files.length> 0) {
Mfile = files [0];
}
IF (mfile == NULL) {
TOAST.MAKETEXT (Decodeh264 TotextureViewActivity.this, "" Video file does not exist, Mr. "", toast.length_short.show ();
Return;
}
MavcDecoder = New avcdecodertosurface (Mhandler,
Decodeh264totextureViewActivity.this, mfile.getabsolutePath (),
New Surface (Surface), 1920, 1080, 30);
mavcdecoder.initcodec ();
}
}
@Override
Public void OnsurfaceTextureSizechanged (SurfaceTexture Surface, Int Width, Int Height) {
}
@Override
Public Boolean OnSurfaceTextureDestroyed (SurfaceTexture Surface) {
IF (mavcdecoder! = null) {
MAVCDECODER.STOPDECodingthread ();
}
Return False;
}
@Override
Public void OnsurfaceTextureUpdated (SurfaceTexture Surface) {
}
});
}
I thought it was possible to witness the effect, the video was indeed turned 90 degrees "positive". But the display is high or not, and it is seriously pulled. What kind of, the picture describes:
I feel that this problem has been seen from the source. I have to correct from the output. My video decoding is only passed from the preview data Demo test from Camera. So review the relevant Camera's associated wide code
....
// Set the size of the preview size onpreviewFrame
Parameters.SetPreviewSize (Mbestsize.Width); MbESTSIZE.HEIGHT
// Set the photo output picture size
Parameters.SetPicturesize (Mbestsize.width) and Mbestsize.Height;
Int rotationdegrees = getcameraDisplayorientation (Activity) MCONTEXT, MCAMERAID;
Log.e (tag, "" INITCAMERA: ROTATION Degrees = "+ rotationdegrees);
Mcamera.SetDisplayorientation (RotationDegrees);
Parameters.SetPreviewFormat (Imageformat.nv21);
....
@Override
Public void onpreviewframe (byte [] data, camera badra) {
Switch (mstate) {
Case State_Preview:
IF (MAVCENCODER! = NULL) {
MAVCENCODER.STOPTHREAD ();
MAVCENCODER = NULL;
TOAST.MAKETEXT (MCONTEXT, "Stop Recording Video Success" "", Toast.length_Short) .show ();
}
Break;
Case State_Record:
Log.e (Tag, "" OnPreviewFrame: Record Video ");
IF (MAVCENCODER == NULL) {
MAVCENCODER = New Avcencoder (Mbestsize.width,
Mbestsize.height, mframerate,
getOutputmediafile (Media_Type_Video), TRUE);
MAVCENCODER.STAR
Private void decodeloop () {
/ / Store data for the target file
Bytebuffer [] InputBuffers = McODec.getinputBuffers ();
// Decoded data, contain metadata information of each buffer, such as deviation, effective data size in the correlation decoder
Mediacodec.bufferinfo info = new mediacodec.bufferinfo ();
Long Startms = System.currentTimeMillis ();
Long Timeoutus = 10000;
Byte [] marker0 = new byte [] {0, 0, 0, 1};
Byte [] DummyFrame = new byte [] {0x00, 0x00, 0x01, 0x20};
Byte [] streambuffer = null;
Try {
StreamBuffer = getBytes (minputstream);
} catch (ioexception e) {
E.PrintStackTrace ();
}
INT BYTES_CNT = 0;
While (mstartflag == true) {
BYTES_CNT = streambuffer.length;
IF (Bytes_cnt == 0) {
streambuffer = DummyFrame;
}
INT StartIndex = 0;
INT Remaining = bytes_cnt;
While (true) {
if (Remaining == 0 || StartIndex> = Remaining) {
Break;
}
Int nextframestart = kmpmatch (Marker0, StreamBuffer, StartIndex + 2, Remaining);
IF (NextFrameStart == -1) {
Nextframestart = Remaining;
} else {
}
ININDEX = McODec.dequeueInputBuffer (TIMEOUTUS);
IF (inindex> = 0) {
Bytebuffer Bytebuffer = InputBuffers [inindex];
BYTEBUFFER.CLEAR ();
Bytebuffer.put (Streambuffer, StartIndex, NextFrameStart - StartIndex);
/ / Call this function to the decoder to the decoder to the decoder after the data is filled with the data of the INDEX's inputBuffer []
McOdec.queueInputBuffer (InIndex, 0, NextFramestart - StartIndex, 0, 0);
StartIndex = nextframestart;
} else {
CONTINUE;
}
Int outindex = McODec.dequeueoutputBuffer (Info, Timeoutus);
IF (OutIndex> = 0) {
// Frame control is not working in this case, because there is no PTS H264 available
While (Info.PresentationTimeus / 1000> system.currenttimemillis () - startms) {
Try {
Thread.sleep (100);
} catch (interruptedexception e) {
E.PrintStackTrace ();
}
}
Boolean Dorender = (Info.Size! = 0);
// After the processing of OutputBuffer, call this function to re-return the buffer to the CODEC class.
McOdec.releaseoutputBuffer (OutIndex, Dorender);
}
}
MStartflag = false;
Mhandler.sendemptyMPTYMESSAGE (0);
}
}
}
Stop decoding thread
Public void stockDecodingsthread () {
MStartflag = false;
IF (McOdec! = null) {
McOdec.stop ();
McOdec = NULL;
Try {
MDECodethread.join ();
MDECodethread = NULL;
} catch (interruptedexception e) {
E.PrintStackTrace ();
}
}
} 12345678910111213
Key code reference for the above H.264 decoded from this blog post
SurfaceView calls decoding code
@Override
Protected void oncreate (bundle savedinstancestate) {
Super.onstance (Savedinstancestate);
SetContentView (r.layout.Activity_Decode_h264_to_surface_view);
MsurfaceView = FindViewbyid (r.id.surfaceview);
MsurfaceView.setkeepscreenon (TRUE);
Surfaceholder Holder = MsurfaceView.getHolder ();
Holder.addCallback (New Surfaceholder.callback () {
@Override
Public void SurfaceCreated (Surfaceholder Holder) {
IF (mavcdecoder == null) {
File file = getExternalFileSDir (Environment.directory_movies);
FILE [] files = null;
IF (file.exiss ()) {
FILES = file.listfiles (new filefilter () {
@Override
Public Boolean Accept (File Pathname) {
Return Pathname.GetabsolutePath (). Endswith ("" "" "" "" ");
}
});
}
IF
Our other product:
