Month: June 2017

GSoC 2017 video stuck issue fixed

After last week’s work the decoder is now able to play videos normally. The video getting stuck at the end was fixed along with other improvements that I talk about below.

Signalling EOS

Earlier I had left signalling end of stream for later to do. This only needed to propagate the flag on the ouput buffer as

p_prc->p_outhdr_->nFlags |= OMX_BUFFERFLAG_EOS;

But it was a little tricky because of the sliding window. The decoder should not release the output buffer before all the buffers in the window have been decoded when EOS is reached. https://github.com/gpalsingh/mesa/commit/69985535972593a4ccd9630efcb3106f8737a8d2 fixed that issue.

That along with the error correction in https://github.com/gpalsingh/mesa/commit/97995b031149ad7d709aca908fd1d263579c1560 made the video close smoothly on ending.

No more dependence on gst-omx for port configuration

For development earlier we were using workarounds for nStride other parameters. Also the component depended on the client to update the corressponding ports. With the patch in https://bugzilla.gnome.org/show_bug.cgi?id=783976 that behaviour was changed. Removing the workaround and using this patch the component stopped working. It was because the outport needed to be enabled first by the component before sending any output. https://github.com/gpalsingh/mesa/commit/32f535834fb7c11095830e49380a3612dc6e8563 made it able to start the decoding process but later on I was again faced another problem.

The decoder assumed that the input buffer is cleared after it’s read. But now in case the out port is disabled the input buffer can’t be cleared. In that case the decoder needs to wait until the right time to release the claimed buffer. https://github.com/gpalsingh/mesa/commit/868577f36ebb6435615938ccae19e90abee237bb fixed that issue by adding checks in decode_frame and get_input_buffer and separating the buffer shifting logic from decode_frame.

Reading stream parameters from headers In https://github.com/gpalsingh/mesa/commit/32f535834fb7c11095830e49380a3612dc6e8563 the decoder sends a “fake” event to just enable the out port as

    tiz_srv_issue_event ((OMX_PTR) p_prc, OMX_EventPortSettingsChanged, OMX_VID_DEC_AVC_OUTPUT_PORT_INDEX, OMX_IndexParamPortDefinition, /* the index of the struct that has been modififed */ NULL);

while reading the header. https://github.com/gpalsingh/mesa/commit/0d79ca5b760ee4b9c3655c94a56ef75065a329fa changes this by actually reading the values and updating them. This also removed dependency on the h264d_SetParameter workaround which was removed.

Moving on

The component works fine with normal streams but fails with streams which change parameters. Also while seeking there is a possibility that the out port will fail to flush and a timeout will occur. The major goal now is to add support for OMX_UseEGLImage which allows faster decoding process.

Advertisements

GSoC 2017: Working H.264 decoder

Last week we had the decoder reading the stream but the output was all wrong. It was just a green screen as below. The reason was the output buffer being filled with 0s.

omxoutput.png

After this week’s work the decoder is able to play the video

videorun

 

Now let’s discuss briefly what I’ve been working on this week.

Buffer management

The first thing that I did was to add proper buffer management to the decoder. At first all the work related to clearing buffers was being done either in h264d_prc_buffers_ready or in decode_frame. Now that is done in h264d_manage_buffers in a single place. It currently doesn’t handle the case when when stream reaches  the end.

Port disable and enable

Another little addition was the ability to enable and disable ports. This change was done in parallel to tizonia’s vp8dec which is being used as guide for the new mesa/gallium component. https://github.com/gpalsingh/mesa/commit/6b33081cb3bfdd90b2d2944276a51386e06472a2

Correcting the video output

The main task was to show the correct output instead of the green screen. This took majority of my time and involved two steps.

Debugging green screen

First we needed to find out why it was happening. After checking the inputs I found that the decoder was doing the decoding work right so the the error most probably lied in the output process. As Julien pointed out, in YUV plane output from a buffer with all 0s is green. Following this clue I found that the one of the stride values was 0.

https://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/auxiliary/util/u_surface.c#n104

for (i = 0; i < height; i++) {
         memcpy(dst, src, width);
         dst += dst_stride;
         src += src_stride;
      }

Which meant that the it was failing to write anything to the buffer.

Both tizonia’s vp8 decoder (vp8dec) and mesa/gallium’s bellagio omxh264dec have different methods to update the value.

Tizonia’s vp8dec reads the stream information from the first buffer in frame, updates the nStride (along with other parameters) and notifies that the output port settings have been changed.

omxh264dec replaces the component handle’s SetParameter function to set the nStride whenever the index is OMX_IndexParamPortDefinition.

Adding the ability to set nStride and nFrameWidth to correct values

The ideal way to fix it was to do it like how vp8dec does (read info from stream and update port parameters). This needed inspecting the gst-omx/gstreamer code to dig out how the information is parsed since omxh264dec currently does not do so. This turned out to be pretty time consuming so I came up with a quick fix to replace the component handle’s SetParameter with a different version like omxh264dec does. The update needs to be done on input port since the output port parameters get overridden due to slaving behaviour.

What still doesn’t work

Even thought it works there are still some improvements still to be made. The video gets stuck at the end and the output also has unexpected thin black strips on top and bottom sometimes. In addition to that the way the decoder sets the output port parameters currently is not ideal which I’ll discuss in detail later on. The decoder shows wrong output for stream where the resolution changes on the fly.

Moving forward

This week our focus will be first on fixing the video being stuck at the end issue. Next will come finding a better solution to set the output port parameters.

GSoC 2017 progress so far

Hi,

This post serves as my first main post relating to the GSoC project I’ve been working on. Earlier I discussed how to set up the development environment.

Introduction

Recently I got accepted as a student for Google Summer of Code 2017 to work on Mesa to add a new OpenMax IL state tracker that uses tizonia instead of bellagio that the current state tracker uses.

Project link: https://summerofcode.withgoogle.com/projects/#4737166321123328

Project proposal: https://docs.google.com/document/d/1BM1LdSPVAQRsle1O_jj9Ssq52Z0HHlY8XwYY5y9j6Lk/edit?usp=sharing

Progress so far

Most of the effort till now has been to make tizonia compatible with gst-omx so that we can test out the new component.

Changes in gst-omx

The commits are are on my github fork https://github.com/gpalsingh/gst-omx/commits/tizonia

The first thing needed to use gst-omx to test for OMX IL 1.2 was to add tizonia as a target. https://github.com/gpalsingh/gst-omx/commit/0aff53c9f01f04a09b41b01a313afa7255ad20a1 is the last of the three commits to do the same. They simply involve adding tizonia target in addition to bellagio, generic and rpi. Right now it needs external IL headers unlike other targets that can fall back to internal headers in case no external IL headers are available.

Next we worked on making it work with the mp3 decoder component that comes with tizonia. Luckily that needed only small change to ignore an error https://github.com/gpalsingh/gst-omx/commit/2f8be06754077847c3f0a0862e6336fe37907e5e

Right now work on getting the tizonia vp8 decoder to work with gst-omx is in progress. It is possible to play the video but there are still some improvements to be made.

Changes to tizonia

Since tizonia is more aligned to audio cases it is was still missing and AVC port. As Juan suggested in our discussion here https://github.com/tizonia/tizonia-openmax-il/issues/319#issuecomment-304410741, I added the new AVC port to tizonia which was merged after Julien fixed some minor errors in it https://github.com/tizonia/tizonia-openmax-il/commit/9c9e6d0f74b6c80984b48d71c46ae52d46d83b7d.

This mostly involved changing the existing code from the VP8 port according to the AVC port as given in the OMX IL specifications. It would need still more work later on for initialising the structures that depend on different profile levels which is currently marked as TODO.

New component

This work is the main project work involving adding the new component to mesa/gallium.

Main branch

On this branch I keep the final work that is (almost) ready to be commited. It is located at https://github.com/gpalsingh/mesa/tree/gsoc

Currently it is complete up to adding an empty component to the omx-tiz state tracker, the openmax state tracker that uses tizonia. It can be used by passing the –use-omx-tiz argument to autogen.sh script.

Dev branch

This is the branch that I do all the experimental work on. It is located at https://github.com/gpalsingh/mesa/tree/gsoc-dev

To avoid confusion this branch uses omx_tizonia instead of omx_tiz in all places.

I’ve added an H.264 decoder which works with tizonia. It uses most of the code from the existing bellagio H.264 state tracker. Bellagio uses a private class (priv) to hold the decoder data while tizonia uses a processor (p_prc) for similar purpose. So it just needed to replace priv with p_prc.

Currently I’m working on improving the decode_stream function that decodes the input stream and shows the ouput https://github.com/gpalsingh/mesa/blob/gsoc-dev/src/gallium/state_trackers/omx_tizonia/h264dprc.c#L1242

In bellagio the decoding and buffer management take place in two different functions (vid_dec_DecodeBuffer and vid_dec_FrameDecoded) whereas in tizonia only one function does the work of both (*_buffers_ready).

I’ve used h264d_frame_decoded as replacement for bellagio’s base_port_SendBufferFunction since while checking with gdb I noticed it invokes vid_dec_FrameDecoded which performs the output and is the buffer management function for bellagio. https://github.com/gpalsingh/mesa/blob/gsoc-dev/src/gallium/state_trackers/omx_tizonia/h264dprc.c#L1296

For replacing port->ReturnBufferFunction (https://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/state_trackers/omx/vid_dec.c#n519) I’m trying out h264d_release_input_buffer.

Trying with that gst-omx compiled for tizonia shows a blank window and the execution stops after that (without any error thrown). My focus will be on fixing this in the upcoming week and hopefully have a working decoder soon.

Setting up development environment for Gallium state tracker development

In this post we will be looking at the process to set up the development environment that I used for working with state trackers in Mesa/Gallium. We will be using gst-uninstalled as the uninstalled environment.

Get the main packages

These are the packages we’ll be testing. We’ll be using Mesa and libva. Mesa has the state trackers and libva because Mesa uses it. We won’t be installing them now as the testing environment still needs to be set up.

First make a folder where all the stuff goes.

$ mkdir ~/tutorial
$ cd ~/tutorial

Then clone mesa into it.

$ git clone git://anongit.freedesktop.org/git/mesa/mesa

Next is libva.

$ git clone https://anongit.freedesktop.org/git/vaapi/libva.git

Install bellagio

$ sudo apt install libomxil-bellagio*

Install tizonia (originally from https://github.com/tizonia/tizonia-openmax-il#installation)

$ curl -kL https://github.com/tizonia/tizonia-openmax-il/raw/master/tools/install.sh | bash
# Or its shortened version:
$ curl -kL https://goo.gl/Vu8qGR | bash
$ sudo apt-get update && sudo apt-get upgrade

Setting up testing environment

Get Gstreamer

(From the official homepage) GStreamer is a library for constructing graphs of media-handling components. Applications can take advantage of advances in codec and filter technology transparently. Developers can add new codecs and filters by writing a simple plugin with a clean, generic interface. Here we will be using the gst-omx and some other plugins for testing.

First build the dependencies for Gstreamer.

$ sudo apt-get build-dep gstreamer1.0-plugins-{base,good,bad,ugly}

Get the installer script

$ curl -O https://cgit.freedesktop.org/gstreamer/gstreamer/plain/scripts/create-uninstalled-setup.sh

Edit the script and add gst-omx and gstreamer-vaapi to “MODULES”

MODULES=”gstreamer gst-plugins-base gst-plugins-good gst-plugins-ugly gst-plugins-bad gst-libav gstreamer-vaapi gst-omx”

or you can clone them later manually. Other options like BRANCH and UNINSTALLED_ROOT can also be changed to your liking.

Run the script and wait for it to finish.

$ sh create-uninstalled-setup.sh

This will install the plugins in ~/gst directory.

Using the environment

The script will give you additional information before exiting which you should take note of. The instructions here are general.

Enter the environment with

$ ~/gst/gst-master.

This will take you a to ~/gst/gst-master in the uninstalled environement (a bit similar to python’s venv).

Use

$ exit

to exit out of the uninstalled environment anytime.

To make it easier to use you can link it to bin in your home.

$ mkdir ~/bin; ln -s ~/gst/gst-master ~/bin/gst-master

And then add this line in your bashrc / bash_profile

export PATH=$PATH:~/bin

Load the changes

$ source ~/.bashrc

Now you can simply use it like a command

$ gst-master

More details could be found here: https://arunraghavan.net/2014/07/quick-start-guide-to-gst-uninstalled-1-x/

Firmware installation (Optional)

You need to do this only if you are using NVIDIA graphics card like I did.

Use the following commands to install the firmware for your NVIDIA graphics card:

$ mkdir /tmp/nouveau
$ cd /tmp/nouveau
$ wget https://raw.github.com/imirkin/re-vp2/master/extract_firmware.py
$ wget http://us.download.nvidia.com/XFree86/Linux-x86/325.15/NVIDIA-Linux-x86-325.15.run
$ sh NVIDIA-Linux-x86-325.15.run --extract-only
$ python2 extract_firmware.py  # this script is for python 2 only
# mkdir /lib/firmware/nouveau
# cp -d nv* vuc-* /lib/firmware/nouveau/

More details could be found on the official page: https://nouveau.freedesktop.org/wiki/VideoAcceleration/

Installing modules

Now that we have the base set up, we can proceed to installing the packages to finish the setup process. Note: You should be uninstalled environment before proceeding.

Install Mesa

First move to the mesa directory

$ cd ~/tutorial/mesa

Then use the autogen.sh script to configure everything

./autogen.sh \
 --enable-texture-float \
 --enable-gles1 \
 --enable-gles2 \
 --enable-glx \
 --enable-egl \
 --enable-gallium-llvm \
 --enable-shared-glapi \
 --enable-gbm \
 --enable-glx-tls \
 --enable-dri \
 --enable-osmesa \
 --with-egl-platforms=x11,drm \
 --with-gallium-drivers=nouveau,swrast \
 --with-dri-drivers=nouveau,swrast \
 --enable-vdpau \
 --enable-omx \
 --enable-va \
 --prefix=/home/gurkirpal/gst/master/prefix/

Note the the prefix is ~/gst/master/prefix. It is necessary to use it otherwise mesa will be installed in default location. In case you used a different UNINSTALLED_ROOT and/or BRANCH you should change it accordingly. Also nouveau is in case for NVIDIA graphics card which is the driver.

Finally make and make install.

$ make -j8
$ make install

This will install mesa in ~/gst/master/prefix and could be used in the environment. This means even if you break something during testing your system wide installation will be unharmed.

Install libva

Similar to Mesa, we just need to give the prefix before configuring

$ cd ~/tutorial/mesa
$ ./autogen.sh --prefix=/home/gurkirpal/gst/master/prefix/
$ make -j8
$ make install

Instal Gstreamer and modules

The modules downloaded by the gst-uninstalled script still need to be installed before using them.

Just need to move to the ~/gst/master directory and use the automated script in ~/gst/master/gstreamer/scripts/git-update.sh which will do all the work for you.

You will have to edit the script to add extra modules like gstreamer-vaapi (only those which don’t need extra arguments)

EXTRA_MODULES="\
 gstreamer-vaapi \
 gst-editing-services \
 gst-rtsp-server \
 gst-python"

or you can choose to do it manually, starting with gstreamer then gst-plugins-base then any others in any order

$ ./autogen.sh
$ make -j8

We still need to install gst-omx manually because we will be using different omx targets

$ cd gst-omx
$ ./autogen.sh --with-omx-target=bellagio
$ make -j8

Note that we need to use “make install” with the gstreamer modules.

Checking the environment

Here are some tips to check if you have set up the environment correctly.

Checking mesa install

Running

$ glxinfo

should give different info from when invoked outside the environment.

For example, Outside the env

$ glxinfo | fgrep "OpenGL core profile version string"
OpenGL core profile version string: 4.3 (Core Profile) Mesa 17.2.0-devel - padoka PPA

In the uninstalled env

$ glxinfo | fgrep "OpenGL core profile version string"
OpenGL core profile version string: 4.3 (Core Profile) Mesa 17.2.0-devel (git-5ff4858)

Checking firmware install

In case you installed the NVIDIA firmware, after installing the firmware you should have lots of files in /lib/firmware/nouveau/. For brevity I’m only showing the number of files that I have.

$ ls -l /lib/firmware/nouveau/ | wc -l
221

The number of files you have should need not be exactly same.

Checking Gstreamer install

Just check if you are using the uninstalled version of gst-*

$ which gst-inspect-1.0
 /home/gurkirpal/gst/master/gstreamer/tools/gst-inspect-1.0
$ which gst-launch-1.0
 /home/gurkirpal/gst/master/gstreamer/tools/gst-launch-1.0

Checking gst-omx install

$ gst-inspect-1.0 omxh264dec

should show information about the omxh264dec plugin. Otherwise you’ll get an error message like this

$ gst-inspect-1.0 omxh264dec
No such element or plugin 'omxh264dec'

Checking VA-API installation

Running vainfo should give output similar to as shown below

$ LIBVA_DRIVER_NAME=nouveau vainfo
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'nouveau'
libva info: Trying to open /home/gpalsingh/gst/master/prefix/lib/dri/nouveau_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.40 (libva 1.8.0.pre1)
vainfo: Driver version: mesa gallium vaapi
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :    VAEntrypointVLD
      VAProfileMPEG2Main              :    VAEntrypointVLD
      VAProfileVC1Simple              :    VAEntrypointVLD
      VAProfileVC1Main                :    VAEntrypointVLD
      VAProfileVC1Advanced            :    VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:    VAEntrypointVLD
      VAProfileH264Main               :    VAEntrypointVLD
      VAProfileH264High               :    VAEntrypointVLD
      VAProfileNone                   :    VAEntrypointVideoProc

You can use the following pipeline to test if gstreamer-vaapi is working fine:

$ gst-launch-1.0 filesrc location=any_video.mp4 ! qtdemux ! h264parse ! vaapidecodebin ! videoconvert ! ximagesink

This should bring up a video window without any audio.

To quit switch to terminal and press Ctrl + C.

You can find clips for testing at http://www.h264info.com/clips.html