Month: August 2017

GSoC 2017: Project Summary (Success!)

Hi, in this post I summarise all the work done on this project. Since our biggest goal to add EGLImage support was met so the project has been a success!

GSoC 2017 Project Link

Final product

The individual patches can be found in this Google drive shared folder:


You can test out the patches directly from this branch on my mirror of the Mesa repository on github:


I use the following command line to compile it:

./ --enable-texture-float --enable-gles1 --enable-gles2 --enable-glx --enable-egl --enable-llvm --enable-shared-glapi --enable-gbm --enable-glx-tls --enable-dri --enable-osmesa --with-platforms=x11,drm --with-gallium-drivers=radeonsi,swrast --with-dri-drivers=radeon,swrast --enable-vdpau --enable-omx-bellagio --enable-omx-tizonia --enable-va --enable-debug --prefix=/home/gpalsingh/gst/master/prefix && make -j8


Below I explain each individual patch. The first two commits are in preparation for the new state tracker:

  1. st/omx_bellagio: Rename state tracker and option : In response to Christian’s review the older omx state tracker was renamed. The option “–enable-omx” was disabled. Only “–enable-omx-bellagio” is accepted now.
  2. gallium: Refactor out vl_put_screen and vl_get_screen : This change moves the screen functions to gallium/auxiliary/vl to decrease code duplication.
  3. st/omx_tizonia: Add –enable-omx-tizonia flag and build files : Adds option to enable tizonia state tracker. This only adds “st/targets/omx-tizonia” and related changes to enable compilation. The “st/omx_tizonia ” directory containing the state tracker is added in the next commits.
  4. st/omx_tizonia: Add entrypoint : This commit adds common structure for all state trackers in “st/omx_tizonia”.
  5. st/omx_tizonia: Add H.264 decoder : First big change. Adds the H.264 decoder component.
  6. st/omx_tizonia: Add H.264 encoder : Another major change. Adds the H.264 encoder.
  7. st/omx_tizonia/h264d: Add EGLImage support : Adds a new feature that wasn’t available previously in the bellagio based state tracker. Makes changes to decoder to allow directly decoding to EGLImage. For the performance improvements check out my other post in which I compared the CPU usage.

After these changes both the state trackers can coexist i.e. “–enable-omx-bellagio” and “–enable-omx-tizonia” both can be used at the same time.

Future plan

The GSoC project might have been over but there are still improvements to be made to the state tracker.

  1. Refactor code before merge : After some discussion it was generally decided that the code should be refactored before committing it to the main mesa branch. The state tracker is supposed to replace the current state tracker completely. But since it still needs more work to add the other components the duplicated code should be refactored out. This will most likely a temporary (until it has all the components that are in the bellagio omx sate tracker) but required change. This will allow the users to directly use the code from main mesa git without having to use the patches or the project branch.
  2. Add other components : The state tracker is still missing some components like mepg decoder and HEVC decoder.
  3. Fix remaining issues : There are a number of small bugs / improvements still left that we’ve put to fix later on. You can check the issues’ current status on github.

I plan to keep on working on the state tracker in the future but won’t be able to give it as much time so only my free time updating the mesa clone before sending in patches.


Special thanks to Julien (GSoC mentor) and Juan (Tizonia OpenMAX IL owner) for providing constand guidance and help. Also thanks to everyone else involved for making this project a success!


GSoC 2017: Preprations before project submission

This week we sent the decoder for review on the mesa-dev mailing list. Julien helped me fix the more advanced edge cases / bugs in the H.264 encoder while I made changes that were asked for by the reviewers. I’ll talk about all that in below in brief.

Code review

This week we had two rounds of reviews made for the decoder part only.

First review

The first set of patches consisted of two changes:

  1. st/omx_tizonia: add –enable-omx-tizonia flag and build files : Makes changes to enable the build flag for tizonia. Also adds targets/omx-tizonia.
  2. st/omx_tizonia: Add AVC decoder : Adds everything under st/omx including the decoder.

These patches were not committed since there were a few issues that I tried to address in second round of reviews.

Second review

The patches that were sent after addressing the issues were the following:

  1. st/omx_bellagio: Rename state tracker and option : Renames old bellagio state tracker and option. I had trouble sending this patch because of it being too large so I ended up sending all the patches again by mistake.
  2. st/omx_tizonia: Add –enable-omx-tizonia flag and build files : Same patch as #1 in earlier patches.
  3. st/omx_tizonia: Add entrypoint : Split from patch #2 from earlier to add only st/omx_tizonia/entrypoint.* and related files.
  4. st/omx_tizonia: Add H.264 decoder : Split from patch #2 to add only decoder.

There were generally no issues with patch #1 and was acked by Christian König.

About the other patches, after some discussion, it was decided that it’s better for the merging to be postponed until both omx_bellagio and omx_tizonia have the shared code refactored out to avoid duplicacy. The changes about general functions like “put_screen” and “get_screen” were sent for review here to check if this was the right way. About the other changes we tried out some experimental changes in this commit. This didn’t quite work since we need to pass the private types to “slice_header” and there doesn’t seem to be any elegant way yet to do so without having to adding OMX IL bits to gallium/auxiliary/vl which will be undesirable. So Julien decided to postpone this task until after monday.

Bugs and fixes

While the patches were being reviewed Julien helped with fixing some issues / bugs in the project.

EGLImage wrong colours fixed

The wrong colours while using EGLImage were finally fixed. The fix is to select a matrix that does YUV to RGB conversion.

EOS buffer clear error fix

Also related to EGLImage was the issue that it failed to clear the video buffers at the end of decoding process. The fix was to increment reference counter of the resource texture.

EGLImage hook issue false negative

The EGLImage hook failing issue was in fact not an issue. I used wrong pipeline to run the decoder. The right pipeline is

MESA_ENABLE_OMX_EGLIMAGE=1 GST_GL_API=gles2 GST_GL_PLATFORM=egl gst-launch-1.0 filesrc location=vid.mp4 ! qtdemux ! h264parse ! omxh264dec ! glimagesink

Artefacts at start of video fixed

The issue involving artefacts was indeed related to overriding the SetParameter. The whole discussion can be found here. A similar change was made to tizonia’s vp8dec.

Deadline next week

With EGLImage working the project has been a success. The work left is to address the edge cases if possible before submitting the project. Merging the commits to the project is also good to have. The next post will be about the whole project summary which will also serve as final submission for this GSoC project.

GSoC 2017: Delays in code cleanup

This week the some of the issues from last time were fixed, some are still under work and some new ones surfaced. Julien has been busy this week and has been unable to provide much input which slowed down some work. I’ll discuss this week’s developments in brief below.

Seeking issue (mostly) fixed

The most important issue form last week was seeking fail. After some digging I found that it was simply because the output buffer was reset before being cleared in “reset_streams_parameters” when the output port was disabled. Simply removing the line that resets the buffer fixed the crash that happened while seeking. The only little problem left is that now the decoder spawns lots of errors similar to this in terminal when trying to seek.


This however does not affect the video playback. The same behaviour is also seen with the existing bellagio based OMX IL state tracker. Being a joint problem with gst it shouldn’t be a blocking the project.

Overriding input port SetParameter

The first go we had was on #2 . Though the exact reason not yet known, the issue could be related to dropped buffers so we decided to override the component’s SetParameter. In bellagio you can simply replace component’s pointer for the SetParameter function like this example from h264enc.c

comp->SetParameter = vid_enc_SetParameter;

In tizonia, to do so you need to add a separate port with it’s SetParameter function overridden with the custom version. The full commit can be found here.

Though this change didn’t fix the issue, it would be good in general such as avoiding unnecessary reconfiguration when there is no resolution change. Since this is an advanced feature it shouldn’t block the decoder patches from getting reviewed.

Encoder crash

Previously I assumend that the crash in the encoder was related to the changes I made recently. After some digging and checking I found that the commits that used to work earlier have stopped working with the same error. So this error might be due to external factors like Gstreamer. We are still waiting to find a fix after the decoder is done.

Preparing commits

Originally the plan was to get the decoder reviewed soon. The first task was to rebase the branch on the latest mesa master branch and resolve the conflicts. The result after resolving conflicts can be found on the gsoc-rebase branch which is separate from gsoc-dev to validate the rebase. It configures successfully but fails to compile. The related issue can be found here. After this hopefully the decoder will be ready for review.

Up next

Since the project end period is looming near the priority is to get the decoder and the encoder reviewed as soon as possible. The issues still present and the new ones that surfaced are slowing down this process. The next few weeks will surely require some hard work.

GSoC 2017: Third phase starts

Majority of last week was spent on cleaning up the decoder for review. The numerous commits were merged to form 3 commits here. The focus has been on H.264 decoder to be working without EGLImage which will be added later in a single commit once both H.264 decoder and encoder are finalised. I’ll talk about the developments in brief below.

Writing to EGLImage

One of the big problems from the last time was to write to the EGLImage. Although everything looked fine the output was still blank. As I said in the last post, the reason was that the code responsible for decoding was never being executed.

The fix that I added to avoid the crash wasn’t actually the right way to avoid the crash we had earlier. Thanks to help from Julien we fixed the crash with the simple change

- if ((!next_is_eos) && ((p_prc->p_outhdr_->nFilledLen > 0) || p_prc->eos_)) {
+ if ((!next_is_eos) && ((p_prc->p_outhdr_->nFilledLen > 0) || p_prc->use_eglimage || p_prc->eos_)) {

The reason was that nFilledLen remains always 0 for EGLImage so the decoder failed to free the output buffer which ultimately lead to crash.

After this fix the decoder was ultimately able to show the output but there arose a new problem: the output colours were wrong. The issue is being tracked in this thread. The output was almost black and white with no colours. For this we asked Christian (author of the bellagio based state tracker) about support. The fix we tried was using “vl_compositor_set_buffer_layer” instead of looping over the layers. The commit with complete changes is available here.

After applying these changes the output changed, but not as expected. The output still has wrong colours but not same as before as shown below


Performance comparison

Even though the output is still not perfect Julien suggested that it would be good to have some data about comparison about CPU usage when using EGLImage and when not using it.

I tested the decoder using three videos of different types using the “top” command:

I took the min, max and average of the CPU usage data I got. Following are the tabulated results:


CPU usage percentage

Let’s visualise the average CPU usage as it is more significant:


Average CPU usage when using EGLImage vs when not

As can be seen using EGLImage is much more efficient compared to when not using. The CPU usage also increases dramatically when in non EGLImage mode with increasing video quality. For the other case it’s only about 1% increase when rendering a 4k movie compared to a 720p movie. Also the peak usage is 6.7% for when using EGLImage whereas it is as high as 91.3 for the other case.

Commits cleanup

Other than the work involving EGLImage we spent some significant amount of time on cleaning up the commits for final review. The focus as of now is the decoder. After that comes the encoder and finally the changes involving EGLImage support. The current progress can be checked on the gsoc branch.

Following individual changes were made:

Including some other minor changes.

Moving forward

The top priority issue at the moment is the seeking failure in H.264 decoder. Other than that there are a bit of artefacts at the start of video and the decoder fails to play with gst-play which also need attention. Ultimately the goal is to get the decoder reviewed.

Other than that a new bug popped in the H.264 encoder with the introduction of the change involving FreeBuffer which also needs to be fixed soon.