Month: July 2017

GSoC 2017: H.264 encoder improvements and EGLImage

With the H.264 encoder component working we moved to the big goal of the project i.e. adding EGLImage support in the H.264 decoder. In the following sections I’ll talk about the work done this week and future goals briefly.

H.264 encoder improvements

We earlier had problem with the input port that it tried to free buffer which was already NULL. The only check it had was for the “is_owned” property (from tizport.c)

if (OMX_TRUE == is_owned)
      OMX_PTR p_port_priv = OMX_DirInput == p_obj->portdef_.eDir
                              ? ap_hdr->pInputPortPrivate
                              : ap_hdr->pOutputPortPrivate;
      free_buffer (p_obj, ap_hdr->pBuffer, p_port_priv);

The is_owned value can’t be set from outside since the struct tiz_port_buf_props_t is defined in tizport.c and not exposed through any header.

The workaround was, as pointed out by Julien, to use super_UseBuffer instead of super_AllocateBuffer inside h264e_inport_AllocateBuffer.  In fact super_Use or super_AllocateBuffer do the same thing except that the latter allocates some memory and sets the flag ”owned” that is later checked while clearing the buffer. This change also has the advantage that it avoids making Tizonia to allocate buffer and to free them , which is inefficient.

Along with that there were some minor improvements too.

EGLImage support

Most of this week was spent on figuring out how to add the EGLImage support to the H.264 decoder component. We went through various approaches and it still isn’t finalised how it will be done but we have made some progress in getting the video_buffers.

From the 2 approaches to get the “struct pipe_resource *” from the EGLImage I used the first approach since, according to Julien, is safer because it calls lookup_egl_image which
seems to go all the way down and internally calls _eglCheckResource so it
does some validity check on the input pointers.

static struct pipe_resource *
st_omx_pipe_texture_from_eglimage_v1(EGLDisplay egldisplay, EGLImage
  _EGLDisplay *disp = egldisplay;
  struct dri2_egl_display *dri2_egl_dpy = disp->DriverData;
  __DRIscreen *_dri_screen = dri2_egl_dpy->dri_screen;
  struct dri_screen *st_dri_screen = dri_screen(_dri_screen);
  __DRIimage *_dri_image = st_dri_screen->lookup_egl_image(st_dri_screen,

  return _dri_image->texture;

The first commit can be found here.

This change only served as getting the pipe_resource. It was also not working since the decoder was trying to make a target from pipe_resource which wasn’t available since the output port was disabled. To prevent that case I added the changes here. This made the error to disappear but still the decoder failed later when trying to end the frame.

Next we dropped how targets were made. After some discussion it was decided to store the video buffers made from the pipe resource and use them as needed.

video_buffer = vl_video_buffer_create_ex2 (p_prc->pipe, &templat, resources);


util_hash_table_set(p_prc->video_buffer_map, p_prc->p_outhdr_, video_buffer);

Another  problem that I was getting that the decoder was getting RGBA EGLImage (1 target) whereas the decoder renders to NV12 which is 2 pipe_texture per frames. The following was added as fix for that.

assert (p_res->format == PIPE_FORMAT_R8G8B8A8_UNORM);

templat.buffer_format = PIPE_FORMAT_R8G8B8A8_UNORM; // RGBA
templat.chroma_format = PIPE_VIDEO_CHROMA_FORMAT_NONE;
templat.width = p_res->width0;
templat.height = p_res->height0;
templat.interlaced = 0;

memset(resources, 0, sizeof resources);
resources[0] = p_res;

video_buffer = vl_video_buffer_create_ex2 (p_prc->pipe, &templat, resources);

The video_buffer is then retrieved later on when h264d_fill_output needs to write to the output.

dst_buf = util_hash_table_get(p_prc->video_buffer_map, output);

The full commit message is here.

Next week’s work

After these changes the decoder compiles and runs without error but fails to write the output to the screen. I found that instead of making more than one video_buffers the decoder makes only one video_buffer. The decoder runs normally when the video isn’t running but when the video starts the main function “h264d_prc_buffers_ready” is never called by the IL core. This is the reason behind the black screen / no output from the decoder. Many parts of the decoder are still to be done/finalised like clearing the video_buffers. Next week we expect to get the decoding to work normally and some more progress.


GSoC 2017: Vid enc working

This week we got the encoder but not without some workarounds. I’ll talk briefly about the fixes for the last week’s problems, the improvements still to be made and the future plan below. The commits can be seen here.

Registering  Multiple components

This blocking issue was fixed this week thanks to Juan’s support. The root of the problem was that Tizonia does not have specialised loaders to load multiple components. You can check out Juan’s much in depth answer here. So instead, we had to provide a generic name to the component and add the other “components” as roles of that component. The following code from entrypoint.c registers the generic component name:

/* Initialize the component infrastructure */
tiz_comp_init (ap_hdl, OMX_VID_COMP_NAME);

And you set the roles for the components as usual. The difference here is that the roles do not have names like they do in bellagio. For example, in vid_dec.c the three video dec roles have different names:

strcpy(comp->name, OMX_VID_DEC_BASE_NAME);
strcpy(comp->name_specific[0], OMX_VID_DEC_MPEG2_NAME);
strcpy(comp->name_specific[1], OMX_VID_DEC_AVC_NAME);
strcpy(comp->name_specific[2], OMX_VID_DEC_HEVC_NAME);

strcpy(comp->role_specific[0], OMX_VID_DEC_MPEG2_ROLE);
strcpy(comp->role_specific[1], OMX_VID_DEC_AVC_ROLE);
strcpy(comp->role_specific[2], OMX_VID_DEC_HEVC_ROLE);

So in gst-omx we can directly use the role name to select the component role as


which is the value of OMX_VID_DEC_MPEG2_NAME.

For the tizonia based component we have to provide both component name and the role, which is also more accurate:

So with these changes the problem with loading multiple components was almost solved. Almost because we still haven’t tested the per role hook registration API which I’ll talk about later below.

Working H.264 encoder

The encoder is able to encode the video. There was a need to add new ports with some changes in functionality. The h264einport and h264eoutport are derived from tizavcport and tizvideoport respectively. The main need for them is that the h264einport replaces the pBuffer of the buffer header with a custom pointer. The replacement occurs here:

r = enc_AllocateBackTexture(ap_hdl, idx, &inp->resource, &inp->transfer, &(*buf)->pBuffer);

The other changes are regarding management of this custom pointer.


The encoder still uses some workarounds which will need to be addressed later on.

Freeing the buffer pointer

The new pBuffer pointer can’t be simply cleared with free(). Trying to do so causes an error to be thrown. In the bellagio based encoder simply setting the buffer to NULL solves the problem.

buf->pBuffer = NULL;

return super_FreeBuffer(typeOf (ap_obj, "h264einport"), ap_obj, ap_hdl, idx, buf);

Inside super_FreeBuffer, a check makes sure that it doesn’t try to clear an already empty buffer:


But in tizonia there doesn’t exist any check for the same. From tizport.z

if (OMX_TRUE == is_owned)
      OMX_PTR p_port_priv = OMX_DirInput == p_obj->portdef_.eDir
                              ? ap_hdr->pInputPortPrivate
                              : ap_hdr->pOutputPortPrivate;
      free_buffer (p_obj, ap_hdr->pBuffer, p_port_priv);

The only property it checks is the is_owned property which is internal to tizport.c. To avoid that I used this patch in the dev environement

commit f3e3f40611129c9d3f942b05b7eb66e37d198ace
Author: Gurkirpal Singh <>
Date: Mon Jul 17 00:12:06 2017 +0530

tizport: check ap_hdr->pBuffer exists before clearing

diff --git a/libtizonia/src/tizport.c b/libtizonia/src/tizport.c
index 58bac53..8d48b3a 100644
--- a/libtizonia/src/tizport.c
+++ b/libtizonia/src/tizport.c
@@ -1187,7 +1187,9 @@ port_FreeBuffer (const void * ap_obj, OMX_HANDLETYPE ap_hdl, OMX_U32 a_pid,
OMX_PTR p_port_priv = OMX_DirInput == p_obj->portdef_.eDir
? ap_hdr->pInputPortPrivate
: ap_hdr->pOutputPortPrivate;
- free_buffer (p_obj, ap_hdr->pBuffer, p_port_priv);
+ if (ap_hdr->pBuffer) {
+ free_buffer (p_obj, ap_hdr->pBuffer, p_port_priv);
+ }

p_unreg_hdr = unregister_header (p_obj, hdr_pos);

Another way to avoid the crash could be to provide a fake buffer by allocating memory to the pBuffer with malloc() / calloc() just before releasing. What approach to use we’re still deciding upon it.

Releasing the input header

Another problem that arose while adding the encoder was clearing the input header. With bellagio, like in vid_enc.c you can set

priv->BufferMgmtCallback = vid_enc_BufferEncoded;

where vid_enc_BufferEncoded writes to the output buffer. The omx_base_filter_BufferMgmtFunction takes care of providing the buffers and calling the BufferMgmtCallback at the right time.

The following call sends the input buffer to the management function

return base_port_SendBufferFunction(port, buf);

Tracing the program, I found that the vid_enc_BufferEncoded is called two times before the buffer is cleared. The first time the whole function runs. The second time it hits

if (!inp || LIST_IS_EMPTY(&inp->tasks)) {
input->nFilledLen = 0; /* mark buffer as empty */
enc_MoveTasks(&priv->used_tasks, &inp->tasks);

which marks the buffer to be cleared.

With tizonia, the component has to implement it’s own buffer management. Currently the component doesn’t uses queue for holding buffers like bellagio internally does so it has to clear the the buffer before requesting a new one. To provide similar functionality I made changes to call h264e_manage_buffers until the condition “if (!inp || LIST_IS_EMPTY(&inp->tasks))” is satisfied in this patch. Unfortunately this didn’t quite work as expected. While running it with gst-launch with this pipeline

gst-launch-1.0 filesrc location=~/mp4_tests/10\ Second\ countdown.mp4.mp4 ! qtdemux ! h264parse ! avdec_h264 ! videoconvert ! omxh264enctiz ! h264parse ! avdec_h264 ! videoconvert ! ximagesink

The video runs for a bit then stops running and hangs. I tried debugging it with gdb, setting breakpoints on all functions in h264eprc.c and adding the command “continue” to all of them so that I could just check how the control flows. Strangely while debugging in this manner the video never hanged. Instead it reached the end and gave some port flush errors. Again trying it without gdb I got same results. This made it really hard to debug. In the end I reverted the commit to try other things. The crashes caused by it didn’t help either.

The second approach I used was to just move the block that clears the buffer at the end. The commit can be found here. This assumes that the inp->tasks is empty which has been in the case from the bellagio st/omx/h264enc traces. Still this could be considered as a workaround which might need work later. With this change the video plays.

What’s next?

Since the enc is almost in working state, next week, Julien will review it while I’ll focus around adding supporting EGLimage support. Juan is working on adding a per role EGLImage registration API that would allow different roles to have different hooks. Julien and Christian provided inputs on how the task could be accomplished.

GSoC 2017: H.264 encoder in the works

This week the project moved into the harder part. I’ll explain the progress briefly below.

EGLImage support in H.264 decoder delayed

We are still stuck on the issue of how to output to the EGLImage. The task was a bit more difficult than we expected. My question in the mesa-dev mailing list is still answered as of now. According to Julien, we might need to add more to add more plumbing/callback in vl_screen or pipe_screen. Since we decided to spend only a week on EGLImage support research we’ve moved on to the next goal.

New component being added: H.264 encoder

The component being developed on the gsoc-dev branch on my clone of the mesa repo.  So far the work has been mainly on the core elements of the encoder i.e. the processor is ready. Still the encoder is unable to encode the data at the moment because the stock tizvideoport that it uses doesn’t provide the same capabilities that are required from the ports. In bellagio, these custom behaviours can be set up by replacing the port or component methods with user made functions. For example, This code from vid_enc.c changes how the port works

port->Port_SendBufferFunction = vid_enc_EncodeFrame;
port->Port_AllocateBuffer = vid_enc_AllocateInBuffer;
port->Port_UseBuffer = vid_enc_UseInBuffer;
port->Port_FreeBuffer = vid_enc_FreeInBuffer;</pre >

With Tizonia, this process is much more organised and decoupled. With Bellagio, ports and components are instantiated in the same constructor but with Tizonia they should have different processors and the component can simply use the new port as a type. Instead of replacing / assigning the functions, Tizonia has predefined values which tell it what function performs what function while registering it. For example, in this code

void *
vp8d_prc_init (void * ap_tos, void * ap_hdl)
  void * tizprc = tiz_get_type (ap_hdl, "tizprc");
  void * vp8dprc_class = tiz_get_type (ap_hdl, "vp8dprc_class");
  TIZ_LOG_CLASS (vp8dprc_class);
  void * vp8dprc = factory_new
    /* TIZ_CLASS_COMMENT: class type, class name, parent, size */
    (vp8dprc_class, "vp8dprc", tizprc, sizeof (vp8d_prc_t),
     ap_tos, ap_hdl,
     /* TIZ_CLASS_COMMENT: class constructor */
     ctor, vp8d_prc_ctor,
     /* TIZ_CLASS_COMMENT: class destructor */
     dtor, vp8d_prc_dtor,
     tiz_srv_allocate_resources, vp8d_prc_allocate_resources,
     tiz_srv_deallocate_resources, vp8d_prc_deallocate_resources,
     tiz_srv_prepare_to_transfer, vp8d_prc_prepare_to_transfer,
     tiz_srv_transfer_and_process, vp8d_prc_transfer_and_process,
     tiz_srv_stop_and_return, vp8d_prc_stop_and_return,
     tiz_prc_buffers_ready, vp8d_prc_buffers_ready,
     tiz_prc_port_flush, vp8d_prc_port_flush,
     tiz_prc_port_disable, vp8d_prc_port_disable,
     tiz_prc_port_enable, vp8d_prc_port_enable,
     /* TIZ_CLASS_COMMENT: stop value*/

  return vp8dprc;

The values in factory_new are like key->value. So vp8d_prc_ctor will be set as the constructor for the new component. With bellagio you’ll have to set it as comp->Construcor=vp8_prc_ctor while loading the component.

The same process works for ports but different functions. Here is an example from the tizavcport.c on which I worked

void *
tiz_avcport_init (void * ap_tos, void * ap_hdl)
  void * tizvideoport = tiz_get_type (ap_hdl, "tizvideoport");
  void * tizavcport_class = tiz_get_type (ap_hdl, "tizavcport_class");
  TIZ_LOG_CLASS (tizavcport_class);
  void * tizavcport = factory_new
    /* TIZ_CLASS_COMMENT: class type, class name, parent, size */
    (tizavcport_class, "tizavcport", tizvideoport, sizeof (tiz_avcport_t),
     ap_tos, ap_hdl,
     /* TIZ_CLASS_COMMENT: class constructor */
     ctor, avcport_ctor,
     /* TIZ_CLASS_COMMENT: class destructor */
     dtor, avcport_dtor,
     tiz_api_GetParameter, avcport_GetParameter,
     tiz_api_SetParameter, avcport_SetParameter,
     tiz_api_GetConfig, avcport_GetConfig,
     tiz_api_SetConfig, avcport_SetConfig,
     tiz_port_check_tunnel_compat, avcport_check_tunnel_compat,
     /* TIZ_CLASS_COMMENT: stop value*/

  return tizavcport;

The big advantage of this is that ports become reusable! This means you can use instances of a port and also modify a port and create new port out of it (similar to subclasses). In the above example, tizavcport inherits from tizvideoport which in turn inherits from tizport which acts in turn acts as base class for ports. Just for the curious ones the hierarchy is: tizavcport -> tizvideoport -> tizport -> tizapi -> tizobject .

For the H.264 encoder a new custom port needs to be made which adds a little extra functionality to the existing tizavcport for the out port and tizvideoport for the in port. The ports will have custom behaviour for tiz_api_AllocateBuffer, tiz_api_FreeBuffer, and tiz_api_UseBuffer (in port only). The code will remain the same as their vid_enc* counterparts in vid_enc.c After this obstacle comes the major blocking obstacle.

Registering multiple components

So far, looking at the Tizonia’s plugins it’s approach has been to provide a single entry point (OMX_ComponentInit) through which it can register it’s roles and types. Each component can have different roles. Each component has it’s own shared object library form which the component can be used. But in gallium, a single provides all the components.

Bellagio provides

int omx_component_library_Setup(stLoaderComponentType **stComponents)

for setting up multitple components. It is invoked two times while setting up the components. First time the argument is NULL, which means the function must tell how many components it intends to register. The return value determines the dimensions of stComponents. For two functions it’s 2×1, for three it’s 3×1 and so on. Each stLoaderComponentType* is a specialised structure to initialise a separate component.

As of now Tizonia only has


as entry point which supports only one component.

This means as of now it’s only possible to add only one component in For testing I’ve enabled only the video encoder. I’ve asked Juan about ways to add multiple components. One possible way might be to add a single component and add other components as it’s roles and check for roles in the hooks which I haven’t tried properly yet. Even if that worked it will be a bit ugly and will require extra work inside the hooks so a different and cleaner solution is more desirable.

Next week’s goals

Making the encoder work takes priority over the multiple component issue even when it is a blocker since it can be solved any time before the project ends. We will also be hoping to make some progress on the EGLImage research.

GSOC 2017 EGLImage

This week we looked into how to add EGLImage support in the new component and other small work.

Adding EGLImage support

An EGLImage is simply a texture whose content can be updated without having to copy the contents to system memory. The format of an EGLImage is opaque to the EGL’s client by design, so any memory allocated through OMX_UseEGLImage macro are not accessible directly by the IL client.

Right now RPI is the only platform where OMX_UseEGLImage code path in gst-omx works at the moment. It is hard to maintain because developing directly on board is harder. After adding this support this code will be able to run on desktop too which will be much easier than developing on the board. The decoder will decode directly in the gpu memory. Without OMX_UseEGLImage, the gst pipeline copies buffers from gpu memory to system memory when decoding and copy back from system memory to gpu memory when rendering with OpenGL, which is quite inefficient.

Components should inspect the EGLImage and determine if the EGLImage is
compatible with the port configuration. A validation hook is registered to check the validity of the returned EGLImage. To get the EGLImage the decoder will have to check OMX_BUFFERHEADERTYPE->pBuffer nullity. In case it is NULL it means it has an EGLImage underneath. Using tiz_krn_claim_eglimage that EGLImage can be retrieved. What still isn’t clear is how to write to the EGLImage in st/omx_tizonia/omxh264dec. For now we are trying to retrieve the mesa pipe screen pointer from the egl_display (pNativeWindow).

Code cleanup

The 42 commits on the gsoc-dev branch were cleaned up and reduced down to 4 commits on the gsoc branch.

GPU Upgrade

Also this week I replaced my older NVIDIA card with AMD RX 460. No blockiness during decoding and the ability to use (older st/omx) and develop the encoder component (new st/omx_tizonia) are some of the benefits from this switch.


Next week’s goals are to find a way to write output to EGLImage and to start adding new components like omxh264enc.