GStreamer 1.6 and OpenGL contexts

With the imminent release of GStreamer 1.6 there has been a change in the mechanism by which OpenGL contexts and Window System (X11, Wayland, Win32, OS X's Cocoa) specific resources are shared with a GStreamer pipeline for elements that need that information. Elements like sinks or any of the GL elements or any global application provided information that is shared between elements.

Previously, in 1.4, there was a property, 'external-opengl-context' available on some(!) elements that one could set on the current set of filter and mixer elements (not sink or source elements though!). While it mostly worked well enough, it was a behaviour difference to how window system handles are shared between the application and the GStreamer pipeline. Internal GL context sharing between elements was occurring in the ALLOCATION query as a value in a GstStructure of the meta:GstVideoGLTextureUploadMeta parameters. The main problem with this is 1. the ALLOCATION query is handled differently by different elements and more critically, 2. the tee element drops ALLOCATION queries entirely. This would result in a performance hit in a pipeline with GL elements containing a tee element like:
gltestsrc ! tee ! glimagesink
In the above pipeline, the GL context's would not be the same between or shared between gltestsrc and glimagesink, so the gltestsrc texture would be downloaded to system memory (a very slow operation) and reuploaded by glimagesink. Evidently we should always attempt to keep data on the GPU if at all possible to be as performant as possible.

GstContext provides a way of sharing not only between elements but also with the application using queries and messages. See https://developer.gnome.org/gstreamer/stable/gstreamer-GstContext.html for more details. Using GstContext we have the ability to share arbitrary data between elements (even across tee's) and the application which is what we want.

For the application side, the required activity is to call gst_element_set_context() with the needed/requested GstContext. This can be at application start up before the pipeline is set up or as a response to a message on the bus.

Example: sharing an X11 display with the bus callback
static gboolean
sync_bus_call (GstBus *bus, GstMessage *msg, gpointer    data)
{
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:
    {
      const gchar *context_type;
      GstContext *context = NULL;
     
      gst_message_parse_context_type (msg, &context_type);
      g_print("got need context %s\n", context_type);

      if (g_strcmp0 (context_type, GST_GL_DISPLAY_CONTEXT_TYPE) == 0) {
        Display *x11_display; /* get this from the application somehow */
        GstGLDisplay *gl_display = GST_GL_DISPLAY (
            gst_gl_display_x11_new_with_display (x11_display));

        context = gst_context_new (GST_GL_DISPLAY_CONTEXT_TYPE, TRUE);
        gst_context_set_gl_display (context, gl_display);

        gst_element_set_context (GST_ELEMENT (msg->src), context);
      }
      if (context)
        gst_context_unref (context);
      break;
    }
    default:
      break;
  }

  return FALSE;
}
Here's an example for passing a user provided GL context:
static gboolean
sync_bus_call (GstBus *bus, GstMessage *msg, gpointer    data)
{
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:
    {
      const gchar *context_type;
      GstContext *context = NULL;
     
      gst_message_parse_context_type (msg, &context_type);
      g_print("got need context %s\n", context_type);

      if (g_strcmp0 (context_type, "gst.gl.app_context") == 0) {
        GstGLContext *gl_context; /* get this from the application somehow */
        GstStructure *s;

        context = gst_context_new ("gst.gl.app_context", TRUE);
        s = gst_context_writable_structure (context);
        gst_structure_set (s, "context", GST_GL_TYPE_CONTEXT, gl_context, NULL);

        gst_element_set_context (GST_ELEMENT (msg->src), context);
      }
      if (context)
        gst_context_unref (context);
      break;
    }
    default:
      break;
  }

  return FALSE;
}
You may have noticed that we don't specifiy how you retrieve these display's or application GL context's. That's very application specific and typically one is using some other library to perform the required setup for the platforms one is running on. However if you're using Gtk+, QML or CoreAnimation directly, there are ready made sink elements (gtkglsink, qmlglsink and caopengllayersink) that perform this for you that are also available in the upcoming GStreamer 1.6 release.

There are also some helpers for the OpenGL side of things if your library doesn't have a sink yet. See gst_gl_context_new_wrapped(), gst_gl_context_get_current_gl_context() and gst_gl_context_get_current_gl_version() in the GStreamer GL stack here and here.

happy GStreamer OpenGL-ing!

Comments

  1. Thanks for the info! Unfortunately - for some reason - I get below message when I build. Do you have any idea? I get the same "Undefined symbols"-error when I try to use the helpers you mention as well (gst_gl_context_get_current_gl_context)...

    I'm using the latest version (1.6.1), Thanks!


    Undefined symbols for architecture x86_64:
    "_gst_gl_context_get_type", referenced from:
    _sync_bus_call in cbridge.o
    ld: symbol(s) not found for architecture x86_64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)


    My bus callback looks like this:

    static gboolean
    sync_bus_call (GstBus *bus, GstMessage *msg, CGstBridge *self)
    {
    switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:
    {
    const gchar *context_type;
    GstContext *context = NULL;

    gst_message_parse_context_type (msg, &context_type);
    g_print("got need context %s\n", context_type);



    if (g_strcmp0 (context_type, "gst.gl.app_context") == 0) {
    GstGLContext *gl_context; /* get this from the application somehow */
    GstStructure *s;

    gl_context = (__bridge GstGLContext*)self.scnView.openGLContext;

    context = gst_context_new (GST_GL_DISPLAY_CONTEXT_TYPE, TRUE);
    s = gst_context_writable_structure (context);
    gst_structure_set (s, "context", GST_GL_TYPE_CONTEXT, gl_context, NULL);

    gst_element_set_context (GST_ELEMENT (msg->src), context);


    }
    if (context)
    gst_context_unref (context);
    break;
    }
    default:
    break;
    }

    return FALSE;
    }

    ReplyDelete
    Replies
    1. You have to link against the libgstgl library. The easiest way is to use pkg-config like so

      gcc $(pkg-config --cflags gstreamer-gl-1.0 gstreamer-1.0 ...) file1.c file2.c $(pkg-config --libs gstreamer-gl-1.0 gstreamer-1.0) -o test

      Delete
  2. Great, thanks! Now I got that working.
    But I do get the same error (Undefined symbols) when I try to use GstGLCAOpenGLLayer... Shouldn't that also be in the libgstgl library?

    Undefined symbols for architecture x86_64:
    "_OBJC_CLASS_$_GstGLCAOpenGLLayer", referenced from:
    type metadata accessor for __ObjC.GstGLCAOpenGLLayer in GameViewController.o
    ld: symbol(s) not found for architecture x86_64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)

    ReplyDelete
    Replies
    1. You're attempting to use the actual layer? or the sink that uses the layer? If you're really set on using the layer, you'll have to get the class using something like http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/ext/gl/caopengllayersink.m#n328

      Delete
  3. Replies
    1. The process is exactly the same. The only change is that you need to create a GstGLDisplay for wayland instead. Use gst_gl_display_wayland_new instead: https://gstreamer.freedesktop.org/documentation/gl-wayland/gstgldisplay_wayland.html?gi-language=c#gst_gl_display_wayland_new

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. It seems like this is the only source of documentation for how to share OpenGL context with GStreamer. Matthew, since it is not documented anywhere else, 8 years later, can you update this blog post? These instructions don't work with GStreamer 1.22. Any child of GstGLBaseFilter on gst_pad_link will call gst_gl_base_filter_find_gl_context, which will look for gst.gl.local_context. As far as I can tell, this context is not mentioned anywhere, not in this post, not in any documentation. I also couldn't find what is the distinction between gst.gl.local_context and gst.gl.app_context and how should I pass a wrapped EGL context for each of them when I respond to NEED_CONTEXT.

    ReplyDelete

Post a Comment

Popular posts from this blog

qmlglsink - GStreamer and Qt's QML

New Qt6 QML (OpenGL) elements