GStreamer 1.6 and OpenGL contexts

With the imminent release of GStreamer 1.6 there has been a change in the mechanism by which OpenGL contexts and Window System (X11, Wayland, Win32, OS X's Cocoa) specific resources are shared with a GStreamer pipeline for elements that need that information. Elements like sinks or any of the GL elements or any global application provided information that is shared between elements.

Previously, in 1.4, there was a property, 'external-opengl-context' available on some(!) elements that one could set on the current set of filter and mixer elements (not sink or source elements though!). While it mostly worked well enough, it was a behaviour difference to how window system handles are shared between the application and the GStreamer pipeline. Internal GL context sharing between elements was occurring in the ALLOCATION query as a value in a GstStructure of the meta:GstVideoGLTextureUploadMeta parameters. The main problem with this is 1. the ALLOCATION query is handled differently by different elements and more critically, 2. the tee element drops ALLOCATION queries entirely. This would result in a performance hit in a pipeline with GL elements containing a tee element like:

gltestsrc ! tee ! glimagesink

In the above pipeline, the GL context's would not be the same between or shared between gltestsrc and glimagesink, so the gltestsrc texture would be downloaded to system memory (a very slow operation) and reuploaded by glimagesink. Evidently we should always attempt to keep data on the GPU if at all possible to be as performant as possible.

GstContext provides a way of sharing not only between elements but also with the application using queries and messages. See https://developer.gnome.org/gstreamer/stable/gstreamer-GstContext.html for more details. Using GstContext we have the ability to share arbitrary data between elements (even across tee's) and the application which is what we want.

For the application side, the required activity is to call gst_element_set_context() with the needed/requested GstContext. This can be at application start up before the pipeline is set up or as a response to a message on the bus.

Example: sharing an X11 display with the bus callback

static gboolean
sync_bus_call (GstBus *bus, GstMessage *msg, gpointer    data)
{
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:
    {
      const gchar *context_type;
      GstContext *context = NULL;
     
      gst_message_parse_context_type (msg, &context_type);
      g_print("got need context %s\n", context_type);

      if (g_strcmp0 (context_type, GST_GL_DISPLAY_CONTEXT_TYPE) == 0) {
        Display *x11_display; /* get this from the application somehow */
        GstGLDisplay *gl_display = GST_GL_DISPLAY (
            gst_gl_display_x11_new_with_display (x11_display));

        context = gst_context_new (GST_GL_DISPLAY_CONTEXT_TYPE, TRUE);
        gst_context_set_gl_display (context, gl_display);

        gst_element_set_context (GST_ELEMENT (msg->src), context);
      }
      if (context)
        gst_context_unref (context);
      break;
    }
    default:
      break;
  }

  return FALSE;
}

Here's an example for passing a user provided GL context:

static gboolean
sync_bus_call (GstBus *bus, GstMessage *msg, gpointer    data)
{
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_NEED_CONTEXT:
    {
      const gchar *context_type;
      GstContext *context = NULL;
     
      gst_message_parse_context_type (msg, &context_type);
      g_print("got need context %s\n", context_type);

      if (g_strcmp0 (context_type, "gst.gl.app_context") == 0) {
        GstGLContext *gl_context; /* get this from the application somehow */
        GstStructure *s;

        context = gst_context_new ("gst.gl.app_context", TRUE);
        s = gst_context_writable_structure (context);
        gst_structure_set (s, "context", GST_GL_TYPE_CONTEXT, gl_context, NULL);

        gst_element_set_context (GST_ELEMENT (msg->src), context);
      }
      if (context)
        gst_context_unref (context);
      break;
    }
    default:
      break;
  }

  return FALSE;
}

You may have noticed that we don't specifiy how you retrieve these display's or application GL context's. That's very application specific and typically one is using some other library to perform the required setup for the platforms one is running on. However if you're using Gtk+, QML or CoreAnimation directly, there are ready made sink elements (gtkglsink, qmlglsink and caopengllayersink) that perform this for you that are also available in the upcoming GStreamer 1.6 release.

There are also some helpers for the OpenGL side of things if your library doesn't have a sink yet. See gst_gl_context_new_wrapped(), gst_gl_context_get_current_gl_context() and gst_gl_context_get_current_gl_version() in the GStreamer GL stack here and here.

happy GStreamer OpenGL-ing!