Vulkan in GStreamer

Today, Vulkan 1.0 as released. For those of you that don't what Vulkan is, the TL;DR is essentially that it's cross-platform/device lower level 3D API than OpenGL (both the desktop and the ES variants) of a similar scope to DX12, Metal or Mantle. As a result, the driver does less work and the application/engine takes over some of the management of resources which hopefully leads to a smaller driver divergence and therefore bugs while at the same time providing increased access to the capabilities of the GPU. Centricular graciously sponsored some of my time to have a look at Vulkan and implement a basic working prototype of a GStreamer sink element that would display video frames onto the screen. I hereby present to you, vulkansink, the result of that labour.
gst-launch-1.0 videotestsrc ! vulkanupload ! vulkansink

Disclaimer: this is still alpha-quality code and it hasn't had the same extensive testing as the OpenGL elements and there are a number of short fallings:
  1. It doesn't scale the input video to the output surface size (yet).
  2. There's very little locking (as required by the Vulkan specification, e.g. VkCommandBuffer's and VkCommandPool's).
  3. There aren't any effects/colour conversion implemented.
  4. Performance has not been a primary concern so there are some inefficiencies in the current design.
All that being said, my impressions of Vulkan are relatively good. It promises to fit infinitely better with GStreamer's extensive use of threads than an equivalent OpenGL implementation. That extra flexibility and functionality comes at the price of increased complexity however I think that is a necessary cost.

The hard API change also provides a higher baseline for implementations and applications to depend against which resets the amount of legacy code that would need to be maintained for older GPU's and systems.

Implementation Details

The current implementation of libgstgl was used extensively as a basis for the structure of how Vulkan is intended to be used in GStreamer.

Memory

Vulkan provides a high level overview of accessing GPU memory through VkDeviceMemory objects. These are simply wrapped in GstMemory objects as GstVulkanMemory with allocation and destruction routines for creating and destroying GPU memory as well as mapping with gst_memory_map() for CPU access to the data (if supported at creation).
Vulkan provides VkBuffer objects which are somewhat equivalent to OpenGL's Buffer objects which are used for storing vertices, draw indices, uniforms, texels, etc and is wrapped up in GStreamer into the GstVulkanBufferMemory object (a subclass of GstMemory). As a backing store, it allocates a GstVulkanMemory object.
GstVulkanImageMemory provides the same abstraction for VkImage objects which also have a backing store of a GstVulkanMemory.

Integrating with applications

The GstContext infrastructure is used extensively to not only share necessary state between elements but also providing application specific state into GStreamer. The current list of GstContext-able objects is, VkInstance, VkDevice, VkDisplay and the VkQueue which all have GStreamer wrappers to account for refcounting and in the future, Vulkan's locking requirements.

Comments

Post a Comment

Popular posts from this blog

GStreamer RTP Session Handling in Rust

Recently we at Centricular had the opportunity to write some new GStreamer RTP elements in Rust as part of a grant from the Sovereign Tech Fund . We of course jumped at the chance. The existing RTP elements are all currently written in the C programming language and have been around for over a decade. The age is showing somewhat with some of the architecture decisions and user visible interfaces. There is also a general desire within the GStreamer community to write parsers and other elements handling potentially untrusted data in a much more memory safe language, like Rust. As a bonus, Rust comes with better tooling out of the box for developers than C/C++. In this blog post we will be looking into how RTP sessions are managed in GStreamer, independent of the specifics of a particular RTP payload format. The Prior Art: rtpbin In GStreamer, the pre-existing interface for handling RTP sessions is managed by the rtpbin element. The rtpbin element is over 15 years old and ha...
Read more

GStreamer 1.6 and OpenGL contexts

With the imminent release of GStreamer 1.6 there has been a change in the mechanism by which OpenGL contexts and Window System (X11, Wayland, Win32, OS X's Cocoa) specific resources are shared with a GStreamer pipeline for elements that need that information. Elements like sinks or any of the GL elements or any global application provided information that is shared between elements. Previously, in 1.4, there was a property, 'external-opengl-context' available on some(!) elements that one could set on the current set of filter and mixer elements (not sink or source elements though!). While it mostly worked well enough, it was a behaviour difference to how window system handles are shared between the application and the GStreamer pipeline. Internal GL context sharing between elements was occurring in the ALLOCATION query as a value in a GstStructure of the meta:GstVideoGLTextureUploadMeta parameters. The main problem with this is 1. the ALLOCATION query is handled diffe...
Read more

qmlglsink - GStreamer and Qt's QML

Windows Support - NEW!! Just recently, I added the missing piece for adding Windows support for qmlglsink for QtQuick 2 applications (tracked by this bug ) with this commit which means that qmlglsink is now usable on all major platforms 1 . With the list of supported platforms (Android, iOS, OS X, Windows, Linux) for qmlglsink growing, I thought I'd write a little bit about it and why you should use it. qmlglsink is a GStreamer element that will render a video stream into a QtQuick QML scene. As you can see from this example , qmlglsink does this by providing a GstGLVideoItem QML item that can be placed anywhere in the scene. The GstGLVideoItem is given a unique name to be able to link the element and the QML item. Without this link, the element cannot render the stream anywhere and will fail changing state. In the C++ code, we then retrieve the GstGLVideoItem by name like so . Then we set the pipeline to playing, start Qt's mainloop and watch the show. Why Use qml...
Read more