GStreamer WebRTC in 2020

A little while ago, we at Centricular developed a plugin for easier WebRTC with GStreamer. For the details on the initial release, see Nirbheek's blogpost about the new GStreamer WebRTC implementation. The initial release gathered a lot of interest and sparked a fair bit of activity around some of the needed features. This is a birds eye view of the new features since Nirbheek's blog post.

New features

RTP Bundling

In a WebRTC connection there may be multiple streams that need to be sent and received. Without bundling, they would require opening a connection for each stream to be sent (and another for some control packets if not muxing RTCP with the RTP stream). This may consume precious ports and TURN resources if relaying is in use with the network topology. A WebRTC stream with bundling will instead send all of the streams over a single connection. GStreamer already supported bundling of streams as required by the WebRTC and related specifications and an implemention only needed to be added to webrtcbin. There is still some work to do on the fallback when the peer does not support bundling when the bundle-policy on webrtcbin is set to balanced. Thanks Mathieu Duponchelle for this improvement! Also thanks Pexip for their various fixes to bundling and RTX.

Data channels

WebRTC Data channels for being able to send arbitrary data are supported starting from the release of GStreamer 1.16 and implemented as part of this bug. Fixes and interoperability with browsers and other WebRTC features have improved since. I also presented a short talk on this at the 2018 GStreamer conference and a recording is available from here. Thanks to the OpenWebRTC project for the GStreamer SCTP elements used here.

RTP FEC (Forward Error Correction)

For all those spotty networks, one mitigation method is to send multiple copies of the same data. The specific form of FEC currently used by the major web browsers is ULPFEC (Uneven Level Protection Forward Error Correction) and is has an implementation in GStreamer using new rtpulpfecenc and rtpulpfecdec elements. These ULPFEC elements have been hooked up within webrtcbin and are enabled with the "do-nack" property on each transceiver. Thanks Mathieu Duponchelle for this improvement! Also thanks to Pexip for the original code!

RTP RTX (Retransmissions)

Another strategy for dealing with spotty networks is to attempt retransmitting late or lost data by having the receiver request the late or lost data from the sender. GStreamer has had RTX elements for some time and these have been integrated into webrtcbin and are enabled with the "do-nack" property on each transceiver. Thanks Mathieu Duponchelle for this improvement! Also thanks Pexip for their various fixes to bundling and RTX.

Limited Renegotiation Support

As of the merging of this MR , a limited form of renegotiation is supported by webrtcbin. This included a simple example showing the feature. The major limitations of this feature are related to the transport of the streams where ICE or DTLS changes are not supported. The underlying transports are also not removed when a stream is removed so if not bundling the streams over a single connection, all removed stream transports are still kept alive. Changing audio or video codecs used or RTX/FEC usage is also not supported.

Miscellaneous

Various other small improvements have also been made to state handling:
  • The peer connection state (connection-state), ICE connection state (ice-connection-state) and ICE gathering state (ice-gathering-state) now more closely match the webrtc specification. Thanks to Sebastian Dröge for that improvement!
  • Improved DTLS state handling fixing stream start up in certain scenarios. Thanks to Sebastian Dröge and Jan Schmidt for this improvement!

Testing

So far, most of our testing has been very much ad-hoc with some unit tests testing webrtcbin's negotiation against itself, some manual testing of the canonical examples against major browsers and testing against very specific private test scenarios where necessary and available. I am very much interested in trying to expand the testing coverage using webrtcbin and having publicly runnable and reproducible tests. I have some preliminary work in progress involving Selenium, GstValidate and the existing signalling server in python3 and websockets available in this MR. I plan to write another blog post about that adventure at some point.

Other Future Work

Congestion Control

This is the largest missing piece for feature parity with the existing browser implementations. The latest developments from the browsers on this front is the introduction of TWCC (Transport Wide Congestion Control). There is an implementation in GStreamer of the necessary RTCP packet formats at the RTP layer that has not yet been plumbed up into the GStreamer WebRTC layer. Once TWCC is implemented in webrtcbin, it would be very much a possibility to control various aspects of the input RTP stream to meet the changing restrictions of the network.

Statistics

While there are some statistics available that match the webrtc-stats specificaion, others are missing either due to the underlying libraries not supporting the necessary values or there is a missing implementation for translating to the required format for webrtc-stats. Improving the statistics that are available from webrtcbin is a long-term goal.

DTMF

There are some GStreamer elements for performing DTMF that need to be plumbed into the WebRTC layer.

Other (Maybe Advanced) Scenarios

There are some scenarios with transport restarts, DTLS fingerprints, audio/video codec changes on renegotiation, transceiver reuse, and other scenarios that would need checking and implementing as needs arose.

Comments

  1. DarKingDooM12 July 2021 at 14:07

    Hey,
    About congestion control, why TWCC ? why not GCC or BBR or PCC ?

    ReplyDelete
    Replies
    1. Matthew Waters29 May 2023 at 01:46

      TWCC provides the necessary information (data) to implement the relevant bandwidth estimation algorithms. TWCC provides per-packet arrival times and whether packets were received. Bandwidth estimation algorithms use the TWCC information to vary the bandwidth used by the connection.

      For specific algorithms, the GCC algorithm been implemented based on the TWCC information within the `rtpgccbwe` GStreamer element. The `webrtcsink` element can use the `rtpgccbwe` element to provide the relevant bandwidth estimation.

      Delete

Post a Comment

Popular posts from this blog

GStreamer RTP Session Handling in Rust

Recently we at Centricular had the opportunity to write some new GStreamer RTP elements in Rust as part of a grant from the Sovereign Tech Fund . We of course jumped at the chance. The existing RTP elements are all currently written in the C programming language and have been around for over a decade. The age is showing somewhat with some of the architecture decisions and user visible interfaces. There is also a general desire within the GStreamer community to write parsers and other elements handling potentially untrusted data in a much more memory safe language, like Rust. As a bonus, Rust comes with better tooling out of the box for developers than C/C++. In this blog post we will be looking into how RTP sessions are managed in GStreamer, independent of the specifics of a particular RTP payload format. The Prior Art: rtpbin In GStreamer, the pre-existing interface for handling RTP sessions is managed by the rtpbin element. The rtpbin element is over 15 years old and ha...
Read more

GStreamer 1.6 and OpenGL contexts

With the imminent release of GStreamer 1.6 there has been a change in the mechanism by which OpenGL contexts and Window System (X11, Wayland, Win32, OS X's Cocoa) specific resources are shared with a GStreamer pipeline for elements that need that information. Elements like sinks or any of the GL elements or any global application provided information that is shared between elements. Previously, in 1.4, there was a property, 'external-opengl-context' available on some(!) elements that one could set on the current set of filter and mixer elements (not sink or source elements though!). While it mostly worked well enough, it was a behaviour difference to how window system handles are shared between the application and the GStreamer pipeline. Internal GL context sharing between elements was occurring in the ALLOCATION query as a value in a GstStructure of the meta:GstVideoGLTextureUploadMeta parameters. The main problem with this is 1. the ALLOCATION query is handled diffe...
Read more

qmlglsink - GStreamer and Qt's QML

Windows Support - NEW!! Just recently, I added the missing piece for adding Windows support for qmlglsink for QtQuick 2 applications (tracked by this bug ) with this commit which means that qmlglsink is now usable on all major platforms 1 . With the list of supported platforms (Android, iOS, OS X, Windows, Linux) for qmlglsink growing, I thought I'd write a little bit about it and why you should use it. qmlglsink is a GStreamer element that will render a video stream into a QtQuick QML scene. As you can see from this example , qmlglsink does this by providing a GstGLVideoItem QML item that can be placed anywhere in the scene. The GstGLVideoItem is given a unique name to be able to link the element and the QML item. Without this link, the element cannot render the stream anywhere and will fail changing state. In the C++ code, we then retrieve the GstGLVideoItem by name like so . Then we set the pipeline to playing, start Qt's mainloop and watch the show. Why Use qml...
Read more