Streams are buffering when server is busy

jessedewild · March 13, 2024, 9:52pm

Hi there,

We have a Janus server set up with no issues at all. We use the videoroom plugin. When our app is not very busy, every video stream is working fine. No buffering issues, or streams falling out.
But when the Janus server gets more busy, so more streams, buffering issues start occurring.

The weird thing is, that the RAM and CPU usage on the Janus server is still very low.

We are using H264 streams. Mainly 720x720 30fps and 960x720 30fps with the ‘42e01f’ profile for H264.

What can we change on the server, or in the video room itself to solve these buffering issues and streams falling out?
Maybe set a max bitrate on the video room? And what would you advice, something around 1.5 mbps bitrate as a max for those two resolutions?

These are our video room settings:

        let create = {
            request: 'create',
            destroy_if_empty: true,
            empty_timeout: 3600,
            videocodec: 'h264',
            audiocodec: 'opus',
            h264_profile: '42e01f',
            is_private: false,
            publishers: 10,
        };

Maybe set the publishers to a lower number? Do not think it will fix anything but maybe!

We do have all the plugins enabled on our server. Maybe only enabling the videoroom plugin will help?

These are our server settings:

  "janus": "server_info",
  "version": 1202,
  "version_string": "1.2.2",
  "author": "Meetecho s.r.l.",
  "log-to-stdout": true,
  "log-to-file": true,
  "log-path": "/var/log/janus.log",
  "data_channels": false,
  "accepting-new-sessions": true,
  "session-timeout": 60,
  "reclaim-session-timeout": 15,
  "candidates-timeout": 45,
  "server-name": "MyJanusInstance",
  "ipv6": false,
  "ice-lite": false,
  "ice-tcp": false,
  "ice-nomination": "aggressive",
  "ice-consent-freshness": false,
  "ice-keepalive-conncheck": false,
  "hangup-on-failed": false,
  "full-trickle": false,
  "mdns-enabled": false,
  "min-nack-queue": 200,
  "nack-optimizations": false,
  "twcc-period": 200,
  "dtls-mtu": 1200,
  "static-event-loops": 0,
  "event_handlers": false,
  "opaqueid_in_api": false,

Kind regards,
Jesse de Wild

lorenzo · March 14, 2024, 10:59pm

The VideoRoom has a few bottlenecks in critical paths, which is why it’s not really meant for many viewers out of the box. It’s why we’re adding helper threads as we did for the Streaming plugin:

github.com/meetecho/janus-gateway

Add support for helper threads to VideoRoom

meetecho:master ← meetecho:videoroom-helper-threads

opened 03:02PM - 15 Sep 22 UTC

lminiero

+372 -3

This patch adds support for the so-called "helper threads" to the VideoRoom. If …you're familiar with the Streaming plugin, in that context helper threads allow mountpoints to leverage some threads to distribute the incoming RTP packets to the indended audience, rather than take care of the process in the same thread that receives the packets themselves: that feature was added a few years back in #1316, and helped the Streaming plugin greatly increase performance in the presence of a lot of subscribers. The VideoRoom plugin, instead, never got the same treatment: as such, there's always been a critical path when handling incoming media packets from a publisher to relay to subscribers. More precisely, this was the path: 1. the publisher's ICE handle loop receives a packet, in `janus_ice_cb_nice_recv`; 2. from there, `incoming_rtp` is called on the VideoRoom plugin for that publisher; 3. that function, among other things, iterates on all subscribers to perform a `relay_rtp` of the packet for each of them; 4. for each subscriber, `relay_rtp` queues the packet for delivery in the Janus core. All four steps happen within the context of the same thread (the one handling publisher's core loop). In case the publisher has too many subscribers, that sequence may take longer than it takes for the following packets to arrive, causing the thread to struggle. This was exactly the main cause of bottlenecks in the Streaming plugin, which led us to add helper threads: the moment helper threads are added, the only entities `incoming_rtp` must pass packets to are the helpers themselves, which will then take care of steps 3. and 4. on their own. As a consequence, the publisher's loop is never kept busy, and can keep on processing incoming packets in a timely way. As it did for the Streaming plugin, this patch is supposed to increase the performance of the VideoRoom plugin in case a single publisher has a lot of subscribers. Notice that this doesn't magically allow infinite subscribers for a publishers: it just makes the process more efficient, allowing to better use the available CPU resources; leveraging multiple Janus servers (e.g., via #3014) is still needed in case you want to increase an audience beyond the capabilities of a single machine. In this first iteration, I kept the configuration very simple: if you add a `threads: <number>` property when creating a room, then all publishers that connect to the room will automatically spawn that number of threads. This happens the moment a publisher joins, not when they publish, and they stay there until the publisher handle is closed: this means it's probably suboptimal as it is, because if you take into account users that will only subscribe, if they create a publisher handle to receive signalling events (as they should) helper threads will be created for them too, needlessly. I'll probably revisit this later on, before or after this is merged (possibly allowing only for specific publishers to have threads, e.g., those that will actually need them), but for the time being that's good enough to experiment with this feature. I tested this briefly and it seems to be doing its job, but I haven't really stressed it out. I don't plan to merge really soon, so in case you're interested in increasing the performance of the VideoRoom because you'll have many users receiving media, please do take the time to test this and report problems.

Please test and let us know if it helps. We haven’t received enough feedback on it, and I’d like it to be merged sooner or later.

jessedewild · March 14, 2024, 11:11pm

Hi Lorenzo, how can I add this to the videoroom plugin?

lorenzo · March 14, 2024, 11:16pm

It’s a pull request. You need to pull Janus from that branch and use that version.

lorenzo · March 14, 2024, 11:19pm

Note it’s disabled by default. You’ll need to pass a threads property when creating a room (via config or API) to specify a number of threads to use to assist (default will be 0, so disabled).

atoppi · March 20, 2024, 10:22am

configure janus.jcfg “event_loops” properly, setting it to the number of available cores or a multiple of it
For the described h264 stream, putting a limit of 1 mbps is enough imho
Lower log verbosity if not needed

As Lorenzo mentioned, the videoroom is not intended to be used with a large number of subscribers. If that is your scenario, consider implementing a different architecture where feeds are being forwarded to streaming mountpoints and attendees use those to fetch the streams.

jessedewild · April 20, 2024, 9:15am

Hi Lorenzo,

What is an ideal starting point for the threads? And do we need to enable it on our server as well?

lorenzo · April 20, 2024, 9:42am

There’s no ideal starting point, since it’s a new feature and not benchmarked. You’ll have to experiment with that, possibly doing some benchmarking of your own.

Topic		Replies	Views
10 Gbit Saturation General	3	172	August 29, 2023
Low-end devices have trouble with streaming, which codec to use? General	1	57	April 20, 2024
Please help we're getting tons of slow links General	1	148	June 12, 2023
Stream via GStreamer acting weirdly on old devices, 50/50 chance of showing streams General	0	29	April 29, 2024
Publisher playout delay General	3	131	June 22, 2023

Streams are buffering when server is busy

Related Topics