Janus streaming plugin freezes and restarts

General
1

Hello, and thank you in advance for your help! I’m working on streaming a Raspberry Pi camera into a web browser using Janus’s streaming plugin. I am able to see the video feed using the streamingtest.html demo app, but I’m having two problems (maybe related) with the feed:

  • the video stutters / pauses for about a second every 5-10 seconds and then jumps forward to the latest frame
  • the video drops entirely and Janus restarts the connection, leading the web page to show a “No remote video available” message. When that happens, I get the following error messages from Janus’s log output:
[WARN] [rtsp-test] 5s passed with no media, trying to reconnect the RTSP stream
[WARN] No c-line or source for RTSP video address, resolving server address...
[rtsp-test] Reconnected to the RTSP server, streaming again

I have tried viewing the video feed using Chrome on both the Raspberry Pi and on a separate computer on the LAN – same result on both devices.

More information on my setup below:

I am converting the Raspberry Pi camera’s data into an RTSP stream using VLC and have configured the Janus streaming plugin to read from that stream.

This is the command I use for creating the RTSP stream:

$ libcamera-vid -n  -t 0 --inline -o - | cvlc stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/stream1}' :demux=h264

(libcamera-vid is a Raspberry Pi-specific command, more documentation here: Raspberry Pi Documentation - Camera software)

This is the configuration I’m using in my janus.plugin.streaming.jcfg for the RTSP stream:

rtsp-test: {
        type = "rtsp"
        id = 99
        description = "RTSP Test"
        audio = false
        video = true
        url = "rtsp://127.0.0.1:8554/stream1"
        #rtsp_user = "username"
        #rtsp_pwd = "password"
        #secret = "adminpwd"
        rtsp_reconnect_delay = 5
        rtsp_session_timeout = 0
        rtsp_timeout = 10
        rtsp_conn_timeout = 5
}

I am using a Raspberry Pi 4B with 4GB of RAM (I have confirmed using htop that the device isn’t CPU or memory-limited, there’s plenty of both on device). My Janus commit is 0ae1c6d286bd59efe41d85efd5186dbad880a6eb (from Oct 30).

Some things I have tried already:

  • I have confirmed using ffplay that I am able to view the RTSP stream smoothly without any stutters / pauses / restarts. The command I used: ffplay udp://192.168.86.27:8888 -fflags nobuffer -flags low_delay -framedrop
  • I have checked that I don’t have a distro-version of libnice installed, so I assume that the version I built from source (following the janus-gateway install instructions) is the one being used by Janus.
  • I’ve increased my UDP buffer size and still see the same behavior:
$ sysctl net.core.rmem_max
net.core.rmem_max = 26214400
  • The Chrome browser console shows these messages being printed periodically. The timing of the logs don’t correspond to the freezes in the video feed:
Remote track muted: Event {isTrusted: true, type: 'mute', target: MediaStreamTrack, currentTarget: MediaStreamTrack, eventPhase: 2, …}
Remote track flowing again: Event {isTrusted: true, type: 'unmute', target: MediaStreamTrack, currentTarget: MediaStreamTrack, eventPhase: 2, …}

When the video fully drops and restarts, I see a log message Removing remote track.

  • I’ve looked around in the default admin.html page to see if anything stands out and haven’t found anything unusual. This is a dump of the handle information for the video session:
{
    "session_id": 8885558412361600,
    "session_last_activity": 188643232182,
    "session_timeout": 60,
    "session_transport": "janus.transport.http",
    "handle_id": 5160968596738015,
    "opaque_id": "streamingtest-5HeJEwQ4XyAr",
    "loop-running": true,
    "created": 188519449260,
    "current_time": 188666096853,
    "plugin": "janus.plugin.streaming",
    "plugin_specific": {
        "state": "watching",
        "mountpoint_id": 99,
        "mountpoint_name": "rtsp-test",
        "mountpoint_viewers": 1,
        "media": [
            {
                "type": "video",
                "mindex": 0,
                "mid": "v"
            }
        ],
        "hangingup": 0,
        "started": 1,
        "dataready": 0,
        "paused": 0,
        "stopping": 0,
        "destroyed": 0
    },
    "flags": {
        "got-offer": true,
        "got-answer": true,
        "negotiated": true,
        "processing-offer": false,
        "starting": true,
        "ice-restart": false,
        "ready": true,
        "stopped": false,
        "alert": false,
        "trickle": true,
        "all-trickles": true,
        "resend-trickles": false,
        "trickle-synced": false,
        "data-channels": false,
        "has-audio": false,
        "has-video": false,
        "new-datachan-sdp": false,
        "rfc4588-rtx": true,
        "cleaning": false,
        "e2ee": false
    },
    "agent-created": 188522976803,
    "agent-started": 188523024310,
    "ice-mode": "full",
    "ice-role": "controlling",
    "sdps": {
        "profile": "UDP/TLS/RTP/SAVPF",
        "local": "v=0\r\no=- 1699492306537239 1 IN IP4 192.168.86.27\r\ns=Mountpoint 99\r\nt=0 0\r\na=group:BUNDLE v\r\na=ice-options:trickle\r\na=fingerprint:sha-256 3A:52:E4:10:FC:EA:EC:E2:5E:A3:1C:E8:F5:BA:90:71:14:69:21:D0:3F:E8:29:7C:33:D8:4D:06:C9:15:8B:21\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS *\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 97\r\nc=IN IP4 192.168.86.27\r\na=sendonly\r\na=mid:v\r\na=rtcp-mux\r\na=ice-ufrag:ihsl\r\na=ice-pwd:Dc/C+XuX39l3g0TShM3Jd3\r\na=ice-options:trickle\r\na=setup:actpass\r\na=rtpmap:96 H264/90000\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=fmtp:96 packetization-mode=1;profile-level-id=640028;sprop-parameter-sets=J2QAKKwrQFAe0IAAAAMAgAAAHnJQABMSwAAvrze9wDxImoA=,KO4fLA==;\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=ssrc-group:FID 1310522026 2030374357\r\na=msid:janus janusv\r\na=ssrc:1310522026 cname:janus\r\na=ssrc:2030374357 cname:janus\r\na=candidate:1 1 udp 2015363327 192.168.86.27 45689 typ host\r\na=end-of-candidates\r\n",
        "remote": "v=0\r\no=- 2287696398829555488 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE v\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 97\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:NtO7\r\na=ice-pwd:ShhK+nJfJwBogtWLpd3hHTQ2\r\na=ice-options:trickle\r\na=fingerprint:sha-256 73:93:72:FB:0D:1B:08:B0:0D:D2:55:FE:26:8C:15:2D:AE:E2:A1:53:4C:AF:85:BE:08:3A:B1:B2:83:B5:7D:A5\r\na=setup:active\r\na=mid:v\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=recvonly\r\na=rtcp-mux\r\na=rtpmap:96 H264/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\n"
    },
    "queued-packets": 0,
    "webrtc": {
        "ice": {
            "stream_id": 1,
            "component_id": 1,
            "state": "ready",
            "gathered": 188522980838,
            "connected": 188523050644,
            "local-candidates": [
                "1 1 udp 2015363327 192.168.86.27 45689 typ host"
            ],
            "remote-candidates": [
                "remote1 1 udp 1845501695 192.168.86.25 52365 typ prflx raddr 192.168.86.25 rport 52365\r\n",
                "2025671863 1 udp 2113937151 192.168.86.25 52365 typ host generation 0 ufrag NtO7 network-cost 999"
            ],
            "selected-pair": "192.168.86.27:45689 [host,udp] <-> 192.168.86.25:52365 [host,udp]",
            "ready": -1
        },
        "dtls": {
            "fingerprint": "3A:52:E4:10:FC:EA:EC:E2:5E:A3:1C:E8:F5:BA:90:71:14:69:21:D0:3F:E8:29:7C:33:D8:4D:06:C9:15:8B:21",
            "remote-fingerprint": "73:93:72:FB:0D:1B:08:B0:0D:D2:55:FE:26:8C:15:2D:AE:E2:A1:53:4C:AF:85:BE:08:3A:B1:B2:83:B5:7D:A5",
            "remote-fingerprint-hash": "sha-256",
            "dtls-role": "passive",
            "dtls-state": "connected",
            "retransmissions": 0,
            "valid": true,
            "srtp-profile": "SRTP_AEAD_AES_256_GCM",
            "ready": true,
            "handshake-started": 188523050714,
            "connected": 188523095551,
            "stats": {
                "in": {
                    "packets": 3,
                    "bytes": 891
                },
                "out": {
                    "packets": 2,
                    "bytes": 1255
                }
            }
        },
        "extensions": {
            "urn:ietf:params:rtp-hdrext:sdes:mid": 4,
            "http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time": 2
        },
        "bwe": {
            "twcc": false,
            "twcc-ext-id": 0
        },
        "media": {
            "v": {
                "type": "video",
                "mindex": 0,
                "mid": "v",
                "do_nacks": true,
                "nack-queue-ms": 204,
                "ssrc": {
                    "ssrc": 1310522026,
                    "ssrc-rtx": 2030374357
                },
                "direction": {
                    "send": true,
                    "recv": false
                },
                "codecs": {
                    "pt": 96,
                    "rtx-pt": 97,
                    "codec": "h264"
                },
                "rtcp": {
                    "main": {
                        "base": 90000,
                        "rtt": 136,
                        "lost": 0,
                        "lost-by-remote": 440,
                        "jitter-local": 0,
                        "jitter-remote": 67,
                        "in-link-quality": 100,
                        "in-media-link-quality": 100,
                        "out-link-quality": 100,
                        "out-media-link-quality": 100
                    }
                },
                "stats": {
                    "in": {
                        "packets": 0,
                        "bytes": 0,
                        "bytes_lastsec": 0,
                        "nacks": 8576,
                        "retransmissions": 0
                    },
                    "out": {
                        "packets": 11858,
                        "bytes": 13923180,
                        "bytes_lastsec": 116481,
                        "nacks": 0
                    }
                }
            }
        }
    }
}
2

This is very likely the cause of your problems. It means that the Streaming plugin didn’t receive any RTP packet from the RTSP server for more than 5 seconds, and thus assumes the RTSP session is broken and tries to restart it. Not sure why the RTSP server stops sending media: it may be it’s sending packets in chunks, sometimes, and so ffplay playing it smoothly may mean it’s performing buffering, which the Streaming plugin never does (WebRTC is real-time).

You can check what’s happening by monitoring/debugging the RTSP/RTP traffic with tcpdump or wireshark. If you see RTP packets stopping for some time, then that’s indeed the cause, and you should double check your vlc RTSP server setup.

1 Like
3

That was indeed the problem – turns out the VLC RTSP server has a timeout where it sends an RTCP goodbye packet then stops streaming. Removing the timeout fixed the video dropping and restarting issue (for posterity, adding a --rtsp-timeout 0 argument to the cvlc command). Thanks!

I do still see lots of pauses / jumps in the video feed. Is this because there isn’t any buffering in WebRTC? If so, is there a recommended way of adding buffers to help smooth out the frames?

4

Closing the loop here, I ended up switching to Gstreamer, which now provides native support for Raspberry Pi camera as a source. The new command I’m using:

gst-launch-1.0 -v libcamerasrc ! video/x-raw, width=1920, height=1080, framerate=30/1 ! videoconvert ! videoscale ! x264enc tune=zerolatency ! rtph264pay ! udpsink host=54.201.78.55 port=8554