Newbie question: Video codec negotiations between clients?

General
1

Hi there,
I’m using the “high performance backend” for Nextcloud Talk, which at its core utilizes Janus for WebRTC. By default, the only Video codec enabled for rooms is VP8.
Rooms are created by said backend with the following command, part of this go file

                "request":     "create",
                "description": id + "|" + streamType,
                // We publish every stream in its own Janus room.
                "publishers": 1,
                // Original: Do not use the video-orientation RTP extension as it breaks video
                // orientation changes in Firefox.
                // I commented this setting:
                // "videoorient_ext": false,
                 // Offer more video & audio codecs
                "audiocodec": "opus,g722,pcmu,pcma,isac32",
                // original: 
                // "videocodec": "vp8",
                // my edit
                "videocodec": "av1,vp9,vp8,h264",
                // added by me 
                "video_svc": true, 
                // added by me 
                "vp9_profile": "0",
                // added by me 
                "h264_profile": "42e01f",
        }

What I see is the following:

  • Firefox and Safari Desktop both happily jump on VP9, and it works fine for them
  • Chromium and Nextcloud Talk on iOS happily use AV1 and the quality is spectacular (so is the battery drain)

But if I try to add Chromium to a running session with Firefox or Safari:

  1. Chromium sees everyone
  2. Firefox does not see the video of Chromium

Firefox sees AV1 as an offered candidate and answers with VP8 – but Chrome does not appear to switch to neither VP9 nor VP8.

This is what Firefox is offered:

v=0
o=- 1683701183979447 1 IN IP4 <redacted>
s=VideoRoom 2033888814411901
t=0 0
a=group:BUNDLE 0 1 2
a=ice-options:trickle
a=fingerprint:sha-256 29:EA:F2:CE:32:BA:79:3F:DE:18:A8:16:4E:CC:49:AA:25:4D:6E:C3:2F:D1:D6:DE:91:CA:B0:01:66:10:51:64
a=extmap-allow-mixed
a=msid-semantic: WMS *
m=audio 9 UDP/TLS/RTP/SAVPF 111
c=IN IP4 <redacted>
a=sendonly
a=mid:0
a=rtcp-mux
a=ice-ufrag:3Uuv
a=ice-pwd:<redacted>
a=ice-options:trickle
a=setup:actpass
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=fmtp:111 useinbandfec=1
a=msid:janus janus0
a=ssrc:2856068009 cname:janus
m=video 9 UDP/TLS/RTP/SAVPF 98 99
c=IN IP4<redacted>
a=sendonly
a=mid:1
a=rtcp-mux
a=ice-ufrag:3Uuv
a=ice-pwd:<redacted>
a=ice-options:trickle
a=setup:actpass
a=rtpmap:98 AV1/90000
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:13 urn:3gpp:video-orientation
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=ssrc-group:FID 1295272686 4248073925
a=msid:janus janus1
a=ssrc:1295272686 cname:janus
a=ssrc:4248073925 cname:janus
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 <redacted>
a=sendonly
a=mid:2
a=sctp-port:5000
a=ice-ufrag:3Uuv
a=ice-pwd:<redacted>
a=ice-options:trickle
a=setup:actpass

This is its local response:

v=0
o=mozilla...THIS_IS_SDPARTA-99.0 1727597345351158116 0 IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 F7:9F:17:CF:89:E3:D0:91:53:08:4D:F2:01:F8:24:87:1D:81:70:B3:69:F0:8C:5F:B5:D4:FE:EA:EE:00:84:9D
a=group:BUNDLE 0 2
a=ice-options:trickle
a=msid-semantic:WMS *
m=audio 9 UDP/TLS/RTP/SAVPF 111
c=IN IP4 0.0.0.0
a=recvonly
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=fmtp:111 maxplaybackrate=48000;stereo=1;useinbandfec=1
a=ice-pwd:<redacted>
a=ice-ufrag:edad8eed
a=mid:0
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=setup:active
a=ssrc:3058767046 cname:{228b0dd6-c3ab-4499-a3f3-051679b7c311}
m=video 0 UDP/TLS/RTP/SAVPF 120
c=IN IP4 0.0.0.0
a=inactive
a=mid:1
a=rtpmap:120 VP8/90000
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=recvonly
a=ice-pwd:<redacted>
a=ice-ufrag:edad8eed
a=mid:2
a=setup:active
a=sctp-port:5000
a=max-message-size:1073741823

My question: Is a negotiation between clients during a videoroom session even possible? Would they agree on the smallest common denominator, like VP8? Or is a codec picked at the start of a session?

Sorry for the newbie question.

2

I cannot help on Nextcloud integrations, sorry, you may want to ask them.

3

Hi @lorenzo, totally get that, and I’m not asking you to tell me how to improve the Nextcloud infrastructure – I was just interested if in principle this should work. That clients negotiate the codec that works best for everyone and switch during the session between codecs.

4

No, in Janus (and the VideoRoom) once you pick a codec it stays like that. The fact that in the VideoRoom you give options, it just means Janus will accept more than one, but once a publisher offers something (A, B or C?) and one is chosen (B) it will be used for the whole session by that publisher. Subsribers interested in subscribing to the publisher will need to support that codec (B) or it won’t work for them.

5

Thank you, that explains the issue then. Good to know.
I guess then, at this time, VP8 is the best choice – so that all can play together.