Simulcast change client substream not working

General
1

Hi, I am following the article Simulcast and Janus: what’s new? (and where’s my SSRC?) | Meetecho Blog to create a simulcast stream with 3 different qualities setting it on the transceiver side. The videotrack is h264. The sdp is generated correctly and everything looks good.

v=0
no=- 3383397139837300255 2 IN IP4 127.0.0.1
ns=-
nt=0 0
na=group:BUNDLE 0 1
na=extmap-allow-mixed
na=msid-semantic: WMS e23ac07d-a5e6-4c2b-a93e-c7ab09c9ffe2
nm=video 9 UDP/TLS/RTP/SAVPF 123 122 121 120
nc=IN IP4 0.0.0.0
na=rtcp:9 IN IP4 0.0.0.0
na=ice-ufrag:Y1Pj
na=ice-pwd:w0W80QYe7Q+ZdOxPw4pyaXJs
na=ice-options:trickle
na=fingerprint:sha-256 38:0C:1B:60:89:94:A4:2D:E9:28:41:5C:A0:99:72:A4:2C:79:36:34:E2:73:4E:9E:C4:93:5A:14:54:F2:61:08
na=setup:actpass
na=mid:0
na=extmap:1 urn:ietf:params:rtp-hdrext:toffset
na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
na=extmap:3 urn:3gpp:video-orientation
na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
na=sendonly
na=msid:e23ac07d-a5e6-4c2b-a93e-c7ab09c9ffe2 9ead9e35-677b-4a09-a325-9a96cfcd9f39
na=rtcp-mux
na=rtcp-rsize
na=rtpmap:123 H264/90000
na=rtcp-fb:123 goog-remb
na=rtcp-fb:123 transport-cc
na=rtcp-fb:123 ccm fir
na=rtcp-fb:123 nack
na=rtcp-fb:123 nack pli
na=fmtp:123 implementation_name=NvCodec;level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e033
na=rtpmap:122 H264/90000
na=rtcp-fb:122 goog-remb
na=rtcp-fb:122 transport-cc
na=rtcp-fb:122 ccm fir
na=rtcp-fb:122 nack
na=rtcp-fb:122 nack pli
na=fmtp:122 implementation_name=NvCodec;level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=420033
na=rtpmap:121 H264/90000
na=rtcp-fb:121 goog-remb
na=rtcp-fb:121 transport-cc
na=rtcp-fb:121 ccm fir
na=rtcp-fb:121 nack
na=rtcp-fb:121 nack pli
na=fmtp:121 implementation_name=NvCodec;level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640033
na=rtpmap:120 H264/90000
na=rtcp-fb:120 goog-remb
na=rtcp-fb:120 transport-cc
na=rtcp-fb:120 ccm fir
na=rtcp-fb:120 nack
na=rtcp-fb:120 nack pli
na=fmtp:120 implementation_name=NvCodec;level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0033
na=rid:h send
na=rid:m send
na=rid:l send
na=simulcast:send h;m;l
nm=audio 9 UDP/TLS/RTP/SAVPF 96 97 98 99 102 103 104 9 0 8 100 101 107 108 109 114 110 112 113 126
nc=IN IP4 0.0.0.0
na=rtcp:9 IN IP4 0.0.0.0
na=ice-ufrag:Y1Pj
na=ice-pwd:w0W80QYe7Q+ZdOxPw4pyaXJs
na=ice-options:trickle
na=fingerprint:sha-256 38:0C:1B:60:89:94:A4:2D:E9:28:41:5C:A0:99:72:A4:2C:79:36:34:E2:73:4E:9E:C4:93:5A:14:54:F2:61:08
na=setup:actpass
na=mid:1
na=extmap:14 urn:ietf:params:rtp-hdrext:ssrc-audio-level
na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
na=sendrecv
na=msid:e23ac07d-a5e6-4c2b-a93e-c7ab09c9ffe2 82071518-d86e-40e0-8bac-08fe5d96df7b
na=rtcp-mux
na=rtpmap:96 opus/48000/2
na=rtcp-fb:96 transport-cc
na=fmtp:96 minptime=10;sprop-stereo=1;stereo=1;useinbandfec=1
na=rtpmap:97 red/48000/2
na=fmtp:97 96/96
na=rtpmap:98 multiopus/48000/6
na=fmtp:98 channel_mapping=0,4,1,2,3,5;coupled_streams=2;minptime=10;num_streams=4;useinbandfec=1
na=rtpmap:99 multiopus/48000/8
na=fmtp:99 channel_mapping=0,6,1,2,3,4,5,7;coupled_streams=3;minptime=10;num_streams=5;useinbandfec=1
na=rtpmap:102 ILBC/8000
na=rtpmap:103 ISAC/16000
na=rtpmap:104 ISAC/32000
na=rtpmap:9 G722/8000
na=rtpmap:0 PCMU/8000
na=rtpmap:8 PCMA/8000
na=rtpmap:100 L16/8000
na=rtpmap:101 L16/16000
na=rtpmap:107 L16/32000
na=rtpmap:108 L16/8000/2
na=rtpmap:109 L16/16000/2
na=rtpmap:114 L16/32000/2
na=rtpmap:110 telephone-event/48000
na=rtpmap:112 telephone-event/32000
na=rtpmap:113 telephone-event/16000
na=rtpmap:126 telephone-event/8000
na=ssrc:1720815888 cname:UViF8F+6ZRrj/g6V
na=ssrc:1720815888 msid:e23ac07d-a5e6-4c2b-a93e-c7ab09c9ffe2 82071518-d86e-40e0-8bac-08fe5d96df7b

But the problem is on the client, when I consume the stream, the highest quality is played by default, as you can see on substream-target after attaching.

"plugin": "janus.plugin.videoroom",
         "data": {
            "videoroom": "attached",
            "room": 1234,
            "id": 113450249,
            "warning": "deprecated_api",
            "display": "unity",
            "streams": [
               {
                  "type": "video",
                  "active": true,
                  "mindex": 0,
                  "mid": "0",
                  "ready": false,
                  "send": true,
                  "feed_id": 113450249,
                  "feed_display": "unity",
                  "feed_mid": "0",
                  "codec": "h264",
                  "h264-profile": "42e033",
                  "simulcast": {
                     "substream": -1,
                     "substream-target": 2,
                     "temporal-layer": -1,
                     "temporal-layer-target": 2
                  }
               },
               {
                  "type": "audio",
                  "active": true,
                  "mindex": 1,
                  "mid": "1",
                  "ready": false,
                  "send": true,
                  "feed_id": 113450249,
                  "feed_display": "unity",
                  "feed_mid": "1",
                  "codec": "opus"
               }
            ]
         }
      },

Then I receive an event that the substream has been set correctly to 2.

[
   {
      "janus": "event",
      "session_id": 618328994457738,
      "sender": 3828716482194333,
      "plugindata": {
         "plugin": "janus.plugin.videoroom",
         "data": {
            "videoroom": "event",
            "room": 1234,
            "mid": "0",
            "substream": 2
         }
      }
   }
]

I can see the video perfectly. Then, I try to change the substream following the videoroom API:

this.remoteVideoRoomHandles[remoteFeedId].send({
      message: {
        request: 'configure',
        substream: substream,
      },
    });

The request is sent:

image
image786×160 17.3 KB

And janus responses with:

{
   "janus": "ack",
   "session_id": 2375912949686871,
   "transaction": "FgIRUykGTLPK"
}

But the problem is that after doing this, nothing happens. I am not able to change the substream. I’ve also tried to send the configure to the videoroom handle, instead of the remoteFeedHandle, and the response is:

configured: "ok"
room: 1234
videoroom: "event"

But again, nothing happens. I still see the highest quality substream. Of course, the 3 substreams are clearly distinguishable, they are set to (it’s in C#):

RTCRtpEncodingParameters parameterH = new RTCRtpEncodingParameters();
parameterH.maxBitrate = ConvertToBits(10.5f);
parameterH.scaleResolutionDownBy = 1;
parameterH.maxFramerate = 60;
parameterH.active = true;
parameterH.rid = "h";

RTCRtpEncodingParameters parameterM = new RTCRtpEncodingParameters();
parameterM.maxBitrate = ConvertToBits(4.5f);
parameterM.scaleResolutionDownBy = 1;
parameterM.active = true;
parameterM.maxFramerate = 15;
parameterM.rid = "m";

RTCRtpEncodingParameters parameterL = new RTCRtpEncodingParameters();
parameterL.maxBitrate = ConvertToBits(0.5f);
parameterL.scaleResolutionDownBy = 1;
parameterL.active = true;
parameterL.maxFramerate = 1;
parameterL.rid = "l";

What am I doing wrong?

Thank for your help.

Edit: The janus version I am using is 1.1.0

2

You need to specify the subscriber’s mid of the simulcast video too, when sending the configure. A subscription may have multiple video streams, so just saying which substream you want will be ignored if you don’t specify which video the request refers to. Please refer to the documentation for the proper configure syntax.

3

Notice that this is required in 1.x, which is what you’re using. The blog post was written for 0.x, and so doesn’t include that requirement.

4

Hi, thank you for the quick reply. I just tried to specify also the mid, as you can see in the request.

image
image873×212 18.8 KB

The mid is ‘0’, I get it from the attached info (it’s always 0 as I only have 1 video):

"streams": [
               {
                  "type": "video",
                  "active": true,
                  "mindex": 0,
                  "mid": "0",
                  "ready": false,
                  "send": true,
                  "feed_id": 121206956,
                  "feed_display": "unity",
                  "feed_mid": "0",
                  "codec": "h264",
                  "h264-profile": "42e033",
                  "simulcast": {
                     "substream": -1,
                     "substream-target": 2,
                     "temporal-layer": -1,
                     "temporal-layer-target": 2
                  }
               },

But again, nothing happens. I still see the high resolution substream… Anything else I can try to change?

This is how I send the request from the client-side:

this.remoteVideoRoomHandles[remoteFeedId].send({
      message: {
        request: 'configure',
        mid: '0',
        substream: substream,
      },
    });

Thank you.

5

I’ve also tried passing it as an array:

image
image756×225 22.6 KB

Like it is specified on the videoroom API VideoRoom plugin documentation. Not working neither.

The videoroom response for whatever I request is configured: ok

6

Are you sending this on the right subscriber handle? It will not work if you send it on another handle (eg., publisher). If so, try the official VideoRoom demo with simulcast enabled and check the messages there as a reference.

7

Yes, I’m sending to the correct subscriber handle, I get the id from the feed.id here:

success(pluginHandle) {
          instance.remoteVideoRoomHandles[feed.id] = pluginHandle;
          logJanusVideoRoom(
            `[speaker ${feed.id}] Joining feed as subscriber...`,
            feed.id
          );
          const subscribe = {
            request: 'join',
            room: room.id,
            ptype: 'subscriber',
            feed: feed.id,
            private_id: room.privateId,
            subscriber: 0,
            pin,
          };

          instance.remoteVideoRoomHandles[feed.id].send({ message: subscribe });
        },

I’ve already checked the demo with simulcast, on EchoTest no mid or request is sent:

image
image863×148 18.3 KB

And the videoroom demo does not seem to do anything with the ?simulcast=true query string. (I’m using the latest Chrome version).

What should I do? The Janus responses are not showing any error…

8

Btw, this is the videoroom.jcfg, just to make sure that this is correct

general :
{
};
room-1234 :
{
  description = "Demo Room";
  secret = "**********************************";
  publishers = 30;
  bitrate = 1024000;
  fir_freq = 5;
  record = true;
  videocodec = "vp8,h264";
  rec_dir = "/mnt/nfs/wct-98e88453-21f2-4bc1-ade0-5d8ef6b25612/recorded";
  audiolevel_ext = true;
  audio_active_packets = 20;
  audiolevel_event = true;
  audio_level_average = 50;
  notify_joining = true;
};
9

fir_freq 5 is way too low, at the very least it should be 10 but can be even higher.
1024000 is probably too low if you want simulcast (depending on how you configured the simulcast envelope on the client side).

10

I changed the videroom cfg to:

general :
{
};
room-1234 :
{
  description = "Demo Room";
  secret = "********************************";
  publishers = 30;
  bitrate = 1024000000;
  fir_freq = 500;
  record = true;
  videocodec = "vp8,h264";
  rec_dir = "/mnt/nfs/wct-af764060-418f-42f6-a844-31862e7e92aa/recorded";
  audiolevel_ext = true;
  audio_active_packets = 20;
  audiolevel_event = true;
  audio_level_average = 50;
  notify_joining = true;
};

and I have the same results, still showing the highest quality. It would be really helpful if I could see some Janus log to know what is it doing or if it’s having some error…

Also, I’ve seen that if I set active=false on m quality, Janus changes the substream randomly between the highest (12Mb bitrate) and the lowest (0.5Mb bitrate) quality (any janus event is received on that quality changes), and my network bandwith is:

image
image910×922 30.9 KB

Anything else I can do? Thank you.

Btw the bitrate value on the cfg is on Bytes, Mb, MB, kb…? It doesn’t make sense to be bits because the sample has a value of 128000 and it would be too low.

11

It’s bits, and yes, the default room has a very low bitrate by default.

12

Ok, so 1024000000 bitrate that I set on last cfg would be still too low because it’s 1Mb, and I have the simulcast configured to h - 12Mb, m - 4.5Mb, l - 0.5Mb. I will set it to 1024000000000 = 100Mb. Do you agree with that?

13

Browsers will ignore that, and those ridicolously high values make little sense anyway. Browsers never go beyond 2 or 3 mbps by default. Just use 0 as the bitrate when creating the room and that will be used as a “no limit”.

14

Ok that makes sense. I think that 0 is the correct value, as we control the users bitrate. Anyway, the simulcast is still not working. I have a setInterval of 2s on client sending this request:

image
image754×223 22.5 KB

And the stream doesn’t change, it is playing always the highest quality and consuming 12Mb bandwith.

As always, Janus responses configured: “ok”. What should I do?

15

As I think I already said, check the default demo and make sure simulcast works as expected there (after you ensure the default room has no bitrate constraint as well, of course). If it does, check the messages they exchange to see if you’re doing something wrong in your code.

16

Where can I find videroom simulcast working on any demo? Because as I said, it doesn’t seem to work for the videroom demo, the only demo I saw that works with simulcast is EchoTest plugin, and the API is different.

17

The default demo page with your own Janus server sounds like a good start :stuck_out_tongue_winking_eye:

1 Like
18

Hi again Lorenzo, sorry to bother you, I would just like to know if this configuration is correct. I find it strange that simulcast: -1 and it is impossible to change it by sending requests to the server. Also the substream-target is always 2 and it seems to keep in a loop trying to change that value. This happens with the simplest possible config.

                 "simulcast": {
                     "substream": -1,
                     "substream-target": 2,
                     "temporal-layer": -1,
                     "temporal-layer-target": 2
                  }

What is the reason for this value? I don’t see any reference to this in the api. Thank you very much again and sorry for the inconvenience.

19

Is that an Admin API dump for a subscriber? If it’s -1 it means the server never detected any substream. It may mean the publisher actually did not negotiate a simulcast stream properly at all, and so the server just relays the only (non simulcast) stream it sees to subscribers. You can check the Admin API for the publisher, as in the stats you should be able to see if simulcast was detected, and which substreams it sees data coming in on.

20

Ok that makes sense thank you.

In the janus admin log: janus.log I see that:

            "in": {
              "packets": 0,
              "bytes": 0,
              "bytes_lastsec": 0,
              "nacks": 0,
              "retransmissions": 0,
              "video-simulcast-2": {
                "packets": 6005196,
                "bytes": 2626644010,
                "bytes_lastsec": 1951136,
                "nacks": 0
              }
            },

This means that only one substream is detected, right?