Hi, I use simple WHIP client and simple WHIP server to ingest an AV1 Stream. On WHIP side everything seems to work, also on Janus gateway everything seems to work. But there is no WHEP playback of the stream. In SDP no videocodec is found. Is AV1 already supported?
VP9 works perfect, both on WHIP ingest and WHEP playback. I defined the AV1 codec both in videoroom and streaming plugin.
And a general issue on the WHEP player. Like most of the webplayers, only Mono playback is possible on Chrome and Chromium forks. The SDP misses the stereo=1 flag (known issue for years now). On Firefox it works perfectly in Stereo.
Yes, AV1 has been supported in Janus for a long time. I’m not sure there’s support for AV1/RTP in GStreamer though? So I don’t know how you’re using the WHIP client to send AV1.
hi Lorenzo, this is my gstreamer pipeline for testing:
./whip-client -u https://janus.XXXXXX.de:7080/whip/endpoint/test -A "audiotestsrc is-live=true wave=red-noise ! audioconvert ! audioresample ! queue ! opusenc ! rtpopuspay pt=100 ssrc=1 ! queue ! application/x-rtp,media=audio,encoding-name=OPUS,payload=100" -V "v4l2src device=/dev/video4 ! video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1 ! videoconvert ! queue ! qsvav1enc rate-control=1 gop-size=60 bitrate=4500 low-latency=true ! av1parse ! rtpav1pay pt=98 ssrc=2 ! queue ! application/x-rtp,media=video,encoding-name=AV1,payload=98"
I don’t know if that’s corrects, but it makes no errors.
[WHIP] Initializing the GStreamer pipeline:
webrtcbin name=sendonly bundle-policy=3 v4l2src device=/dev/video4 ! video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1 ! videoconvert ! queue ! qsvav1enc rate-control=1 gop-size=60 bitrate=4500 low-latency=true ! av1parse ! rtpav1pay pt=98 ssrc=2 ! queue ! application/x-rtp,media=video,encoding-name=AV1,payload=98 ! sendonly. audiotestsrc is-live=true wave=red-noise ! audioconvert ! audioresample ! queue ! opusenc ! rtpopuspay pt=100 ssrc=1 ! queue ! application/x-rtp,media=audio,encoding-name=OPUS,payload=100 ! sendonly.
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_18
libva info: va_openDriver() returns 0
[WHIP] Configured jitter-buffer size (latency) for PeerConnection to 200ms
[WHIP] Starting the GStreamer pipeline
[WHIP] Creating offer
[WHIP] Offer created
[WHIP] Setting local description
[WHIP] Sending SDP offer (1254 bytes)
[WHIP] Resource URL: https://janus.XXXXXXX.de:7080/whip/resource/47DIyFPtgrTqzpnX
[WHIP] Received SDP answer (940 bytes)
[WHIP] Setting remote description
[WHIP] ICE gathering started...
[WHIP] ICE connecting...
[WHIP] PeerConnection connecting...
[WHIP] ICE gathering completed
[WHIP] ICE connected
[WHIP] ICE completed
[WHIP] DTLS connecting...
[WHIP] DTLS connected
[WHIP] PeerConnection connected
The videoroom config:
room-1234: {
description = "GGRN"
secret = "ggrn"
publishers = 6
bitrate = 5000000
fir_freq = 10
audiocodec = "opus"
videocodec = "av1"
svc=false
record = false
}
The streaming plugin config:
rtp-test: {
type = "rtp"
id = 1
description = "Opus/AV1 live stream coming from WHIP source"
metadata = "GGRN Broadcasting Server Test"
audio = true
video = true
audioport = 5002
audiopt = 111
audiocodec = "opus"
audiofmtp = "sprop-stereo=1;stereo=1"
videoport = 5004
videopt = 98
videocodec = "av1"
videosvc = false
secret = "ggrn"
}
If you share the SDPs, we can check what the client offered, and what Janus sent back as an answer.
Can you see the WHIP video when you join the same VideoRoom from a browser?
No, there is no video in videoroom from WHIP, but the local video is there.
Is there a chance to debug the simple whip server to see what SDP’s are exchanged?
Then something’s wrong in the publishing.
Add -l 5.
[WHIP] Sending SDP offer (1254 bytes)
v=0
o=- 2792513406252914092 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=group:BUNDLE video0 audio1
m=video 9 UDP/TLS/RTP/SAVPF 98
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:yeX7Uo77YZU9e+r20k6CBeeFOLW1sHDm
a=ice-pwd:kfKp5/bKVb+JQKdSSkiKaPPESAUC4P95
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:98 AV1/90000
a=rtcp-fb:98 nack pli
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 transport-cc
a=framerate:30
a=ssrc:2 msid:user1883973515@host-1d6ab5fd webrtctransceiver0
a=ssrc:2 cname:user1883973515@host-1d6ab5fd
a=mid:video0
a=fingerprint:sha-256 C8:18:03:72:A6:76:16:F8:3B:AA:90:B6:F9:07:D8:EC:97:30:31:EB:89:D5:E0:3C:39:E5:5C:86:19:46:3F:2D
a=rtcp-mux-only
m=audio 0 UDP/TLS/RTP/SAVPF 100
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:yeX7Uo77YZU9e+r20k6CBeeFOLW1sHDm
a=ice-pwd:kfKp5/bKVb+JQKdSSkiKaPPESAUC4P95
a=bundle-only
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:100 OPUS/48000/2
a=rtcp-fb:100 transport-cc
a=fmtp:100 sprop-stereo=0;sprop-maxcapturerate=48000
a=ssrc:1 msid:user1883973515@host-1d6ab5fd webrtctransceiver1
a=ssrc:1 cname:user1883973515@host-1d6ab5fd
a=mid:audio1
a=fingerprint:sha-256 C8:18:03:72:A6:76:16:F8:3B:AA:90:B6:F9:07:D8:EC:97:30:31:EB:89:D5:E0:3C:39:E5:5C:86:19:46:3F:2D
a=rtcp-mux-only
[WHIP] Resource URL: https://janus.XXXXXX.de:7080/whip/resource/AJuynlwh4xVSwDTd
[WHIP] Received SDP answer (1107 bytes)
v=0
o=- 1699175397437505 1 IN IP4 XXX.XXX.XXX.XXX
s=VideoRoom 1234
t=0 0
a=group:BUNDLE video0 audio1
a=ice-options:trickle
a=fingerprint:sha-256 F2:82:92:B8:62:35:6B:7A:7F:E6:19:95:73:6A:E0:EE:F3:28:FE:DD:7B:04:DF:FD:BD:AD:4E:B0:23:51:2C:29
a=extmap-allow-mixed
a=msid-semantic: WMS *
m=video 9 UDP/TLS/RTP/SAVPF 98
c=IN IP4 XXX.XXX.XXX.XXX
a=recvonly
a=mid:video0
a=rtcp-mux
a=ice-ufrag:c3O/
a=ice-pwd:12MViSf6MGX5wbNx2UufFR
a=ice-options:trickle
a=setup:active
a=rtpmap:98 AV1/90000
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=msid:janus janusvideo0
a=ssrc:1364538497 cname:janus
a=candidate:1 1 udp 2015363327 XXX.XXX.XXX.XXX 54427 typ host
a=end-of-candidates
m=audio 9 UDP/TLS/RTP/SAVPF 100
c=IN IP4 XXX.XXX.XXX.XXX
a=recvonly
a=mid:audio1
a=rtcp-mux
a=ice-ufrag:c3O/
a=ice-pwd:12MViSf6MGX5wbNx2UufFR
a=ice-options:trickle
a=setup:active
a=rtpmap:100 opus/48000/2
a=msid:janus janusaudio1
a=ssrc:1094284014 cname:janus
a=candidate:1 1 udp 2015363327 XXX.XXX.XXX.XXX 54427 typ host
a=end-of-candidates
-- Found candidate: candidate:1 1 udp 2015363327 XXX.XXX.XXX.XXX 54427 typ host
[WHIP] Setting remote description
[WHIP] ICE gathering started...
[WHIP] ICE connecting...
[WHIP] PeerConnection connecting...
[WHIP] ICE gathering completed
[WHIP] ICE connected
[WHIP] ICE completed
[WHIP] DTLS connecting...
[WHIP] DTLS connected
[WHIP] PeerConnection connected
[WHIP] Sending candidates: candidate:1 1 UDP 2015363327 2003:e3:7f15:dd00:fdd7:ec37:e109:5fc8 44992 typ host
[WHIP] Sending candidates: candidate:2 1 TCP 1015021823 2003:e3:7f15:dd00:fdd7:ec37:e109:5fc8 9 typ host tcptype active
[WHIP] Sending candidates: candidate:3 1 TCP 1010827519 2003:e3:7f15:dd00:fdd7:ec37:e109:5fc8 57653 typ host tcptype passive
[WHIP] Sending candidates: candidate:4 1 UDP 2015363583 2003:e3:7f15:dd00:94ef:8bcd:6980:b207 33072 typ host
[WHIP] Sending candidates: candidate:5 1 TCP 1015022079 2003:e3:7f15:dd00:94ef:8bcd:6980:b207 9 typ host tcptype active
[WHIP] Sending candidates: candidate:6 1 TCP 1010827775 2003:e3:7f15:dd00:94ef:8bcd:6980:b207 59129 typ host tcptype passive
[WHIP] Sending candidates: candidate:7 1 UDP 2015363839 2003:e3:7f15:dd00:9ce1:a091:127:239e 45249 typ host
[WHIP] Sending candidates: candidate:8 1 TCP 1015022335 2003:e3:7f15:dd00:9ce1:a091:127:239e 9 typ host tcptype active
[WHIP] Sending candidates: candidate:9 1 TCP 1010828031 2003:e3:7f15:dd00:9ce1:a091:127:239e 36969 typ host tcptype passive
[WHIP] Sending candidates: candidate:10 1 UDP 2015364095 2003:e3:7f15:dd00:3b25:841d:4f2f:a5d0 42120 typ host
[WHIP] Sending candidates: candidate:11 1 TCP 1015022591 2003:e3:7f15:dd00:3b25:841d:4f2f:a5d0 9 typ host tcptype active
[WHIP] Sending candidates: candidate:12 1 TCP 1010828287 2003:e3:7f15:dd00:3b25:841d:4f2f:a5d0 39903 typ host tcptype passive
[WHIP] Sending candidates: candidate:13 1 UDP 2015364351 192.168.178.111 43765 typ host
[WHIP] Sending candidates: candidate:14 1 TCP 1015022847 192.168.178.111 9 typ host tcptype active
[WHIP] Sending candidates: candidate:15 1 TCP 1010828543 192.168.178.111 36331 typ host tcptype passive
[WHIP] Sending candidates: candidate:16 1 UDP 2015364607 fe80::8a6d:f779:c070:4d1 56121 typ host
[WHIP] Sending candidates: candidate:17 1 TCP 1015023103 fe80::8a6d:f779:c070:4d1 9 typ host tcptype active
[WHIP] Sending candidates: candidate:18 1 TCP 1010828799 fe80::8a6d:f779:c070:4d1 50399 typ host tcptype passive
[WHIP] Sending candidates: end-of-candidates
Please learn how to use the code/quote functionality when sharing logs, for better readability: I fixed the post for you.
That said, negotiation seems ok. As such, it may be the way GStreamer encodes or packetizes AV1 in RTP that is broken, in the sense that browsers don’t like it. Unfortunately I don’t have any rtpav1pay element in my GStreamer installation, so I can’t test this myself. Try enabling recordings in the VideoRoom, and converting the resulting video mjr with janus-pp-rec: if that one complains too, or the resulting video is unplayable, then that’s indeed the problem.
1 Like
Thank you for fixing 
rtpav1pay element is part of the rs plugins.
I built gstreamer myself using cerbero. Quite easy to build. The rs plugins are built by default. Only three commands are needed.
git clone https://gitlab.freedesktop.org/gstreamer/cerbero
./cerbero-uninstalled bootstrap
./cerbero-uninstalled package gstreamer-1.0
After that, you can easily use the gstreamer build inside a cerbero shell. No need to install it systemwide.
./cerbero-uninstalled shell
I’ll record some content and try to play it then.
I recorded the videoroom, made mp4 and the av1 playback with ffplay works perfect.
So I think the encoding and rtpav1pay seem to work fine.
Can you make this mjr file available somewhere? I’ll check if it works with Record&Play. Notice that not all browsers support AV1 in WebRTC: if you’re using Firefox as a WHEP viewer, for instance, I think it’s normal for it not to work.
Yes, sure… here we go
The mjr file
The resulting mp4
Works fine with Record&Play, when I use Chrome to watch it.
1 Like
Yes, but no WHEP playback and no video in videoroom…
You can check yourself WHEP Player
It may be a missing keyframe (maybe GStreamer sets one at the beginning and then doesn’t for a long time?), or too frequent ones. Maybe try different gop-size values? You may also want to limit the size of RTP packets, since with the high bitrate you have they may be too big, especially with the SRTP overhead added later: I think this can be done by setting the mtu property in rtpav1pay (1200 is a reasonable value, in case).
I set gop-size to 60… should be a good value for a 30fps stream and mtu=1200. But I think you are right. When the WHEP player is connected BEFORE we start WHIP, it’s all working. But when we start WHIP first, it’s not working. I think it’s the missing keyframe option. But i can set GOP size to any value, it makes no difference.
No, at 30fps a gop-size=60 forces a keyframe every 2 seconds, which is absolutely overkill and NOT recommended on WebRTC (keyframes so frequent are things you usually do on HLS and make things much worse on WebRTC).
That said, I verified streaming the mjr to a Streaming mountpoint that video works if you open the mountpoint before it’s live, and doesn’t if you join later. This confirms that the problem is that, if you start receiving RTP packets after the first ones are sent, the browser never receives the information needed to decode the video (keyframe, usually). So yes, it’s an encoder problem, and that’s where you have to work to fix it. If you can get a keyframe out every 5 or 10 seconds, that’s usually a good tradeoff for new viewers.
Well, honestly, I don’t know how to fix it, so I go for vp9 instead. That works really good, also with qsvvp9enc. Seems only av1 has this issue on encoding side.
Thank you very much lorenzo for your time and work. Helped me a lot.
You can see the result here. It’s also Stereo audio now, as I use the bugfixed vdo.ninja WHEP player.
That’s easy to fix in our demo too, you just need to munge the stereo attribute back in the SDP on both offer and answer. The Streaming demo in the Janus repo does that.
1 Like