I suspect you’re simulating all those users from the same browser instance, and using HTTP as a transport for the Janus API. Browsers have a limit to how many TCP connections you can keep up for HTTP to the same server (I think 6?), and since long polls will keep one busy, new requests will need to wait for a connection to become available.
As such, this is a limitation of your testbed, not Janus. Just use different browser instances or profiles to simulate more than 3-4 users.