Mersoft has been developing solutions to test network technology and web applications for over 15 years. Load testing any web application requires the ability to start up large quantities of virtual clients, run a series of scripted test scenarios, and collect the results. The challenge is compounded in WebRTC load testing with a live streaming media application where the quality of all the streams between the end points must be measured as well as client behaviors, server components, load balancers, STUN and TURN servers, etc.
Most testing apps (and there are plenty) will start up the prescribed number of load-generator clients and test app functions and the stream in the same way – from a user’s perspective. There are a few downsides to this approach:
- Full Windows+browser virtual clients are resource-intensive and take time to set up
- The complex environment is error-prone because the odds of changes to the client are high. They come from operating system patches, browser updates, and the web application being tested. It gets worse if the application exists in a complex enterprise environment where different groups support the authentication, middleware or other components used by the app. Teams spend time investigating failures and updating scripts instead of testing the product.
- Many users will access the app server via a native iOS or Android app, or a set-top box. Emulating a browser on Windows doesn’t serve any purpose in these circumstances.
When Mersoft decided to put its testing chops to work for WebRTC load testing, we took a different approach. First, we created a proprietary ‘headless webRTC’ client that runs extremely lean and fast. Plus, it can gather much more information about the WebRTC media. Here’s a screen shot of recent test session we did with our own Mersoft move™ and stream™ products.
In this case, it was a small test with only 20 clients and running 3 minutes each. Scale capacity is technically limitless, but regularly can handle thousands. It takes only a few seconds for each virtual client to get started and only require a small footprint for the OS and other overhead to execute the functions. The load tests run for as long as needed and, since they are easy to repeat, we can set it up to run in cycles while we intentionally break things to ensure each part reacts properly. When issues do arise, each client has detailed PCAP files, log files and a video recording.
Since Mersoft test™ is in the cloud, it replicates actual user experiences by accessing the apps from the public internet across whatever geography is needed. This ensures that the firewalls, load balancers and network connections are verified, not just the application and it’s servers. Mersoft test™ can also run in a customer’s data center, or even a combination.
Testing with a browser is supported too.
I just said how full browser clients are fat and slow, and error-prone, and offer less visibility to performance, but sometimes you still need them. For complete WebRTC load testing, it’s valuable to test user interface functions along with WebRTC functions. So, that’s why Mersoft test™ can create Microsoft Windows virtual machines and can run Chrome, Firefox or Internet Explorer/Edge. Use full Selenium scripting capability alongside the WebRTC scripting for a comprehensive testing approach. Soon, we will provide virtual Android and iOS devices as well.
In practice, we find that using a full browser client is helpful for other types of testing as well. This includes performance monitoring in production or test environment verification as part of a continuous integration process. So, it helps to think about the type of testing scenario and then decide the best approach from there.
For those with experience in load and performance testing in network services, web applications, e-commerce, traditional VoIP, or other areas may be veterans in their own domains but may be somewhat new to WebRTC. In these domains, best practices and tools of the trade are established. With WebRTC load and performance testing, there are some unique challenges and complexities to overcome and ensure the application is ready to handle anything that customers throw at it.
At Mersoft, we’ve been building and running testing, diagnostic, performance and analytics tools for over 15 years. So since we started working with WebRTC several years ago, the experience showed we needed some unique tools to make sure our customers’ services were solid. They include applications that leverage real time communications for certain functions, integrations of multiple vendor unified communications (UC) products, live streaming social media and other platforms. Mersoft has several of our own products that use WebRTC so we needed tool for our internal teams too.
When it came time to identify the WebRTC load and performance testing features most important to include in Mersoft test™, we listened carefully to our customers, drew from own experiences building WebRTC products as well as our years of testing and analytics work in other domains. Here’s what we found matters most:
- Test the entire platform: signaling (the network), application functions, media streams. Most teams test only one of these.
- Understand the topology. Check the call setup. Are the SDPs being exchanged successfully? Are there firewall issues?
- Keep track of all the ICE candidates and their various media streams in every direction. When some fail and others succeed, intermittent problems occur, and are tough to find. Cover paths through TURN servers and peer to peer. See how many TURN servers are used, if it uses a TURN-first policy and that affects startup or latency.
- Understand the nature of each client. If it’s a browser, consider brand, version, security, local settings. If it’s a native app (consider for example, if it’s node.js, is it “stomping” on its variables when under load?).
- Test both by using the client itself and by running “headless” for faster speed and richer analytics
- Gain insights. Don’t just get reams of log files. Instead, generate useful information that helps engineers find problems faster
- Draw conclusions about the performance of a group of tests rather than just looking at each test separately
- Include WebRTC testing as part of a continuous integration process. That way, the testing team can spend its time investigating problems and making the testing even better, rather than running tests
At Mersoft, we are passionate about helping our customers develop scalable and robust WebRTC products, whether we are working with global video surveillance companies, startup live streaming social media companies, enterprises managing complex unified communications platforms or others.
- An intuitive dashboard delivering insights as events happen.
- A notification engine configured for SMS, email, Slack and HipChat.
- Microservices to post events to Splunk or AppDynamics.
Want to talk to one of our engineers or see a demo? Email: firstname.lastname@example.org or call 913-871-6200.
New Case Study Published:
Executive Summary – WebRTC Analytics Video Chat Case Study
A major U.S. communications provider worked with Mersoft on load testing and production monitoring. The focus was their video chat services that use WebRTC. The services include 2-way video, audio, text messaging and screen sharing. It was a fairly new product, but gaining in popularity. They wanted to make sure it was ready for large-scale use.
Mersoft and the client collaborated to set up Mersoft test™ for production monitoring and several load tests to identify improvement areas. They quickly found and resolved several issues from node.js issues with the client, web server capacity issues, SDP (session description protocol) exchange issues, ICE candidate issues, and problems when a selective forwarding unit was utilized versus a peer-to-peer connection. They resolved each problem quickly, improving the customer experience.
The engineers used Mersoft test™ to emulate the client experience so as new versions are released, the scripts can be easily rerun. The client now has the capability to manage the unique complexities of a large-scale WebRTC product and can save both time and labor in addition to getting insights unavailable from other products. In addition, all the graphs in this case study were taken directly fromMersoft test™. More graphs are available to help teams visualize the data effectively.
Download it here.
KANSAS CITY – APRIL 3, 2017 – FOR IMMEDIATE RELEASE. Today, Mersoft announces the latest release of its popular and innovative Mersoft stream™. The engineering team of this Kansas City-based technology company has disrupted the market yet again with new features.
A new SFU or “Selective Forwarding Unit”: It creates a 1-to-many relationship for media streams that range from a few connections to thousands and does so without transcoding. As a result, Mersoft stream™ outperforms other technologies with its much lower latency and a very small processing footprint.
HLS integration with CDN integration: Customers can now create even more versatile products by selecting either a WebRTC or HLS stream for use in their client application. This option is especially useful when leveraging a content delivery network (CDN). Now, customers can leverage Mersoft to build apps that can broadcast not just to hundreds or thousands of clients, but now millions. With it’s stateless architecture, Mersoft servers can be horizontally scaled enabling both load distribution and geographic redundancy across the globe.
Mersoft stream™ still runs both as a SaaS solution managed by the excellent pro services team at Mersoft, or hosted in a client’s datacenter. With flexible pricing options and a talented system integration team, it’s never been easier for companies to integrate new live media.
Mersoft CEO Ron Sloop commented, “The Mersoft product team continues to impress. After months of work, this is one of the biggest releases of new features in quite some time. We’re very proud of the team and already getting significant attention especially from companies needing to scale with large volume.“
In any buy-versus-build decision, of course, every product vendor wants a business to buy their product. Don’t try and implement it yourself they say. It‘s just like a frozen dinner. Why labor over a hot stove when you can microwave a delicious meal in just 7 minutes! A WebRTC live streaming buy or build decision is similar.
Companies—especially technology companies— love to build things themselves. However, time is money. We see this so often in WebRTC live streaming. Companies full of smart engineers learn about WebRTC. They try a few reference implementations (Google’s AppRTC is a popular choice), pick some Platform-As-A-Service APIs, and an engineering team will think, “How hard could it be?” Then, three months later, they realize it is more challenging than anticipated. The hypothesis seems to be that WebRTC is a complete solution. But really, it is just a really good tool. There’s still significant WebRTC experience needed to use that tool to create a successful solution in addition to a deep understanding of audio and video protocols, performance and best practices.
Some of the challenges people discover:
- Reference implementations are simple browser-based video chat, not useful for an actual product.
- WebRTC does not specify signaling and there are many ways to do it, each with pros and cons depending on the situation.
- Hard Costs: WebRTC has its own challenges and takes time and R&D labor to master.
- Opportunity Costs: In many cases, WebRTC is not delivering a critical part of the business. Those smart engineers could have focused on strategic features that make the business unique, rather than on WebRTC.
At Mersoft, we’ve invested a lot of research in our products that use it. Furthermore, we specialize in specific use cases focused on delivering the best technology to convert live media to WebRTC, HLS and DASH. Mersoft has the fastest startup times, the lowest lag, and does it all without transcoding. Our SDK eliminates the need for Flash, browser plug-ins or ActiveX controls. These are the areas we deliver real value to customers so they can jump over the learning curve and focus on their core strategy.
A few things that set Mersoft apart:
- We don’t force customers to use our client apps (the SDK is powerful)
- We don’t force customers to use our cloud (can be installed in your datacenter)
- We have the ABSOLUTE BEST technology to convert video to WebRTC, HLS and DASH. The fastest startup times, the lowest lag, and no transcoding using Mersoft stream™.
It seems to be a fairly common situation that Mersoft will engage after a customer started to do it on their own. Occasionally in a sales visit, they ask, “Who are your competitors?” We will say, “Sometimes, you are!”
While Mersoft is focused on delivering key value in specific areas, we do it for some pretty diverse use cases including home security, video surveillance, social media live streaming, unified communications and more. So, contact Mersoft today and learn about how we can help you get your WebRTC live streaming product to market faster, and don’t labor over a hot stove.