1.) Make sure your test runs long enough
This isn’t necessarily a specific measurement of time, but CDN traffic loads are pretty elastic. Everything runs great until it doesn’t. Schedule your test for a weekend when a popular new game update drops (like COD) or a large Xbox update is being served. These are events that historically stretch CDNs to their limits and can cause performance problems for other users. Don’t just run your test for four good days and assume that it will always be that good.
2.) Measure the 99th percentile
At this point, all CDNs are good, at least most of the time. Measuring averages tells you very little about how a CDN performs, if anything. Focus on the worst performance. It doesn’t matter that 50% of downloads were served sub 50 seconds as much as it matters that 1% of downloads were served at 800 seconds. You care about 100% of your end-users; find a CDN that cares about them too.
When you test for the right length of time and measure the 99th percentile, you get an accurate representation of a CDN’s weaknesses and can setup a multi-CDN configuration where the CDNs complement each other. It’s outstanding to have 1 CDN in your mix that serves a popular update and has lowered throughput during that time. It’ll hurt your QoE if every CDN in your mix serves the same content and has the same bad days. Don’t let noisy neighbors dictate the experience your users get.
3.) Don’t fall into the latency trap
You know how your game works. In most cases, an asset won’t load, and users can’t use it until it’s fully downloaded. Measuring the time to first byte (latency) tells you very little about the performance end-users can expect. It’s time to the last byte (throughput) that matters and gives you a complete picture of what the QoE will be in-game.
“The average of an elephant and a mouse is a cow, but you won’t learn much about either elephants or mice by studying cows.”
Lawrence Drowsky
4.) Always evaluate based on your use case
When possible, it’s best to test each CDN independently, but if you’re going to use 3rd party measurement tools and statistics, make sure you’re tweaking them to represent your setup. If you are serving large file downloads, don’t rely on small file delivery numbers to tell you how a CDN will perform with your content. Be sure to test each region you’ll deliver based on the right file sizes and parameters.
“The deep, fundamental question in statistical analysis is Compared with what?”
Edward Tufte
5.) Compare your options
3rd party measurement tools can be useful in setting a baseline. If you set proper parameters for the data, you can get a decent idea of the “industry average.” If you don’t have time to evaluate every CDN in the market for your game, it makes sense to set benchmarks and test the contenders independently.
