And we’re back with Philip Cote who is with us again to talk about load testing for game server hosting. If you just got here, be sure to check Part 1 where we introduce the reasons game studios should do load testing before launching their game. Now let’s get right into it!

Is there anything special about video game traffic? What differentiates it from standard web traffic, especially during a load test?

The truth is web traffic has evolved a lot. With the way it currently works, you could start a session on Amazon but end up being connected via 40 different servers around the globe. Each of these servers caters to different aspects of your experience on that website. And with load balancers shifting traffic between different machines, you don’t have to worry about persistence between you and the server. Game traffic is entirely different. In an online game, the player needs to be online from the start of the session till its end. In this scenario, if you try to cut the connection, the game will be lost.  Because of this peculiar feature, it’s impossible to redirect game traffic from one server to another – at least, not yet. Load balancers cannot be used to even out the load across five different servers. And unlike web traffic, it’s not possible to use clusters in game hosting. It’s just not how game servers work. Since game servers never leave any room for plan B, we at Edgegap strive for stability in our load tests. We check our servers to see if they can last the entire duration of a match without any problems. So far, they can – and we mostly attribute that feat to the containers which fuel Edgegap’s solution.

How long do your load tests usually last?

It depends on what we’re trying to test. If our focus was on long-term stability, we’d run tests for seven days straight. If we wanted to test the logic of our tool, three hours is more than enough. The time frame for each test depends on the component that’s under scrutiny.

You mentioned earlier that you use Artificial Intelligence for your load tests. Could you shed more light on these AI?

Switching to AI for our load tests is a decision that has greatly improved the quality of metrics we get. Under normal circumstances, we aren’t able to receive data from the players themselves. But our AI technology allows us to gather all sorts of first-party data from the robots that play our simulated games. Resorting to AI also helps us know if we made the right decision at all times. By comparing our parameters at the end of each load test, we can score each player. These scores help us determine if the game selection was the best one and if the latency didn’t suffer throughout the game. These are the kind of metrics that validate load tests and they can only be gotten through AI.

Are there any specific things that you look for in a load test?

Of course! Right now, we’re at 250 locations around the world, which means that we can deploy any type of game over all of those sites. Our goal is to get up to 1000 by the end of the year. That would be 1000 sites around the world that need to be up and running. When you group a thousand servers within the same data center, it becomes a challenge. Scaling in 1000 different places at the same time becomes a nightmare because you need to monitor all the sites that are involved. At the same time. Accomplishing this requires a reliable internet connection, so we usually assign some internal tools to validate the connections and ensure that all servers are responsive. We use an internal module to validate the health of each access point at every minute. Frequent checks like these ensure that our services don’t deteriorate.  If there’s a site with any of these issues, our systems automatically stop sending games to that location. And at the end of the day, we take inventory of all the locations that failed during deployment. We also use machine learning to look out for sites that repeatedly give us errors during deployment. Blacklisting those locations is our way of ensuring our network only comprises problem-free servers.

Will load testing for 1000 locations take longer than for 250?

Not necessarily. We would just need to incorporate more players into our load tests and have more game servers deployed.

What do you do when a certain location fails?

During the load test, the automated part of our system removes the failed sites from the selected location. This way, we don’t send traffic to those sites anymore. After that, I do follow-ups with the faulting vendors. Since multiple vendors provide multiple portions of our infrastructure, that’s a lot of follow-ups. We urge the vendors to fix what needs to be fixed so that we can bring the site back online. Our solution is based on having hundreds of decentralized servers so it’s always best to have as many sites as possible available for deployment at any given time.

Do you track each load test and compare results between them?

We do. The variables that we track range between hundreds to several thousands. Again, it all depends on the focus of each load test. But the components we mostly keep our eyes on include: the speed of deployment, its quality, the number of sites available, latency, jitter, packet drop, and of course, cost. In game hosting, cost is always a crucial factor. At Edgegap, we protect the viability of our solution by ensuring our servers always pay the right amount of money for available sites. We also compare the sizes of the machines that we use for each load test, along with their type and efficiency. So yes, we do a lot of comparison with past benchmarks to determine if each load test met our expectations.

What third-party tools do you use to monitor the load tests and your environment?

Overall, we’re spread out over a lot of tools to monitor our environments. For everything pertaining to CPU, memory, and network, we use our internal monitoring tool. Whereas we store all our generated logs on our BigData repository. We also use this tool to analyze the conclusions of our tests.  The good thing about doing a lot of load tests with the same tools is we can prepare some dashboards as templates and use them repeatedly. We’ve found this tactic to be resource-efficient. However, we’re always on the lookout for new tools that pinpoint the minor, seemingly insignificant factors that contribute to a player’s bad experience.

How do you make sure you learn from each load test?

Once my team and I gather all the metrics, we do a debriefing. We exchange notes. We earmark what went wrong or right. But most importantly, we make an effort to answer the question ‘What could we do better?’ Making sure that my team is fully involved in our load tests is something I consider non-negotiable. Why? Because I find that keeping everyone in the loop always spurs them to go above and beyond. So whenever we gather for some brainstorming, I share my ideas on what we can do differently for the next load test. The developer contributes his two cents as well. And so does the next – it goes on and on. When you have a team that’s committed to constant improvement, like the one here at Edgegap, you find that this collaborative effort snowballs into better and more insightful load tests.

Conclusion

Like ancient methods of divination, load tests are a way of seeing into the future. They are carefully-planned, data-oriented processes that make sure your game doesn’t crash midway into an online match, leaving your players stranded and unhappy. If you’d like to learn more about how you can host your game on decentralized servers that have been tested by thousands of AI players, reach out to us to Get Started!