Mobile Game Testing: Industry Leaders’ Challenges and Strategies

Leaders from Playtika, Unity Technologies, and Infinity Games share some of the challenges they’ve faced and solutions for delivering highly engaging games.

What Are Some of the Biggest Mobile Game Testing Challenges You’ve Faced?

Melissa shared that one of the biggest challenges teams find themselves tackling first is testing on the myriad of different devices and operating systems that are a direct result of the growing fragmentation within the Android ecosystem. For example, the emergence of newer devices every year, makes testing more complex than ever before.

  • Testing ads and third party integrations, and
  • How to create games with high-quality, using advanced features such as inter-player chat, matching, awards, etc.

What Are Some Ways You’ve Overcome These Challenges?

For Robson, he has found ways to help his team overcome the challenge of not having dedicated testers. He assumed the role of the QA lead and also adopted Apptim, which has helped enable anyone on the team to run functional and performance tests on their games.

“Being able to have a tool (Apptim) that recorded the play session of the person and automatically got all of the logs and generated a performance report in the end, is something that for us, as a small company, was pretty huge. We’ve improved the quality of the games a lot because of the crashes or exceptions that we couldn’t catch before, and now we have that access.”

Aliaksandr shared that, to deal with the high cost of end-to-end mobile game performance tests, he found you can reuse existing automation tests. He commented, “It will also save you time in such cases because almost every time you have automation already created for your functionality, why not add some markers or some [performance] measurements, and it will help you to also understand what’s going on where you are.”

What is Your Advice for Reducing Testing as a Bottleneck?

For Melissa, the short answer may seem controversial to testers, but she shared that the best thing she’s learned to do is to get out of the developers’ way. She said,

“Removing myself has been the biggest, reducing that bottleneck, putting automation in place, letting the developers really collaborate on those ideas and moving the production cycles and everything faster because I am not stationed on a team, I’m not embedded, but I’m there as somebody that can coach them and help them.”

She explained that she’s a big advocate of modern testing principles, which give a lot of credence and leverage to the teams themselves. “Developers know the code the best, the more they interact with the code and the more they understand the code, the more they can test it, and they can test it on a level that oftentimes as someone who was traditionally in the role of a tester might not be able to get to.”

“Automation always makes things easier as much as it makes it harder, but you have to balance the positive with the negative, right?”

The first answer that came to Robson’s mind about reducing testing as a bottleneck was, “You can always ask the developer to stop creating bugs. That works very well.”

“Using a tool like Apptim for that can also help catch some underlying bugs and reduce the friction between the tester and the developer because sometimes there is a lot of back and forth.”

To illustrate the point, he said, “Today, I had an issue where someone reported a bug. ‘Hey, this is happening at the end of level six of the game.’ I checked the video from Apptim because he was using it for that session, and it was actually at level 10. So if I didn’t have the video to back it up, I might have lost quite a lot of time chasing a bug that didn’t really exist.”

How is Testing a Mobile Game Different Than Testing Other Mobile Apps?

The consensus among the panel was the added complexity that games have at different levels and understanding how they interact. Melissa stated,

“Mobile apps are complex as it is, but you have multiple platforms you’re looking at, and then you multiply that by the game complexity and the algorithms and the performance you have to maintain.”

Melissa shared several other factors that add more complexity in mobile game testing:

  • Dealing with game performance, device performance, and network performance.
  • Testing various ad layouts and formats: For example, you can have a full page static ads, banner ads, video ads, etc.
  • Non-linear usage and behavior patterns from users open a wide array of possibilities. Depending on the order that the user does something, you might find a different bug.

What Aspects of Non-Functional Testing Like Performance Do You Prioritize in Your Games and Where?

For Aliaksandr, when choosing where to prioritize performance, he stated areas like the game lobby and anything a brand new player will see first are top priority.

How Do You Set Mobile Game Performance Benchmarks?

Aliaksandr volunteered to expand upon this follow up to the previous question, saying that it can be measured from the number of users. But first, it’s important to understand business critical transactions. Then step by step, you can do benchmarking, load testing, and so on. He also shared, “Use such a tool as Apptim to set markers and see what’s wrong with these events, with this transaction, this feature, deep dive, and use code profiling… and so on.”

Product-led growth enthusiast. Life-long learner and traveler. Basic San Diegan with a love of tacos and stand-up paddle boarding.