How test automation helped the FA achieve peak performance

The FA is the world’s largest governing football association, with thousands of stakeholders interacting with various systems - between 100 and 120 different applications supporting its operations - from members of grassroots teams, professional leagues, and the national team, to those responsible for managing Wembley Stadium and those participating in its education operation such as teaching potential coaches.

With an increasingly heavy load, the performance of the applications was beginning to suffer.

Poor performance results in negative feedback

Nick Sewell, Head of Application Development at the FA, explains the situation: “We received lots of negative feedback about the slow speed of our system’s response. This can prove challenging during the peak and busiest time of the year. For example, we have a player registration window in which millions of people playing football at the grassroots level need to be registered. So system response time is incredibly crucial.

“In addition, we frequently announce squads for the England men’s and women’s teams, which increases visitors to the website. Therefore we needed to make sure that we’re ready for those seasonal demands.”

“Historically, all our systems have been on-premises,” Sewell adds. “Now, we’re moving them to the cloud. Part of that journey is to improve our processes and tools when it comes to software development and testing. We needed a tool that allowed us to test performance to address the issues around speed and response.”

The FA had previously been introduced to Tricentis’s NeoLoad enterprise performance testing tool by its technology services partner Cognizant, but Roy Botley, Head of Quality Assurance (QA), extended its use in a more methodical, iterative manner as part of the overall transformational process. According to Sewell, “Roy looked at the process around performance testing and kind of ‘topped and tailed’ it. As a result, we made NeoLoad the standard tool to use with a standard process.”

Eliminating performance issues

During the busiest period, it could take up to 10 minutes to register a single player on the old system which, in Sewell’s words, was “just not an acceptable level.” On average, there are 15 steps in the registration process, which sometimes increases to 30, with around 300 checkpoints to measure. “Now we’ve reduced each step of the player registration to under three seconds,” says Sewell. “And we’re able to do bulk registrations. Instead of doing 100 registrations, one at a time, we can now speed up and do 100 in one go – in under 30 seconds.”

“Our biggest issue – system response time – has been eliminated,” he continues. “We’re now at a level where the business is confident in what we’re launching. Systems Performance is in a very good place. NeoLoad just reduces the risk – and provides peace of mind it for the business and the technical delivery team.”

Doing performance testing properly

Previously, a queuing tool was needed to throttle the volume of traffic, so the system could handle more than 400 people at a time. Now, though, the FA’s modern, iterative approach to performance testing means it can handle 10 times as many users accessing its website concurrently.

“We’ve tested up to 4,000 users,” says Sewell. “We’re confident in that number. The benefit of NeoLoad is that we don’t have to worry about performance issues; visitors to our website are doing what they intend to do at the speed they expect.”

“One of the key changes is that we’ve got the business involved [in performance testing],” says Botley. “We’d hear from people within the business that the performance was ‘below expectations’ – but what did that mean? We asked them to define the non-functional requirements in terms of how long something should take, and provide some critical workflow scenarios to enable us to run tests and measure the results against those SLAs.”

The FA takes the same iterative approach to infrastructure optimisation, too. The association is migrating its on-premise systems to the cloud. “We went through a process of experimentation on the different settings – CPU, memory, and so on,” say Sewell. “NeoLoad would test against these different settings, and we’d keep tuning and adjusting until it reached the point of being efficient and working in the way we needed it to.”

“This is the first place I’ve worked where we do performance testing properly,” comments Sewell. “NeoLoad allows us to do that for every step, for all non-functional requirements.”