The real world impact of Serious Games

Game-based learning has become increasingly prevalent in academic and

corporate environments over the past ten years. The market is forecast to more

than triple in size between 2022 and 2028 to become a $50bn industry (Research

and Markets, 2021).

Although the pandemic was undoubtedly good news for this sector – as isolated

individuals found meaningful social connections in digital gameplay against

others – this growth has primarily been driven by research, which consistently

shows game-based learning to be highly effective for learner engagement and

impact. For example, studies by authors like Zafar et al (2022), Perotta et al

(2013) and Sitzmann (2011) have found that game-based learning when compared

to traditional training methods:

• Improves knowledge retention by up to 90%

• Improves training completion rates by more than 300%

• Improves learner satisfaction by more than 80%

How to measure impact?

Despite these encouraging findings, the perennial challenge faced by most

training providers and Learning & Development (L&D) departments is to connect

the cause and effect of a particular initiative with a meaningful business impact.

This assumes that the training has overcome the initial challenge

supplier and the buyer is aligning effectively on intended outcomes. It can be

useful here to apply James Kirkpatrick’s (2016) model of training evaluation:

Level 1 – Reaction:

How did participants like the training?

Level 2 – Learning:

Did the participants learn something new?

Level 3 – Behaviour:

Did the participants’ behaviour change after and due to the training?

Level 4 – Results:

What business outcomes did the training bring for the organisation?

Level 1 impact assessment

Level 1 assessment is quickly done with the familiar post-session “happy sheet.”

In their own right, enjoyable and engaging learning experiences are becoming

increasingly central to a positive employee experience (LinkedIn, 2021). Given

the immersive, competitive and social formats, satisfaction and engagement

scores for PBS games regularly exceed 90%. PBS' Net Promoter Scores, often used for cross-industry benchmarking, regularly exceed 70 (versus traditional training averages of 30).

Comments like this follow many courses:

"By far the best interactive training I've been involved in." - Armando Garcia, Visible Value Participant, 2022

This sort of feedback is more than just good marketing material: L&D buyers like

to be reassured that a solution has been enjoyable – especially in an era of

unengaged workforces – and are more likely to re-buy as a result. However, what

happens as a result of the training is more important.

Level 2 impact assessment

Impact assessment should then progress to looking at knowledge acquisition,

Level 2 in the Kirkpatrick Model. This is best done either with follow up quizzes or with pre/post learning impact data. Here's an example of this in action for a Visible Value program run by PBS in October 2023:

Qualitative data on learning impact can also be shown in participant feedback:

"A wonderful experience – financial terms are no longer 'Greek' to me and I have a better understanding of the impact my operational decisions have on the financial side of my division." - Renisha Singh, Bespoke course participant from JCB, 2021

Level 3 impact assessment

In determining whether knowledge acquisition leads to behaviour changes, educators rely on wider and longer-term data points. Factors like internal observation, coach/manager feedback, and additional self assessment are often beyond the control of a training supplier. After each course, PBS asks all participants "As a result of this training, what are you going to do differently in your job?" Their responses then serve as a mini-action plan, which

should lead to relevant and trackable changes in behaviour. For example:

"As a result of the experience, I’m going to collaborate more with sales, supply chain and finance. We need a good marketing plan based on collaboration with sales in order to get their customer feedback and information on market conditions – as well as motivating them to sell the right products." - Marvin Mateo, Visible Value participant, 2022

More useful here though is behaviour data provided by the client. A good example of this in action is from PBS' work with Cargill on a Strategic Pricing learning program for commercial managers. An objective of the program is to increase the consultations that commercial team members book with the strategic pricing team. Cargill report that these consultations have increased by more than 30% since the program was launched virtually in 2021.

Level 4 impact assessment

Whilst learner engagement, knowledge retention and behaviour change might be

“Key Performance Indicators” of training impact, the holy grail remains a positive

effect on business results. Of course, connecting cause and effect in organisations is rarely simple or linear.

Nevertheless, it is crucially important that the training supplier and buyer are

aligned on the real business outcomes the solutions seek to support. The regular

critique of HR, L&D and Talent departments is that they are not connected

enough to the business (Blind, 2018). Undertaking this alignment for every

solution will result in more win-win situations.

Developing the case study of Cargill's Strategic Pricing program, sponsors report that participants of the program have generated approximately $10 million more in opportunity pipelines compared to non-participant control groups.

Elsewhere, PBS’ serious games have a track record of contributing to measurable business outcomes in some of the world’s largest businesses:

• New joiners to Compass Group were taking too long to understand how their day-to-day decisions led to business outcomes. PBS has trained 1000s of Compass employees to attain a common understanding of "value," which has supported a 700% increase in share price since the program started in 2006.

• A North American division of Ford was underperforming. The plant managers participated in a customised PBS commercial acumen experience. 12 months later, the division reported a $34 million increase in profits.

• If a drug filing process for Roche is delayed, it can be worth up to $80 million per month in losses. Since working with PBS in 2014 to minimise this problem, Roche has completed all drug filings on time.

Seeing the business impact of a solution can take many months. This is especially

the case when the goal is people-related, such as reducing employee attrition,

increasing diversity, or even boosting the promotion rates of participants. In all

cases, therefore, it is important that clients track learning impacts related to

engagement, knowledge retention and behaviour change.

Relationships matter

Almost all L&D buyers and suppliers like to claim that they are relentlessly

focused on the learning that impacts organisational outcomes. Even if objectives

can be agreed on, the measurement and attribution of actual impacts is

extremely challenging in this fast-moving, complex and interconnected

environments.

As ever, the unmeasurable "gut feeling" of a buyer about a solution can be just as

important. The longevity of PBS’ client relationships reflects its ability to operate

as one with the client whilst delivering on their ever-evolving needs.

The perspectives of the buyers themselves provide the best evidence of impact: "We were very impressed with the detail that PBS went into when designing our dealership simulation with us. They were eager to learn about our operating model, so they were well-placed to deliver the training too. This has achieved our aims of creating the understanding that we sought. Our dealer-facing team are now much better placed to support our dealers."

Jon Nixon, General Manager at JCB 2022

"We evaluated several partners to create our sustainable finance game. PBS had the most innovative and relevant approach to understanding and meeting our needs. Participants’ feedback has been incredibly positive. The game experience really engages participants and enables them to understand the complex challenges and interdependencies of sustainable finance in a playful and social way. We have found PBS to be an extremely capable, proactive and innovative partners and we look forward to expanding our relationship with them." Klaus Woeste, CEO at Attain Learning 2022

"We decided to work with PBS to co-create our talent management game because they have a proven record in designing and delivering high-impact L&D solutions. Importantly, we knew that PBS would enter into a partnership with us and be able to design something that fit PepsiCo’s culture and unique context. The game has already resulted in higher learner engagement scores and significantly increased our Managers’ confidence and ability to make better talent decisions."

Andrew Collier, L&D Director at PepsiCo 2022

#seriousgames #trainingimpact #learninganddevelopment #corporatetraining #experientiallearning #measuringtrainingimpact