In Capture the Flag, Secure Your Knowledge we documented some of the technical underpinnings of how Salesforce uses gamification, specifically a capture the flag (CTF) style competition, to help Salesforce admins and developers learn how to secure their Salesforce instances. By competing in the game, participants are challenged to optimize security settings and controls and fix security bugs in code. The more hands-on security challenges you complete, the higher your score in the friendly competition. At the end of the challenge, participants walk away with points, bragging rights, possible prizes, and, most of all, a set of skills that have been reinforced by direct interaction with a live Salesforce instance.
We run the Secure The ‘Force challenges for multiple reasons. It is fun to participate in a CTF — as a participant, you are presented with challenges that can stretch your knowledge and skills. If you are motivated by competition, you can measure your progress against other players as you complete challenges and gain points. Our main goal for the CTF is to teach participants how to increase the security of their own systems and to reinforce learning through hands-on, interactive techniques.
Effective Learning
Secure The ‘Force is fun, but there is also solid science behind what we do and how we run it. Salesforce is a proponent of continual training and skills growth, not just for our employees but also for our customers, partners and fellow technical organizations.
Studies, including the Cisco-funded Multimodal Learning Through Media: What the Research Says study, document that the most significant increase in performance of higher order skills is achieved through Multimodal Learning.
Source: Multimodal Learning Through Media: What the Research Says
The study lists the major principles that are used to define learning optimizations and how they can be applied.
1. Multimedia Principle: Retention is improved through words and pictures rather than through words alone.
2. Spatial Contiguity Principle: Students learn better when corresponding words and pictures are presented near each other rather than far from each other on the page or screen.
3. Temporal Contiguity Principle: Students learn better when corresponding words and pictures are presented simultaneously rather than successively.
4. Coherence Principle: Students learn better when extraneous words, pictures, and sounds are excluded rather than included.
5. Modality Principle: Students learn better from animation and narration than from animation and on-screen text.
6. Redundancy Principle: Students learn better when information is not represented in more than one modality — redundancy interferes with learning.
7a. Individual Differences Principle: Design effects are higher for low-knowledge learners than for high-knowledge learners.
7b. Individual Differences Principle: Design effects are higher for high-spatial learners rather than for low-spatial learners.
8. Direct Manipulation Principle: As the complexity of the materials increases, the impact of direct manipulation of the learning materials (animation, pacing) on transfer also increases
Implementing Learning Principles
The genesis of our Secure The ‘Force capture the flag project is based on Principle 8, the Direct Manipulation Principle. By giving participants a purposefully misconfigured Salesforce instance and challenging them to fix it, our players are required to perform the same security actions that they need to take back to their day jobs. The gamification and point earning aspects of the CTF help to keep players interested in continuing the game, and the interactive challenges help to transfer the required knowledge and skills to the players long-term memory for future use.
The underlying components of the Salesforce CTF system are actively developed, and we regularly make enhancements to improve the effectiveness of the tooling. The learning principles are also used to direct innovation and development. After the debut of the CTF system at TrailheaDX 2019, the Salesforce team scoped out a set of improvements based on user feedback and observed user behavior in conjunction with the learning principles.
Secure The ‘Force games are run on the exhibition floor of a conference, which is a high distraction environment.
In order to maximize the ability of players to focus on the content and successfully accomplish the challenges, we had to minimize the friction of players needing to interact with both the CTF scoring system and their challenge system.
Increase Focus, Decrease Distraction
Guided by Principle 2: Spatial Contiguity and Principle 3: Temporal Contiguity, we implemented a new custom challenge type in code called an “interactive challenge.” An interactive challenge uses custom code to call out to the challenge evaluation APIs and parse the result of the evaluation. By using interactive challenges, we were able to keep players in the moment by having the description of what they needed to accomplish be directly inline with the functionality to evaluate the challenge and issue their flag. Using technology to execute well-defined and repetitive tasks allowed the participants to focus on solving the challenges, rather than the minutia of claiming a flag value.
We also leveraged Principle 7, Redundancy Principle, to guide our development efforts. By building an automated interaction flow, we were able to reduce redundant information, allowing the players to use their time and energy to work on the details of the challenge tasks without needing to perform a separate series of actions to claim their flags
Results
We ran another successful Secure The ‘Force game at Dreamforce ’19 and had over 480 people play the game over three and a half days.
As part of our continual improvement process, we analyzed the feedback we gathered from the event. In addition to the very positive feedback from our players during the event we also looked at the hard data that we gathered.
Given that the Secure The ‘Force took place in the Trailhead Zone expo floor of Dreamforce ’19, where we shared space with hundreds of other booths, talks, and activities, we are very proud of what our metrics show. Over the 3.5 days of play, our players spent an average of over 40 minutes interacting with the game. Players also worked on an average of 20 questions. For a security-focused game that took place directly in the middle of the hustle and bustle of a massive conference, we feel that our players were able to walk away with a deeper and more internalized understanding of the security options available to them in Salesforce products.
Looking Forward
With 2019 behind us, we are planning on a bigger and better 2020! We will be expanding the scope and scale of our externally facing CTF competitions as well as expanding our use of CTFs for our internal training. Keep an eye on this feed for more articles as we continue to build and refine both our tooling and our methodologies.
If you are interested in how you can use learning principles and gamification tooling to increase the interest and retention of security topics for non-security professionals, I will be leading a hands on learning lab at RSAC 2020: Everyone Can Play! Building Great CTFs for Non-Security Folks. I hope to see you there!
