Sarah Bradley | Symphony

The Problem

MyCompliance Admin Homepage displaying Symphony Banner

Creating campaigns or courses for different groups depending on their job role, or their seniority is time consuming. Many organisations do not have a dedicated person to create and send this vital training, and they may find it difficult to know how to segment their userbase so that they receive the most meaningful training.

Organisations also need to be able to assess how effective their cyber security training is, and identify areas where they need to improve.

Research

Symphony Dashboard — Symphony Configuration Screen

We conducted unmoderated user interviews with admin users to discover their concerns and painpoints. Summary of the findings:

Admins relayed that the most time consuming part of creating a year long campaign was selecting the right content to ensure that they were covering all topics relevant to their companies cyber security strategy.
Admin users reported wanting to have the abilty to find out at a glance how well their userbase understood different cyber security areas and how departments compared against eachother.
We found that they wanted an easy way to tell if users were improving.
Companies wanted to be able to send training with either a heavy or light touch depending on the knowledge level of the user around a given cyber security topic.
Companies were interested in how the method of content delviery effected engagement rates. For example, were users more likely to complete a course delvered through MS Teams vs the MyCompliance Platform.

Requirements

After user interviews and market research, it was decided that training the machine learning algorithm to assign a user a risk score would be valuable to organisations. It would also be really compelling to an end user - finding out you pose a high risk to your organisation is likely to encourage you to take training more seriously and complete in a timely manner. We could also use this risk score to assign varying levels of content. The requirements were:

Allow the ability to gauge the baseline knowledge of a user on the platform, wether they are a new user or existing.
Assign an organisational risk score - seeing at a glance the levels of high, medium and low risk amoung the userbase, as well as an overall risk level of the entire organisation.
Display engagement across the organisation, and seeing how this affects risk levels.
Ability to automatically assign users content based on their role, department, level of seniority and risk score.
Ability to automatically serve users content that they were more likely to engage with, eg. if they have a preference for podcasts vs video content.
Display to admins how their organisation compares to others in the same industry.

We also knew that there is a reluctance for users to fully trust AI/Machine learning at first, so we would need to take the approach of making reccomendations based on the algorithm, rather than making decisions. Lean towards making suggestions instead of decisions. In further phases, as the admin builds more trust in how Symphony will work, we can automate more and ask for permission less.

Design Process

We decided that in order to establish the users baseline knowledge of various cyber security topics, we would target them with a questionnaire. Based on the questionnaire results and there acivity on the platform over the last 12 months, they would be assigned a risk score.

The user would then be sent on a year long learning path, which was tailored to their seniority, department and role, with the amount of training delivered increasing with risk.

The users actions on the platform such as gaining a high score on a quiz or successfuly reporting a phishing simulation email for example, would feed back into the data model and update the users risk score.

We wireframed and tested the questionnaire with users to ensure that we had the correct balance of asking enough questions to accurately establish a baseline, yet short enough so that the user doesn't abandon.

Initial idea generation - how would the admin segment their users?

We conducted several rounds of user testing at the wireframe stage and iterated heavily so that we were sure that we had the balance right between automation and control. We also wanted to ensure that the reporting gave admins the information the required.

After user testing we decided that we would include the ability to drill through the dashboard reports to display stats on how individual users performed on the baseline questionaire and training topics.

Symphony - Dashboard

Responding to user feedback, we decided to allow admins to re-order the training topics on the learning pathway. We wanted to allow them to send relevnat training sooner if the threat had become more important to them. For example, if a company had recently had a physical security breach, they could decide to send the training on that area the following month.

Outcome

We have not yet launced Symphony, development is still in progress. We have conducted remote user feedback sessions with customers and the feedback has been very positive so far, though there have been some suggestions of further features that would enhance the product.