Measuring effectiveness of VR through evaluation and feedback

Being immersed in virtual reality is an exciting and unique experience, but after the initial novelty has worn off, how can we measure the effectiveness of this experience?

Participants often remember the first time that they put the goggles on and become immersed in virtual reality. It’s an amazing, futuristic experience and the novelty and incredulity of the situation is often clear from verbal and physical reactions. The vast majority of people will take the headset off and say “Wow! That was amazing!” which is great feedback – but not enough to gauge the effectiveness of the intervention. To do this, you will need to ask questions of your viewers to establish what they saw, how they felt and what they learnt. There are various ways in which this can be achieved:

Printed feedback cards – This is a great method for feedback if you are short of time or if you have a high turnover of participants. Thinking of the questions you want to ask participants beforehand allows you to consider what feedback and evaluation is most important to you, and to carefully construct both closed and open questions that will yield valuable answers. Having a standard set of questions, particularly questions which ask the participant to rate something out of 10 or to give a yes/no response, will allow you to benchmark responses against each other and easily analyse the data statistically. Answers like this can also help you spot trends in the data when answers are plotted against participant gender, age or background.

Group discussion – A group discussion can work well if you are showing the film to a whole class, for example, and have the time to discuss the film as a group afterwards. Asking questions in this way can provoke peer-to-peer discussion about what they saw in the film (and what they may have missed!) and establish social norms between participants. For example, if a participant believes their friend behaves in a certain way, they are more likely to behave that way as well. Asking leading questions can also illicit the feedback you require, much like with printed feedback cards, but with the added benefit of a peer-to-peer discussion as well. It also gives you the opportunity to correct a participant if they have something wrong or even allow you to identify those who should perhaps watch the film again because they haven’t understood or have missed something critical. Make sure that someone can take notes to keep a record of what the participants say.

Direct participants to website/SurveyMonkey survey to give feedback – Much like printed feedback cards, it is possible to set up a microsite or run a SurveyMonkey survey to direct participants to in order to get feedback. Getting feedback in this way has its advantages – it is all stored online safely and at a click of button, you can get reports and statistics without having to manually input data. Issues can arise when considering how to direct people to the site or online survey – either by printed cards or an e-mail – and whether participants will forget or not bother to complete it. If feedback and evaluation is important to the project, then this may simply be too risky.