Evaluation

After introducing our prototype to users, we ran a benchmark evaluation with three notable goals:

1. Understand how varying levels of experience with video games, specifically 3D ones, affected the user’s understanding of the controls.

   1. We used this information of develop our tutorial room more with important instructions.

2. See what elements of the room drew the user’s attention.

   1. Our goal was to have several points of interest be available from the start so that larger groups could split up and discover things individually, before calling their partners over for support.

   2. This test informed our decisions about how to indicate which elements were intractable.

3. Understand how users made associations between two game objects. When the users break the vase, the statue in the opposite corner becomes angry and begins turning to track the player. This correlation is important since it guides the user to the next step of the puzzle.

   1. We initially used spacial audio cues to convey this effect to the users. After completing ICAT day, we believe additional indicators would be beneficial, such as an evil laugh to indicate the connection, glowing red eyes for in-world understanding, and an ambient glow to draw more attention.

      
      

The table below (Table 1) is taken from our first benchmark evaluation, and contains information about the users and their experience. This data is purely quantitative, however the evaluation also included open-ended questions that gave us the insight outlined above.