The focus of this session was on the value of proper usability studies to improve games, and how developers without $250,000 testing suites can get the most of out it. Divided into two parts - speaking lectures, and smaller group interaction - we had our main speakers Marcos Nunes-Ueno, John Davis and Kevin Goebel. In addition I had the pleasure of meeting Philip Hove, an experimental psychologist (most of Microsoft’s game usability has some background in psychology).
After some lessons in PowerPoint usability, follow by what seemed like our room blowing one of the conference fuses running 8 Xbox 360’s on one circuit, we got into the meat of what usability testing is:
Designers Vision –> Game -> Users Experience
What the usability research attempts to do is get the design across to the player - the game may still be fun, but if the player doesn’t get everything the designer was trying to convey, the game had usability problems.
Normally new usability researchers at Microsoft get a 3 week crash course, but for the purposes of the GDC that course was compressed into 6 hours. Microsoft has approximately 20 people exclusively working on game usability, testing games like Age of Empires III and Halo 2. Over 120 researcher engineers working on usability across its campus, with a new game usability lab being established in Japan. Since its establishment in 1997, the game usability team has logged over 4,000 studies involving approximately 40,000 individuals.
A normal usability study is conducted in the lab - a relaxed, living room style room for the subject, and an attached control room behind a one way mirror from which the controller and several observers can watch. The study attempts to find problems in the games interface and presentation that impede the subject from realizing the games vision. We watched as subjects playing “Midtown Madness” proceeded to follow traffic laws and avoid hitting other cars, despite being in last place. The subjects reported the game was fun, but they where not getting the designers vision of a mad race through city streets.
Usability problems usually fall under three categories:
Poor Discoverability - When the player cannot find out how to do something on his own, or an even bigger problem, doesn’t realize the feature exists. A video showed subjects presented with a sword on the ground and an ogre in the distance. Some players ran away because they didn’t have any weapons, others tried to fight the ogre with their fists and died. Why? “Do you see anything you could use” – “No, not really. There are some sticks [the sword] on the ground but nothing I could fight this guy with”?
Unintended Challenges - Designers want their game to provide a challenge to the player, but usability problems can occur when players are frustrated by a game mechanic that falls outside of the envisioned game skills. An example we saw was a driving segment in an ice level of “Halo 2” - drivers would proceed down slopes and through tunnels until they saw a bright cave in the distance, straight ahead. There was a problem however - it was a pit that resulted in instant death. To make matters worse the pit was several minutes from the checkpoint, so subjects often forgot about the put that looks like a cave by the time they got back there.
Unclear goals - The last major categories of usability problems in games are when players don’t understand the goals. They don’t understand them, don’t know when they’ve completed them, or don’t know when they have them. An example we where shown was the first good or evil quest from “Fable”, where the player comes on a bully and a kid. In the initial version of the quest, the dialog given by the characters was not sufficient to suggest that the player should have any interest or that there might be a task to perform. Subjects who encountered the bully dismissed both characters in seconds and went on.
How a usability study works
Before any subjects are involved, the researchers need to determine both who the game is targeted at, to recruit appropriate testers, and what elements of the game need to be tested. This is broken down into a task list. For each task list there will be 5-8 subjects who will all attempt to do the same things.
When subjects are brought in it’s important to make them feel at ease and work them into the process. The controller/researcher is automatically perceived as an authority figure, and this can influence how the subject will behave. Giving the subject a tour of the lab and assuring them that it will be a study of how well the game works, not how well the subject plays it.
Usability studies work best when the subject speaks aloud what they are doing and thinking. To get them into this mode of thought the researcher will have them read aloud all the instructions they receive. Once the subject is ready to begin testing the controller retreats to the control room to monitor and when needed prompt the subject. The separation from the subject is important, to allow them to act naturally. Typically a group of observers from the development team will be in the observation booth, however this is not revealed to the subject.
During testing the subject will be asked to read aloud the task that they will perform and then try and do it. Tasks are designed to test elements of the game for the 3 main usability problems. Wording of tasks is important – they should not include keywords from the HUD or other parts of the game. The subject should not be lead, either by the wording of the task or by the controller, to avoid tainting the test.
Typically the study is video taped, and this can be linked with notes through time stamps. A full test suite can have multiple video captures to record the game screen itself, and the player’s hands. The controller room has one or more video screens and space for whiteboards and other note taking.
Less expensive usability labs can be created. The “budget” lab would require two adjoining rooms, such as a conference room and an adjoining office or conference room, one or more cameras, and one or more monitors, two speaker phones, and lots of cable and duck tape. Cameras are setup to send a video feed back to the makeshift control room, while the speaker phones provides two way communication, with the mute button allowing the controllers/observers to speak without being heard by the subject.
The least expensive setup is simply having the controller sitting beside the subject taking notes and prompting when needed. This has many drawbacks however – the subject is receiving feedback of what you think (such as when you take notes), you can’t have additional observers as they would further stress the subject, and it makes it much harder not to give the subject help, which taints the results.
After the test data is collected and the issues identified are examined in detail to determine the exact cause and suggest a solution. Time stamped video/notes can help immensely in this instance.
Unite 2010
