Previous Table of Contents Next


5.4.1. Product Implementation for CNN@Work

At this point in the development of our process of Systematic Creativity, we were still working on the best way to format information in the spreadsheets and, given the time restrictions we were under, we decided that we needed to share the information quickly with the product team. We did this by having the team as a whole do a walkthrough of several of the critical tasks, step by step. We used overheads of the interface as it currently existed. We put up the main window and asked the team members to write down what they would do to achieve a particular task. We asked, after each choice in the process, how many had selected the correct choice and which, if any, other choices had been selected. Then we went on to the next step. We continued this process for the subset of tasks we had selected. Team members quickly got a different view of the interface when they looked at the tasks users were expected to do in order to accomplish the ultimate goal. This process succeeded in convincing team members of the value added by viewing the system from the users’ perspective. In fact, we would recommend that product revision teams start by doing a walkthrough of the previous product to quickly understand the extent of the changes that should be made and to start off with the users’ view in mind.

We were working closely with the developers during product implementation. This was a relatively small project and the head developers had been convinced of our added value during our design work. Therefore, major decisions during implementation were not made without consulting one of the human factors staff on the team.

5.5. USABILITY TESTING

There are two aspects to usability testing: what to test and how to evaluate the results. The Systematic Creativity framework helps with both of these issues. We use the prioritized goals to address what to test. It is also worth noting that the goals form the basis for the user scenarios needed for the test plan. While some elaboration is needed, the framework contains the basic information needed for constructing user scenarios. If tapes have been made of the user interviews, usability people can refer to these if help is needed in refining the test plans. Looking at the facilitators and obstacles in the new user tasks also help us construct questionnaires to be used in testing. We want to know if we have sufficiently reduced the obstacles so that users perceive our product as beneficial in achieving a goal.

The framework is vital to evaluating the results of usability testing. From the framework we can classify failures in the following manner:

1.  Failure to support a desired goal.
2.  Failure to sufficiently reduce obstacles.
3.  Failure to make visible the support for a desired goal.
4.  Failure to provide a necessary action or object.
5.  Failure to provide meaningful feedback on task progress.
6.  Failure to provide initiators that are visible or meaningful to the user.

The first failure is catastrophic but should not happen if we have done sufficient design testing. Questions of supporting goals should be identified during our design iterations and incorporated into the product. However, if our involvement with the product occurs late in the cycle, this may be a possibility. The development team will need to decide if the lack of support for certain goals still results in a viable product.

Failure to sufficiently reduce the obstacles associated with the tasks is also a very serious error. Again, this should have been identified during design testing. Producing a more efficient user task can be a large redesign problem that can seriously affect the product schedule. The Systematic Creativity framework can be used to identify the importance of this task which can help the team members decide if the time for redesign needs to be taken.

Failures 3 and 4 are major failures and again should have been detected earlier in the design iteration. If little up-front work has been done, these failures will be detected at this time. Using the framework we can see what goals and tasks need the particular action or object. We can then evaluate the seriousness of this failure based on the priority of the affected goals and the difficulty of adding the visibility or the action or object needed.

Failures 5 and 6 are the least severe — at least from the standpoint of how much redesign needs to be done to correct them. Having the Systematic Creativity framework in place, the team can locate an individual action or object and see how changes made in the context of that particular task will affect other tasks using that particular action or object. The tasks in which the action or object is used can be traced back to user goals, thus allowing the team to make the corrections in a more global fashion, rather than making a change and subsequently discovering effects in other portions of the interface.

5.5.1. Usability Testing of CNN@Work

It is always difficult to determine when design testing stops and usability testing begins. We tend to think of usability testing as more formal testing later in product implementation. In usability testing we concentrate on testing that our usability goals have been verified. In defining the usability goals for this product, we specified percentages of users that should be able to accomplish basic tasks during their first encounter with the system. We also specified what percentage of users should rate each task within a specified range we had defined to represent “perceived ease of use”. Specifying these usability goals is an extremely difficult task, especially when the product or the functionality is new. We were able to use some information from usability tests on the first version to produce specifications for previously supported functionality. We based the specifications for new functionality on the priorities of user goals in relationship to priorities of previously supported goals. More important than having “exact” specifications in the requirements document, we now had agreement from the team that these requirements were as real as functionality requirements.

Basic tasks for this product included:

  Changing channels.
  Setting up a simple filter from scratch.
  Identifying where a story was captured and retrieving that story.
  Viewing a story currently available.
  Viewing a story currently available and storing it.
  Viewing a story currently available and creating a filter based on this story.

A usability test determined if our product met the specified usability goals we had established. The usability test focused on the same tasks that we concentrated on during design, those that were of the highest priority to users.


Previous Table of Contents Next