Previous Table of Contents Next


3.4. MALFUNCTION ANALYSIS OF CURRENT SYSTEM

A malfunction analysis of the current system identifies usability problems with the existing system (including manual and/or computerized tasks), as well as gaps in functionality and problems in the business process as it is currently carried out. If a business process re-engineering exercise has been conducted, much of the process- and task-related malfunctions will already have been identified, and the usability analysis can be limited to usability problems at the user model, and at the dialogue and detailed level of the user interface. If no business process re-engineering has been done, the usability engineers frequently identify the opportunities for process and task improvements. Often this implies significant changes to functional requirements/specifications and business policies and procedures, and hence, rework and project delays.

For example, at the XYZ Finance & Insurance Company, the malfunction analysis revealed a fair amount of duplicate data entry: if a customer bought more than one financial or insurance instrument, basic customer information had to be entered for each instrument. Access to the existing legacy system functionality was through a large number of transaction codes. Experienced users knew the most frequently used codes from memory and used them efficiently for quick customer service. Novice users required the use of a reference booklet for several weeks, until they memorized the frequently used codes. Users felt that having to use a reference booklet in the presence of a customer did not project a professional image.

Malfunction analysis is essential for application redesign so that usability problems will not be repeated. When carrying out a malfunction analysis, it is also very useful to identify those parts or features of the application that users deem particularly useful so that they can be preserved where appropriate.

Malfunctions are rated according to their severity and classified as critical, serious, or minor. The severity of a usability problem is a combination of three factors:

  Frequency: how common is the problem?
  Impact: how easy or difficult will it be for the users to overcome the problem?
  Persistence: is it a one-time problem that users can overcome once they know about it or will users repeatedly be bothered by the problem?

These three components are combined into a single severity rating as an overall assessment of each malfunction. The ratings (described below) are provided to assist in prioritizing improvements and decision making:

  Critical problems make successful task completion impossible, unacceptably long, or are fraught with significant (costly or embarrassing) errors or they have a detrimental effect on how an organization conducts its business.
  Serious problems have a major impact on the user’s performance and/or the operation of the system overall, but do not prevent users from eventually performing the task.
  Minor problems do not greatly affect the user’s job performance, but when taken as a group, negatively affect the user’s perceptions.

In many cases, it is difficult to separate the causes of malfunctions and to rate them only on an individual basis, as submalfunctions often also contribute to, or are part of the cause of, these malfunctions. Therefore, ratings are given to malfunctions which may also include less severe sub-malfunctions, though these submalfunctions are not rated individually. For example, malfunctions with navigation may be rated as Critical. However, these also include related submalfunctions, such as problems with the wording of menu choices which, taken on their own, may have been rated as only Minor. Critical malfunctions must be addressed in the redesign; serious malfunctions should be addressed in the redesign.

Usability performance objectives can often be derived from malfunctions. When evaluating user interface designs during iterative prototyping and testing in the usability design phase, user interface designs are assessed in terms of how well they address the malfunctions and the derived usability performance objectives (see Section 4.2). The process of understanding and rating malfunctions forms an important part of the foundation of the bridge.

If the project involves commercial-off-the-shelf software (COTS) software, it is essential that a quick malfunction analysis be conducted, at a minimum as a heuristic evaluation and preferably as a quick co-operative evaluation (Nielsen, 1992; Monk et al., 1993). The purpose is to assess which aspects of the COTS user interface should or should not be replicated in the extensions and other applications in the set. The objective is to determine whether, or to what extent, to adopt the user’s model, navigation style, and widget operation into the extensions and other applications in the set. Frequently, COTS software is chosen early on in a project for its functionality and price, with little or no regard for its usability (which frequently leaves a lot to be desired).

Malfunctions are documented in a report and posted in the usability engineering room. In the case of a fire prevention assignment, the malfunction analysis is typically carried out as a co-operative evaluation augmented by the analysis of data from user surveys, hot line support services and comments collected by, for example, the user departments. In the case of a fire fighting assignment, the malfunction analysis is carried out in two JAD sessions with the User Committee and the Steering Committee.


Previous Table of Contents Next