Developing the Checklist

Our checklist for Websites was modeled on the checklist for user guides that we presented in 1996 at the SIGDOC96 conference [3]. When we considered the design of our checklists, we looked at two options: a series of questions that could be answered with a Yes or No, or a scoring system based on a set of guidelines.

Our checklists use the question and answer method because we believe it more accurately tracks continuous enhancements to a Web site over the life cycle of the product or service it describes. The question and answer method is also more consistent when only one evaluator tests a Web site at different times in the development life cycle. A scoring system is open to interpretation by the evaluators and we often found a wide variance of scores among different evaluators.

Determining the Questions

For the questions themselves, our strategy was to apply the usability concepts developed by Chignell, Mehlenbacher, and Nielson, as well as ideas from existing checklists and apply them to Web sites.

Usability Categories

The Nielson research helped us realize the importance of grouping the many guidelines, while the Chignell and Keevil checklist [3] provided the following five usability categories or metrics:

Based on Chignell's findings, we included navigation strategies in our Usability Index checklist, including items on use of a contents, headings, index, and glossary. These are the "book" metaphor access devices that users employ for both the online and printed documents. The following list is an example of the questions for the Headings category that appears in the section on Finding the Information.

Headings (Choose one topic at random)

The following list is an example of the questions for the Organization of the Site section that is in the category of Understanding the Information. Organization of the Site

From the Mehlenbacher research, task orientation became a requirement for the Usability Index checklist. The following lists are examples of the questions for the User-Oriented Tasks and Information Updates categories that appear in the category on Supporting User Tasks.

User-Oriented Tasks

Information Updates

For the Evaluating the Technical Accuracy category, the questions included the following.

Technical Content

Determining the questions takes time because you have to research and summarize the usability issues in a single question. For example, the category Presenting the Information, includes questions such as:

These questions are based on user interface research by Neilson [11] and others. The research shows that 10 seconds is about the time limit for keeping the user's attention focused on the dialog. For longer delays, users will want to perform other tasks while waiting for the page to display, so they should be given feedback indicating when the computer expects to be done. Feedback during the delay is especially important if the response time varies, because users do not know what to expect.

The checklist is available on the Web site of Keevil & Associates. You can view the checklist in HTML table format or you can download it in Microsoft Excel format. If you use the Excel format, Excel calculates the usability index based on your answers. You can also add and delete questions to better fit the requirements of your own Web sites.

In summary, our approach for developing the checklist was:

After applying the research to develop our checklist, the checklist was used for to measure the usability index of an example Web site.

Example Web site

We used the Web checklist to measure the usability index of the Keevil & Associates site [6]. In May 1998, the measured usability index was about 70 per cent because the site was missing a glossary, search facility, index, interactive feedback, and weekly updates. It is essential to update your Web site weekly to attract return customers. Also, the main purpose of the site - to sell technical writing services is not clear because the site contains many other topics.

The results of the usability index calculation for the Keevil & Associates site are recorded in the checklist in Excel format that you can download (60 K, takes about 30 seconds with a 33.6 modem).

To measure the usability index of your own site, follow the Measuring the Usability steps.