There’s been a fascinating discussion surrounding User Interface Engineering’s recent articles and panel discussions around how many user tests are enough for modern (primarily web-based) software systems.
In reading the discussions, which provide a rich body of emerging research clarifying industry folklore on the subject, I was repeatedly struck by the claim that modern consumer web sites are inherently more complex than traditional software applications, from which the five to eight users rule of thumb emerged.
I sent Jared and the other wonderful folks the following e-mail posing my question about their assumptions on traditional applications. I look forward to continued discussion.
——–
Dear friends,
I’ve read with great interest your articles in UIEtips about how many users are needed for testing. It’s great to see research clarifying and focusing the myths of our craft and help it move toward engineering.
One of the concepts that is repeated a number of times in the discussion is that content-rich web sites like online sales sites are more complex than traditional software applications. Having worked for the last 10 years on large-scale, highly complex desktop software systems – and smaller applications prior to that – I was hoping you could clarify what sorts of applications you are referring to.
A software architecture model that I’ve been developing (okay, really dabbling with on and off), identifies that there are (at least) three different architectural aspects to a particular software application or system:
– technical architecture – the technology and mechanisms that support the activity of the system
– data architecture – the storage, retrieval, and management of the data manipulated by the system
– presentation architecture – the tasks, workflow, interaction, visual, and other design elements of the users’ interface to the system (aspects are nicely modelled by Jesse James Garrett’s information architecture model)
Different software systems have more complexity and scope in different proportions of these architectures. For example, a huge online bookstore like Amazon has a large complexity and scope in its data architecture and somewhat smaller in its technical and presentation architectures. An IDE for visual engineering languages, like RSLogix 5000, the product that I have worked on for the last 7 years, has large technical and interactive components, but smaller data components. (A CAD package might have similar proportions.) A massively parallel scientific computing system may be hugely complex in the technical dimension, but be rudimentary in its presentation and data architectures.
In your discussions, are you referring to the day-to-day consumer software tools that each of us has on his or her desktop computer? Or software in general? Admittedly, usability work is focused on those systems with large presentation architecture components, but web sites are not inherently more complex in that dimension except – possibly – for the size of the data repository and the number of users that interact with it at one time.
In fact, presentation complexity is much more constrained on a web site than a traditional desktop application in its interactive/manipulative components. (As an example, how many web-sites have had to deal with the interactive semantics of drag-and-drop of an arbitrary selection set of named graphical objects into another context, resolving name collisions with objects that may already exist in the new context? Our product does, and we’ve done extensive usability testing on the interaction.)
The number of users question is just as relevant to real desktop software systems as it is to web-sites. Five to eight doesn’t jive in my products either, although for a specific, bounded task, it usually suffices. But you’ll need to cover a couple of hundred of those tasks in testing if you ever want a thorough, quantitative understanding that you’ve discovered 80% of the problems. Now, if only I could find someone in my organization who would fund those tests…
Thanks again, and sorry for the soliloquy – I got on a roll, I guess.
Yours,
Jim Jarrett
Rockwell Automation
