Usability testing without a budget

How many times have you been thinking to yourself, or even said out loud to your team, that “We really should do some usability testing on this project”, only to be knocked down by the inevitable answer: “We don’t have the budget for it”? If you’re anything like me, the answer is dozens. Well, it doesn’t have to be like that.

Even larger projects can often lack a dedicated budget for usability testing. Maybe the client doesn’t think it’s necessary, maybe the budget is really tight, or maybe there are other reasons. Whatever. The good news is that you can squeeze in a low budget, quick and basic test anyway, which is much better than not testing at all.

It doesn’t really matter how much you know about usability – after working on a project for a couple of weeks you’ve become blind to many of the problems. Sure, experience will help you avoid making many design mistakes in the first place, but something will slip through. Every time. That’s why doing even a very basic usability test will improve your site. And here’s one way of doing it.

Something to test

First of all, you need something to test on. In some cases this will be a clickable prototype, in others it’s a more or less finished site (which is not ideal – the earlier you test, the better).

In either case, you need a few scenarios to test. A scenario in this context is a task that the user can perform on the site. I usually try to test at least three scenarios. If you’re testing on a prototype, you need to build dummy HTML pages for every step of the way for those, and if you’re testing on the site that’s being developed you need to make sure everything needed to complete the tasks has been implemented. Try to make any dummy content as “real” as possible – many people tend to get hung up on things like spelling mistakes or illogical content, which is not what you’re testing.

Possible tasks can be “Find the telephone number to the person in charge of sales”, “Place an order for product X”, or “Find out when and where such and such lecture is held”. If you know which areas of the site are the most critical, make sure to come up with tasks that involve those areas.

Finding people to test with

Once the scenarios are in place, you need some people to test them on. Ideally you would find people that are representative for the site’s target demographics. In low budget usability testing, you don’t have the time or the money to find those people. So what is quick and cheap enough? Look around you. Ask some people around the office, ask your team to check with their spouses or parents, ask a friend. Anyone who uses the web and isn’t too involved in the project will do. Let them know that it will only take half an hour, and that they’ll be helping you improve the site you’re building.

You don’t need a whole lot of people. I usually settle for three or four persons. After testing only one or two you’ll probably be getting a feel for where any problematic areas are.

Preparation

For the actual testing you obviously need a networked computer (or a local copy of the site), preferrably in a reasonably quiet room or area of the office. You need one person to conduct the test – this will be the person talking to the users. If possible, one more person from the team should be present to help take notes.

Before testing, you should do the test yourself to make sure it really is possible to perform the tasks you’ll be testing.

The test

Start by telling the test subject what the test is all about and how it works. Let them know that you’re testing the site you’re working on, not how good they are at using the web, and if they make a mistake or fail to accomplish any of the tasks you’ll be giving them it’s because you need to fix the site, not because they are stupid.

Some background info on the subjects is also good to have, especially when it comes to their web usage. A person that uses the web 12 hours a day is likely to be more used to finding workarounds than someone using the web only half an hour daily.

Ask the test subjects to think out loud during the test. It helps you understand why they choose to click on link A instead of on button B. It could also give you ideas for improvement. I find this part very interesting. Some people talk a lot, while others more or less just do what you ask them to (complete the tasks) without saying a whole lot.

Avoid asking leading questions. Try to leave them as open as possible to avoid influencing the test subject’s answer or actions.

Also ask the testers if they have any general opinions on the site, the design, the structure, etc, and if they have any suggestions for improving the site. You’ll be surprised by how much some people will tell you about what they think is good and bad in web design.

Make sure to take notes of what the test subjects do during the test. Don’t rely on remembering everything that surfaces during the test – you won’t. It’s very easy to forget small but important details.

Debriefing and drawing conclusions

When testing is done, go through the notes you made and clean them up, typing everything in a text document. This step is for documentation and to keep things fresh in your memory.

Summarise the tests and try to find any common problematic areas. If all test subjects were having similar problems you know where to start improving the site.

Meet with the team and discuss the results. Decide what changes to make, and what to prioritise. There will often be some really low-hanging fruit that will only take minutes to fix and can be done right away. Other changes may require talking to the client.

It doesn’t have to be difficult or expensive

If you’ve been unable to do any usability testing because of time or budget constraints, try the very basic method I’ve outlined here in your next project. It isn’t comparable to full scale testing with carefully selected participants, video cameras and all that, but it doesn’t require much of anything, is very cheap, and will alert you to any major problems in your design and/or information architecture without breaking the budget. Obviously you can do much better and more detailed testing than this, but as stated before: if you have no budget, minimal testing is much, much better than no testing at all.

Posted on May 2, 2005 in Usability

Comments

  1. An excellent write-up about simple usability testing. Reminds me a lot of Steve Krugs classic Don’t make me think. I should really read it again, it’s been a while. Thanks for the reminder Roger!

  2. Although usability testing doesn’t necessarily cost a lot of money, it’s expensive in another respect: time.

    Even a small usability test takes time to prepare, set up and evaluate. In today’s world with streamlined organisations and projects that need to be delivered yesterday, that is often the clincher.

  3. Family members and friends can make good test subjects - the occasional bomb comment can be a real eye opener when you’ve already fallen in love with a design so much so that loopholes go unnoticed.

    In addition to taking notes, a voice recorder can come handy. It’s less intrusive than a video camera, which makes the test more realistic.

  4. May 3, 2005 by Roger Johansson (Author comment)

    Jaakko: Yes, that is one excellent book, and he does describe a similar approach to this in one chapter.

    Tommy: It takes a little time, but it doesn’t have to be much. If you can afford to spend a day on testing you can squeeze it in. You could probably cut some corners and get it done even quicker. But I get your point, and for some projects, in some organisations, that will make usability testing very difficult.

  5. We are fortunate to have guys in the office who are not directly involved in the site production, so we can occasionally test the interface. I also like to test with friends and relatives in a more easy going manner. This is usually better than prepared test, because the least I want is that they suppress criticism trying not to hurt my feelings. In my opinion — the more honesty, the better and having said that, the formal test setup can be distracting.

  6. Good idea is also to use a camcorder/webcam to record the subject’s behavior + do some screen capturing to see his/her mouse movements.

  7. May 8, 2005 by Roger Johansson (Author comment)

    Jan: Absolutely. That does take a bit more time though, and many people get kinda nervous when there’s a camera around. The main idea here is to make testing as quick and easy as possible to increase the chances of actually getting it done.

  8. It’s always important to at least show it to someone, either your friend or family member, so they will give you some quick feedback… And as Prabhath mentions, it is not possible to test the website by its author/frequent user, since they really miss many details that could help.

  9. Yep, I call it coffee shop testing - I have done this a few times. Great results, although I have had to buy a fair amount of coffee once or twice because the people who turned up were simply not close enough to the profile. My bad - maybe I need to be tougher and turn them away :) It is important when using strangers to get a good profile of them - demographic group, and internet experience covers it mostly.

  10. Roger,

    Your comments on how to enact a small scale usability test are very useful. However, I think it’s important to stress how much usability norms can differ across different audiences and regions, and how much adaptability and approachability after publication should be more of a factor in determining requirements. A company/organisation/community with no prior experience of its audience will have no conception of the capabilities permissible, and so would have to offer the bottom line. It may turn out that that’s insufficient, because your majority audience turns out to be extremely competent and technologically advanced - they may desire the complexity, because they can use it.

    See the (admittedly badly formed) article at The Register on Google and scope for some brief comments on the different perceptions that east and western cultures have of what the internet itself means, let alone usability of it.

    Any designer should aim to follow the simplest approach possible, adding on optional extras as the user goes through the process, without complexifying things too much. Usability in such a case might therefore be reducible to a set of criteria, or dare I say it, standards.

    But it can’t, of course, and we know why. Usability, and accessibility, are individual characteristics particular to each and every user, and so depend on a combination of user knowledge and education, and designer prediction and “leeway” to allow a vaguely balanced experience. (let’s exclude perhaps one of the least usable sites on the web, Jakob’s, which fails not by structure but by design)

    All of which brings me to: usability and accessibility studies which involve “real life” users are immensely important until the designer gets it. At that point, unless they pay no attention to web trends and societal change, they have a reasonable - while not expert - understanding of what to provide.

    Furthermore, reasonable is perhaps the best stage to be at. Nielsen may have an excellent understanding of usability, but he has no idea how to implement it. Joe Clark has a fabulous understanding of captioning and many accessibility issues, yet he still frightens everyone off by being Joe Clark, hence prohibiting access. Experts are absolutely necessary, but not in proliferation; here the jack of all trades is more or less enough.

  11. Another quality article. It’s worth mentioning the role of usability reports in your project. I have found that it’s valuable to make a list of every task that caused confusion with each of the test participants. Conduct the same test with four or five people, and you might find different people stumbling on the same task.

    In my experience, the best use of these informal usability tests is to find a small handful of the most serious usability flaws, and fix them. In such circumstances, a simple list of items that need fixing is often the best report type.

    I also like to archive all of the notes, recordings, etc… They can come in handy later as a teaching aid for general usability lessons.

Comments are disabled for this post (read why), but if you have spotted an error or have additional info that you think should be in this post, feel free to contact me.