As part of the product, testing documentation seems like an obvious thing to do, but what does it really mean? I’ve fielded the question in a few different places now and it’s always interesting to delve deeper and understand the rationale behind the request.
“We should test the documentation”, seems like a harmless enough statement and I’ve heard it uttered in more than one place yet I’ve never worked in a company that actually tested the documentation.
Some clarification is usually required, of course, as there are many ways that you could test the documentation, for example:
- For online help, test that any hooks from the application to the documentation work properly
- Check that cross-references are corrected and, if hyperlinked, work properly
- Test the content matches exactly what is on-screen
- Test the flow of information is correct and makes sense from a users point of view
Now, we do test that our online help works correctly, that the correct page is launched from the application and that a sampling of internal links work correctly.
However, the last two items in the list can be a bit of a thornier issue. Testing the content is a much bigger job than most realise, presuming it is being done correctly and it also raises the fun issue of is this a bug in the documentation, or in the application.
Every time I’ve had this question asked I’ve said yes we should test the documentation, but what I really mean is that we should use the documentation to test the application. One method I’ve thought about is to have one person would read out the instructions, with another piloting the application. Just an idea but it would flag up some issues in both areas, I think.
Of course such resource heavy requirements rarely see the light of day and the simple fact is that we don’t test the content of our documentation. Yes we get it reviewed but they are separate (if closely related) things.
So, in an effort to close the gap between reviewing the documentation and testing the documentation, it is probably worthwhile running a short workshop for your development team, a 10 minute session that shows how best to review documentation. I’m planning such a workshop at the moment, so more on that, soon.
Thanks, Gordon, for raising one of the trickiest issues in creating and maintaining high-quality documentation!
If your online help is a “regular” web site, you can actually automate the second point with a broken link crawler…
I’ve found that you *can* actually use the documentation to test an application, but it works best if (a) there’s an agreement that this will be done and (b) the documentation is ideally written using the development spec rather than the application. Else you frequently get into the kind of arguments you mentioned.
And I’ve found that training colleagues (where present) are more suited to testing documentation than developers, if only because they feel the pain more when the documentation is faulty or plain bad…
Just my 2 cents, Kai.
Kai, thanks for the comment, a couple of responses.
In our agile development process there is no development spec so we have nothing to go by but conversations and builds of the application.
As for using training, we have a good system here during the final testing phase of swapping development teams (team A will test team B’s work and vice versa), so the docs could be used at that point, it’s something we are hoping to try out this time around.
Duh – I forgot you were doing agile… 🙂 My comments apply to slower, more “conventional” software development efforts…
We do a version of this more or less routinely for our docs that are aimed at customer sysadmins, particularly install docs. True, it’s a resource-intensive step, and it accordingly gets fudged sometimes in emergencies, but at least for install docs we pretty much insist on it. We can’t afford to do it for GUI help content, unfortunately.
There are two parts to the agreement we hammered out with our QA folk, which has worked reasonably well for a year or so:
1. Writers must successfully install and operate the software, using their own instrux, before sending text to QA.
2. QA must follow the steps in the docs to carry out their test cases. (QA-specific steps aren’t allowed.)
Hi there,
I know its a loong time since you wrote the blog item, but having read it, it makes sense. However, it has just wetted my appetite and i was wondering if you wrote more on it? I would really like to read it please.
Regards,