How to Report Bugs Effectively

How to Report Bugs Effectively

by Simon Tatham, professional and free-software programmer

Introduction

Anybody who has written software for public use will probably havereceived at least one bad bug report. Reports that say nothing ("Itdoesn't work!"); reports that make no sense; reports that don't giveenough information; reports that givewrong information. Reports ofproblems that turn out to be user error; reports of problems that turn out tobe the fault of somebody else's program; reports of problems that turn out tobe network failures.

There's a reason why technical support is seen as a horrible job to be in,and that reason is bad bug reports. However, not all bug reports areunpleasant: I maintain free software, when I'm not earning my living, andsometimes I receive wonderfully clear, helpful, informative bug reports.

In this essay I'll try to state clearly what makes a good bug report.Ideally I would like everybody in the world to read this essay before reportingany bugs to anybody. Certainly I would like everybody who reports bugs tometo have read it.

In a nutshell, the aim of a bug report is to enable the programmer to seethe program failing in front of them. You can either show them in person, orgive them careful and detailed instructions on how to make it fail. If they canmake it fail, they will try to gather extra information until they know thecause. If they can't make it fail, they will have to ask you to gather thatinformation for them.

In bug reports, try to make very clear what are actual facts ("I wasat the computer and this happened") and what are speculations ("Ithinkthe problem might be this"). Leave out speculations if you want to, butdon't leave out facts.

When you report a bug, you are doing so because you want the bug fixed.There is no point in swearing at the programmer or being deliberatelyunhelpful: it may be their fault and your problem, and you might be right to beangry with them, but the bug will get fixed faster if you help them bysupplying all the information they need. Remember also that if the program isfree, then the author is providing it out of kindness, so if too many peopleare rude to them then they may stop feeling kind.

"It doesn't work."

Give the programmer some credit for basic intelligence: if the programreally didn't work at all, they would probably have noticed. Since they haven'tnoticed, it must be working for them. Therefore, either you are doing somethingdifferently from them, or your environment is different from theirs. They needinformation; providing this information is the purpose of a bug report. Moreinformation is almost always better than less.

Many programs, particularly free ones, publish their list of known bugs.If you can find a list of known bugs, it's worth reading it to see if the bugyou've just found is already known or not. If it's already known, it probablyisn't worth reporting again, but if you think you have more information thanthe report in the bug list, you might want to contact the programmer anyway.They might be able to fix the bug more easily if you can give them informationthey didn't already have.

This essay is full of guidelines. None of them is an absolute rule.Particular programmers have particular ways they like bugs to be reported. Ifthe program comes with its own set of bug-reporting guidelines, read them. Ifthe guidelines that come with the program contradict the guidelines in thisessay, follow the ones that come with the program!

If you are not reporting a bug but just asking for help using the program,you should state where you have already looked for the answer to your question.("I looked in chapter 4 and section 5.2 but couldn't find anything thattold me if this is possible.") This will let the programmer know wherepeople will expect to find the answer, so they can make the documentationeasier to use.

"Show me."

One of the very best ways you can report a bug is by showing it to the programmer.Stand them in front of your computer, fire up their software, and demonstratethe thing that goes wrong. Let them watch you start the machine, watch you runthe software, watch how you interact with the software, and watch what thesoftware does in response to your inputs.

They know that software like the back of their hand. They know which partsthey trust, and they know which parts are likely to have faults. They knowintuitively what to watch for. By the time the software does something obviouslywrong, they may well have already noticed something subtly wrong earlierwhich might give them a clue. They can observe everything the computer doesduring the test run, and they can pick out the important bits for themselves.

This may not be enough. They may decide they need more information, andask you to show them the same thing again. They may ask you to talk themthrough the procedure, so that they can reproduce the bug for themselves asmany times as they want. They might try varying the procedure a few times, tosee whether the problem occurs in only one case or in a family of relatedcases. If you're unlucky, they may need to sit down for a couple of hours witha set of development tools andreally start investigating. But the mostimportant thing is to have the programmer looking at the computer when it goeswrong. Once they can see the problem happening, they can usually take it fromthere and start trying to fix it.

"Show me how to showmyself."

This is the era of the Internet. This is the era of worldwidecommunication. This is the era in which I can send my software to somebody inRussia at the touch of a button, and he can send me comments about it just aseasily. But if he has a problem with my program, he can't have mestanding in front of it while it fails. "Show me" is good when youcan, but often you can't.

If you have to report a bug to a programmer who can't be present inperson, the aim of the exercise is to enable them toreproduce theproblem. You want the programmer to run their own copy of the program, do thesame things to it, and make it fail in the same way. When they can see theproblem happening in front of their eyes, then they can deal with it.

So tell them exactly what you did. If it's a graphical program, tell themwhich buttons you pressed and what order you pressed them in. If it's a programyou run by typing a command, show them precisely what command you typed.Wherever possible, you should provide a verbatim transcript of the session,showing what commands you typed and what the computer output in response.

Give the programmer all the input you can think of. If the program readsfrom a file, you will probably need to send a copy of the file. If the programtalks to another computer over a network, you probably can't send a copy ofthat computer, but you can at least say what kind of computer it is, and (ifyou can) what software is running on it.

"Works for me. So what goeswrong?"

If you give the programmer a long list of inputs and actions, and theyfire up their own copy of the program and nothing goes wrong, then you haven'tgiven them enough information. Possibly the fault doesn't show up on everycomputer; your system and theirs may differ in some way. Possibly you havemisunderstood what the program is supposed to do, and you are both looking atexactly the same display but you think it's wrong and they know it's right.

So also describe what happened. Tell them exactly what you saw. Tell themwhy you think what you saw is wrong; better still, tell them exactly what youexpected to see. If you say "and then it went wrong", you have leftout some very important information.

If you saw error messages then tell the programmer, carefully andprecisely, what they were. Theyare important! At this stage, the programmeris not trying to fix the problem: they're just trying to find it. They need toknow what has gone wrong, and those error messages are the computer's besteffort to tell you that. Write the errors down if you have no other easy way toremember them, but it's not worth reporting that the program generated an errorunless you can also report what the error message was.

In particular, if the error message has numbers in it, do let theprogrammer have those numbers. Just because you can't see any meaning in themdoesn't mean there isn't any. Numbers contain all kinds of information that canbe read by programmers, and they are likely to contain vital clues. Numbers inerror messages are there because the computer is too confused to report theerror in words, but is doing the best it can to get the important informationto you somehow.

At this stage, the programmer is effectively doing detective work. Theydon't know what's happened, and they can't get close enough to watch ithappening for themselves, so they are searching for clues that might give itaway. Error messages, incomprehensible strings of numbers, and even unexplaineddelays are all just as important as fingerprints at the scene of a crime. Keepthem!

If you are using Unix, the program may have produced a core dump. Coredumps are a particularly good source of clues, so don't throw them away. On theother hand, most programmers don't like to receive huge core files by e-mailwithout warning, so ask before mailing one to anybody. Also, be aware that thecore file contains a record of the complete state of the program: any"secrets" involved (maybe the program was handling a personalmessage, or dealing with confidential data) may be contained in the core file.

"So then I tried . . ."

There are a lot of things you might do when an error or bug comes up. Manyof them make the problem worse. A friend of mine at school deleted all her Worddocuments by mistake, and before calling in any expert help, she triedreinstalling Word, and then she tried running Defrag. Neither of these helpedrecover her files, and between them they scrambled her disk to the extent thatno Undelete program in the world would have been able to recover anything. Ifshe'd only left it alone, she might have had a chance.

Users like this are like a mongoose backed into a corner: with its back tothe wall and seeing certain death staring it in the face, it attacksfrantically, because doing something has to be better than doing nothing. Thisis not well adapted to the type of problems computers produce.

Instead of being a mongoose, be an antelope. When an antelope isconfronted with something unexpected or frightening, it freezes. It staysabsolutely still and tries not to attract any attention, while it stops andthinks and works out the best thing to do. (If antelopes had a technicalsupport line, it would be telephoning it at this point.) Then, once it hasdecided what the safest thing to do is, it does it.

When something goes wrong, immediately stop doing anything. Don'ttouch any buttons at all. Look at the screen and notice everything out of theordinary, and remember it or write it down. Then perhaps start cautiouslypressing "OK" or "Cancel", whichever seems safest. Try todevelop a reflex reaction - if a computer does anything unexpected, freeze.

If you manage to get out of the problem, whether by closing down theaffected program or by rebooting the computer, a good thing to do is to try tomake it happen again. Programmers like problems that they can reproduce morethan once. Happy programmers fix bugs faster and more efficiently.

"I think the tachyonmodulation must be wrongly polarised."

It isn't only non-programmers who produce bad bug reports. Some of theworst bug reports I've ever seen come from programmers, and even from goodprogrammers.

I worked with another programmer once, who kept finding bugs in his owncode and trying to fix them. Every so often he'd hit a bug he couldn't solve,and he'd call me over to help. "What's gone wrong?" I'd ask. He wouldreply by telling me his current opinion of what needed to be fixed.

This worked fine when his current opinion was right. It meant he'd alreadydone half the work and we were able to finish the job together. It wasefficient and useful.

But quite often he was wrong. We would work for some time trying to figureout why some particular part of the program was producing incorrect data, andeventually we would discover that it wasn't, that we'd been investigating aperfectly good piece of code for half an hour, and that the actual problem wassomewhere else.

I'm sure he wouldn't do that to a doctor. "Doctor, I need aprescription for Hydroyoyodyne." People know not to say that to a doctor:you describe the symptoms, the actual discomforts and aches and pains andrashes and fevers, and you let the doctor do the diagnosis of what the problemis and what to do about it. Otherwise the doctor dismisses you as ahypochondriac or crackpot, and quite rightly so.

It's the same with programmers. Providing your own diagnosis might behelpful sometimes, but always state the symptoms. The diagnosis is an optionalextra, and not an alternative to giving the symptoms. Equally, sending amodification to the code to fix the problem is a useful addition to a bugreport but not an adequate substitute for one.

If a programmer asks you for extra information, don't make it up! Somebodyreported a bug to me once, and I asked him to try a command that I knewwouldn't work. The reason I asked him to try it was that I wanted to know whichof two different error messages it would give. Knowing which error message cameback would give a vital clue. But he didn't actually try it - he just mailed meback and said "No, that won't work". It took me some time to persuadehim to try it for real.

Using your intelligence to help the programmer is fine. Even if yourdeductions are wrong, the programmer should be grateful that you at leasttriedto make their life easier. But report the symptoms as well, or you may wellmake their life much more difficult instead.

"That's funny, it did it amoment ago."

Say "intermittent fault" to any programmer and watch their facefall. The easy problems are the ones where performing a simple sequence ofactions will cause the failure to occur. The programmer can then repeat thoseactions under closely observed test conditions and watch what happens in greatdetail. Too many problems simply don't work that way: there will be programsthat fail once a week, or fail once in a blue moon, or never fail when you trythem in front of the programmer but always fail when you have a deadline comingup.

Most intermittent faults are not truly intermittent. Most of them havesome logic somewhere. Some might occur when the machine is running out ofmemory, some might occur when another program tries to modify a critical fileat the wrong moment, and some might occur only in the first half of every hour!(I've actually seen one of these.)

Also, if you can reproduce the bug but the programmer can't, it could verywell be that their computer and your computer are different in some way andthis difference is causing the problem. I had a program once whose windowcurled up into a little ball in the top left corner of the screen, and satthere and sulked. But it only did it on 800x600 screens; it was fine onmy 1024x768 monitor.

The programmer will want to know anything you can find out about theproblem. Try it on another machine, perhaps. Try it twice or three times andsee how often it fails. If it goes wrong when you're doing serious work but notwhen you're trying to demonstrate it, it might be long running times or largefiles that make it fall over. Try to remember as much detail as you can aboutwhat you were doing to it when it did fall over, and if you see any patterns,mention them. Anything you can provide has to be some help. Even if it's onlyprobabilistic (such as "it tends to crash more often when Emacs isrunning"), it might not provide direct clues to the cause of the problem,but it might help the programmer reproduce it.

Most importantly, the programmer will want to be sure of whether they'redealing with a true intermittent fault or a machine-specific fault. They willwant to know lots of details about your computer, so they can work out how itdiffers from theirs. A lot of these details will depend on the particularprogram, but one thing you should definitely be ready to provide is versionnumbers. The version number of the program itself, and the version number ofthe operating system, and probably the version numbers of any other programsthat are involved in the problem.

"So I loaded the disk on tomy Windows . . ."

Writing clearly is essential in a bug report. If the programmer can't tellwhat you meant, you might as well not have said anything.

I get bug reports from all around the world. Many of them are fromnon-native English speakers, and a lot of those apologise for their poorEnglish. In general, the bug reports with apologies for their poor English areactually very clear and useful. All the most unclear reports come from nativeEnglish speakers who assume that I will understand them even if they don't makeany effort to be clear or precise.

• Be specific. If you can do the same thing two different     ways, state which one you used. "I selected Load" might mean     "I clicked on Load" or "I pressed Alt-L". Say which     you did. Sometimes it matters.
• Be verbose. Give more information rather than less. If you     say too much, the programmer can ignore some of it. If you say too little,     they have to come back and ask more questions. One bug report I received     was a single sentence; every time I asked for more information, the     reporter would reply with another single sentence. It took me several     weeks to get a useful amount of information, because it turned up one     short sentence at a time.
• Be careful of     pronouns. Don't use words like     "it", or references like "the window", when it's     unclear what they mean. Consider this: "I started FooApp. It put up a     warning window. I tried to close it and it crashed." It isn't clear     what the user tried to close. Did they try to close the warning window, or     the whole of FooApp? It makes a difference. Instead, you could say "I     started FooApp, which put up a warning window. I tried to close the     warning window, and FooApp crashed." This is longer and more     repetitive, but also clearer and less easy to misunderstand.
• Read what you     wrote. Read the report back to     yourself, and see ifyou think it's clear. If you have listed a     sequence of actions which should produce the failure, try following them     yourself, to see if you missed a step.

Summary

• The first aim     of a bug report is to let the programmer see the failure with their own     eyes. If you can't be with them to make it fail in front of them, give     them detailed instructions so that they can make it fail for themselves.
• In case the     first aim doesn't succeed, and the programmer can't see it failing     themselves, the second aim of a bug report is to describe what went wrong.     Describe everything in detail. State what you saw, and also state what you     expected to see. Write down the error messages, especially if they     have numbers in.
• When your     computer does something unexpected, freeze. Do nothing until you're     calm, and don't do anything that you think might be dangerous.
• By all means     try to diagnose the fault yourself if you think you can, but if you do,     you should still report the symptoms as well.
• Be ready to     provide extra information if the programmer needs it. If they didn't need     it, they wouldn't be asking for it. They aren't being deliberately awkward.     Have version numbers at your fingertips, because they will probably be     needed.
• Write     clearly. Say what you mean, and make sure it can't be misinterpreted.
• Above all, be     precise. Programmers like precision.

 

转载：http://www.chiark.greenend.org.uk/~sgtatham/bugs.html