论文写作注意事项

Tips and Guidance for Students Writing Papers and Reports
Gernot Heiser

---

Contents

• Introduction
• General Advice on Technical Writing  
• About (Honours) Thesis Writing    
  • General  Advice
  • Structure and Coherence
• About Paper Writing    
  • Paper vs thesis
  • General rules
  • Expectations on systems papers
  • Structure
  • Formatting
• Some Things People Frequently Get Wrong
• Microsoft Word
• Some TeXniques
• Auxiliary Material    
  • My talk on paper writing
  • On evaluation and benchmarking
  • A nice example

---

Introduction

This document originally grew out of frustration of having to fight my way through poorly written thesis and paper drafts, and was targeted at my own students. It developed (mostly grew) over time as I found new issues to address.

In the meantime, externals seem to use it too. I have consequently started to make it less specific to my own students, in the hope that more people benefit from the (accumulated over time) significant effort that went into this. Pleasesend me feedback if you find any errors or have suggestions for improvements.

I get to read a fair amount of student prose, and the quality is highly variable, and some of it  everything but a pleasure to read. In fact, some of it isvery poorly written. Sometimes it's an early draft (where somehow the author thinks things will magically improve later), sometimes it is even a final (submitted) version of an undergraduate thesis.

I also do a lot of reviewing, mostly as a member of program committees of conferences. And sadly, some of those papers submitted for publication are not much better than drafts my students give me to read.

Such poor writing is annoying and counter-productive:

• I'm wasting my time correcting stuff students should be doing  themselves. My time would be better used commenting about the  technical contents. But they are often hard to get to in a poor  manuscript.
• Maybe I read the thing carefully even if it's poorly  written, maybe not. Most wouldn't. I certainly wouldn't if it came from  somewhere else.
• If a paper submitted to a conference is poorly written, this  will reduce its chance of acceptance, like it or not. But it's also  costing me (as your supervisor, or as a reviewer) extra time and effort, so I'd be much happier if all  submissions were written well.
• Poor written communication skills will most likely impede your  career, whether you'll be working in industry or academia. Hence  it's important to do something about it while there is still time:  
  • Maybe you're just too lazy to do a proper job. In this case    you're really saying that you consider your time more valuable    than mine. I don't agree, and it'll cost you.
  • Maybe you have a real communication problem. In this case it    is important that you realise this, and that you do something    about it. Get professional help, communication skills are    important for your future! There are places on your campus which    can help you.

I've written up some general hints on technical writing, followed by more specific guidelines aimed atstudents writing their thesis. (This is mostly aimed at undergraduate thesis students, although I've assessed PhD theses which got a lot of the same things wrong, so PhD students may find this useful too.) I separately discuss writingconference papers.

Finally there is a section listing style and grammar issues I encounter most frequently in student prose, and some guidelines on how to do better. However, there is much more to get wrong, and I recommend getting a good style book. I generally follow:

Pam Peters. The Cambridge Australian English Style Guide. Cambridge University Press, 1995.

This describes the “official” rules in place in Australia (as far as there is such a thing). It often isn't specific enough for the purposes of technical prose.  The following is an excellent book, geared towards folks like us. The author knows computing jargon, and knows her nerds:

Lyn Dupré. Bugs in writing: A Guide to Debugging Your Prose. Addison-Wesley, 1995

This book's main drawback is that it uses American rules, which are some times conflicting with Australian/British rules. She follows the official American rules to the dot, including where no-one else does, and occasionally produces bizarre results. The book is nevertheless very useful.

It is acceptable to use American rules (for papers), but only if you use them consistently. That means using all American spelling as well as grammar and style rules. Don't mix!

An interesting case is program vs. programme. The former is American while the latter used to be the British (and Australian) spelling. However, theMacquarie  Dictionary since at least the beginning of the century considersprogram the correct spelling for all cases (not only computer programs), while theOED treats both equally. Facit: useprogram.

Another good (although highly incomplete) reference are the Notes for Authors editor Peter Salus wrote for the (long-defunct) Usenix Computing Systems journal. You can tell from the examples that this was written in the '80s, but the advice (except as it relates to the troff typesetting program) is still current.

The below hints are generally consistent with British/Australian rules, but sometimes narrower, particularly if this makes them consistent with American rules as well.

General Advice on Technical Writing

One of the most crucial aspects of writing a good technical paper is what I callmaintaining user state. Like a good operating system, the writer should ensure that the (mental) state of the user (i.e. reader) is kept coherent. A good writer is fully aware of the relevant state in the mind of the reader at any point of the paper/report.

What do I mean with this? Basically it means that the paper systematically builds up the reader's understanding and knowledge of the work, starting from a reasonable initial state. This means you need to put yourself into the reader's shoes (or, rather, brain) and ensure that they can follow at each instance. One of the characteristics of good writers is that they do this well. Here is what this means:

• Make reasonable assumptions about the initial state  (i.e. prior knowledge). A common fault of theses and papers is  too much assumed knowledge. You're breathing this stuff daily, and  somehow assume everyone else does. They don't.

  I frequently encounter the opposite problem in paper manuscripts:  too much information is given that is considered general knowledge  of the discipline. This is understandable in the case of junior  students, who do not yet have a good understanding of what is  considered the “general knowledge” of the discipline.

  It is important to understand that what is “reasonable”   assumed knowledge depends on the kind of  report or paper you're writing. For an undergraduate thesis the  initial state is the knowledge of the field you can expect from one of  your peers: students at the same stage, but who haven't done  their thesis in your area. You can assume them to have passed the  basic operating-systems course (with no more than a credit) but know  nothing about the area beyond that.

  For papers it can be anything between an informed and intelligent  generalist to a subject expert, depending on the publication  venue. In particular, reviewers are experts in the discipline area  of the conference, but they may not be the top experts in the  (potentially) narrow problem your paper addresses. This becomes more  pronounced as you progress through your PhD: in the end (when you're  finished) you'll bethe expert, and there may only be a  handfull others at a comparable level of expertise. And not all of  them will be your reviewers, so you can expect most reviewers to  have a somewhat lesser degree of expertise (but you can also assume  them to be very smart!)

• Make sure the paper/report is self-contained. While it is  important to reference prior work (yours as well as others), don't  expect the reader to have read all those papers! In fact, a person of  the right target group can in general not be expected to have read any  but the most seminal papers, and even if they read them, they can't be  expected to remember every detail (relevant as it may be to you). For  example, if your work is about system virtual machines, you can expect  the reader to be familiar with the Xen work, especially the Cambridge  SOSP paper. But don't expect them to remember every trick described  there!

  Also, don't assume that the reviewers don't know some work  intimately, there's always a likelihood that someone will. So don't  think you can hide anything!

• Ensure that at any point in the paper, you don't expect more  knowledge from the reader than the union of the initial state and what  you've told them so far. This seems obvious, but is violated  frequently. Remember, the reader normally reads the work  sequentially. It doesn't help them if a term you're using right now is  explained ten pages later.Define-before-use isn't just important in  programming, it's important in writing just the same.

  Occasionally it is necessary to leave a detailed explanation for  later. In this case you must provide a forward reference at the place  where it is first used. However, this isonly acceptable if  an approximate understanding suffices at the point of the forward  reference, and you can reasonably expect that approximate  understanding to exist in the reader's mind. Examples are concepts  that are widely used (and you use them consistently with the reader's  expectation), or you have given at least a brief (potentially informal  or anecdotal) explanation.

• Occasionally you have circular dependencies: Explaining A requires  understanding B, and explaining B requires understanding A. What do  you do here? The usual way out of this is to first give a brief,  informal explanation of all terms, and follow it up with a rigorous  definition/explanation later. Particularly when the definitions are  highly formal or tricky, this is a good idea anyway.

• Remember, human memories aren't perfect, and therefore the user  state is lossy. Like with DRAM, you need to refresh it. If a term has  been explained early on, and then not used for many pages, chances are  that the reader has forgotten. (Remember, a 50–100-page report is  unlikely to be read in one go, the reader may have read that section a  week ago. Even a conference submission may not be read in one go,  reviewers get interruptions too!) So, give them a little refresher. Examples:

  • In Section 2 we had defined gizmo to be something    every cool kid wants. Now we will... [Gently re-hashes definition]
  • The concept of a trusted computing base has been    introduced in Chapter 3. We will now have a closer look at the    components of the TCB in our system. [Unintrusively reminds    readers of the definition of TCB without seeming repetitive]

• Be consistent in your own terminology. This again sounds too  obvious to mention, but is violated all the time. For example, don't  use two different terms interchangeably,unless they  really have the same meaning, and you have made that point  explicitly. For example:

  • Hypervisor and virtual-machine monitor are    traditionally interchangeable, but don't use them interchangeably,    as you're likely to confuse the reader. It's ok (in fact good) to    mention interchangeable names, but thenstick to one for the    rest of the document! Also, note that these two terms are    starting to take on different meanings, “hypervisor”     is increasingly used only for the part of the virtual-machine    monitor which executes in the most privileged mode, while there    may be less privileged parts (such as the usermode virtualization    support in L4, or Dom0 in Xen).
  • In a capability system you invoke operations by supplying a    capability. You may talk of using a capability toinvoke an    object, orinvoking a capability. Both are reasonable    terminology, but they can't co-exist, as a capability is different    from the object that gets invoked through it. Decide which    terminology you want to use, and stick to it. Don't confuse the    reader with loose language!

There are probably more rules of this sort, I'll add them as I think of them (and feel free to suggest some to me). In summary, the more you worry about maintaining user state, the more readable people will find your work.

About (Honours) Thesis Writing

This section focuses on honours (undergraduate) theses, as they tend to be the ones needing advice. However, there are late-stage PhD students who are also well advised to read this (I've seen examples both when reading drafts from internal students as well as when assessing theses from other unis).

General  advice

First rule is think before you write. Have an outline, know what you want to write about in each part, and how to approach it. If you start off with a brain dump, the final thesis will probably look like a brain dump. Not a good position from which to get a high mark... The section on structure tries to help you with this.

Also, be careful how you write. Ensure that the thesis is  well-readable. This implies following the general style and grammar  rules, violating those detracts the reader and makes the text harder  to follow. These rules have developed for a reason. Also, check below for the proper use of “we”.

You may think I'm petty for insisting on proper prose. The reason I  do it is becausea report that ignores these rules is hard to read  and annoys the reader. Making it hard to read wastes my time,  and I don't feel I've got time to waste.

Many students have the attitude of “I'll write it down quickly  and worry about the details later.” That's fine, as long as you  worry about the “details”before you present a  draft for feedback. Experience is they don't, and sloppy remains  sloppy.I am yet to experience a case where I got to read a  sloppily/carelessly written draft which ended up being a well-written  thesis. These cases may exist, I just haven't seen them. Avoid  starting off in the wrong direction! Thesection on  typical mistakes tries to help you with this. Read it before you  start, and read it again before you hand out a draft for feedback!

Also, take feedback on your draft seriously. This means not only  blindly fixing marked-up issues, butthink about the  comments. Particularly if the same mistake gets highlighted  repeatedly, think about why you make this mistake, and how you can  avoid making it again. How else do you want to learn good writing?  Writing your thesis is your job, not mine. I only provide feedback so  you can learn!

Obviously, having a close look at a number of good thesis reports is a good idea. However, there are at least two problems with this: which of the reports posted on theDiSy thesis page (only accessible from within the cse.unsw.edu.au and nicta.com.au domain) are good, and, given that none is perfect, what should one look out for? There are obviously no marks posted, and even if you know that a particular thesis achieved a high mark, you can never be sure whether that was because, or despite the writeup.

There are no firm rules on how to write a thesis, and there is certainly a lot of advice available. I'll try to concentrate on a few main points (which tend to apply to a lot of technical writing, not only to theses but also to conference papers).

Structure and coherence

Make sure your thesis is well structured, that each major section does what it is supposed to do, and that the whole thing hangs together. The basic structure is often as follows (but other structures are possible).

In particular, don't think you need to have exactly as many major sections or chapters as there below list implies; sometimes it makes sense to merge things, sometimes it makes sense to move things (e.g. the literature review is in many papers deferred until after the results), sometimes it makes sense to split a logical part into several individual sections. Also, PhD theses often integrate multiple broad issues which are covered in separate chapters, each with its own background and related work. Use common sense!

Title

Use a descriptive title for your work. Don't use a title that promises more than you'll deliver, don't use a title that implies something different from what you've done. (The focus of a thesis often shifts in the course of a year, don't be afraid to adjust the title, in consultation with your supervisor.)

Abstract

A short (1–3 paragraph) summary of the work. Should state the problem, major assumptions, basic idea of solution, results. Avoid non-standard terms and acronyms. The abstract must be able to be read completely on its own, detached from any other work (e.g. in collections of paper abstracts). Don't use references in an abstract.

Introduction

Introduce the problem (gently!) Try to give the reader an  appreciation of the difficulty, and an idea of how you will go about  it. It's like the overture of an opera: it plays on all the relevant  themes.

Make sure you clearly state the vision/aims of your work, what  problem you are trying to solve, and why it is important. Especially,  make clear you highlight the challenges you need to overcome. Having  made it through the intro, the reader should have a clear idea of what  to expect in the remainder of the work. This applies to the problem,  the author's hypothesis, and (at a very high level) the approach  taken. Leave out any of these at your peril!

While the introduction is the part that is read first (ignoring title  and abstract), and it is generally a good idea to write it first, in  order to define the roadmap for the thesis, it isimportant  that you revise (and more often than not, re-write) the introafter  the document is essentially finished, and all the results are in and  understood. This is essential to ensure consistency.

Remember, the intro is the first  thing that is being read, and will have a major influence on the how the  reader approaches your work. If you bore them now, you've most likely  lost them already. On the other hand, if you make outrageous claims, pretend to solve the  world's problems, etc, you're likely fighting an uphill battle later  on.

Make sure you pick up any threads spun in the introduction  later on, to ensure that the reader thinks they get what they have been  promised. Don't create an expectation that you'll deliver more than you  actually do. That is calledbait and switch, and tends to  leave the reader highly dissatisfied. Remember, the reader may be your marker (of a thesis) or  referee (of a paper), and you don't want to piss them off.

It's also important that you provide enough context and  indication the limitations/assumptions of your work to avoid  uprising (and disappointing) your reader.

Exposition of problem

The basic problem should have been stated in the intro, here is the  place to go into detail.

Make it clear you know what you are talking about (and this includes  being complete, don't jump right into things, give the reader a chance  to follow). Give a thorough and complete discussion of the problem,  enough so an educated reader whose speciality is outside yours can  appreciate that you're trying to attack an interesting problem, and also  appreciates what's interesting about it. (Remember to maintain user state!)

Btw, don't call this section“exposition of the problem”, or you'll  be immediately exposed as someone who can only follow recipes. Same  applies to the next bit.

Literature review (often called “related work”)

This is   really important. If you cannot demonstrate that you know, and  understand, what others have done, you only demonstrate that you're  clueless. For an undergraduate thesis this, together with a thorough  understanding of the problem, should be the result of the first  session's work. It is an unfortunate fact that many students do very  little work during the first session of their thesis. It usually shows  here (and is usually reflected in their mark). Don't think you can fool  your thesis supervisor/assessor. And don't even dream about fooling the  referee of a paper. If you haven't done your homework here, it's  probably not worth going any further.

In this part you demonstrate that you are aware of what's going on in  the field, and how it relates to your particular problem. In an undergraduate thesis  (unlike a conference paper) it may be ok to repeat work that has  already been done elsewhere (usually in somewhat different  circumstances). Be open, and explain why what you're doing is still  worthwhile. In the more normal case that you're doing something that  hasn't already been done, convince the reader that this is actually the  case. One of the less convincing arguments goes along the line “a  Google search on `frying giblets on StrongARM-driven toasters' didn't  turn up anything”. You might as well pack up here. The way to convince  the reader that your work hasn't been done before is to explain what has  been done, what's different about what has been done, and, if you're  good, why it hasn't been done already. There is always related work, and  the more vague you are about it, the more obvious it is that you haven't  done your homework. (And, no, looking at all the Google hits isn't  enough.)

Sometimes some relevant background work is quite old; the  discipline goes in cycles and it isn't all that infrequent that  people rediscover things that have been done 30 years ago (virtual  machines are an example). In such case please note that the language  has changed a fair bit in the meantime. You're not doing your reader  a favour of reporting an old paper's findings in that paper's  language (and in the informed reader's mind you'll raise the  suspicion that you don't understand what's going on). Talk about the  work of the paper in contemporary systems language. This makes it  easier to compare to other work, including yours.

Design of your solution

Having explained the problem, and what others have done in  similar situations, now explain your approach. Again, give a general  overview of your design first, and then go into detail  (i.e. use a top-down approach). Make sure that the document  (particularly a thesis) is self-contained: It should be possible for  a reader familiar with the general area (that means operating  systems, not methods for implementing free-block lists) to  understand your design. (Remember to maintain user state!)

Discuss design tradeoffs before you present the design you have  settled on, don't use the backward approach of “I'm doing it  this way. I could have done it that way, but...” This smells of  having been added as an afterthought. Show that you have thought  things through, and convincingly show how and why you have arrived at  the best solution.

Note that this may be an inversion of the approach you have taken in  reality: You might have tried something, run into problems and then  changed the design. Remember: your thesis isn't an activity report, it  is the presentation of research. Which detours you took to arrive at  the destination is primarily irrelevant (and in many cases just an  admission of not having thought things through before you  started).Focus on the outcome, not the journey!

It's not necessarily wrong to point out what traps you fell  into, but present that in the context of a discussion of design  tradeoffs. Sometimes the correct design may be impossible to  determinea priori, making some early experiments  essential. But that doesn't mean it should be presented as a history  lesson. Discuss the alternatives, say what you did to investigate the  implications, and then present your design decision.

Importantly, be forthright about the limitations and assumptions of your  design. Also, make sure you justify any shortcuts/limitations  convincingly

Implementation

In many (not all cases) there is a clear difference between the  general approach (design) and its implementation in your particular  circumstances. The design may be more general than what you can do given  time and resources. Or you have developed a general design, and are now  implementing a prototype on particular hardware. Or the design is  relatively high-level but leaves open a lot of implementation  questions. Avoid mixing up discussions of design and implementation!  Design is first, implementation later.

Give all required  details. It should be possible to understand all this without  referring to the source code. (I generally refer to the source code to  check whether the code is consistent with the report, I shouldn't have  to do this in order to understand the report.)

This will, in general, include extracts of actual source code (or  pseudo-code), basic algorithms, function prototypes etc. Don't list  pages of C code, an electronic copy of the source should accompany  the submission and should be available to the marker, so there's no  point in killing extra trees to put it into the report.

Make sure you describe your implementation in enough  detail. (Maintain user state!) Someone  who has nothing else but your thesis report to go by should be able to  repeat your work, and arrive at essentially the same  implementation. Reproducibility is an important  component of scientific  work. Also, clearly state the limitations of your implementation, and  justify them.

Experiments

A thesis almost always has an experimental part, typically some  benchmarking. This is usually its weakest part. Many students debug  their code less than a week prior to the submission deadline (typical  indication of having started too late) which makes it hopeless to do any  real benchmarking. Benchmarking takes time, for running the experiments,  but also for thinking them up in the first place, and for analysing the  results (and, inevitably, decide you'll have to do more benchmarks to  clear things up).

Probably the majority of theses I mark is really deficient in this  part, typically for lack of attention (often resulting from a late  start). Think about what makes sense to measure, what you want to learn  from your measurements. Think about what is really the relevant  contribution of your thesis, and how you can prove that you have  achieved your goals. Think about what you can measure in order to get a  good insight into the performance of various aspects of your design, how  you can distinguish between systematic and accidental effects, how you  can convince yourself that your results are right.Most of this  should have been done during Part A of your thesis, together with your  project plan you should have decided what your success criteria are, and  how to establish that you have met them.

If you get surprising  results, don't just say "surprise, surprise, performance isn't as good  as hoped". Find out why. Surprises without explanation indicate either  that you are clueless about what's going on, or that you have made a  mistake (most likely both). Unconvincing results tend to imply  unconvincing marks. (Of course, this could be avoided if the results  were available more than a couple of days prior to the thesis  deadline.)

It is amazing how few students have even the faintest clue of the  most basic statistics and their use. Measurements always have  statistical (sampling) errors. Owing to the deterministic nature of  computers these are sometimes very small in our area, particularly  in the case of micro-benchmarks, where disturbing factors can be  minimised. However, the reader should be given an indication of how  statistically significant the results are. This is done by  providing at least astandard deviation in addition to  averages. Whenever the results of several runs are averaged, a  standard deviation can (and must) be supplied. After all, you  average to reduce statistical errors.

The reproducibility argument applies here just as much as for the  implementation. Give enough detail on what you measure, and  how you measure it, so that someone who has your  implementation (but not your test code) or has re-done your  implementation independently, should be able to repeat your  measurements and arrive at essentially the same results. I read many  theses which contain results which seem outright wrong. In most  cases not enough detail is provided to allow me to pinpoint the  likely source of the error. In many cases the cause is systematic  errors resulting from an incorrect measurement technique. If it  seems wrong, and the text doesn't convince me that it isn't  wrong, I will assume that itis wrong.

Discussion

Discuss and explain your results. Show how they support your thesis  (or, if they don't, come up with a damned good reason real quick). It is  important to separate objective facts clearly from their  discussion (which is bound to contain subjective opinion). If the reader  doesn't understand your results, you probably do neither. And this will  be reflected in the assessment.

Conclusions

Don't leave it at the discussion: discuss what you/we can learn from  the results. Draw some real conclusions. Separate  discussion/interpretation of the results clearly from the conclusions  you draw from them. (So-called “conclusion creep” tends to upset  reviewers. It means surrendering your scientific objectivity.)

Identify all shortcomings/limitations of your work, and discuss how  they could be fixed (“future work”).

I repeat: don't stick slavishly to this structure. Also, remember that the thesis must be:

• honest, stating clearly all limitations;
• self-contained—don't write just for the locals, don't assume that the reader has read the same literature as you, don't let the reader work out the details for themselves.

Also, a thesis isn't called “thesis” by accident: It is supposed to present a thesis you are making about some system, and your justification and confirmation of that thesis. This means that a thesis isnot anexperience report. You may have taken a few detours and explored a few blind alleys. Some of that might be valuable to document, but only for what general truths can be learned from it, e.g. what the pros and cons of particular design decisions are.

So, explain the facts (and what's behind them) but don't bother the reader with the details of you got to the end. I repeat,focus on the outcomes, not on the journey!

Kevin Elphinstone has written an excellent guide on how to write a thesis, which also contains further references. My physics colleague Joe Wolfe has written aPhD thesis guide from a somewhat different angle. And there is a wealth of info at theOnline PhD Guide.

 

 

 

 

 This article first appeared in ACM SIGOPS Operating Systems Review, Vol. 17, No. 3 (July, 1983), pages 35-40. Every effort has been made to keep the text in this document identical to that of the original article. The text in this file was scanned using OCR technology and has been carefully proofread, but some scanning errors may remain. This document is being made available with the permission of the authors.

An Evaluation of the Ninth SOSP Submissions
or How (and How Not) to Write a Good Systems Paper

Roy Levin and David D. Redell
Ninth SOSP Program Committee Co-chairmen

Introduction

On March 21, 1983, the program committee for the 9th Symposium on Operating System Principles, having read the eighty-three papers submitted, selected sixteen for presentation at the symposium. This acceptance ratio of about one in five approximates those of past SOSPs, although the number of submissions was somewhat lower than in recent years. Several members of the program committee found it surprisingly easy to separate the good papers from the bad ones; indeed, the ten committee members quickly agreed on the disposition of over 80% of the papers. As the acceptance ratio indicates, most of these were rejections.

After the committee had completed its selection process, several members expressed disappointment in the overall quality of the submissions. Many of the rejected papers exhibited similar weaknesses, weaknesses that the committee felt should have been evident to the authors. In the hope of raising the quality of future SOSP submissions, and systems papers generally, the committee decided to describe the criteria used in evaluating the papers it received. This article combines the criteria used by all of the members of the committee, not just the authors.

To try to avoid sounding preachy or pedagogic, we have cast this presentation in the first and second person and adopted a light, occasionally humorous style. Nevertheless, the intent is serious: to point out the common problems that appear repeatedly in technical papers in a way that will make it easier for future authors to avoid them. As you read this article, then, suppose yourself to be a prospective author for the 10th SOSP or for TOCS. You've done some work you would like to publish, so you sit down to write a paper. What questions should you be asking yourself as you write? These are also the questions that we, the reviewers of your paper, will be asking to determine its suitability for publication.

Classes of Papers

Your paper will probably fall naturally into one of three categories:

• It presents a real system, either by a global survey of an entire system or by a selective examination of specific themes embodied in the system.
• It presents a system that is unimplemented but utilizes ideas or techniques that you feel the technical community should know.
• It addresses a topic in the theoretical areas, for example, performance modelling or security verification.

Obviously, a single set of evaluation criteria cannot be applied uniformly across these categories; nevertheless, many criteria apply equally well to all three. As we describe each one below, we will try to emphasize the classes of papers to which it applies. Often it will be evident from context.

Criteria for Evaluation of Submissions

Original Ideas

Are the ideas in the paper new? There is no point in submitting a paper to a conference or journal concerned with original work unless the paper contains at least one new idea.

How do you know? You must be familiar with the state of the art and current research in the area covered by your paper in order to know that your work is original. Perhaps the most common failing among the submissions in the first category (real systems) was an absence of new ideas; the systems described were frequently isomorphic to one of a small number of pioneering systems well-documented in the literature.

Can you state the new idea concisely? If your paper is to advance the state of knowledge, your reader must be able to find the new ideas and understand them. Try writing each idea down in a paragraph that someone generally versed in the relevant area can understand. If you can't, consider the possibility that you don't really understand the idea yourself. When you have the paragraphs, use them in the abstract for the paper.

What exactly is the problem being solved? Your reader cannot be expected to guess the problem you faced given only a description of the solution. Be specific. Be sure to explain why your problem couldn't be solved just as well by previously published techniques.

Are the ideas significant enough to justify a paper? Frequently, papers describing real systems contain one or two small enhancements of established techniques. The new idea(s) can be described in a few paragraphs; a twenty-page paper is unnecessary and often obscures the actual innovation. Since construction of a real system is a lot of work, the author of the paper sometimes unconsciously confuses the total effort with the work that is actually new. ("My team worked on this system for two years and we're finally done. Let's tell the world how wonderful it is.") If the innovation is small, a small paper or technical note in a suitable journal is more appropriate than an SOSP submission.

Is the work described significantly different from existing related work? An obvious extension to a previously published algorithm, technique, or system, does not generally warrant publication. Of course, the label "obvious" must be applied carefully. (Remember the story of Columbus demonstrating how to make an egg stand on end (by gently crushing it): "it's obvious once I've shown you how".) You must show that your work represents a significant departure from the state of the art. If you can't, you should ask yourself why you are writing the paper and why anyone except your mother should want to read it.

Is all related work referenced, and have you actually read the cited material? You will have difficulty convincing the skeptical reader of the originality of your efforts unless you specifically distinguish it from previously published work. This requires citation. Furthermore, you will find it harder to convince your reader of the superiority of your approach if he has read the cited works and you haven't.

Are comparisons with previous work clear and explicit? You cannot simply say: "Our approach differs somewhat from that adopted in the BagOfBits system [3]." Be specific: "Our virtual memory management approach uses magnetic media rather than punched paper tape as in the BagOfBits system [3], with the expected improvements in transfer rate and janitorial costs."

Does the work comprise a significant extension, validation, or repudiation of earlier but unproven ideas? Implementation experiences supporting or contradicting a previously published paper design are extremely valuable and worthy candidates for publication. Designs are cheap, but implementations (particularly those based on unsound designs) are expensive.

What is the oldest paper you referenced? The newest? Have you referenced similar work at another institution? Have you referenced technical reports, unpublished memoranda, personal communications? The answers to these questions help alert you to blind spots in your knowledge or understanding. Frequently, papers with only venerable references repeat recently published work of which the author is unaware. Papers with only recent references often "rediscover" (through ignorance) old ideas. Papers that cite only unpublished or unrefereed material tend to suffer from narrowness and parochialism. Remember that citations not only acknowledge a debt to others, but also serve as an abbreviation mechanism to spare your reader a complete development from first principles. If the reader needs to acquire some of that development, however, he must be able to convert your citations into source material he can read. Personal communications and internal memoranda fail this test. Technical reports are frequently published in limited quantities, out-of-print, and difficult to obtain. Consequently, such citations as source material should be avoided wherever possible.

Reality

Does the paper describe something that has actually been implemented? Quite a few of the SOSP submissions proceeded for fifteen pages in the present tense before revealing, in a concluding section (if at all), that the foregoing description was of a hypothetical system for which implementation was just beginning or being contemplated. This is unacceptable. Your reader has a right to know at the outset whether the system under discussion is real or not.

If the system has been implemented, how has it been used, and what has this usage shown about the practical importance of the ideas? Once again, a multiple man-year implementation effort does not of itself justify publication of a paper. If the implemented system contains new ideas, it is important to explain how they worked out in practice. A seemingly good idea that didn't pan out is at least as interesting as one that did. It is important to be specific and precise. "Our weather prediction system is up and running and no one has complained about its occasional inaccurate forecasts" is much less convincing than "everytime we fail to forecast rain, the users hang their wet shirts over the tape drives to dry". In the latter case, at least we know that people are using and depending on the system.

If the system hasn't been implemented, do the ideas justify publication now? This can be a difficult question for an author to answer dispassionately, yet any reviewer of the paper will make this judgment. It is always tempting to write a design paper describing a new system, then follow it up in a year or two with an "experience" paper. The successful papers of this genre nearly always include initial experience in the closing sections of the design paper. The subsequent experience paper then deals with the lessons learned from longer-term use of the system, frequently in unanticipated ways. Reviewers are very skeptical of design-only papers unless there are new ideas of obviously high quality.

Lessons

What have you learned from the work? If you didn't learn anything, it is a reasonable bet that your readers won't either, and you've simply wasted their time and a few trees by publishing your paper.

What should the reader learn from the paper? Spell out the lessons clearly. Many people repeat the mistakes of history because they didn't understand the history book.

How generally applicable are these lessons? Be sure to state clearly the assumptions on which your conclusions rest. Be careful of generalizations based on lack of knowledge or experience. A particularly common problem in "real system" papers is generalization from a single example, e.g., assuming that all file system directories are implemented by storing the directory in a single file and searching it linearly. When stating your conclusions, it helps to state the assumptions again. The reader may not have seen them for fifteen pages and may have forgotten them. You may have also.

Choices

What were the alternatives considered at various points, and why were the choices made the way they were? A good paper doesn't just describe, it explains. Telling your readers what you did doesn't give them any idea how carefully considered your choices were. You want to save future researchers from following the same blind alleys. You also want to record potentially interesting side-streets you didn't happen to explore. Make sure to state clearly which is which.

Did the choices turn out to be right, and, if so, was it for the reasons that motivated them in the first place? If not, what lessons have you learned from the experience? How often have you found yourself saying "this works, but for the wrong reason"? Such a pronouncement represents wisdom (at least a small amount) that may benefit your reader. Many papers present a rational argument from initial assumptions all the way to the finished result when, in fact, the result was obtained by an entirely different path and the deductive argument fashioned later. This kind of "revisionist history" borders on dishonesty and prevents your readers from understanding how research really works.

Context

What are the assumptions on which the work is based? The skeptical reader is unlikely to accept your arguments unless their premises are stated. Make sure you get them all; it's easy to overlook implicit assumptions.

Are they realistic? For "unimplemented systems" papers, this amounts to asking whether the assumptions of the design can hope to support a successful implementation. Many paper designs are naive about the real characteristics of components they treat abstractly, e.g., communication networks or humans typing on terminals. For theoretical studies, it must be clear how the assumptions reflect reality, e.g., failure modes in reliability modelling, classes of security threats in security verification, arrival distributions in queuing systems.

How sensitive is the work to perturbations of these assumptions? If your result is delicately poised on a tall tower of fragile assumptions, it will be less useful to a reader than one that rests on a broader and firmer foundation.

If a formal model is presented, does it give new information and insights? Simply defining a model for its own sake is not very useful. One deep theorem is worth a thousand definitions.

Focus

Does the introductory material contain excess baggage not needed for your main development? "Real system" papers are particularly guilty of irrelevant description. If your subject is distributed file systems, the physical characteristics of the connection between computer and communication network are probably not germane. Avoid the temptation to describe all major characteristics of your system at the same level of depth. Concentrate instead on the novel or unusual ones that (presumably) will be the focus of the original technical content of the paper.

Do you include just enough material from previously published works to enable your reader to follow your thread of argument? Do not assume that the reader has read every referenced paper within the last week and has them at his fingertips for instant reference. If you want your reader to get past page three, avoid introductory sentences of the form "We adopt the definition of transactions from Brown [4], layering it onto files as described by Green [7, 18], with the notions of record and database introduced by Black [10] and White [12] and later modified by Gray [6]". On the other hand, don't burden your reader unnecessarily with lengthy extracts or paraphrases from cited works.

Presentation

Are the ideas organized and presented in a clear and logical way?

Are terms defined before they are used?

Are forward references kept to a minimum? Readers get annoyed when they repeatedly encounter statements like "Each file consists of a sequence of items, which will be described in detail in a later section". The reader has to remember the technical term "item", but the term has no semantics yet. It's all right to ask him to do this once or twice, but only when absolutely necessary. Even if you can't afford the digression to explain "item" at this point, give the reader enough information to attach some meaning to the term: "Each file consists of a sequence of items, variable-sized, self-identifying bit sequences whose detailed interpretation will be discussed below under 'Multi-media Files'." Your reader may not yet understand your concept of files completely, but at least he has some glimpse of the direction in which you are leading him.

Have alternate organizations been considered? Theoretical papers, particularly of a mathematical character, are generally easier to organize than papers describing systems. The expected sequence of definition, lemma, theorem, example, corollary works well for deductive argument, but poorly for description. In "real system" papers, much depends on the intent: global survey or selective treatment. Frequently, difficulties in organization result from the author's unwillingness to commit to either approach. Decide whether you are surveying your system or focusing on a specific aspect and structure the paper accordingly.

Was an abstract written first? Does it communicate the important ideas of the paper? Abstracts in papers describing systems are sorely abused. The abstract is more often a prose table of contents than a precis of the technical content of the paper. It tends to come out something like this: "A system based on Keysworth's conceptualization of user interaction [4] has been designed and implemented. Some preliminary results are presented and directions for future work considered." No reader skimming a journal is likely to keep reading after that. Avoid the passive voice (despite tradition) and include a simple statement of assumptions and results. "We designed and implemented a user interface following the ideas of Keysworth and discovered that converting the space bar to a toe pedal increases typing speed by 15%. However, accuracy decreased dramatically when we piped rock music instead of Muzak (tm) into the office." Leave discussion and argument for the paper. It helps to write the abstract before the paper (despite tradition) and even the outline, since it focusses your attention on the main ideas you wants to convey.

Is the paper finished? Reviewers can often help you to improve your paper, but they can't write it for you. Moreover, they can't be expected to interpolate in sections marked "to be included in the final draft". In a mathematical paper, a reviewer regards the statement of a theorem without proof with suspicion, and, if the theorem is intended to culminate prior development, with intolerance. Similarly, in a paper describing a system, a reviewer cannot tolerate the omission of important explanation or justification. Omitting sections with a promise to fill them in later is generally unacceptable.

Writing Style

Is the writing clear and concise?

Are words spelled and used correctly?

Are the sentences complete and grammatically correct?

Are ambiguity, slang, and cuteness avoided?

If you don't have sufficient concern for your material to correct errors in grammar, spelling, and usage before submitting it for publication, why should you expect a reviewer to read the paper carefully? Some reviewers feel that this kind of carelessness is unlikely to be confined to the presentation, and will reject the paper at the first inkling of technical incoherence. Remember that you are asking a favor of your reviewers: "Please let me convince you that I have done interesting, publishable work." A reviewer is more favorably disposed toward you if he receives a clean, clear, carefully corrected manuscript than if it arrives on odd-sized paper after ten trips through a photocopier and looking like it was composed by a grade-school dropout. Even if you aren't particularly concerned with precise exposition, there is certain to be someone in your organization who is. Give your manuscript to this conscientious soul and heed the resulting suggestions.

Summary

These thirty-odd questions can help you write a better technical paper. Consult them often as you organize your presentation, write your first draft, and refine your manuscript into its final form. Some of these questions address specific problems in "systems" papers; others apply to technical papers in general. Writing a good paper is hard work, but you will be rewarded by a broader distribution and greater understanding of your ideas within the community of journal and proceedings readers.

 

What Makes a Good System Paper - by Gernot Heiser in APSys 2013

Aug 1st, 2013                 | Comments              

There’s a session in APSys 2013, where Professor Gernot Heiser discussed about how to write a good system paper. It is really helpful, and I would like to share some of his suggestions here.

He introduced 4 rules of writing:

reviewers are pot luck

Most problems raised by reviewers may be:

I'm not convinced you're solving a real problemI'm not convinced you're solving the problemI don't understand - too badly writing

And one important thing we need to know is:

papers without a PC "champion" have a hard stand

Hence we must make sure there’s something which at least one reviewer will think cool!

a paper has a story

That means each paper should have one main message, and the author must know clearly what the message is, to make sure that the reader gets it, and make sure it’s an interesting one.

A paper has a narrative. It starts from zero and then works on transmitting the message. Everything you write must support the message.

The other important notes you should know are: maintain user state! ( be conscious of what the reader knows/remembers)

imited real estate: the two “c” s

be “clear”

Which means every sentence, paragraph, section should has a clear purpose, which should be clearly communicated, and the overall message is consistent.

be “concise” (brief but complete)

Don’t waffle! Be precise! and make sure it’s readable, lucid, enjoyable.

Other things need to consider are:

maintain reader statedefine before usebe aware of what the reader has learnedrecall/remind if necessary

presentation matters - paper engineering

Some important bits here:

introduction: sell the idea, the significance and the approachbuild tension, make reader interestedconvincing argumentationtop-down, not bottom-upmaintain reader stateconvincing evaluationstate assumption/limitations honestly

And notes about each section of one paper:

Introduction: most important part of the paper!

It mostly issues an idea to come, explains the problem you’re solving,outlines your approach, indicates results/outcomes, states contributions, andcorrectly cites PCs’ work important!!!

Meanwhile, you should capture the reader’s interest to sell your idea, and be concise (stay within about one page), make sure the paper delivers what you promise. Remember: reviewer hate “bate and switch”

other parts:

background: set the scene in more detail

In backgound, you can cite related work, describe problem in detail, explain solution in detail (honest with limitations and assumptions)

abstract

it mostly used to steer to the right reviewers

what, why, achievement, implication (four sentences)important: redo for camera ready

One interesting statement is that abstract usually has two purposes:

for published: rewrite for people to readfor reviewers: to select paper to review, get right reviewers!

evaluation: often largest part

It shows your solution actually works

progressive: significant improvements in important situationsconservation: no (or insignificant) degradation elsewhere

Be careful about the scenarios of your benchmark

artificial/constructed best cases will be discountedthink of ways in which your approach could fail/deterioratego out of your way to be fair, anticipate any scepticism of your work

---

Other notes:

style and form

Write in engaging style, lead reader though the paper

avoid tobbom-up structure, present idea top-downfollow style rulesuse active voiceavoid buzzwords("novel", "mobile social supercomputing in cloud")     

Be mindful of reader’s brain state (which is lossy)

maintain reader statedon't assume every reviewer is expert in your narrow areabut don't think you can hide stuff from reviewers

Follow formatting rules

don't play with margin, baseline skip etc.don't use microscopic fonts > 40y olds have problems with <8pt font

Spell-check, proof-read, proof-read

get native speaker to proof-read if you aren'tget outsider to read it - great way to spot holes before it's too late!

Mechanics

use revision controldon't use MS worduse BitTex

---

And some other materials:

• Levin & Redell: An evaluation of the 9th SOSP submissions or how to write a good systems paper.
• Tips and Guidance for Students Writing Papers and Reports

---

The trip to Singapore for APSys 2013 really make me harvest a large. It is my first time to submit a poster, and the first time to apply for the Travel Grant. And I’ve meet, talked to and discussed with a lot of genius and excellent people there. Although I’m not good at social, attending such excellent meeting let you learn from others a lot, and make you much more familiar with the System community.

Finally is my photo in presenting my poster work - RemoteBinder. The audience is the lengendary people Yandong Mao, graduated from Fudan University, and now PhD of MIT CSAIL.