The Art of Unix Programming

  • More of the Unix philosophy was implied not by what these elders said but by what they did and the example Unix itself set. Looking at the whole, we can abstract the following ideas:
    1. Rule of Modularity: Write simple parts connected by clean interfaces.
    2. Rule of Clarity: Clarity is better than cleverness.
    3. Rule of Composition: Design programs to be connected to other programs.
    4. Rule of Separation: Separate policy from mechanism; separate interfaces from engines.
    5. Rule of Simplicity: Design for simplicity; add complexity only where you must.
    6. Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.
    7. Rule of Transparency: Design for visibility to make inspection and debugging easier.
      … A software system is transparent when you can look at it and immediately understand what it is doing and how. It is discoverable when it has facilities for monitoring and display of internal state so that your program not only functions well but can be seen to function well.
      … For a program to demonstrate its own correctness, it needs to be using input and output formats sufficiently simple so that the proper relationship between valid input and correct output is easy to check.
      … The objective of designing for transparency and discoverability should also encourage simple interfaces that can easily be manipulated by other programs — in particular, test and monitoring harnesses and debugging scripts.
    8. Rule of Robustness: Robustness is the child of transparency and simplicity.
    9. Rule of Representation: Fold knowledge into data so program logic can be stupid and robust.
    10. Rule of Least Surprise: In interface design, always do the least surprising thing.
      … demand the least new learning from the user.
      … most effectively connect to the user’s pre-existing knowledge.
      … Pay attention to your expected audience. They may be end users, they may be other programmers, or they may be system administrators. What is least surprising can differ among these groups.
      … Pay attention to tradition. The Unix world has rather well-developed conventions about things like the format of configuration and run-control files, command-line switches, and the like. These traditions exist for a good reason: to tame the learning curve. Learn and use them.
    11. Rule of Silence: When a program has nothing surprising to say, it should say nothing.
      … When your program’s output becomes another’s input, it should be easy to pick out the needed bits.
      … Well-designed programs treat the user’s attention and concentration as a precious and limited resource, only to be claimed when necessary.
    12. Rule of Repair: When you must fail, fail noisily and as soon as possible.
      … Software should be transparent in the way that it fails, as well as in normal operation. It’s best when software can cope with unexpected conditions by adapting to them, but the worst kinds of bugs are those in which the repair doesn’t succeed and the problem quietly causes corruption that doesn’t show up until much later.
      … Therefore, write your software to cope with incorrect inputs and its own execution errors as gracefully as possible. But when it cannot, make it fail in a way that makes diagnosis of the problem as easy as possible.
      … Consider also Postel’s Prescription:10 “Be liberal in what you accept, and conservative in what you send”. Postel was speaking of network service programs, but the underlying idea is more general. Well-designed programs cooperate with other programs by making as much sense as they can from ill-formed inputs; they either fail noisily or pass strictly clean and correct data to the next program.
    13. in the chain.
    14. Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.
      … most applications would be written in higher-level languages like Perl, Tcl, Python, Java, Lisp and even shell.
    15. Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
      … In the Unix tradition, code generators are heavily used to automate error-prone detail work. Parser/lexer generators are the classic examples; makefile generators and GUI interface builders are newer ones.
    16. Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
      … “Prototyping is important for system design as well as optimization — it is much easier to judge whether a prototype does what you want than it is to read a long specification.” MikeLesk
      … “Make it run, then make it right, then make it fast” Kent Beck
      … “Using prototyping to learn which features you don’t have to implement helps optimization for performance; you don’t have to optimize what you don’t write. The most powerful optimization tool in existence may be the delete key. One of my most productive days was throwing away 1000 lines of code.” KenThompson
    17. Rule of Diversity: Distrust all claims for “one true way”.
      … Even the best software tools tend to be limited by the imaginations of their designers. Nobody is smart enough to optimize for everything, nor to anticipate all the uses to which their software might be put. Designing rigid, closed software that won’t talk to the rest of the world is an unhealthy form of arrogance.
      … Therefore, the Unix tradition includes a healthy mistrust of “one true way” approaches to software design or implementation. It embraces multiple languages, open extensible systems, and customization hooks everywhere.
    18. Rule of Extensibility: Design for the future, because it will be here sooner than you think.
      … Never assume you have the final answer.
      … Always, always either include a version number,
      or compose the format from self-contained, selfdescribing clauses in such a way that new clauses can be readily added and old ones dropped without confusing format-reading code.
      … When you design code, organize it so future developers will be able to plug new functions into the architecture without having to scrap and rebuild the architecture.
  • Compactness
    • K.I.S.S. keep it simple
    • size of a module 200 – 400 logical lines of code (x2 for physical lines)
    • entry points 7 (+-2)
    • semi-compact C and Python [and PHP –Ed.]
  • Orthogonality
    • In a purely orthogonal design, operations do not have side effects; each action (whether it’s an API call, a macro invocation, or a language operation) changes just one thing without affecting others. There is one and only one way to change each property of whatever system you are controlling.
    • D.R.Y. don’t repeat yourself [for code –Ed.]
  • Top-Down versus Bottom-Up
    • In self-defense against this, programmers try to do both things — express the abstract specification as top-down application logic, and capture a lot of low-level domain primitives in functions or libraries, so they can be reused when the high-level design changes.
  • Doug McIlroy, the inventor of Unix pipes and one of the founders of the Unix tradition, had this to say at the time [McIlroy78]:
    • Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features.
    • Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.
    • Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.
    • Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them.
      • He later summarized it this way (quoted in A Quarter Century of Unix [Salus]): This is the Unix philosophy:
        1. Write programs that do one thing and do it well.
        2. Write programs to work together.
        3. Write programs to handle text streams, because that is a universal interface.
  • Rob Pike, who became one of the great masters of C, offers a slightly different angle in Notes on C Programming [Pike]:
    1. You can’t tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don’t try to second guess and put in a speed hack until you’ve proven that’s where the bottleneck is.
    2. Measure. Don’t tune for speed until you’ve measured, and even then don’t unless one part of the code overwhelms the rest.
    3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don’t get fancy. (Even if n does get big, use Rule 2 first.)
    4. Fancy algorithms are buggier than simple ones, and they’re much harder to implement. Use simple algorithms as well as simple data structures.
    5. Data dominates. If you’ve chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.
  • Applying the Unix Philosophy
    • Everything that can be a source- and destination-independent filter should be one.
    • Data streams should if at all possible be textual (so they can be viewed and filtered with standard tools).
    • Database layouts and application protocols should if at all possible be textual (human-readable and human-editable).
    • Complex front ends (user interfaces) should be cleanly separated from complex back ends.
    • Whenever possible, prototype in an interpreted language before coding C.
    • Mixing languages is better than writing everything in one, if and only if using only that one is likely to overcomplicate the program.
    • Be generous in what you accept, rigorous in what you emit.
    • When filtering, never throw away information you don’t need to.
    • Small is beautiful. Write programs that do as little as is consistent with getting the job done.
  • Attitude Matters Too
    • When you see the right thing, do it — this may look like more work in the short term, but it’s the path of least effort in the long run.
    • If you don’t know what the right thing is, do the minimum necessary to get the job done, at least until you figure out what the right thing is.
    • You have to believe that software design is a craft worth all the intelligence, creativity, and passion you can muster.
    • If someone has already solved a problem once, don’t let pride or politics suck you into solving it a second time rather than re-using.
    • And never work harder than you have to; work smarter instead, and save the extra effort for when you need it.
    • Lean on your tools and automate everything you can.
    • stop and think; ask yourself what you’ve forgotten.
    • Why do you design software instead of doing something else to make money or pass the time? You must have thought software was worthy of your passion once.
    • You need to care. You need to play. You need to be willing to explore.
  • Unix has a couple of unifying ideas or metaphors that shape its APIs
    • “everything is a file”
    • and the pipe metaphor built on top of it