Software Engineering

Click here to show the site navigation frame

I see a lot of poorly-written software, both as an end-user and as a developer. Probably I see more than my share, because the tool I use most often (Visual Basic/VBA) is so accessible. People learn how to program with these tools before they learn anything about the art and craft of programming. The result is often sloppy, unmanageable, unmaintainable code.

I'm firmly convinced that it doesn't have to be this way. There is an art to producing good software, and it can be learned. To that end, I'd like to use this page to pass along references to books and web sites that might help. Perhaps it will do no good at all, but at least I'm trying.

Books

By the way, I almost always have one book on software development in the bathroom, where I get to browse through it and read short chunks several times a day. This keeps development issues on my mind and, I think, helps me be a better developer. If I ever have to set up workspace for a major development effort again, I'm going to have the facilities people install bookshelves in the stalls.

• After the Gold Rush, by Steve McConnell. Subtitled "Creating a True Profession of Software Engineering", this book reminds me a bit of the classic The Mythical Man-Month. It's a series of essays examining the state of software engineering today, and arguing that the time has come for accreditation of college courses, a code of ethics, and licensing of software engineers. Whether you agree or not, it's worth reading for a glimpse at one future we might be headed for.
• AntiPatterns, by William J. Brown, Raphael C. Malveau, Hays W. "Skip" McCormick III and Thomas J. Mowbray. "Refactoring Software, Architectures, and Projects in Crisis". A rather shallow book that seems to be capitalizing on the popularity of "patterns" and "refactoring" by rehashing stuff you could find elsewhere in more concise format. An antipattern consists of identifying a problem caused by ignoring a best practice, and then suggesting the best practice as a refactoring. Skip it and go read a book that just gives you the best practices instead. 
• Code Complete, by Steve McConnell. "A Practical Handbook of Software Construction," this fat book includes hundreds of examples of good and bad code in a variety of languages and extensive discussion of why some code is better. I find myself going back to this book regularly, and especially appreciate the frequent pointers to actual hard data. An excellent distillation of a sprawling and confusing field.
• Debugging the Development Process, by Steve Maguire. Tales from the trenches of the early years of Microsoft, with some lessons for software team management. I rate this as a fairly minor work, certainly nowhere near as good as his Writing Solid Code (see below). Worth reading if you're just starting to think about the tradeoffs involved in software development.
• Dynamics of Software Development, by Jim McCarthy. Jim was a Program Manager for the Visual C++ team at Microsoft, and this slim book summarizes some of the lessons he learned with that sometimes-troubled product. It tends to be a bit on the light side, and the lessons here are mostly available elsewhere. I found the discussion of handling schedule slips pretty valuable, though.
• Extreme Programming, by Kent Beck. Overall I'm pretty skeptical about this one. You may gain a few ideas, but I think there's not enough meat in this book to warrant the price. More a polemic that programmers can do good work without planning than an actual methodology here, little indication that it won't scale or won't work for everyone, and no measurement. If it comes your way, read it for ideas you can use, but I'm pretty lukewarm about "XP" after reading it. 
• An Introduction to Data Structures with Applications, by Jean-Paul Tremblay and Paul G. Sorenson: Excellent, but alas, out of print. But if you're writing programs and don't understand things like linked lists, hash tables, stacks and binary trees, you need to read some data structures book. There's a bunch of 'em on Amazon. Reinventing the wheel is not only time-consuming, it's also error-prone. There's no excuse for not taking advantage of things that other people have figured out and published.
• Mathematical Structures for Computer Science, by Judith L. Gersting. This is another textbook, and again, it's not important that you read this particular book, but if you're going to be developing software you need some of this background. Computer science majors are routinely exposed to this stuff, but those who come into the field in other ways aren't. If you don't know what a finite-state machine is, for example, you really do need to explore the field a bit.
• The Mythical Man-Month, by Frederick P. Brooks, Jr.: This is a classic in the field, now nearly 25 years old. Nevertheless, many of the software management principles learned on IBM mainframes still apply directly to writing PC software. The book's title essay makes the point that adding more developers usually makes a late project even later. In my experience, this is true, and most managers still don't understand how it works. If you need to manage software developers at all, you need to read this book.
• Peopleware, by Tom DeMarco and Timothy Lister. Another classic, now expanded with 8 more chapters (be sure to get the 2nd edition). This is a book for managers of software developers. So much of this seems obvious these days: good working conditions, letting teams gel, the effect of quality on productivity...so how come so many managers still seem oblivious? A good first book if you are new to managing developers and want to know how to do a better job.
• The Pragmatic Programmer, by Andrew Hunt and David Thomas: A short book of advice subtitled "From Journeyman to Master". It's aimed more at individual consultants than big organizations, and language-independent (though biased towards command-line tools; you'll have to read the book to see the argument they make). Some of the advice is obvious, but there is enough of it that you'll find a few worthwhile gems. Also notable is that their bibliography includes a list of web sites, and the back of the book has a tear-out card with their checklists and tips for handy reference. There's also some good stuff at their web site. 
• Rapid Development, by Steve McConnell: A giant book of rapid development that makes the point that there are many tradeoffs between risk and schedule. With case studies and hard data McConnell evaluates a variety of tools and techniques for rapid development and lays out clearly what you can expect from each. If you ever have a project to deliver under schedule pressure you must read this one. A new classic.
• Refactoring: Improving the Design of Existing Code, by Martin Fowler (with contributions by Kent Beck, John Brant, William Opdyke, and Don Roberts): I ran across this one just recently. This is the first book I've found to take seriously the notion that sometimes it pays to rewrite code to make it clearer, even if there's no new functionality to be added. Such thoughts are enough to bring joy to a maintenance programmer's heart. Fowler discusses why one would want to refactor, how to test refactored code (his thoughts on classes containing their own tests are also very worth reading), and gives a catalog of refactorings that he has used over the years. All of the examples are in Java, but the book is worth reading even if you're working in another language so long as you understand the terminology of object-oriented programming.
• Software Project Survival Guide, by Steve McConnell: A high-level overview of the development process targeted at managers and technical leads. McConnell comes out of Microsoft and you can sometimes see the influence of their way of doing things on his thinking. A good overview, but I think not as good as his other two books. But if your organization has no discipline in developing software and you need a place to start, this is very accessible.
• Writing Solid Code, by Steve Maguire: Despite a subtitle that many people find very funny ("Microsoft's Techniques for Developing Bug-Free C Programs"), this is a very solid work concentrated on the mechanics of improving your actual coding skills. About half of it is specific to programmers working with the C language, but the other half applies to any language. Even if all you learn from this is how and when to write assertions, and the importance of single-stepping through every single line of code you write, buying this book will be money well spent.

For more short book reviews in this same general arena, hop over to Joel's site. Also worth reading is the discussion thread in his forum on the topic.

Web Sites

• Agile Modeling Home Page : An extensive collection of material related to lightweight software modeling techniques. Lots of good stuff here. 
• Alistair Cockburn, Humans and Technology: Sprawling website full of papers on object-oriented design and cooperative programming. Check out his papers on pair programming, methodology, people as active devices and more. Lots here to provoke thought.
• Big Ball of Mud: An essay that takes a look at some real-world software patterns and considers why so many projects end up taking the messy form of a big ball of mud, despite our best efforts to learn and apply solid software engineering principles. Includes some advice about salvaging projects that have reached this state. 
• Build Better Software: Good set of basic notes (presented as a slideshow in Flash) on basic build and release management.  (New 1/9/03)
• The Case Against Extreme Programming: A thoroughgoing attack (or perhaps hatchet job) on XP, together with rebuttal and discussion.
• Catalog of Non-Software Examples of Design Patterns: If you have trouble understanding the abstract language sometimes used in describing software design patterns, take a look at this site. It provides a set of real-world analogs to patterns, ranging from customer service to building a Happy Meal, that may help set some of the concepts in your mind.
• Center for Software Engineering: This outfit is located at USC, and as you might expect it takes an academic approach to the field. They tend to study large projects, formal models, and the analysis of requirements for complex systems. Of note are the COCOMO-based tools for estimating the time and effort that large projects will require.
• Construx Software: A commercial outfit founded by author Steve McConnell. They offer professional consulting, seminars, checklists and estimation software. They're more focused on macro-scale management than the micro-scale optimization of code.
• CostWright is a software project estimation tool from Premia, makers of the CodeWright editor (which I heartily recommend). I haven't tried CostWright, but it looks interesting.
• Ed Yourdon's Home Page: Ed made rather a splash as one of the people publicly worried about the Y2K bug, but before and after that he's been a steady practitioner of good software engineering. There are lots of good pointers to other sources here, and a great sense of humor. The only software engineering web site I read for fun.
• Extreme Programming: This is a wiki site, which means that anyone (I think) can contribute to the discussion. In this case many of the proponents of XP are contributing, which is resulting in a sometimes-fascinating, sometimes-preachy bull session on the merits of XP practices.
• Extreme Programming: A Gentle Introduction covers the basics of XP, just in case you're too busy to read the book.   
• Extreme Programming FAQ - The basics about XP, with pointers to further information. Good comparisons with other programming practices.
• Guide to the SWEBOK - That stands for Software Engineering Body of Knowledge. This is an IEEE project to codify what software professionals ought to know. Plenty to chew on here if you're looking for ways in which to increase your professional knowledge.
• High-Level Best Practices in Software Configuration Management: One of a number of white papers on the Perforce web site. Perforce are also the makers of an excellent source code management system.
• How to Write Unmaintainable Code: When you think all is lost and no one will ever write good software again, it's time to go browse this site. Tongue in cheek (I hope!) and in the context of java it recommends dozens of worst practices.
• In Defense of Coding Standards is an essay subtitled "How to Create Coding Standards That Work." It argues for a minimalist approach that looks likely to succeed in bringing some form of (often much-needed) standardization to nearly any development organization. 
• An Introduction to Extreme Programming - On the O'Reilly Network, this article introduces the concepts of XP and considers possible synergies between XP and open source. Includes reader comments.
• Joel On Software - Joel Spolsky has done software management at Microsoft and Juno. His essays on the software management process are well worth reading even if you don't agree entirely with his management style (I don't, but I still learn from them). It's a Manila site, so it can be hard to find things, but browsing through the links in his daily entries will bring you to many interesting things.
• Literate Programming -- Propaganda and Tools: "Literate programming is defined as the combination of documentation and source together in a fashion suited for reading by human beings." This is an idea introduced by Donald Knuth over twenty years ago that hasn't caught on widely yet. However, I think it shows promise. This page will point you at basic definitions and tools and give you an entry point to the Literate Programming Web Ring. I'd love to see someone implement these ideas for VB/VBA.
• Little Nybbles of Development Wisdom - Good set of bullet points on building a robust commercial web product from someone who's actually done it. Straightforward language that's likely to teach you a few new things and remind you of a few old things you've forgotten about.
• Manifesto for Agile Software Development: A statement of position from practitioners of a bunch of "lightweight" development practices such as XP and Scrum that picks out the common themes of responding quickly to change and valuing results over process. Perhaps most useful for the "About the Authors" page that gives links to a bunch of other interesting sites.
• Marotz Cost Xpert is another piece of estimating software that supports both COCOMO and Function Points (two popular ways to estimate the size of a software project). I haven't tried it yet, but they have a 45-day evaluation version available.
• Mob Software: The Erotic Life of Code is a wandering essay arguing that we can get more and more interesting code by turning vast mobs of programmers loose on it.
• Open Source Software and Best Practices: A very short essay that you can read in a few minutes. Jeffrey Taylor spent some time thinking about the connections between the anarchy that is open source and the best practices recommended by software researchers. Open source comes off pretty good in his account. This one might get you thinking "out of the box."
• Pairprogramming.com: A site devoted to research about and cheerleading for the practice of pair programming, where two developers work at the same time on the same source code. Several interesting articles here.
• Patterns Home Page: Software patterns have had a big impact on the field of software engineering in recent years. The basic notion is to learn by abstracting the pattern of workable solutions. This page provides links to hundreds of resources, from tutorials to mailing lists to book reviews to online pattern repositories.
• Process Builder: I'm somewhat embarrassed to be recommending a site that's this butt-ugly. Read it for the content. Actually, what you should do is follow the link to subscribe to the Building Processes newsletter. This is a low-volume, friendly thing put out by Alan Corwin. Of all the technical newsletters I get via email, I think this is the one that most consistently includes both articles that make me think and links that enrich my software development efforts. 
• Process Impact is a consulting house focused on "practical software process improvement". Check out the Publication page, which includes a number of excellent articles on software culture, risk management, project management and related topics. 
• Refactoring Home Page: From Martin Fowler, who wrote the book on refactoring. Lots of good stuff here, including extra material and errata for the book, links to other sites and commercial tools, and a news section. Well worth a visit.
• Scrum is another methodology for hyper-rapid software development. Looks similar to Extreme Programming. Reading (and implementing) this sort of thing can be inspirational to organizations that are so mired in process that they can no longer get anything shipped.
• SGI Software Usability II - A leaked 1993 memo from an engineer at Silicon Graphics, analyzing what went wrong with a large release. There are some lessons to be learned here about both writing software and managing the software process.
• Software Engineering: A page of stories from Tom Van Vleck. These range from simple parables to technical journal articles about Multics, with which he was deeply involved. Most are short, and you can read an ponder one while you're waiting for your next compile to complete. 
• Software Engineering Institute: Located at Carnegie-Mellon, this is a federally-funded outfit that does research into serious management of serious software projects. They make many of their research reports available online. Most of it is not very accessible, though, or applicable to the sort of small-scale development most people I know work with. Worth a look if you get deeply interested in this stuff.
• Software Engineering Laboratory: This is a NASA effort that makes available fat PDF publications of what's worked at Goddard Space Flight Center. They're proponents of an extremely structured approach -- perhaps the antithesis of the extreme programming folks. "Manager's Handbook for Software Development" is one of the seminal documents of software engineering management. 
• Software Engineering Professionalism Website: Another effort from Steve McConnell, this one complements his book After the Gold Rush (mentioned earlier on the page). Links to much useful information on licensing, codes of ethics, reading lists, and other professional development topics.
• Software Program Managers Network: This is a group of software manager for the Department of Defense, very concerned with how to keep track of large, out-of-control projects. Even if you're working on a small, relatively in-control project it's worth taking a look at the Risk Radar and Project Control Panel products; both are free software that use Microsoft Office components to help keep track of software projects.
• Stackframe: This weblog is devoted to "Serving up information, opinion and hot-links about hard-core software development". Looks to be shaping up as a good place to keep up on all sorts of development news.
• Strengthening the Case for Pair Programming: A paper reporting on productivity and quality gains from the practice of having pairs of programmers working together. Includes some qualitative data.
• Test-Driven Development in .NET: An excellent introduction to this useful practice. (New 3/7/03)
• Trends in Software Engineering Process Management: The title is a mouthful, and you have to go through the "Subscribe" link to get to the content, but what you'll find is short articles on managing the software process. They're a good size for sharing with your boss (you know, the one with the limited attention span). Lots of links here too, which I need to mine when I get the chance).
• Tutorial: Functional Specification: Though aimed at web designers, this one has some good advice on how to proceed for anyone who suddenly ends up being responsible for a functional spec.
• XProgramming.Com: A new site for devotees of Extreme Programming. I haven't dug into this much yet but it looks useful. There's a book by Kent Beck that I've got on order that appears to be the seminal document on the subject.

Did I miss your favorite software engineering/software management book or web site? Write and tell me about it! I'll be happy to pass on more resources on this page. We all benefit from more awareness of these issues.