Agile, Post-Agile, Scrum, XP, Heavyweight, Waterfall, SDLC, Oh My!

All of these (and many more) are software development methodologies with supporters and detractors alike. The definitions of what makes up each one varies as much as the people who profess to use them; sometimes the labels themselves are hardly indicative of how they really work.

I won't even attempt to define every variation of these. One common dividing point is often how they iterate; a single iteration from specification to completion or multiple iterations (i.e. breaking down a project into smaller steps). Generally people talk about Waterfall (and its other names like SDLC, Heavyweight) for the former and Agile (with its many variations) for the latter; however this is a fairly limited divisor.

In the beginning most people practiced Waterfalls as it seemed to have the most promise for orderly development and seemed more engineering like. When you built a house or bridge you generally followed a step by step process and thus it seemed a good idea to do the same for the "young science" of creating software. With enough experience people began to notice that this didn't really work well since unlike physical structures, software is infinitely malleable and far less constrained. Software is bits, which can be changed any time, and given a big enough environment to run in, can be made to do anything someone might want. There are of course limitations on speed, storage, memory and i/o but even these can be modified. So naturally people wanted to take advantage of these flexibilities. Often the features decided on early in the process had to change since the marketplace was changing rapidly, new technologies became available, hardware improved, and people's expectations grew. Unlike bridge building, where principles of dealing with gravity were well know for hundreds of years, and advanced in materials took decades, software moved much faster than the process allowed for. The process began to look mighty incompatible with what people wanted.

I covered the famous book in a post The Mythical Man-Month - 32 Years Old And Still Relevant so I won't go deep into this again. Suffice to say it made people think that building software might be quite different from building bridges, and somehow we need to rethink how it's done.

When I started coding in 1981 at a defense contractor, I didn't even know what a methodology was, the word itself wasn't used. I do remember we had some informal process that was certainly a waterfall. My first project certainly was, a six month project to build a code formatter (for Jovial J73). It seemed odd to me that I spent months in meetings to determine the requirements, wrote the code (in Fortran no less!), did a formal delivery, and then watched absolutely no one ever use it. Apparently the need that existed initially vaporized during the development and I never heard it. My mind filled with various swear words...

In starting the team that was to develop Trapeze (covered in these two posts) we decided to work a lot looser. First we spent enough time to build out the basic ideas (since no one had ever built a spreadsheet program without fixed rows and columns), decide on a division of labor (I did the UI, Ken did the engine, and Bob did the functions, plus some other pieces) with a reasonable interface between the main areas. Then we started coding rather independently (but within our agreed boundaries) until we began to connect up. At one point I realized my UI wasn't good enough, and rebuilt all of it from scratch, but do to the good interfaces I was able to maintain the old and new UI so that the other two could continue. At regular intervals we would sit down and discuss where we were and what still needed to be done. The code always ran so progress was always visible. Eventually we shipped it in January of 1987.

What's odd about this "agile" process was we just did it, without really thinking about it as a process. We were working in C, no unit tests, no code repository other than a hard drive, primitive 512K Macs and one Lisa (Mac XL), not even a QA person until right before the end (who doubled as the tech support guy). No project manager or anything managerial (I was nominally the leader, since it was original idea, but we were equal). It worked well for us as a process, and generally continued this way for the other projects as well (Persuasion 1.0, and Deltagraph for 5 years). We did 10 major releases of all of these, generally in a 6-8 month cycle. No internet, no email, eventually we did have a project manager at Deltapoint but the process was still more or less the same.

Yet we didn't call it anything; it was just the way we worked.

Today it clearly has many of the same roots as Agile; emphasis on code instead of documentation, just enough requirements to get started, then discover them during iterations (our cycles were not that well defined though); constant feedback from the "customer" as the app was always available (even if it meant Fedex!).  I remember distinctly our releases for Deltagraph always started with no more than 2 pages of feature requests; on that 6-8 months of work was built.

I never heard the terms "unit test", "integration test", "methodology" and don't remember "waterfall" until about 12 years ago. Agile was unknown to me until this century sometime, yet on reading about it, everything seemed obvious and familiar.

Sadly none of the places I've worked this century have been agile, most were waterfall bound either explicitly (one place in a phone conference a VP bragged to the customer how we used the great SDLC process and my eyes rolled back into their sockets several times) or implicitly. Oddly enough most of the places I worked also had great difficult with completing projects, had much customer dissatisfaction, and generally wasted many $ in worthless projects. Yet it's so hard to explain how mythical their process is and get people to even consider something else.

I think back to those 10 years where we in ignorance made something work so well without even knowing that any formal methodologies existed. It makes it hard to see people struggle to develop software today and know it doesn't have to be that hard.

Of course I am not blind to the fact that we were a small team with real dedication, supportive customers (for the most part) and minimal management, which may have added greatly to the success. Although the applications were extremely complex and large, the technologies used were fairly few, so we could focus on coding instead of learning (unlike building a modern J2EE app for example) but I still think there were benefits in using a very lightweight process.

So what does that make me, Pre-Agile?