Agile or Waterfall?

At school / university, they mostly teach and (we) practise software development by the book. I mean, requirements before design, before architecture, before development, before testing, ... Waterfall. Any project that requires documentation values that phased way of building software.

Management is oriented pretty much by that same principles. Documentation and well-defined process. After all, this same style of management has always served the various engineering disciplines with proved results, so it must be right, right?.

When companies are competing in the enterprise market, they abide by these rules. It shows how reliable and professional they are. There are even the CMMI and other kinds of enterprise certifications, which seem to push the company in the more-process-and-documentation direction, which in turn makes them more enterprise-ish.

But there's the flipside of the coin

In the past few months, I've been reading pretty much everything I could find about agile methodologies. Heck, I've even used agile management practises and an agile development process in the last project (namely, Feature Driven Development, FDD). However, as it was pretty much self-instructed, we always felt like there was something not quite right there (nevertheless, the project was very successful from the client's point of view and development run pretty smoothly).

Looking for defined and more complete agile methodologies, I've been reading about SCRUM lately. I've read soft-evangelism and user experiences, mostly. The various articles, testimonies and other resources on the Web were a very good source of information, of course, but it was too much to process in such a short time. I'll be digesting it for the next months. Maybe I'll have a chance to try it in a class I'm taking this semester, called Agile Software Development Methodologies.

Summarizing, SCRUM advocates short development iterations, nearly constant user and/or customer interaction, daily team synchronization (yes, management can be present), among other practises. It doesn't exclude things like Extreme Programming (XP) or other development practises, but upwards (in the management layers) some action must be taken to accommodate SCRUM.

And that brings me back to the initial subject. Management likes status reports, documentation and organizations like processes. By this perspective, SCRUM probably brings additional risks to a project. And many times, it's the case that a proposition must be made with a closed budget, for which the agile method might not seem so suitable. So, it's not easy to convince management. Some of the articles I've read about SCRUM mention this transition difficulty, and recommend starting by the small projects, not upfront disrupting the existing practises (they wouldn't let you, anyway).

Adopting SCRUM would be a whole other topic, and besides that, I don't think I could provide meaningful knowledge about it, as I've never been there doing that (I actually met someone at BarCamp Portugal who does it since February -- Ricardo Mestre, we didn't talk much but he promised that next year he would make a talk or workshop about that).

Why can't they co-exist?

However passionate or reluctant opinions I read, I was never completely convinced that one approach or the other was completely right or completely wrong. I believe there are cases where Waterfall might be the best option, but I was mostly convinced by Agile approaches.

I was always someone who tries to bring different perspectives together, especially when their representatives are passionate about it, because it means that there must be some value in their ideas. So, today I was thrilled when I read a post by Richard Banks stating that SCRUM and CMMI not only can work together, but yield extraordinary results! The organization in question, Systematic Software Engineering, is actually CMMI level 5 certified! And not only are they using SCRUM but also Lean. (The original report on the subject is from Jeff Sutherland, co-creator of SCRUM.)

I believe this is not a coincidence or simply bright people being in the right place and time. CMMI 5 indicates an organization with continuous process improvement and high process maturity. Perhaps only a very mature organization is able to change itself into adopting agile practises, at such a large scale. But wouldn't it be better if your organization changed before it reached such a large scale?

It's hard to list in a logical way all the stuff I've been reading on this matter. If you're interested, just explore my del.icio.us bookmarks regarding the agile and scrum categories.

Buzzwords / Acronyms:

Update: Added the name of the person I met at BarCamp that uses SCRUM.


Mickaël said...

I share yout opinion even if I think waterfall manegement could in some rare cases work well with agile methology. It think they can't coexist, but when they coexist it's because management did not understood agility, hence makes it failing.

See this post.

Mickaël said...

oups sorry, I wanted to say "I think they can coexist, but when they are not coexisting well it's because..."

I let you fix my previous comment.

Joaquim Rendeiro said...

Hi Mickaël, thanks for commenting on this subject and sharing your views. It was interesting to read your post, because of the management perspective. I'm gonna check your blog's archive!

Btw, please correct my name in your post, it's Joaquim (vs. "Joachim"). It's very common to see my name written like that, but since it's going to be archived for posterity... I'd prefer the correct form, please :)

MytyMyky said...

I've been tring this stuff out within possibilities. The last project I worked on (a small interview managemet software) worked pretty well on a "discover the new features as we go" concept. Current project is a lot more complex, but the methology works pretty well.

Just need to be careful to not lose the big picture, and good to maintain a task list of the features that needs to be implemented. DSL and code generation helps alot on reflecting added properties and what not.

Also got to start to use unit testing and the likes. :D

Joaquim Rendeiro said...

Yeah unit tests are such a nice cushion! Once you start doing TDD regularly you'll wonder how could you develop in any other way before :)