OT 2004: The Dangers of Extremism
April 3, 2004
I gave a talk at OT2004, as part of Andy Carmichael's Famous For 15 Minutes session. The talk was on "The Dangers of Extremism", in which (in the brief 15 minutes that I had available) I attempted to summarize the points made in Extreme Programming Refactored, and describe some of the motivations behind writing the book.
Here's the complete transcript from the session:
This talk is about the dangers of extremism in extreme programming, and what to do about it. The short answer is not to be quite so extreme. But to adequately describe how to do that - and why - is more than can realistically fit into a 15-minute talk. So for this talk I'll summarize the problems and some of our proposed solutions. If you want the details, do check out my book, "Extreme Programming Refactored: The Case Against XP," co-authored with Doug Rosenberg.
The book has caused a polarizing reaction - people either love it or hate it. I find this odd, because we're not saying XP is blatantly wrong; we're saying XP has problems. We're not saying "kill XP", we're saying "here's what's wrong with XP, let's fix it." Some people point to the sarcastic tone of the book; but the book's tone is essentially anti-hype, a reaction to the sheer amount of hype surrounding XP.
Although some people in this room have almost certainly worked on an XP project, others might be wondering what XP is - what it represents. So I'll start with a very brief overview. After that I'll describe my analysis of the XP practices - in other words, what I feel is wrong with XP - and finally, I'll suggest some ways of making XP more suited to more software projects.
What is XP?
XP is like the Atkins Diet. Like the Atkins Diet, it turns common wisdom on its head; it's trendy; controversial; it works for some people but there are also horror stories of how it can go wrong.
For many people, XP is a breath of fresh air, because it represents a solution to some age-old problems with more traditional development processes. The industry is littered with failed projects: projects that failed because the team spent months gathering requirements, months doing the design, but then got it wrong anyway, and gave the customer a system that was precisely what he didn't want: or just as bad, a system that kept crashing.
So if traditional processes are broken, what's the fix?
XP's answer is profound in its simplicity. Teams are spending months gathering requirements but still get them wrong? So don't spend months gathering requirements! Instead, do some brief requirements exploration, then get on with the programming, exploring the requirements in further detail as you go along.
Teams are spending months doing design work, but still the design changes straightaway? So don't spend months doing up-front design! Instead, do some brief design work, then get on with the programming, using unit tests, refactoring and UML diagrams to evolve the design as you go along.
XP is about being incremental, inching forward in short baby steps making small adjustments to your direction as you go.
The result, in theory, is that we can give value to the customer by delivering working software very early, and then adding to it using frequent short releases.
It sounds strangely seductive: and that's because it is. If XP gave us this for free, we'd all be doing it.
XP is classed as a "lightweight" methodology because it only consists of 12 practices. These include pair programming; collective ownership; whole team (which used to be known as "on-site customer") - it basically means that the whole team must fit into one room; sustainable pace (which means don't overdo it; clock off at 5PM if you can); test-driven development (via unit tests and customer acceptance tests); simple design; and so on.
There's great synergy between these practices. For example, pair programmers - where two programmers sit at the same PC and work on the same program - could become stale, especially if two junior programmers are paired up, or if your partner has bad breath. So that's countered with pair rotation, where people regularly swap partners.
Pair rotation is problematic because it means you're constantly switching to different tasks. So that's balanced out with collective ownership - where no-one is singularly responsible for any code. Nobody "owns" the code - any pair can dive into any part of the system and refactor it.
Requirements are written as one-line "user stories" - brief, free-form statements of what the system needs to do. The detail is filled in later - and that makes XP agile, because there's a good chance the requirements will change before you actually need the details. This approach could cause delays if you get to the stage where you need the details and no-one's around to tell you. So that's balanced by having a customer representative in the room at all times.
Any process needs to be tailored to suit the local conditions. XP is no exception - in fact, its creators actively encourage you to tailor XP to suit your project. Of course, if the project fails then you can't claim that you were doing XP…
XP is suited to certain types of project, and with a dedicated group of individuals it can work. But there are easier ways of achieving the same set of agile goals.
Teams are using XP successfully. There are success stories, but there are also reports of failed XP projects, and programmers who just don't get on with the XP way. And this can't all be blamed on people not doing XP properly.
What's wrong with XP?
So, what's really wrong with XP? To answer that, let's revisit some of the touted benefits.
First - I mentioned that XP is considered to be a lightweight methodology. Unfortunately, the practices themselves are very high-discipline and require a lot of ongoing attention from the whole team.
XP is synergistic because the practices all support each other. But the reverse is also true: if one practice fails, the other ones are quick to follow. I've seen it happen, particularly with test-driven development. If someone stops writing the tests before the code, it becomes very difficult to refactor the code. So code ends up being hacked. Then it becomes harder to write tests or to extend the system with clean code. You quickly end up with emergent entropy instead of emergent design.
The same problem - that XP's practices are too tightly coupled and difficult to maintain - means that XP is also a difficult process to tailor. You have to have a good understanding of the dynamics, the risks, before you tailor XP. And yet most teams who adopt XP start by only doing some of the practices. (If they're lucky they'll start with unit testing; if they're unlucky they'll start by not designing in detail before coding).
XP is difficult because it requires unfailing diligence and buy-in from everyone on board. As soon as one of the plates stops spinning, the risks escalate. This is true of any development process… but it's particularly true of XP (as we explore in XP Refactored).
In particular, XP places a huge responsibility on two of the team members: the coach, and the on-site customer.
The coach is the XP expert who must keep all the plates spinning at all times. It's a rare breed of person, but if you find the genuine article then your XP project might just stand a chance.
The on-site customer role can work well if you get the right customer. But in many ways it's XP's Achilles heel. If you can get an on-site customer for your project (whether or not you're using XP), then it's definitely worth doing, especially if they're prepared to work side-by-side with your programmers for a large part of the project. The early feedback can be hugely beneficial, and some customers love that level of involvement.
The problem with XP is that, executed to the letter, it relies on there always being an on-site customer representative in the room.
The customer role is also a difficult one. The XP customer has a list of responsibilities as long as your arm. It's a huge job for one person.
e.g. Understanding customer wishes, Participating in the planning of iterations and releases, maintaining regular contact with end-users, talking to developers, clarifying feature requests, understanding some of the developer's technical concerns, Specifying functional tests for user stories, and verifying that these tests run correctly, Participating in the planning of iterations and releases, maintaining good contact with management, explaining progress, and justifying the time spent with the development team.
So it's no surprise really that the on-site customer is now expected to be a team equal to or larger than the team of programmers, and on all of them speaking with one voice. That's the trade-off for not having to spec things out in-depth. It's high risk. If the customers can't be there, or they forget to speak with a single voice, ambiguities and assumptions soon put in an appearance.
XP has a problem which faces the agile movement in general: that with the rush to embrace change, we actively encourage change, and the deadline keeps getting pushed out to keep in line with the changing scope. Except we don't know that, because the scope was never fixed.
How can we fix XP?
So how can XP be fixed? How can it be strengthened? In my book "Extreme Programming Refactored", we discuss this issue in detail, and suggest some alterations to the XP practices to make them less extreme. What you end up with could be called "moderate programming", though that doesn't sound very exciting.
It's important to base your design on a solid foundation. If you're lucky, you'll work on a project where everything goes swimmingly from the beginning. The requirements will be bang-on correct; the architecture will be perfect; the detailed design will be right first time. Okay, you'd have to be astoundingly lucky; or a fortune-teller.
So if we can't guarantee correctness for both requirements and the initial design, we instead follow the 80/20 rule. Get whatever details you can then move on. Maybe a month's requirements work, a month's design, all the while prototyping and exploring the user's goals.
If the requirements are basically right - at a fundamental level - then we can design in enough detail, at an early stage, to begin developing at a rapid pace.
There's nothing to say that we can't do this on an XP project - but deep-down it just isn't in the spirit of XP. After a brief period of exploration, XP is fundamentally about filling in the details as we go along.
What we end up with is a process that is not entirely unlike XP itself: just gently tweaked here and there to make it more robust.
Agile Development with the ICONIX Process
In my next book, "Agile Development with the ICONIX Process" (which I'm co-authoring with Doug Rosenberg and Mark Collins-Cope), we're describing this process in a lot more detail. The book is mainly about the ICONIX Process - a lightweight methodology for object-oriented analysis and design, which is surprisingly well suited to agile projects.
In the new book, we're also expanding on the "Refactoring XP" part of "XP Refactored", and describing how to use the ICONIX Process on an agile project with shifting requirements, short fixed iterations and so forth. It's due out in September.
Do keep in mind that methodologies aren't everything. If you have an experienced team of capable developers then you're 90% there already. Processes exist simply as guidance, and we expect to tailor our process to suit each project. So above all, it's important not to treat XP, or any other methodology, as a religion. Sacred cows are there to be slaughtered, after all…
The Q & A Session
The Q&A session following the above talk was kind of fun. Some interesting questions were asked. One of the questions, "How many XP projects have you actually worked on?" I didn't answer particularly well. There just isn't a simple numeric answer, but the answer I gave (a kind of stumbling "well it depends") I put down to nerves on the day. However, the answer that I should have given - which occurred to me a few minutes after the session had wrapped up - was the following:
I've worked on several agile projects. I'm currently leading a project which has been delivering working versions successfully for about 18 months. At various stages of this project, as a team we've tried out the XP practices (with at most about three quarters of the practices being tried out at one time - although test-driven development, fixed one-week iterations and short releases have been in place throughout). I've also concentrated on tailoring the XP practices to strengthen them in the areas where we found them to be lacking. This project is also used as a case study and the basis for the "refactored XP" process described in the book.
Reality XP Forum
Back to XP Central
to the Front Page