Software Reality
Programming with
a dose of satire.

Site Map Search


Agile Development
 
Extreme Programming
 
Code Generation


Articles
Lifecycle
Design
Programming
Soapbox
Reviews
Cthulhu

Check out our ageing Reviews Section


Use Case Driven
Use Case Driven Object Modeling with UML: Theory and Practice
Get from use cases to working, maintainable source code. Examples use Spring Framework, JUnit and Enterprise Architect

Agile UML
Agile Development with ICONIX Process
A practical subset of agile development techniques, illustrated by example

Get Controversial!
Extreme Programming Refactored
Extreme Programming with a dose of satire
Available now:



ICONIX/Sparx Public Classes to Come to London

ICONIX is planning a series of open- enrollment public classes entitled Hands-On Enterprise Architect for Power Users in collaboration with Sparx Systems.




Design

Emergent Design vs. Early Prototyping

By Matt Stephens
May 26, 2003

Travel straight from A to BThis article compares two approaches to architecting and designing software: emergent design (a cornerstone of Extreme Programming) and early prototyping.

Emergent design (sometimes referred to as emergent architecture) is an evolutionary prototyping technique used to “evolve” a design, through refactoring, from code—with little or no significant up-front design.

"Early prototyping can provide many of the benefits of an agile approach"

Conversely, early prototyping is a technique that traditionally fits into less “agile” methodologies which take a design-up-front approach. However, early prototyping (used correctly) provides many of the benefits of an agile approach (for example early feedback), whilst retaining the more rigorous benefits gained from designing a system before coding it up. (The trade-off, especially when combined with detailed up-front design, is that the first production version of the product might not be ready as soon as with XP - say for another month or two. This may or may not be a problem for your customer...)

Before we get started, it is worth pointing out that I am starting out in favour of early prototyping. These articles (and this upcoming book) might help to explain why. In fact, because this article really argues in favour of early prototyping, the answer to "which is better" may be something of a foregone conclusion.

So, with the disclosure out of the way, let's get on with it!

Actually (sorry!), one other thing before we get started. Many people have wildly varying interpretations on the meaning of “architecture” when it comes to software. For the purposes of this discussion, I'm defining architecture as a high-level design view of an entire system under development. The architecture can be broken down into more detailed designs for individual subsystems or components. Therefore, “zooming in” on an area of the architecture shows us a design from which the system can be programmed.

Non-functional requirements (such as scaling or availability) tend to affect a project at the architectural level, meaning that a large proportion of the project would be impacted by a change in these requirements.

"Early prototyping is an effective means of thinking an architecture through"

Early prototyping is an effective means of thinking an architecture and design through. It can also be used to determine whether a chosen architecture is going to be appropriate. Prototypes can also help with requirements, because they provide early feedback to the customer to help them decide whether their original requirements are correct. The prototype can also help to elicit yet more requirements from the customer, because when they see the product working, this will get them thinking about what else they want. There’s no better visualization than the real thing.

This is one of the reasons why XP takes an evolutionary prototyping approach. If prototyping is so good, well then let’s do it all the time! Effectively, the prototype becomes the real product.

This article examines whether XP’s approach is a realistic extension to the early prototyping technique, or whether it is perhaps taking a good thing too far (again), stretching the technique to breaking point.

If you're unfamiliar with emergent design in XP, a quick overview can be found here (xprogramming.com).

 

Getting the Architecture Right

XPers like to use an over-arching metaphor to describe their architecture. Can the metaphor be evolved in the same way that the design can? The metaphor drives the architecture; therefore if the metaphor evolves, so too does the architecture.

"Changing the architecture is very different to changing the design, because the architecture affects everything"

Changing the architecture is very different to changing an aspect of the design, because the architecture affects everything. In a way, it is everything (at least to do with the project!) It’s all-encompassing; it documents both functional and non-functional design decisions. It includes the choice of operating system, programming language, application platform, database and so on. At the implementation level, the architecture includes the overriding “vision” of the way in which all the pieces are going to fit together.

It is vitally important to get this part of the puzzle right. This is why early prototyping is a vital part of all but the most trivial of projects. Early prototyping highlights many of the technical issues, and helps to establish what is reasonably possible within the bounds of the chosen technology. Early prototyping helps to shape the ideal architecture using “quick and dirty” throwaway code. The production code can then be written with more attention to production details (such as error handling, usability and so on).

Contrast this with the XP approach of emergent design. The theory here is that it should be possible to launch straight into the project without spending time up-front (before production coding begins) thinking the architecture through. Done the XP way, the architecture itself would (in theory, at least) arise spontaneously as the code base evolves.

In a discussion on Usenet regarding emergent architecture, XP author Robert C. Martin said:

“The fact that the architecture arises spontaneously does not mean that the architecture doesn't take work to derive. It does! Lots of work. It’s just that the developers have more and more information with each new iteration. Each new batch of information solves more of the architecture puzzle. I agree, architecture requires up-front planning. However, I don’t define “up-front” as meaning “pre-code”. I think you have to write some code up front to find the real architecture.”

I almost choked on my cup of tea when I read “I don’t define up-front as meaning pre-code”. However, it then occurred to me that this could be about right if it is referring to early prototyping (which of course Martin isn’t—he’s referring to production code).

This reply to Martin’s message was posted by David Van Camp:

“Sorry, I simply don’t buy it. I’ve been though this ‘evolution’ too many times. I call it ‘redesign’. It is painful and labor intensive. To call it ‘refactoring’ is not to give it justice.”

These two messages highlight the differences between XP’s approach, and a less eXtreme approach that I (and many others) have had good success with—early prototyping and design.

XP's emergent design is really an "evolutionary prototyping" approach. With XP's approach, the prototype itself becomes the production code (although to be fair, in addition to "prototyping", the XP approach also promotes code quality through such techniques as pair programming, collective ownership and extensive unit testing).

Conversely, with “traditional” early prototyping, the prototype is largely abandoned early on in the project. It’s kept for reference, but the production version is begun afresh. No time is wasted trying to refactor the prototype code into something more acceptable: its place is simply to explore, to find out what’s possible, and to get some idea of what’s going to be involved in the real thing. That is not to say that some code won’t be cut & pasted from the prototype. However, the main thing that is kept is the knowledge and insight into the solution that the prototype gives us.

Combined with up-front design, my feeling is that this is a much more realistic approach to getting the architecture right than its evolutionary cousin.

Does that mean, then, that we can get away with not making any changes to each piece of production code once it’s written? Hardly. The design will still change a bit—this is inevitable. However, this is very different to the overall architecture changing—because the architecture affects everything.

"XP techniques such as unit testing and refactoring can still be useful"

Because the detailed (low level) design is still likely to change as our understanding of the solution improves, XP techniques such as unit testing and refactoring can still come in useful. However, our level of reliance on them is much lower—hence less time needs to be spent doing them (I still advocate writing lots of unit tests though—and by all means write the tests before the code…)

By taking the time to prototype the proposed solution early, and to spend time designing before you begin production coding, you should find that later on there will be much less need to redesign the existing code base.

Early prototyping and up-front design should take place in parallel. You might do a bit of architecture and design before you begin prototyping, so that what you’re coding up is at least on the right planet. Your understanding of the required design should improve as the prototype progresses. Feed this back into the design straightaway (whether you’re using UML models, free-form lines and boxes, words and sentences, or whatever, to document your design).

Above all, think ahead to what you are likely to be creating as part of this project (that means looking ahead further than the current two-week iteration). Can the overall architecture handle what you need to do in a month’s time? Discover this now, and save yourself the pain later. Note, this is very different to coding for that future requirement before you need it. At this stage, you are simply making sure that the architecture will not need a complete overhaul in a month’s time, simply to accommodate something that no one had spent time thinking about because it happened to be in next month’s release.

How extensive should an early prototype be? Generally, not very. The value is in learning whatever you need to as quickly as possible, and then moving forward with the real thing. Typically, the prototype consists of about 10%-20% of the overall project functionality—just enough to get a feel for the required architecture, and to make whatever basic design mistakes that we all make early in the project.

"This is the difference between a prototype and a production system"

If the prototype’s functionality has been derived from use cases, it will consist mainly of the basic flows (the “sunny day” scenarios), and very few of the alternative flows. The alternative flows are the ways in which you expect things to go wrong, or the things that you expect the user to do differently (“What if the user clicks cancel?” “What if the user submits the form without having entered all the fields?”). When use cases are written properly, there are many, many more alternative flows than basic flows. In some projects, the alternative flows will be about 80% of the overall specification.

This is the difference between a prototype system and a production system. And that’s why, in projects where the alternative flows are important (i.e. all projects), XP’s evolutionary prototyping approach just doesn’t cut it.

 

Is the Prototype the Production System’s Ugly Sister?

The prototype does tend to be pretty ugly, and deliberately so. Absolutely zero time should be spent trying to make it look appealing. There are, however, a few notable exceptions to this:

The prototype’s goal is to prove that a particular technology can be made to look good (e.g. writing a GUI client with Java Swing). In this case, the prototype would be more of a proof-of-concept, and would consist primarily of GUI tricks and best practices to get the best out of the platform and widget toolkit being used. In fact, you would spend less time concentrating on actual functionality, because that’s not the point of the prototype!
The customer has difficulty distinguishing between form and function. If this is the case, it sometimes pays dividends to spend a little extra time making the prototype look good. Don’t go overboard though—remember what you’re creating is going to be abandoned pretty soon, in favor of a brand spanking new, shiny version with all mistakes learnt (at least that’s the plan!)
The prototype will also be used by Sales & Marketing, as an early sales pitch for your upcoming new product. In this case, looks and shininess (the “ooh” factor) probably take priority over real demonstrable functionality. Showing the customer your impressive 20-megabyte auto-generated comma-delimited invoice file might bore them faster than you think.

These are the exceptions, though. Most of the time, the prototype is intended primarily as a means of gaining insight into the ideal architecture, and is written in tandem with the design—before any production code gets written.

 

Feeding the Prototype Into the Design

This aspect of early prototyping is worth emphasizing: if you write the prototype at the same time as you are modeling the design, you can “refactor” the design without having to change any code.

Don’t bother changing the prototype itself—just feed the new insight into the design. Therefore (and this can be a big time-saver) you don’t need to spend any time writing unit tests for the prototype.

The “prototype” itself should be many small, disjointed programs—lots of little prototypes, each one intended to explore a different aspect of the design (similar to XP’s “spike”, but less ad-hoc). Once the purpose of each mini-prototype is finished, you’re ready to abandon it and move on. Of course, keep the code to refer back to (and maybe copy & paste some code snippets into the real version); but don’t lose sight of the original goal of the prototype, which is simply to get you to the next stage of the project as quickly as possible.

 

Using the Prototype as a Requirements Elicitation Tool

"Sketch out the user goals before even the prototype is written"

Another aspect of the prototype is that it can be used as a means of eliciting further (and more realistic) requirements from the customer. As with XP, when the customer actually sees working software doing the things they originally asked for, one of two responses is likely:

“Yes! That’s pretty much what I wanted. But seeing that has made me think of some other things we’re going to need…”
“Yes! Well, kind of. Well actually, not really. Seeing this now, I realize that I really wanted something completely different…”

In both cases, it’s useful to elicit these responses from the customer as early as possible, so that any changes or additions to the requirements can be worked back into the project plan, and so that the architecture can be updated before too much production code has been written.

This type of feedback is one of the ideological foundations of XP. However, our approach is geared around getting effective feedback from the customer and users even earlier—before the production code has been written. In fact, it is advisable to take this a step further, and sketch out the user goals before even the prototype is written. The goals can be revisited once the customer and end-users have provided feedback on the prototype.

There are some dangers with this approach (which affect both XP and early prototyping as a customer requirement tool):

Scope creep. This danger is inherent in any project that has close involvement with the customer. Of course, you want the customer to be closely involved, to give feedback early on what’s being produced. The problem is that the customer might start to get carried away, and to ask for things that gradually have less and less to do with the original objective of the project.

This is more of a danger with XP, because the “prototyping” stage goes on for the entire duration of the project. Handling this problem effectively really needs a skilled analyst who is able to get at the problems that are driving each feature request, and in turn get at the root causes behind each problem.

Any new requirements discovered through prototyping (evolutionary or early) might be based more on technical issues than on business needs. If the customer is closer to the solution as it’s being produced, then he or she is further away from the problem. They might, as a result, base their new requirements on what they think is possible, or on the perceived solution, rather than on the problems that their business really needs to have solved. (“I hear SOAP solves lots of problems, let’s use that!”)

The resultant product will no doubt still work, and the customer will probably be very happy because they’re getting exactly what they asked for, but it won’t be as useful to them as if they had based their requirements purely on their non-technical business needs. This is an insidious problem—what the customer doesn’t know about, they won’t miss. However, it just means that later, the real business problem will be discovered (e.g. “our accounts department is just too inefficient working this way, SOAP or no SOAP”). The result will be yet another iteration to sort out the problem.

In the XP world, this is fine: everyone’s happily embracing change. However, the problem could have been solved the first time around, if it had been uncovered. Always remember, the requirements are based on the problem, not the solution!

 

Keeping it in Perspective

Prototyping is not a 100% cure-all, of course. There are times when a prototype is not worth bothering with. There are also dangers (see this article for some examples).

You may have 6 months before you start the project in earnest and may choose to spend 3 months on design, 3 months on prototyping. Or you might decide you'd rather spend all the time designing and making sure, by talking to everyone involved, that you are on the right track.

If you spend 3 months on a prototype and find it is hated and you've got completely the wrong idea, then you've learnt a valuable experience but at quite a cost. You might have done better putting more rigour into your design. This is why (as discussed earlier in this article) the "prototype" shouldn't be treated as a single monolithic prototype: it should be lots of smaller programs and disposable "program fragments", transitory artifacts produced as an integral part of the design process.

Thus, prototyping as discussed in this article is not the same concept as producing a big proof-of-concept (which can also be useful under certain circumstances, but can also turn into a big waste of money if it turned out to be the wrong design).

 

Conclusion

As I mentioned at the beginning of the article, I'm already in favour of early prototyping as a valuable method that contributes to getting the architecture right early. XP's emergent design approach can help to evolve the design over the course of an ongoing project; but it is an expensive approach because of the extra overhead involved in adhering to the XP practices throughout the project.

A mixture of the two approaches may actually be the best solution. Use early prototyping (plus specific, repeatable design techniques) to help get the architecture right early; and, if necessary, continue to evolve the design at a low level (code level) as the project progresses. Use unit tests and refactoring (plus whichever other XP techniques, in moderation, will fit your organization) to help keep the design clean. And break the project into smaller iterations (e.g. a month each) to avoid the problems normally associated with waterfall projects.

This "less eXtreme" approach to agile development is described further in my upcoming book (co-authored with Doug Rosenberg): Extreme Programming Refactored: The Case Against XP [amazon.com amazon.co.uk]

 

Related Articles:

Development by Demo - how prototyping can go wrong.

Interface Design - one example (out of many) of how to get the technical design right...

The Inmates Are Running the Asylum - ...and how to get the interaction design right (fundamentally important to software design).

 

Elsewhere on the Web:

Essential XP: Emergent Design

<< Back to Design

<< Back to Software Reality

All trademarks and copyrights on this page are owned by their respective owners.
Stories and articles are owned by the original author.
All the rest Copyright 1998 onwards Matt Stephens. ALL RIGHTS RESERVED.