Divide and Conquer - Why IT Administrators Have Become All-Powerful Demi-Gods
By Robin Sharp
November 30, 2003
|How many administrators does it take to tip a donkey?
In the beginning it was simple: there were analysts and developers. PC developers doubled up as administrators, and analysts doubled up as managers.
Developers did various types of administration: system, network, database and runtime administration. Administration was done in a timely manner, sensitive to the needs of the project.
Somehow over the past 10 years, a cancer has been growing inside large organisations - called role fragmentation. As the roles have fragmented, an entire non-industry has bloomed. Network administration, database administration, security administration, component administration, deployment administration, tool administration. The list just goes on and on.
This fragmentation has had some seriously negative consequences for the industry. In particular:
Slower development times
Increased communication overhead
Slower rates of change
De-skilling of the workforce
Extra manpower needed
Why has the administration been allowed to bloom unnoticed in the software industry when it is having all these negative effects?
While the negative cost of administration has hit development times, the positive cost of supporting all those administrative roles has exploded. The reality is that software administration is now a multi-billion dollar industry whose function in the whole can only be described as parasitic. No matter which corporate I consult for, their administrators give me some form of "I don't believe it" moment every day.
Lets go back 10 years to see how we got into this mess...
Once upon a time, developers were promoted if they had the stomach for Gantt charts and paperwork. The engineers delivered functionality to the business: there was a clear power hierarchy. If you were lucky, you had an administrator who did exactly what you asked, and kept the systems running without imposing on development.
Today it's a very different picture. Your typical IT director in a large corporate is surrounded by an entourage of software administrators. It was interesting listening to a famous British filmmaker (David Putnam) comment on how the gaggle of administrators surrounding Hollywood stars tried to make them as paranoid as possible about needing their administrative skills.
In the same way, IT managers have been told they had lost control of their software development. In reality, they had lost control of their understanding of software development, and lost the will to catch up. Managers have been told they need to impose standard administration.
Let's take a look at these different types of administration, and see why they've degenerated to such a fragmented state. After that, we'll examine whether it's possible to do anything about this ungodly mess.
Sun (well actually J2EE - we wouldn't want to taint Sun engineers) had a vision, and that vision was fragmentation of development roles. As with many "visions", the J2EE vision was mostly not based in reality. They came up with many roles including deployment administrators, application server vendors, application server administrators, container vendors plus many more, and last but not least the dear old developer.
Unfortunately the J2EE vision was crosseyed: they imagined roles where roles didn't exist (have you heard from your favourite "container vendor" recently?)
"I've never drunk so much tea in my life."
In their hurry to meet this imaginary demand, the J2EE team created the concept of deployment. For those of you not familiar with deployment, let me explain. When you go live with software it's nice to package it up into a nice bundle - to make life easier for administrators. In fact J2EE plays Russian dolls with archives (WARs inside JARs inside EARs).
Sadly, some wacko at Sun decided that this deployment structure should become part of your compile-run cycle rather than your code-test-deliver cycle. Development times have slowed down by at least half.
One of the consequences of J2EE's deployment strategy is that developers can be seen staring aimlessly around the room for 5 minutes, or wandering off for cups of tea after changing a line of code and waiting for their EJBs to rebuild and redeploy. I've never drunk so much tea in my life; programming has started to become a background task. I now know what it felt like to punch programs into cards and wait my turn in the queue for the mainframe. In thirty years we've gone from computer tapes to red tape.
My view of Database Administrators is polarized. I always view databases as complex engines that need to be treated as such to get the best performance out of them.
"DBAs who understand the database are worth their weight in gold."
Having undertaken some serious SQL on several projects, I understand that those DBAs who understand the details of the database engines are worth their weight in gold. They are in essence developers, human like the rest of us.
However, there are a large number of DBAs who don't understand databases. Their position of authority comes only from them holding onto the database permissions. Any thought of creating or altering a table on even the most prototype of system is firmly resisted. Before you can create a table, it must pass the unwritten naming standards convention. Table names must be upper or lowercase, abbreviated so that they are unreadable and match the mainframe naming conventions of a maximum of 8 characters.
A good example of a DBA who didn't know his subject was on a DB2 database where VARCHARs were officially changed to CHARs because on the Mainframe they are "less efficient". No matter that I'm using a PC.
This degenerative scenario is one reason why "agile database techniques" just don't work well in corporate organisations. Try "evolving" a physical database design in one of these places, with several table changes per day or per week. Every little change must be justified to the sysadmin; then you have to wait while he reluctantly makes the change for you.
This isn't something that can be fixed simply by saying "hey guys, can't we all just be nice to each other?", because there's a deeply rooted culture of spiteful, dog in the manger, prevention. Their biggest joy is controlling everything whilst killing productivity.
Source Code Administration
In the beginning there was CVS, CCS, PVCS and SourceSafe. They were simple and safe. Even today they are the most used source code administration tools and and well integrated into most vendors' tools. Somehow, salespeople have got their feet in the corporate door and managers have been sold the lemon that is heavyweight source control systems.
"The principle of least astonishment."
The tired (but oddly effective) argument spouted by some of these salespeople is that with their system, all the source code is in one place and, hey guys, open-source just isn't safe, ya know? But open-source is widely used, well tested and above all follows the principle of least astonishment -- unlike many of the corporate source control systems I have been forced to use.
Two of my particular least favourite "heavyweight" source control systems are Rational ClearCase and Continuus (Continue-is – or Continue-less as I call it).
My most annoying software experiences come from using astonishing source control systems. Like the first time I used Continuus. I checked a System Architecture document in; then when I checked it out, the document had been trashed. I was told it was because the Continuus database "had not been set up by the administrator to handle that sort of file yet" - oh you mean a file sort of file.
It was then that I discovered that the backup directory hadn't been backed up for 6 months. Another great feature of Continuus is that there's no "official" rollback function - only a delete function - which behaves just like a rollback function except for the first time that you check software in, when it acts like a delete function. Sparkling.
ClearCase is another one of those products where the behaviour is not safe. For example, if you find that another person has checked out a file, then you can check it out "unreserved". When you go to check back in a large batch of files, it checks them all back in except for the code that was unreserved (it’s that remembering state thing again). So the net effect is that the code under source code control can't compile. CVS is free and has this facility: why should we pay a premium to make our code base unstable?
It’s been my experience that source control administrators are some of the biggest (or rather smallest) bottlenecks in the company. They are never available when you need them to explain why a file has been swallowed up. When you finally manage to get them to come to your desk, they perform mental gymnastics as they first try to figure out what has happened to your source, then perform verbal gymnastics as they explain why you shouldn't have checked in your file before clicking an obscure checkbox in a deeply nested set of dialogs.
Most IT directors are senile. I can prove it. They think developers are secretaries. How else do you explain them mandating that developers and secretaries have the same administrative rights on their PCs?
Ever tried to install some mandated software? I have been told by every large corporate I've worked at in the past 5 years that either:
(1) We don't have the licence for the mandated software
(2) You don't have permission to install the mandated software
(3) Somebody will come round next week to install the mandated software; or
(4) You'll get sacked if you install the mandated software
It's as if running InstallShield has somehow got just a bit too tricky for us stupid types.
Another great idea borne of lack of experience is that of roaming profiles. It may be OK for lightweight users who pull single files off a file server then replace them. Back in the real world, I need my megabytes locally.
|How the administrator should have reacted...
At one client I got an administrator to change my roaming profile as they had somehow created two of them. Can you guess what happened? Yep, they deleted my source code!
His air of indifference caught me completely off-guard: "Oh well, never mind, it's only 2 weeks' work." Yes, but it was 2 weeks on the critical path of a team of 5 people.
Stretching contentedly and taking a sip from his cup of tea, he told me, almost as an afterthought, that they did backups. However, the next day (when out of arm's length) the same administrator told me it would take a week to recover. I finally persuaded him to fast-track me, and then it only took 2 days to be told that all the directories under roaming profile hadn't been backed up, as it was "too expensive". Only a few files were backed up. Consequently I undertook the dismissable offence of downloading an undeleter, and brought my software back in 2 minutes.
The interesting thing is that this particular administrator knew that nothing bad would happen to him as a result of his gross ineptitude, therefore he didn't give a damn. This project was incredibly important to the company - with penalties of GBP £1million per month if the deadlines weren't met! But no matter: "Oh well, never mind..."
The corporate ecosystem has become so dysfunctional
that there is no reason to fear failure, nothing to lose no matter how great your crime. The process, no matter how broken, is king.
You'd think that all these rules and mandates would make corporate IT a safe environment in which to program. Well, their sysadmins certainly have lots of "don'ts". They don't:
(1) Use roaming profiles appropriately
(2) Communicate that only specific sub-directories were backed up
(3) Apologise (ever)
(4) Provide quick access to the backups online
(5) Have any undelete software
Administrators also like to further frustrate our lives with the overuse of firewalls. Now let's get this clear: I have nothing against firewalls - especially when they're used properly.
One client I worked at was a pyro-wall-maniac. In our development area (that's development, not UAT or live!) we had six firewalls alone. The administrator kept changing the configuration every few weeks. On Monday your apps don't work. What's more interesting is that there were so many firewalls that they had to rely on a single vendor ("Otherwise we wouldn’t be able to manage them").
"But isn't that a security hole, having just one vendor?" I asked. "If there's a security alert for that vendor then your whole network is open."
"Oh, but it's the best firewall."
Closely related to security administation is the admin of virus checkers and the like on each developer's PC. I'm sure you have, but have you looked at the background processes recently on your PC? You may not have noticed but when administrators aren't deleting your files they're stealing your CPU cycles. Virus checkers, process checkers, mail scanners.
Most virus checkers scan archives every time you open, close or modify them. If administrators can get away with it - and they can because they control your box - they will try to stop you from doing any development work at all.
At one client, every hour my CPU would grind to a halt as the virus checker scanned my machine - yet again - for software that I may have accidentally downloaded. When I pointed out that I had the same build as secretaries running Word, I was told that I shouldn't have cause to complain because nobody else had noticed any problems. The straight answer to that was: why not?
Virus checkers are another great example of administrators making our lives needlessly inefficient. Have you ever stopped to think how virus checkers interact with Russian doll-like J2EE archives?
At another client, I noticed that my 2Ghz CPU was performing like an old 486 when developing J2EE apps. The administrators had no idea that their virus checking software was bringing an already slow process to its knees. Eventually I had to take the law into my own hands, and configure the anti-virus software to exclude ear, jar, war, java and jsp extensions. Deployment finally dropped below 5 minutes and I didn't have to reboot 3 times a day. Joy...
But why was I still frustrated? Because every night the anti-virus software refreshed the exclusion list and I had to commit a sackable offence every morning, making sure I could do development that day. I can think of no better example of senior management adrift from their responsibilities.
Project administration - formerly known as project management. Over the past 5 years I have seen project management go from a job managers enjoyed to one of constant administration. Project managers have been sucked - unknowingly, it seems - into the quagmire of administrative tasks. A lot of project management time is now spent implementing policies that don't benefit the project directly.
"The reality is, of course, different..."
Like any experienced general, an experienced project manager will tell you that even the best plans go out the window as soon as the real work commences. A good example is of an N-tiered system where the project plan is often broken down either by tier or by function. It is presumed that the developers will either complete tier-1, tier-2 etc, or complete all the tiers for function-1, then function-2 etc. The reality is of course different.
What happens instead is that there are functionality dependencies between tiers, so that tier-1 of function-1 and tier-1 of function-2 must be completed before tier-2 of function-1. In other words there are complex dependencies between tasks that unless you've done the same project before can't be foreseen.
Dealing with multiple administrators creates communication problems. Because administrators create bottlenecks, managers spend a large amount of their time chasing them up. If there was only one administrator then the communication would be clearer. If you're the administrator then they're even clearer!
What Has Happened
In the past 5 years, software development has advanced at such a pace that senior management has, for the most part, lost touch with the development process. Senior managers have become more distant from the technology: it's a noticeable trend.
A great deal of the current inefficiency in software development comes from senior IT managers who are persuaded by administrators to impose unnecessary policies on developers. The administrators have been given too much power, and often take up adversarial positions against the development managers. The current operating mode of most organisations is entirely negative. Administrators now utilise development hurdles (such as database installations) to enforce policies.
"It's no wonder that so many projects are being outsourced."
I don't want to be a snob about this, but you really don't hear many administrators saying: "I programmed for 10 years but found admin paid more money and was less boring." What is often overlooked, but is absolutely critical, is that developers are the clients of administrators. They are here to serve us. Yet the power structures have somehow developed so that we're perceived as equals, and now they just frustrate us.
In the past 5 years I have spoken to many developers who feel that the administrative burdens are now halving their development efficiencies. If the trend of administration continues then most projects will not be cost effective. It's no wonder that so many projects are being outsourced.
Developers are forced to spend more and more time hankering to administrators, then administrators increasingly gain an upper hand. The project development cycles spiral out of control. With administrative fever implicitly driving development timescales, departments soon become stuck in a downward spiral of inefficiency. The problems become unrecoverable without radical or prolonged change.
What We Can Do About It
Firstly, the true cost of administration must be accounted for when totalling up the cost of any project. Administration dead-time is a significant and growing aspect of all projects I have witnessed in the past 5 years. Dead-time waiting for software to arrive, files to be undeleted, versions to be mandated: it all needs to be added up and made explicit.
Transparent administration is good administration. Imagine if the business had to stop whenever the IT department had forgotten something. There would be hell to pay. To stop this happening, developers have created something called user acceptance testing. Administrators should therefore have something akin - called Developer Acceptance testing. If we don't like the way administrators behave, then we would be able to send them back to the drawing board - or simply get rid of them altogether.
It is my experience that we are now 60% over-administrated. In the next round of job cuts or process re-engineering, IT directors should not hesitate to eliminate a high proportion of administrators, or move them into programming roles. The huge majority of their job effort is involved in either administering over-engineered tools or in communication with developers - and once the administrators go, so too do those duties (as if by magic...)
Does administration really matter? Yes it does: administration of all sorts. But the number one priority is competitive advantage and delivery of a mature, stable system. But, hand on heart, administration is now out of control, and IT managers let it happen because to them, it looks like management.
Bad Manager Personalities
IT Support Department to Hire Rottweilers as Technicians
Stop the Press: Software Delivered on Schedule!
Obsolete Documents (and what to do about them)
Related Books (Links to Amazon.com):
<< Back to Lifecycle