Programming with
a dose of satire.
Lifecycle
|
| ||||||||
 |
 |
|||||||
|
Lifecycle |
||||||||||||||||||||||||||||
Role FragmentationDivide and Conquer - Why IT Administrators Have Become All-Powerful Demi-Gods By Robin Sharp
In the beginning it was simple: there were analysts and developers. PC developers doubled up as administrators, and analysts doubled up as managers. Developers did various types of administration: system, network, database and runtime administration. Administration was done in a timely manner, sensitive to the needs of the project. Somehow over the past 10 years, a cancer has been growing inside large organisations - called role fragmentation. As the roles have fragmented, an entire non-industry has bloomed. Network administration, database administration, security administration, component administration, deployment administration, tool administration. The list just goes on and on. This fragmentation has had some seriously negative consequences for the industry. In particular:
Why has the administration been allowed to bloom unnoticed in the software industry when it is having all these negative effects? While the negative cost of administration has hit development times, the positive cost of supporting all those administrative roles has exploded. The reality is that software administration is now a multi-billion dollar industry whose function in the whole can only be described as parasitic. No matter which corporate I consult for, their administrators give me some form of "I don't believe it" moment every day. Lets go back 10 years to see how we got into this mess...
Once upon a time, developers were promoted if they had the stomach for Gantt charts and paperwork. The engineers delivered functionality to the business: there was a clear power hierarchy. If you were lucky, you had an administrator who did exactly what you asked, and kept the systems running without imposing on development. Today it's a very different picture. Your typical IT director in a large corporate is surrounded by an entourage of software administrators. It was interesting listening to a famous British filmmaker (David Putnam) comment on how the gaggle of administrators surrounding Hollywood stars tried to make them as paranoid as possible about needing their administrative skills. In the same way, IT managers have been told they had lost control of their software development. In reality, they had lost control of their understanding of software development, and lost the will to catch up. Managers have been told they need to impose standard administration. Let's take a look at these different types of administration, and see why they've degenerated to such a fragmented state. After that, we'll examine whether it's possible to do anything about this ungodly mess.
Sun (well actually J2EE - we wouldn't want to taint Sun engineers) had a vision, and that vision was fragmentation of development roles. As with many "visions", the J2EE vision was mostly not based in reality. They came up with many roles including deployment administrators, application server vendors, application server administrators, container vendors plus many more, and last but not least the dear old developer. Unfortunately the J2EE vision was crosseyed: they imagined roles where roles didn't exist (have you heard from your favourite "container vendor" recently?)
In their hurry to meet this imaginary demand, the J2EE team created the concept of deployment. For those of you not familiar with deployment, let me explain. When you go live with software it's nice to package it up into a nice bundle - to make life easier for administrators. In fact J2EE plays Russian dolls with archives (WARs inside JARs inside EARs). Sadly, some wacko at Sun decided that this deployment structure should become part of your compile-run cycle rather than your code-test-deliver cycle. Development times have slowed down by at least half. One of the consequences of J2EE's deployment strategy is that developers can be seen staring aimlessly around the room for 5 minutes, or wandering off for cups of tea after changing a line of code and waiting for their EJBs to rebuild and redeploy. I've never drunk so much tea in my life; programming has started to become a background task. I now know what it felt like to punch programs into cards and wait my turn in the queue for the mainframe. In thirty years we've gone from computer tapes to red tape.
My view of Database Administrators is polarized. I always view databases as complex engines that need to be treated as such to get the best performance out of them.
Having undertaken some serious SQL on several projects, I understand that those DBAs who understand the details of the database engines are worth their weight in gold. They are in essence developers, human like the rest of us. However, there are a large number of DBAs who don't understand databases. Their position of authority comes only from them holding onto the database permissions. Any thought of creating or altering a table on even the most prototype of system is firmly resisted. Before you can create a table, it must pass the unwritten naming standards convention. Table names must be upper or lowercase, abbreviated so that they are unreadable and match the mainframe naming conventions of a maximum of 8 characters. A good example of a DBA who didn't know his subject was on a DB2 database where VARCHARs were officially changed to CHARs because on the Mainframe they are "less efficient". No matter that I'm using a PC. This degenerative scenario is one reason why "agile database techniques" just don't work well in corporate organisations. Try "evolving" a physical database design in one of these places, with several table changes per day or per week. Every little change must be justified to the sysadmin; then you have to wait while he reluctantly makes the change for you. This isn't something that can be fixed simply by saying "hey guys, can't we all just be nice to each other?", because there's a deeply rooted culture of spiteful, dog in the manger, prevention. Their biggest joy is controlling everything whilst killing productivity.
In the beginning there was CVS, CCS, PVCS and SourceSafe. They were simple and safe. Even today they are the most used source code administration tools and and well integrated into most vendors' tools. Somehow, salespeople have got their feet in the corporate door and managers have been sold the lemon that is heavyweight source control systems.
The tired (but oddly effective) argument spouted by some of these salespeople is that with their system, all the source code is in one place and, hey guys, open-source just isn't safe, ya know? But open-source is widely used, well tested and above all follows the principle of least astonishment -- unlike many of the corporate source control systems I have been forced to use. Two of my particular least favourite "heavyweight" source control systems are Rational ClearCase and Continuus (Continue-is – or Continue-less as I call it). My most annoying software experiences come from using astonishing source control systems. Like the first time I used Continuus. I checked a System Architecture document in; then when I checked it out, the document had been trashed. I was told it was because the Continuus database "had not been set up by the administrator to handle that sort of file yet" - oh you mean a file sort of file. It was then that I discovered that the backup directory hadn't been backed up for 6 months. Another great feature of Continuus is that there's no "official" rollback function - only a delete function - which behaves just like a rollback function except for the first time that you check software in, when it acts like a delete function. Sparkling. ClearCase is another one of those products where the behaviour is not safe. For example, if you find that another person has checked out a file, then you can check it out "unreserved". When you go to check back in a large batch of files, it checks them all back in except for the code that was unreserved (it’s that remembering state thing again). So the net effect is that the code under source code control can't compile. CVS is free and has this facility: why should we pay a premium to make our code base unstable? It’s been my experience that source control administrators are some of the biggest (or rather smallest) bottlenecks in the company. They are never available when you need them to explain why a file has been swallowed up. When you finally manage to get them to come to your desk, they perform mental gymnastics as they first try to figure out what has happened to your source, then perform verbal gymnastics as they explain why you shouldn't have checked in your file before clicking an obscure checkbox in a deeply nested set of dialogs.
Most IT directors are senile. I can prove it. They think developers are secretaries. How else do you explain them mandating that developers and secretaries have the same administrative rights on their PCs? Ever tried to install some mandated software? I have been told by every large corporate I've worked at in the past 5 years that either:
It's as if running InstallShield has somehow got just a bit too tricky for us stupid types. Another great idea borne of lack of experience is that of roaming profiles. It may be OK for lightweight users who pull single files off a file server then replace them. Back in the real world, I need my megabytes locally.
At one client I got an administrator to change my roaming profile as they had somehow created two of them. Can you guess what happened? Yep, they deleted my source code! His air of indifference caught me completely off-guard: "Oh well, never mind, it's only 2 weeks' work." Yes, but it was 2 weeks on the critical path of a team of 5 people. Stretching contentedly and taking a sip from his cup of tea, he told me, almost as an afterthought, that they did backups. However, the next day (when out of arm's length) the same administrator told me it would take a week to recover. I finally persuaded him to fast-track me, and then it only took 2 days to be told that all the directories under roaming profile hadn't been backed up, as it was "too expensive". Only a few files were backed up. Consequently I undertook the dismissable offence of downloading an undeleter, and brought my software back in 2 minutes. The interesting thing is that this particular administrator knew that nothing bad would happen to him as a result of his gross ineptitude, therefore he didn't give a damn. This project was incredibly important to the company - with penalties of GBP £1million per month if the deadlines weren't met! But no matter: "Oh well, never mind..." The corporate ecosystem has become so dysfunctional that there is no reason to fear failure, nothing to lose no matter how great your crime. The process, no matter how broken, is king.
Administrators also like to further frustrate our lives with the overuse of firewalls. Now let's get this clear: I have nothing against firewalls - especially when they're used properly. One client I worked at was a pyro-wall-maniac. In our development area (that's development, not UAT or live!) we had six firewalls alone. The administrator kept changing the configuration every few weeks. On Monday your apps don't work. What's more interesting is that there were so many firewalls that they had to rely on a single vendor ("Otherwise we wouldn’t be able to manage them"). "But isn't that a security hole, having just one vendor?" I asked. "If there's a security alert for that vendor then your whole network is open." "Oh, but it's the best firewall."
Closely related to security administation is the admin of virus checkers and the like on each developer's PC. I'm sure you have, but have you looked at the background processes recently on your PC? You may not have noticed but when administrators aren't deleting your files they're stealing your CPU cycles. Virus checkers, process checkers, mail scanners. Most virus checkers scan archives every time you open, close or modify them. If administrators can get away with it - and they can because they control your box - they will try to stop you from doing any development work at all. At one client, every hour my CPU would grind to a halt as the virus checker scanned my machine - yet again - for software that I may have accidentally downloaded. When I pointed out that I had the same build as secretaries running Word, I was told that I shouldn't have cause to complain because nobody else had noticed any problems. The straight answer to that was: why not? Virus checkers are another great example of administrators making our lives needlessly inefficient. Have you ever stopped to think how virus checkers interact with Russian doll-like J2EE archives?
At another client, I noticed that my 2Ghz CPU was performing like an old 486 when developing J2EE apps. The administrators had no idea that their virus checking software was bringing an already slow process to its knees. Eventually I had to take the law into my own hands, and configure the anti-virus software to exclude ear, jar, war, java and jsp extensions. Deployment finally dropped below 5 minutes and I didn't have to reboot 3 times a day. Joy... But why was I still frustrated? Because every night the anti-virus software refreshed the exclusion list and I had to commit a sackable offence every morning, making sure I could do development that day. I can think of no better example of senior management adrift from their responsibilities.
Project administration - formerly known as project management. Over the past 5 years I have seen project management go from a job managers enjoyed to one of constant administration. Project managers have been sucked - unknowingly, it seems - into the quagmire of administrative tasks. A lot of project management time is now spent implementing policies that don't benefit the project directly.
Like any experienced general, an experienced project manager will tell you that even the best plans go out the window as soon as the real work commences. A good example is of an N-tiered system where the project plan is often broken down either by tier or by function. It is presumed that the developers will either complete tier-1, tier-2 etc, or complete all the tiers for function-1, then function-2 etc. The reality is of course different. What happens instead is that there are functionality dependencies between tiers, so that tier-1 of function-1 and tier-1 of function-2 must be completed before tier-2 of function-1. In other words there are complex dependencies between tasks that unless you've done the same project before can't be foreseen. Dealing with multiple administrators creates communication problems. Because administrators create bottlenecks, managers spend a large amount of their time chasing them up. If there was only one administrator then the communication would be clearer. If you're the administrator then they're even clearer!
In the past 5 years, software development has advanced at such a pace that senior management has, for the most part, lost touch with the development process. Senior managers have become more distant from the technology: it's a noticeable trend. A great deal of the current inefficiency in software development comes from senior IT managers who are persuaded by administrators to impose unnecessary policies on developers. The administrators have been given too much power, and often take up adversarial positions against the development managers. The current operating mode of most organisations is entirely negative. Administrators now utilise development hurdles (such as database installations) to enforce policies.
I don't want to be a snob about this, but you really don't hear many administrators saying: "I programmed for 10 years but found admin paid more money and was less boring." What is often overlooked, but is absolutely critical, is that developers are the clients of administrators. They are here to serve us. Yet the power structures have somehow developed so that we're perceived as equals, and now they just frustrate us. In the past 5 years I have spoken to many developers who feel that the administrative burdens are now halving their development efficiencies. If the trend of administration continues then most projects will not be cost effective. It's no wonder that so many projects are being outsourced. Developers are forced to spend more and more time hankering to administrators, then administrators increasingly gain an upper hand. The project development cycles spiral out of control. With administrative fever implicitly driving development timescales, departments soon become stuck in a downward spiral of inefficiency. The problems become unrecoverable without radical or prolonged change.
Firstly, the true cost of administration must be accounted for when totalling up the cost of any project. Administration dead-time is a significant and growing aspect of all projects I have witnessed in the past 5 years. Dead-time waiting for software to arrive, files to be undeleted, versions to be mandated: it all needs to be added up and made explicit. Transparent administration is good administration. Imagine if the business had to stop whenever the IT department had forgotten something. There would be hell to pay. To stop this happening, developers have created something called user acceptance testing. Administrators should therefore have something akin - called Developer Acceptance testing. If we don't like the way administrators behave, then we would be able to send them back to the drawing board - or simply get rid of them altogether.
It is my experience that we are now 60% over-administrated. In the next round of job cuts or process re-engineering, IT directors should not hesitate to eliminate a high proportion of administrators, or move them into programming roles. The huge majority of their job effort is involved in either administering over-engineered tools or in communication with developers - and once the administrators go, so too do those duties (as if by magic...) Does administration really matter? Yes it does: administration of all sorts. But the number one priority is competitive advantage and delivery of a mature, stable system. But, hand on heart, administration is now out of control, and IT managers let it happen because to them, it looks like management.
About the author:
Related Articles: IT Support Department to Hire Rottweilers as Technicians Stop the Press: Software Delivered on Schedule! Obsolete Documents (and what to do about them)
Related Books (Links to Amazon.com):
Talkback: Have Your Say If you've experienced the same problems as Robin... or found corporate culture to be entirely the opposite... let us know! Message Index:
The Messages: agree 100%Good administrators are like gold dust. If you find one for your project, be happy! But to become a good administrator, admins should spend some time as developers first. They need to see both sides of the coin. Anonymous person Sat Dec 06 17:45:54 GMT 2003 right on but programmers helped create this situation This article is dead on and just reading it felt like some sort of therapy. But the author does not mention the fact that wet-behind-the-ears programmers and incompetent/malicious programmers have contributed to this situation. no one gets sudo access because some kid typed rm -rf * in the wrong directory. someone created a column in dev and the release had to get rolled back because no one ever created the column in production. and don't even mention the mp3 share shutdown last week that was sucking up all the bandwidth. This infrastructure was necessary as a system of checks and balances to keep the losers in line -- at the expense of efficient, intelligent programmers. red tape redtape@redtape.com San Francisco, CA, USA Sat Dec 06 18:30:00 GMT 2003 Been on both sides of the fence I actually am one of the people who did development work for a number of years and then became an admin because it was more interesting to me. I program for fun outside of work on various little projects, and didn't want to burn out on it. That being said, a good administrator is still going to prevent you from doing everything you like. It's not personal. The administrator is the front guy when the piracy boys come around, when another admin emails that a box on his network has broken into theirs, etc. If every user on a network were well trained in the dos and donts then most of the hardline restrictions could be cut out, replaced with passive monitoring. I've been on both sides of the coin. When working with a small engineering company, the users were reasonably competant, didn't install cutesy utilities that are spyware in disguise, and the lack. They had admin rights to their machines. Some users "just didn't get it", and were restricted. Secretaries and the like were restricted by default. However, good communication is essential, going in both directions. The Administrator needs to outline the policies, what they are, and why they are there. The users need to explain to the Admin what their problems or issues are. In a good working environment, this occurs naturally and without the need for outside parties (i.e. management) to be brought in. A /good/ Administrator makes things run fluidly. A /bad/ one will just hinder. This is why /good/ Administrators are a) hard to find and b) hard to hire, because everyone these days has the $2495 MCSE 'certification', which gives the HR people some alphabet soup to filter on. A surprising number of /good/ Administrators don't actually have a lot of those certifications as they are perceived as a waste of time (which they are), so when it comes time to get a new job, they get passed over in favor of the MCSE who couldn't spell computer 2 weeks ago. An Admin 1997@blargh.com Sat Dec 06 19:08:07 GMT 2003 Shut up The I.T. Staff is not there for you. That is where your wrong. They are there for the company, and there are other people in the company. Do you sell the software? do you market the software? Do you take support calls for your software when it breaks? NO you don't, and I.T. is off fixing them too, while you sit at your desk on your high horse bitching all day. I.T. has a bunch of work, and your idea of firing more of them will just slow down the process. Being in I.T. i've seen most developers are rock stupid. All they can do is program their widgets, and everything else is TOOO much for them. The reason you "developers" have so many problems, is because you create them through all the stupid stuff you do. Maybe if you where nice to your I.T. people they would fix your problems faster. I.T. always makes the assholes wait, and thats probally why you never get service. Anonymous person Sat Dec 06 19:17:11 GMT 2003 there are problems on both sides of the fence While I agree that admins do try to slow the developers down...there is a reason. All too often the developers hand us unsupportable, insecure, poorly implemented code. While you are frustrated by incompetent admins, we are frustrated by incompetent developers. A few bad apples on both sides of the fence screw us all. My suggestion would be to get your admins involved with your project early and often. Keep them informed of your design so that the final product is something both sides can live with. Anonymous person Sat Dec 06 19:26:43 GMT 2003 Good start, bad ending. Dear $deity, that's sad. While starting out with valid points, you end up wanting to restrict, mangle, fire, trample and whatnot else your hated nemesis, /the admin/. Even though you yourself point out that not all is actually the fault of /the admins/; the rest is mangagers losing it and crackpot design (/by developers/). Oh, and working in the wrong environment. Both a corporate culture that is hard on everyone (listen to your admins now and then, they're human too), and seemingly the wrong platform. If you hadn't guessed by now, yes, I play admin here. I'm the sole admin where there used to be five or so. Big deployment, lots of little servers, nobody remembers what they're doing, all clustered, all different. Just about the same playground mess as ye average developerfarm. The admin sandbox is mostly cleaned up now, the developer sandbox isn't, yet. And my, do they cry. Mommy! I don't waaaant to clean up my room!!1 Anonymous coward yurop Sat Dec 06 19:28:25 GMT 2003 you are complaining about the symptom, not the cause I work at a company that has spent that last 6 years developing internet software. when I talk to the developers I want to pull my hair out. here are the people who are supposed to be doing the right thing who ask what a socket is, or how TCP works (especially when it's the people who are writing the communications protocols) I've seen the almighty developers turned loose on the database and write querys that took 10 min to run, once a DBA looked at it for 5 min it got modified to take 2 sec to run. don't try to blame this on your network/system/firewall/etc administrators blame it on the fact that the clue level of IT staff (INCLUDING developers) has been dropping steadily. yes there are a lot of clueless admins out there, but there are also a lot of clueless developers and most of the procedures you are complaining about were put in place to defend againt the clueless developer. now your rants about project administration I can agree with for the most part. it seems like the overhead of something being a project is frequently as high or higher then the work of actually developing the code. and as for firewalls in the development environment. I just had to install a half dozen firewalls in development becouse the development management demanded it, they claimied that they couldn't possibly develop code to run in production if their development environment wasn't absolutly identical. in other words they couldn't know what their programs do on the network and stick to standards unless firewalls were in place to enforce those standards Anonymous person Sat Dec 06 20:16:08 GMT 2003 the scope of expertise is different a good programmer can program in any language (they may have to take a few days to learn a new language, but then they can program in it, even if they miss some of the optimizations) a good sysadmin not only needs to be able to program in several languages (even though it's useually called scripting not programming) and they need to look at the whole picture of how all the parts (including those not written by the developers, and those run by other companies) work togeather. overall I find that the sysadmins useually have a much broader skill base then developers. there are exceptions but not many (and in both cases there are a lot of people who don't know what they are doing at all, I'm discounting them for this comparison) Anonymous person Sat Dec 06 20:27:53 GMT 2003 How's the view from the cheap seats? After reading this article all I can say is wow. Talk about misinformed, unjustified, and just plain "out there". Normally I would accuse an author writing something like this of smoking something illegal in most corners of the world, but this is such a pile of steaming fecal matter that I just have to ask what planet the author has been living on, and why couldn't they just stay there? What is their definition of "admin"? Are they talking about Administration such as in an education environment? Are they referring to upper management in a big corporation? Perhaps they're referring to middle management or even the actual grunts that administer technology platforms. Actually, the author groups all of the above into one entity. Project managers are the same as database admins are the same as the kid that replaces your keyboard when you spill coffee on it. In real life, there's a huge seperation between these groups. This is just plain ignorant. It seems funny to me that there isn't any mention of systems administrators. You know, the people that the developers run to crying whenever ANYTHING doesn't work for them. Can't find the compiler? Call the systems guy. Used up all the disk space? Call the systems guy. Need a new development machine to work on because your current one suddenly seems broken after you made all those kernel parameter changes? Call the systems guy. Oh wait, the author has grouped the systems guy in with the PC Admins, because, after all, no one uses UNIX to develop applications, it's all mainframe or Windows. The author leads a very sheltered life. I could go on for days. As I've typed this, developers and "admins" alike that I know have read the article and feel, as I do, that perhaps the author is in the wrong line of work. Perhaps managing a McDonalds is more in line with the author's skillset? Why is the author so bitter towards this grouping of admins? I don't think the author even knows. All I know is I certainly won't be doing any business with jarvis software or whatever the company is that the author works for. Especially when he's the "Managing Director" which falls into the very same group of people he's complaining about. Thor undisclosed Midwest Sat Dec 06 20:28:56 GMT 2003 This article is bad, almost SCO bad..... This article is so bad, Darl McBride must have wrote it. It includes pretty much everything developers have whined about since I've started in the business. First off, as a systems admin, my job is to provide the company I work for a stable production system(s). I get incented for making sure that happens. That means I am not a popular person, since my first response to some whining developer is to say no. This person seems to forget that 60-75% of unplanned downtime is because of undocumented or poorly planned change. If you don't believe me, talk to Gartner or Meta or your favorite research company. Better yet go through your companies system outage records and see what the cause of most of your downtime is. You are keeping track of that aren't you? Because my pay is based on the performance of the systems I take care of, I am very conservative about changes to those systems. It gets even worse when the developers try to go around the corporate mandated change control process or even don't follow the code promotion process they set up. Yet they are more than happy to call me at 3 am when the app they wrote and sneaked onto a system doesn't work the way they think it should. The other scenario is that the app they didn't tell us about is stepping on something else and I have to try to figure out who to contact after taking 2 hours to figure out what is going on. I'm going to insist that you test your app before you put it on a production system, that is why we have a development and test environments. I'm also not going to troubleshoot your code for you, I'm fluent in C, perl, python, expect, shell and dangerous in c++ and java. However you wrote the code, you should understand the environment you run it in. If you need help in understanding that, come see me, we can talk over coffee or something. I'm also going to think you are an idiot when you ask me why the FTP job you set up to transfer mission critical data is failing. Especially when you didn't write a wrapper for it, don't do any sort of logging and expect it to be guarenteed delivery. You may laugh, but this happened to me in the past year. The company I work for has 150 developers, about 170 Unix systems and 6 system admins. We are not only supposed to keep the systems running 7x24, we are supposed to be resources for projects and handle walk up work. We are proud that we are able to juggle the work load. The industry has changed greatly in the past 5-10 years. This industry in fact runs on change. What used to work no longer does. The business expects us to not only keep the systems up, but to interface with them at a non-techie level and get more professional at what we do. It is apparent to me that the author of this article is stuck in the past. S. Scherbinski sski AT speakeasy.net USA Sat Dec 06 20:58:42 GMT 2003 Example I was brought on as a consultant to work on a critical business project for a newer subsidy of a Fortune 500 company. They had built a whole IT process from start that depended on experts in very fragmented compartments. I was to be one of 24 people on this project. My role was developer. I quickly found out that although people loved their titles, few had any real expert knowledge. Thanksfully, over 2 years, 18 of the people moved on to other places and were never replaced--mostly because myself and the project manager didn't complain. I just took over the responsibility. By the end of those 2 years we only had myself, a project manager, a DBA, a data analyst, a system administrator, and a helpdesk SME for my app. All testers, analysts, production monitors, implementors, managers, supervisors, etc. were gone. My customers loved our app because we could literally go from concept to production in a day if something was small enough. I averaged three minor changes a week and two major enhancements per month. Most applications in the company did a minor change a month with major enhancements every six months. During my third year on the app other applications began coming to my application to do work for them. Instead of their developing web applications using the normal process, I could develop the functionality on my application in a fraction of the time and trouble. Their app just called mine and everyone was happy. All good things must come to an end though. All senior consultants were downsized one day and I was one of the more senior. Before I was out the door, six full time employees were already assigned to the application. Two years later and my understanding is that the application is largely unchanged since I left and there are dozens of people assigned to it. Customers are very frustrated because they can't get the simplest of changes in. Management loved my style, ability, and process but not enough to realize they could have it on most other applications by changing their process. Sometimes people just don't get it. Annonymous Overland Park, KS Sat Dec 06 21:02:21 GMT 2003 Admins and Their Management While I'll agree that these things happen too often in the world of software dev, I'd say that responsibility on _both_ sides is key. The approach I've used which worked wonders (once the accounting and management staff saw the result) was charging back time to the departments stalling our work. As the PM, I maintained a log of the time in dev these admins cost us, and handed the results to the managers (and _their_ managers) of those departments. We didn't get actual dollars shifted around the first time we did it, but when the upper-level management saw the logs with details of some of the obstructionist behavior, they instituted a 'wide and deep' review of some of the people and policies. Come departmental budget time, the CIO and her immediate reports made it clear that 'chargeback' money might find its way into future requests by the obstructing departments. As a result, we had two admins reduced in responsibility, and two had their managers breathing down their necks on the next project. The security admin left for grayer pastures within a month of the new policy, and no one missed him. Like it or not, you have to make people see the dollars if you want changes. And it can't all be "those evil admins" - the dev team has to communicate up front what they expect, and then stick to it. Dev staff can't just pull something out of thin air halfway through the project and expect admin staff to comply, either. Sadly, the admin staff pointing out that it's the dev staff 'whining' fundamentally misunderstand their roles in a corporation. Admin staff are _not_ there to make policy - they are there to implement it. Admins are almost by definition a _cost_ center, not a _profit_ center whereas it's just the opposite for dev staff. Cost centers should have input into overall policy, but they should never be put in a position to dictate it. It's not just admin staff, but also Marketing, Sales, HR - any department not directly making the end product. If you aren't producing the widget to sell, your role is supportive in nature, and the weight of your opinion should reflect that. Until CIOs and their peers are made to understand that by pointing it out, production times and dollars will continue to rise until competition shuts you down. JK Cole JK_Cole@hotmail.com Chicago, USA Sat Dec 06 21:04:21 GMT 2003 Administrators are there to manage LIMITED resources It is amazing how people can never see the big picture. Developers think that everybody has unlimited resources and becuase they can hack something together therefore they can do anything. Businesses exist to make money PEROID. To maximize their profits need to miminize their cost. Most of the time that means cutting infra-structure cost so limited resources have to be shared by everybody. Just this week I found a single client was download a single 5Meg web page every minute. We were filling their T1 connection with a single web page! Nobody at that site could do any web surfing because of this massive web page being downloaded every minute. That was not the worst part, the developor had hard coded his userid and password in to the web app! If he left the company or changed his password that app would have broken. We have not checked yet to see what other applications are hacked the same way. Basically millions of dollars to develop business to business services depends on this developer staying with the company and not changing his password. Never mind the security problems of sending his username and password all over the world when ever somebody visit our site. People are complaining about slow performance on the web site, well I found that the switch was behaving like a hub and everything in that vlan could see all the traffic for one of the systems. Half the bandwidth for all the systems is being tied up with a backup. Interesting week so far. We had a developer figured that because he knew how to code therefore he could configure his own PC so he decided he did not need our standard install and managed to get infected with a virus that they took down our MS SQL servers. Lost $2million per day not counting overtime to fix and restore the databases. But hey he is a developer and he knows almost as much as an engineer. Another group who are too good to be told anything. People were complaining about slow internet response, once we investigated we found one developer running streaming media on their special built PC and was chewing up %15 of the bandwidth just to listen to music! The other thousand people had to suffer so the developer could listen to some music! How about the developer who designed a database on a test box and then sreamed to have it rushed into production so he would not miss his deadline. In production the database is as slow as hell. Problem? No index on a 1 terabyte database! But hey he is a developer and too good to be told anything. When we took away administrative rights on people's PC and standardize each system the helpdesk calls went from 8000 per month to just 4000 in the first month. Also the time to fix each problem was cut to minutes to fix in most cases. I could go on with hundreds of example where people do not realize that the company has limited resources and it has to be shared by thousands of users but you probably get the point. The main trouble is that developers never ask any question, they never plan, and they expect that everybody to jump when they say. If you need special software installed on your system then don't wait until the last minute to ask the helpdesk to install, plan ahead. Ask the security, network and DBAs to come to at least some of your planning meetings. Make sure to build in time for testing in a realistic environment with the administrators helping you troubleshooting. The network and security can sniff your appliction while it is doing a test run and that can be feed back to make the appliction work well with the firewalls and network. The good DBAs can help you fine tune your application so it does trash the database sever. Remember there is a big differnce between a test database of a few megs and a production of hundreds of gigabytes. If administrators are saying no to you it is probably because you have not approached them correctly or your mistakes have cost them time and trouble. Realize the limits that your company has to deal with and things will go better. Later. bob bob@aol.com Sat Dec 06 21:08:39 GMT 2003 Whining is not quite as effective as responsible action There's more whining than good sense in this article, but I can't help but feel sympathetic all the same. The overall stupidity level in IT has never been higher, and at the same time the environment is more complex than ever. Of course it's not comfortable. As someone who has been at various times over the past thirty years a senior system administrator, a security architect, and a system developer, I'd like to offer one simple suggestion: make it a priority for each person to fix the problems in his area of responsibility, and don't settle for anything less. Our technical challenges are hard enough without compounding them with workarounds and crosswired responsibilities. It's so basic, and yet so neglected. Most sites I see are a total mess, and the one factor common to all of them seems to be neglect. System administrators have to keep things secure, which means applying the principle of least privilege. They're not doing their job right if they give away privilege needlessly, but neither are they if their configuration choices get in the way of developers doing their job. Both sorts of problems have to be adjusted as needs change. System administration is often done badly. That doesn't mean that developers have a license to hack around the privilege restrictions, it means they have a license to beat on the sysadmins until things are set up correctly. Constructive dialogue is the more responsible route, if you can manage it, but regardless, let the truth speak. Conversely, if you're a developer, don't expect to get access to any resource you don't actually need. Don't ever expect to get root to anything. In a properly managed environment, you don't need it. Something is wrong if you do, and asking for root marks you as a clueless liability. Asking for appropriate access to do your job marks you as clueful. You think that you have to install software on your workstation? Wrong. You're part of a team. If the team needs the software, then everyone gets it, and gets it installed identically and correctly. Identifying and communicating your needs requires intelligent collaboration. Work on that. I can't help it if little of this is recognized by management in many organizations. Sorry about that. It's the blind leading the blind up there. Let's fix this one among ourselves. Supposedly it's what we know how to do. Once thing are really running sweetly, then we can begin to take on the management challenge too. Starfish Sat Dec 06 22:08:24 GMT 2003 The Author is a self-absorbed a**. I have to agree with pretty much none of what the author has said. I have been in systems administration for nearly 10 years, and have primarily worked for software development companies. Many times I have encountered the animosity between developers and systems admins. While many respect one another, and their unique talents, there are some who just plain can't respect anyone else's job because it might disturb their high and mighty perch. The author is one of these people. This article could have been written the exact opposite from the position of IT Administration or IT Management, all about how developers are making systems management unbearable, or creating stability or security problems through sloppy design or code. This has already been mentioned in previous comments. Better would have been an article about how developers and IT Administration can work effectively together. I have always worked with the development heads to build infrastructures that can best support their work, and then their code in production. Yes, there are policies for changes, testing etc. But these are established within windows they can find acceptable, and that the IT budget can afford. Yes, there are policies on changes and production system access, but they are there for the protection of the company's services. Which, at the end of the day, falls upon my shoulders to ensure 24/7 operations. The skill set to develop code is distinctly different from the skill set to administer myriads of systems, software and networking technologies. The key to them working together is communication, mutual respect (which I have often found is harder to get from the developers than from IT to them) and realistic, defined expectations. I love developing the infrastructures to suppor the great products our developers are turning out. When the product goes to market it is the end result of a good deal of work by both departments. The author seems to live in a bubble where he's King Codehead and any other skill is really a subset of his genius. I can't beleive this garbage made it to print. For shame the editors of this publication for not providing a more productive peice than a detailed bitch and rant from a jaded developer. Derek IT Manager DoxEMR www.doxemr.com Anonymous person Sat Dec 06 22:34:58 GMT 2003 admin clients of developers: not true everywhere Your comments are very interesting, and highlight the way it should work, in a box that develops software as a primary product. Yet more and more professional admins start in other fields, and bring practices from other places. Your comments of developers clients of admins only works where the company produces software. Yet the hierarchy between developers and admins is nominally in the market at large. I've certainly seen the exact opposite effect where the Operations Manager(responsable of 7/24 operations of course) of a firm was nothing short of bullied by the Development manager, over a lack of planning by his department about needing 7/24 coverage of an upgrade to systems by developers. (Note to managers: do NOT wait for people on 7/24 rotation over a weekend to be out of town before telling them they might be needed) However, I don't want this example to cover up the fact that your examples certainly match some of my experiences in the "software as a product" field, just not in the "software as part of a service". Now the "software as part of a service" is probably the fastest-growing part of the industry, mostly because IT managers don't always understand the software they buy, but they can certainly understand a service contract, or hire a lawyer that can. That leads me to think that your "problem" is going to become more and more prevalent, until those supervising the admins wise up to the fact that those admins have to be productive, and in a service domain, that means giving the client what he wants, and if the developers are doing what they are paid to do, then the admins DO have to support them(but in this scenario, they ARE equals: development of new systems, vs support/maintenance of existing systems are two aspects of a same thing: making sure a system delivers the service. I'm looking forward to hear other thoughts on this matter, and how it influences the new waves of outsourcing, and how outsourcing CAN just be an attempt to get more control over the development process, by removing developer's job security out of the equation. Eric Robibaro Qc, Canada Sat Dec 06 23:19:36 GMT 2003 from the other side As a developer who started out as an administrator I'd just like to say that any developer who thinks an admin should spend some time as a developer, should probably spend a few months, or years, serving as an admin before they make too many criticisms. Anonymous person Sat Dec 06 23:22:15 GMT 2003 Seems that only admins have the time... ...to write thorough defenses of the fairly justified criticism here. Anyone who assumes they've got a rock solid lock on being "right" and the other guy being "wrong", isn't really getting the point. We all lose when it heads out the door to another country. wryobservation Sat Dec 06 23:40:56 GMT 2003 The writer's is not being a prima donna but straw men are easy to strike down Pulling the worst of the worst offenders from each of these administrative types doesn't make the entire industry worthless. I like to think I am a generalist (and I do know many people who can multitask just fine like I do) but at some point most generalist working in a team of generalist can see the soundness of delegating projects, clients or jobs to teammates in a way where the teammate isn't force to deal with non-related tasks all day. There's something to say for efficiency for specializing. If you need to, swap the job once in a while so that everyone is fully aware of how to everything. What keeps the day interesting will also prevent "tunnel vision". Li-fan Chen obiwantcp@hotmail.com Toronto, Canada Sat Dec 06 23:44:19 GMT 2003 ahem Lets be honest here.. Several aspects of what was set up was necessary.. Look at the dot com era see how many stories of clueless non it managment..would you trust them with unsecured full computer access to say a system like oh eBay? I have seen CEO's run companies into the ground because they insisted that they did not need a full IT staff that they could do it themselves.. Fred Grott badapple@netnitco.net Sun Dec 07 01:12:07 GMT 2003 Just sandbox the developers and make everyone happy I doubt there's ANYTHING that angers and frustrates developers more than having to rely on somebody else to do something they're perfectly capable of doing themselves but for some ***** bureaucrat standing in the way. To the admin who proudly watched helpdesk requests plummet after imposing standard configurations, I'd like to ask... what percentage of the pre-standard helpdesk requests came from developers? I'd guess pretty damn close to zero. I've NEVER seen a developer call for helpdesk support before doing everything possible to fix the problem himself. And it seems like 99% of the problems developers have in companies with nazi-like admins are a direct result of the admins messing with the developers' computers. How bad is it in some companies? Let's just say that at my company, the developers went out and bought their own hard drives and caddies for their laptops. Once a week, we boot them up with the corporate image hard drives snapped in and leave them running over lunch so the admins will get a warm happy illusion of control while the admin/maintenance app generates its weekly compliance report and applies all the Microsoft hotfixes that we ourselves had downloaded from Microsoft and installed 5 weeks ago... then promptly shut them down, put our own drives back in, and do productive work for the rest of the week. I have developer friends who work for another big company who all run Linux on their desktops PRECISELY because it makes them management-proof. The most ironic part is that most of them end up running Windows anyway... under VMware. Of course, sometimes it's necessary to find creative ways around admin roadblocks. I think the grand prize goes to a co-worker who wrote a client-server app that masquerades everything as http requests for script-free pages with lots of images. Big images. That really happen to be the aggregate data represented by a REAL page elsewhere (or a downloadable file linked to one), ready for re-assembly on the client side. Leave it running on a machine at home to which port 80 is forwarded by the router, launch the client on the machine at work & tell Mozilla to use it as the proxy, and life is good again. If the admins feel a legitimate need to keep the rest of their corporate network safe, fine. Get the developers a $50/month dsl line and put THEIR network outside the DMZ entirely. If it saves 90 minutes per month of admin grief and time due to fights with the developers, it paid for itself. Hell, ask the developers, and they'll happily chip in ten bucks apiece as a low-cost way to buy happiness and be left alone. Give them a machine to run their own database server on (to which the admins can periodically dump a representative subset of the data, possibly munged for security purpose). When they're ready to test, deploy it to a test server on the inside. High fences make good neighbors. Big guns just provoke an arms race pitting developers (who, by career definition can write their own custom apps to achieve their own agenda) against admins (who might know how to program too, but are usually too busy and have to rely on prepackaged solutions to which the developers already know the weaknesses). Jason jason@spamproof.net Miami, US Sun Dec 07 01:50:35 GMT 2003 "Sys Admins" Hee, hee, hee. These furious reponses from so-called "system admins" is just what I would expect from people that have built up a fantastic rationlization to conceal the fact that they are three levels removed from whatever it is their business does, and finally have a mirror held up to them. As a developer for, say, a healthcare organization, I can develop the coolest website the world has ever seen, and it will not make a single patient any healther on its own. Developers are always [at least] two levels away from something useful. I am sometimes troubled that in reality, my life's work will amount to little more than changing the alignment of some microscopic iron filings on some number of unseen glass discs. As difficult to rationalize as that is, a service admin is still another level removed. IT has created these great ivory towers, with CIOs, organizational charts, roles and responsibilities, and the unfailing SOPs to try to give the illusion that they are somehow a revenue center that they absolutely aren't. Even though I earn my living through IT, I applaud the downsizing and off-shoring. In my experience 95% of the people in IT positions would better serve society laying bricks, assembling machines, cutting hair or whatever. Anonymous person Sun Dec 07 02:35:55 GMT 2003 You lost me at "De-skilling[sic] of the workforce." I don't have time to wade through illiteracy. Maybe you ought to "skill" your english. Stupid ass. Pete G Switzerland Sun Dec 07 02:57:21 GMT 2003 No Talent Whiners the author hit it on the head perfectly but restrained himself from bluntly stating the real truth: most administrators are no talent a$sholes. reading the comments posted here from all these no talent losers is funny as all hell. i had to take a job as a sysadmin (for 6 mo) because programming jobs were scarce, and experiencing 1st hand what a poor poor job the guy before me did, and how little it takes to keep a lan/wan running - i just have absolutely no respect for the job - it's a glorified help desk position with admin acct rights. any a$$hole could do it. i believe msce stands for "massive scrotum connivance expert" clitlicker clit@licker.com nyc, usa Sun Dec 07 03:07:04 GMT 2003 Complexity has mandated these roles With the increased complexity of programmed systems, as well as the greater business dependency on day to day operations, it has become necessary for more (system, database, network, security, web, whatever) administation work to be done. In a very small business, the work level is so small it can be done by someone who does other things, such as programming. But at some point as you look at businesses that have more and more IT work going on, then you simply have so much administrative work going on that someone has to do it all day long. Or that might be many someones. What programmer wants to stop designing, developing, creating, and/or coding, and just do boring administrative stuff all day, every day, 5 days a week, 50 weeks a year? Enter the professional system administrator. People who for whatever reason do not do programming might very well be happy doing administration instead. Better them than you (the programmer), right? Is that what you want? Think about it. I happen to be a rarity ... a programmer who prefers to work professionally as a systems/network administrator. A friend of mine is in a similar capacity but he prefers being a database administrator. I do know programming, but my decision to do administration professionally is more due to the business environment programmers face as compared to that which administrators face. What I do see in my system administration work is that the amount of work needed to configure and deploy software in a production business environment has greatly increased. Some of that, but by no means all of it, is due to the complexity of the software itself. Many systems require so many decisions by administrators that at times it can be a day-long chore just to get something installed. And there are business needs that increase this workload, too, such as the need to make sure systems stay up 99.999 percent of the time (that's no more than 5 minutes outage a year, including maintenance windows). Other business aspects including coordinating perhaps thousands of servers running in a network to keep everything in sync. Administrators have to consider issues like making sure backups are done, backups can be recovered from, and filesystems are never full and always optimally tuned. And don't forget all the procedural documentation that has to be done. Life as a system administrator can be very intense. Unfortunately, there are also many "low performance" system administrators out there, too. You may well have run into those. Not all system administrators are like that, though. Can anything be done about it? Although increased software complexity has added to the problem, eliminating it won't eliminate the problem since business process complexity is there, too (and plays an even larger role in most cases). What I suggest to programmers and especially development teams is to find a good system administrator who understanding the programming functions while preferring to do system administration, and arrange to work with them (such as hiring them into commercial development teams) for a special role of evaluating system administration functionality of larger software projects. They should work in the design cycle as an analyst (analysing administrative and business impacts of how systems are organized), as well as in a testing role. If you are doing development with Extreme Programming, they would just being doing both of those things all the time until you have a finished project. Phil Howard cqu77448@vcny.arg.rot13 West Virginia, US Sun Dec 07 03:38:54 GMT 2003 Its a two sided street This article hit many truths for me. Ive seen all this come true and scrathed my head in bemusement at times as well. Ive been in this game for 17 years now. Mostly in System Engineering and System Admin roles of different types. I programme for fun in my own time. Yes often coder's do seem to bear the brunt of many disgruntled admins. Sometimes for their stupidity, others for just being in the wrong place at the wrong time. Yes corporate's are now dogged in admin and project do suffer a huge deal because of it, however it's all got to be taken with some view of what is best for the company and the context of it. Sometimes the admin's themselves are to blame for being a bunch of slackers that hold jobs of seemingly importance and power who are so "run off their feet" to respond in a timely manner or be helpful, but in reality surf the Net most of the day and do little until backed into a corner and yelled at. And other times its the developers that went off and wrote the most fancy all-singing and dancing app without thinking (or sometimes even knowing) where the app will run, what resource restrictions it may have, or worse what environment it will run on and impact on. I dont have the answers but this article sure did point to many of the truths that exist in the real world (at least the part of the real world that I work in). In the end I think its a meeting of the minds that smooth the process of the complex corporate domains where understanding and simplication of the problems would go a long way to helping transition talented ideas from development to production. Chris chris@renovo.co.nz New Zealand Sun Dec 07 04:06:45 GMT 2003 Outsourcing is fun Um. This sounds suspiciously like you worked for companies that have no idea what they are doing. I regularly deal with dev folks and don't have any problems with them. Or with the DBAs, Firewall, backup, etc. Perhaps the solution is to have companies that do IT actually DO IT. It seems crazy to me that medical companies, car companies, etc have internal IT departments. They don't generate their own power, they don't make their own pencils, they don't build their own desks. Companies need to concentrate on their core competencies and "outsource" everything else. It sounds like you have contracted for companies that don't do that. Well, ok, those companies are inefficent and will either grow up or die. Its capitalism man, not a socialist dictatorship. The Army shouldn't run everything. andy andy andy@yeah.com CA, USA Sun Dec 07 04:28:27 GMT 2003 Of course that project is the only project, right? This is more in response to: JK_Cole@hotmail.com and charging back 'lost productivity' to the administrators. I see a problem with this -- namely that the systems administration team is generally responsible for ensuring acceptable operability for all projects and functionings of the company, not just your little corner. I have been in the position, standing as gatekeeper against different development projects who wanted to introduce their changes on production systems. A good many times, the changes would have broken other projects, interrupted systems operations, or something even worse. There are reasons for the standard build, test, deploy structure. There are reasons in a good many areas that developers are not allowed priviledged access to the production systems. There are reasons for change management systems. There are reasons that the answer you get is "We'll see if we can get to it during the next maintenance window.' We're not just responsible to the team producing widget A. We also have to support widget B, C, X, Y and Z, accounting, finance, helpdesk, sales and marketing, and management. So we have to be careful. Anonymous person Sun Dec 07 04:50:32 GMT 2003 The real problem is optimization of non-critical resources The frustrations voiced in this article are caused by a company's desire to manage costs, and in so doing, optimizing the wrong thing. Have you ever seen a $90/hour developer sitting on her hands for a day because the company has chosen to optimize the time of the $30/hour support staff by keeping their support queue full (and response time long)? Optimizing the wrong thing. How about a way-too-tight schedule being squeezed even tighter because the company has chosen to squeeze 3 projects onto one test box because an additional box would cost $10k? Sound reasonable? Not if you count the cost of capital going into the project every day that it's late and not delivering revenue. Optimizing the wrong thing again. Why can't Jane work on the required documentation that is holding up the entire project? Because Rajiv is using the documentation tool, and the company doesn't have any more seat licenses. Let's make the project more late and cost the company thousands a day, all to save on a $500 license. Optimizing the wrong thing yet again. If companies were paying attention to the cost of projects being late, in capital costs, in lost revenue, etc., they'd think differently about how they optimize. Now, maybe there's someone thinking about that. But it isn't the manager in charge of support -- he gets rewarded for supporting more boxes for less money, and that happens by keeping the in-box full of work so that none of the support staff are ever idle. And it isn't the manager in charge of purchasing hardware -- she gets rewarded for spending less money. It's called optimizing for the local process at the expense of the global, and it's what causes the kinds of frustrations voiced in the article. It's not the programmers' fault, and it's not the support staff's fault (although they can make the problem better or worse by how they chose to work with each other). It's a problem that derives from how the department heads are rewarded. It's the assumption that by optimizing every little thing, when you join them all together, the whole will turn out good. And that's not correct. There are many situations in business where you must purposefully make something you do less efficient (at least when seen through a microscope) so that you can, as a company, make more money. If we actually gave the support staff enough people, then they wouldn't find they need quite as much of the control they feel they have to have to keep their heads above water. If we actually got enough hardware, we wouldn't have projects sitting idle (with programmers working hard at looking busy) waiting for their test box to become avaialable, and we wouldn't have sysadmins feeling like they have to meter out the resources like water in a desert. Wayne Conrad wconrad@yagni.com Phoenix, AZ, USA Sun Dec 07 06:27:26 GMT 2003 Levels of Admins I do see where alot of his issues come from. I am, myself, a Unix Systems Administrator. (I can do Windows, but choose otherwise.) And, having been through 10 years in this field, I have seen what the "dot com" era has done to it. At the peak of the "dot com" era, the industry could not get enough IT warm bodies. They were sending truck drivers to trade school and hiring them as junior to senior administrators, when they were not. (No offense to truck drivers.) Now with the bubble popped, all of those admins are out there, getting into real companies. I knew of a couple of positions at Google, that went unfilled for along time. This was because, when they list the opennings, they would get about 8 thousand resumes per openning. They would cut that down to about 50 to phone interview, then about 25 for personal interviews. When all said and done, it was mentioned that only 3 were actually worth bringing in for personal interviews. Right now we (sysadmins) have a glut of level 1 admins calling themselves level 2 or 3. And, I believe that things will get better as these people get weeded out, and they take a new career path. This may take awhile. This issue is just making things alot worse for people in a career, where you do things right and you're invisible, do things wrong and the world knows. Robert Lanning lanning@seagate.com San Jose, CA, USA Sun Dec 07 07:02:32 GMT 2003 Sys Admin As any sports team will tell you, the the team should work fluidly as a whole. Developers, Sys admin and business unit should function as one so everybody wins. As whether developers or sys admin are better, it depends on the company's business; if the company develop software as a business, then the developers rule but if the company is a non software company then sys admin rule. But the essence is that both skillls are essential in the IT world. jmm2 jmm2@linux.org Cambridge, US Sun Dec 07 07:43:17 GMT 2003 Examining the wrong problem? Maybe the problem isn't that ALL sys admins are bad, or that ALL developers are bad. What's interesting is that the author brings out worst-case sys admin examples in order to form an image of the general administration picture. Then again, all the admins that replied did the same thing. The problem might just be that the developer or admin in question encountered someone of little skill in his or her trade. The developer screwed up a production database? He or she should have been thinking of the consequences of their actions, or, if they didn't know much what they were doing, their lack of expertise in that area and the cost to the company of tooling around. They should have asked a DBA for advice. It seems that all the moaning in these posts comes from two things. First, people in positions where they lack the skills (whether it be technical, or interpersonal) to effectively work. In other words, perhaps the argument is the same presented by both sides: The computer industry has become saturated with under-skilled workers. However, there are less than skilled workers in many trades. How many times has a construction company not properly built a structure, or a healthcare worker cause an "accident" due to negligence? In many fields, a good, intelligent worker is rare. If they're rare, everybody can't have them. That leaves the rest of us with the less skilled. Second, the general mentality, it seems, of "I know best". This example (and I am mindful that this is possibly a worst case scenario) perhaps illustrates, to some extent, the general mentality of many IT workers. I developed a system, and a customer asked their "hacker friend" to test out the security of the system. The friend's response was "It's okay." When asked what a perfect system would be, he replied "I don't know because I haven't designed it yet". This person's mentality is "I know better than anyone else". I see this as a fairly common mentality in the computer related industry. Is it not presumptuous for the friend to think that he could design a perfect system security wise? Is it not presumptuous of the friend, having never met me or talked to me, to automatically assume he was more intelligent and knowledgeable than I? The friend truly believes that if he worked on it hard enough, he would be able to do it right, and better than me. The developer truly believes that by tooling around on the database, he can "figure it out" and the admin truly believes that by setting the virus checkers to scan every hour, he will prevent viruses. Each is operating in an area not of their expertise in good faith believe that they are intelligent enough to find a best solution without consulting somebody who is directly affected by that action. Then again, might this not all come down to the fact that we are humans and not machines? Might the author have rubbed some DBA wrong on a day when he found out his 16 year old daughter was pregnant? Might the author even have run into a admin who simply had bad personal traits, such as a penchant for vengence or laziness? And generalizing this again, might we not add that any given human, no matter how mature, is not a machine, and operates irregularly? Humans have feelings, emotions, egos, vices etc. All of which might be considered "shortcomings" by people who work with regular computer all day long. Which brings us back to the point. Perhaps this is why this article was written by a "computer person" and not a construction worker. Perhaps the real problem is that "computer people" as a whole perhaps lack the communication skills to properly interface with each other. David R. Pittsburgh, USA Sun Dec 07 08:23:17 GMT 2003 Accountability I think most of the separation of admin and development responsibilities is made by management types who usually do not have any experience in either. The reason is so they can manage blame (one of the key objectives of management). The managers that oversee the development groups play the "cover-my-ass" card by giving control of the systems to the sysadmins. That gives them someone to point the finger at when things go wrong, even if they are at fault. The administrators, on the other hand, have two sometimes-conflicting goals they must balance. One is to keep the systems up, and the other is to provide the services needed by the users/developers. When systems go down, they can blame it on the "extra services" the developers required. I think the following solution would be fair: Developers can run their own systems, as long as they understand that they will be fully responsible for the them. That means installing, configuring, securing, backups, troubleshooting, and any other required tasks. If something goes wrong, they are not to look to the admin group for support. In turn, if the developer proves incapable of running the systems, then they lose the priviledge of being able to be their own admin. Jason Austin, TX Sun Dec 07 08:53:15 GMT 2003 Preach it brother! Greetings, http://www.m-w.com/cgi-bin/dictionary?book=Dictionary&va=de-skill How about Mirriam Webster's definiton of de-skill? How about you improve your own language skill by reading better authors, hmm? In any case, this article hits it right on the head, from developers who AREN'T ALLOWED to write efficient queries against the real database because it takes a 5-day process to do an 'explain plan' on an SQL statement versus the live site, to the 7-10 day process to get a program that is critical to development (and for which we have a site license) onto a new developers machine. (Shouldn't it have been on their machine as part of the standard install that the IT team does? Well, no, because they decided to use the same software setup for (again) administrative assistants and top-notch developers. Every developer in my company is migrating to Linux as their main platform because (1) we develop a Linux product, and so it's justifiable, and (2) the IT people are NOT ALLOWED to do Linux administration. Every single developer in my group knows more about Windows and Linux administration than the combined knowledge of the IT staff, having administered their own machines in startups and medium sized companies where the cost of 'IT' is a burden unbearable. It's only when someone is hired in as a 'CIO' to please investors, or similar crud, that an 'IT' group is started, and productivity rapidly starts dwindling as developers are less and less able to get the information that keeps a company alive. This article hits the nail on the head. While it will be denied up one side and down the other by the IT people whose ox it gores, it is exactly true. Skyrocketing development overhead is the crippling and previously unspoken cost to abusive and intrusive systems, network, and database administration. One perfect example is the mandatory use of Outlook in my company as the email client, and intrusive anti-virus software. Of course, the only reason for needing the anti-virus software is because Outlook propagates viruses like sticking your hand down the throat of someone who just died of Ebola... As a former professional anti-virus author, I know better than to run Outlook, and therefore I am comfortable permanently disabling the anti-virus software. To permanently disable it, I had to kill Tivoli, which is a remote-control software. To kill Tivoli I had to gain Administrator access. That took about 5 minutes, since I had physical access... Only an idiot uses Outlook (the single most insecure email client!) for corporate email, and only the prince of idiots mandates it company-wide. Where does the mandate come from? The CIO, of course. I have met some brilliant database administrators, but for my company, their management handcuffs, trips, and steps on them, by making them only able to respond to developer inquiries through lengthy, five-day turnaround 'official requests'. This means we can't find out the potential performance of a query on our live site without a crippling time delay. When the DBA's sat next to the developers, and our management was the same guy, we could bop over, and sit with them, trying variations until we found the best queries. They were HAPPY to do it, because they knew it meant the site got good queries. Now, instead, we have had some of the worst performing releases ever, because of the artificial seperation of the groups. Network administration is sometimes good, sometimes bad. They are ABSURDLY bad when they change firewall configurations without asking or informing ANYONE who might be needing that firewall configuration to talk to a third party. When our network connections start failing arbitrarily, it's about 75% of the time the fault of some twit who thought, 'Oh, hey, this port is open, maybe we should close it...', without wondering if maybe it was left open for a PURPOSE! This article speaks the truth, although not as well as I'd like. It focuses on Java development, and whines a bit much in places. The fundamental problem is that servants of the system have believed they should take on the role of masters. The goal should be to keep things running smoothly, not to build a power base over as many resources as possible. If a developer can do the install of the software package today, instead of in two weeks when the IT guy can get around to it, then the developer MUST be allowed to. It's not a flipping union job, folks. Installshield makes it pretty damn simple. That's the thing that has been lost, the recognition that * administration is about providing resources, not hoarding them, not doling them out miserly so that people will be grateful for the scraps. The goal is to keep the company running smoothly, and work happening. Anything that interferes with that, is absolutely, without question, a broken process. I am responsible, if I put a program that behaves badly on our network. Equally, the * administrator should be responsible if their response time holds up work. There must be strong consequences in both directions. Don't stop me from putting a program on my machine, but if I screw up, punishment should be evenly handed out. If you do that, you should find, at least I have in all the companies I've worked for, that the developers cause far, far less problems than the process-laden system, network, and database administrators. This may be because I have been lucky to work with exceptional developers in all 8 companies over the 15 years that I've been a professional programmer. I strongly doubt that is the sole reason, however. A frustrated developer San Jose, CA, USA Sun Dec 07 09:18:43 GMT 2003 Interesting article Not a completely bad article, and it outlines the divide that IT departments now suffer badly from. Mainly due to a lack of managerial understanding. Well, after working in a very demanding environment as a senior sysadmin, I saw some pretty disturbing things... Two of our Administrators had entirely the wrong attitude, one regularly called the Developers idiots/morons/timewasters behind their back, and would denigrate them at every opportunity in front of our team; and he was most senior in the team. He also considered himself the Prima Donna of the outfit, he could not be questioned, he NEVER made mistakes (only f*ckups), and breaking one of his myriad self-apointed laws which he himself often failed to follow, was a disciplinary offence. The other basically followed his lead, and was so lacking in administrative and general computing experience should really not have been there. His attitude was also holier-than-thou, whereas his skillset was laughable, and an accident waiting to happen (well actually he had a few accidents). On the other had a very large department of developers from COBOL to J2EE with a wide range of skills, ranging from excellent to middling. We had to deal with bad code changes, which were often put into production without enough examination by the Sysadmin mentioned. This lead to major system problems like running out of resources. Exceeding process limits as one example. Ah yes.. the late night support calls... What I really noticed, was that the previous department, (which had walked out completely due to lack of payrises), had left things in quite a mess indeed. Secondly, the consultation between Developers and the Systems department had broken down to the point of being almost non-existent, whereas previously there were strong bonds which helped forge the original complex operating environment, and kept it in check. Once the simple act of Systems meeting with Development both at a team and managerial level broke down, all fingers were pointed to change control systems as the saving grace. Of course with no idea of what Systems expected or what Development had in mind, completely flawed concepts would be developed into reality then attempted to hack them into shape to put them into production. Departments seriously have to look at how they MANAGE their resources, environment, people effectively, and communicate between departments. Everything else is secondary in IT-land as if your people are clueful they can address any other issues from a technical standpoint. Saying, "Let's fire all the administrators..." is not a decent answer to this one. One could very well say the same about developers when we all know it's just a few individuals causing all the issues. The trick is to hire the right people in the first place, with BROAD knowledge, without massive personality defects and a God-complex, who can work effectively and remember that Developers/Users are their clients and come first. Stop hiring people who's primary interest is in making pots of money, they have no passion for IT, or care for the people in it. Anonymous Coward Australia Sun Dec 07 12:09:25 GMT 2003 Some of those practices are appropriate... ...when done right. Like having all files stored on the file server rather than locally; intelligent file sharing services (like AFS) will keep a local cache with copies of all files a developer is actually using, so that you don't have the performance penalty of transferring a file over the network every time it's accessed. Keeping most developers away from root on their machines is, sadly, a Good Thing. Recently I've had to deal with a department head enraged at IT because one of the developers, abusing root privileges, left the machines of several folks in her department (whom he was supposedly helping) in a state where rebuilding said machines would wipe critical data -- something which could not have happened if he hadn't had those privileges and the system configuration had, as it should have, forced all data to be stored in either (1) a location automamtically backed up, or (2) a location manually transferred on rebuilds. Re the software licensing bit -- I'm sorry you don't like it, but dealing with licensed software is a pain in everyone's behind. We're mostly a Linux shop, and dealing with the Win32 machines (and their licensed office software, and their licensed virus checkers, and their licensed development tools) is a PITA -- and if we let folks do software installs on their own, they *do* perform unlicensed installs (heck, one of the VPs is king of that practice). If we get audited and unlicensed software is found, people get fired to pay the incurred expenses -- simple as that -- and those firings would go well beyond IT. Beyond that, a fair number of the points in this article are reasonable -- I, like the rest of the IT department, have worked as a developer with DBAs and sysadmins as incompetant as those mentioned, and perhaps that's part of why I *do* consider the developers my first set of customers (this often to the chagrin of managerial types who get to wait in line). Just because you don't appreciate them, however, please don't label all IT practices as wrong. Charles Texas, US Sun Dec 07 15:44:24 GMT | ||||||||||||||||||||||||||||
Slower development times
![In Search of Stupidity [amazon.com]](/lifecycle/images/book_stupidity.jpg){kind=small}
![Fd Companies [amazon.com]](/lifecycle/images/book_fdcompanies.jpg){kind=small}
![Oracle 10g [amazon.com]](/lifecycle/images/book_oracle10g.jpg){kind=small}