One reason for failure is a complete misunderstanding (or, more charitably, underestimation) of the complexities involved in a development effort.  This is a crucial issue.  We rarely appreciate how complex computer software really is.  According to Jerry Weinberg, modern software systems are the most complex things ever built by human beings. (Weinberg also said that if carpenters built houses the way programmers write programs, the first woodpecker to come along would destroy civilization.)

Software projects weren't always so complex.  A few decades ago we were writing programs to solve specific logistical problems such as payroll, receivables aging, order entry and so on.  In the process control world the programs involved logic to open and close valves or logic to adjust a control surface or activate a servo of some kind.  So that is what we did.  We developed independent programs to solve what we saw as more or less independent problems.  Information technology had a narrower purpose in those days, one that  was easier to grasp.

Since then, the fundamental problems that we concern ourselves with have changed.  As time passed, information technology began to assume responsibility for groups of programs, or application systems.  Instead of payroll we started implementing human resources systems which included the same basic data but extended the scope to include other programs using overlapping data.  Instead of order entry programs and receivables aging programs, we started dealing with systems that include both and others as part of a single application.  We started developing process control applications that managed entire power trains in automobiles, processes in chemical plants and avionics subsystems in aircraft.

Now, we are beginning to work with systems of systems that integrate applications based on shared data requirements.  We can no longer deal with order entry applications separate from inventory, warehouse control and production applications.  We must not only incorporate these varied applications into integrated ones, we now have to go outside the immediate organization and include the entire supply chain.  We treat an entire automobile as a network of process control nodes and include external navigation and traffic data besides.  This extension of scope requires us to concern ourselves with issues not previously considered to be in the domain of information technology, namely the organization itself.  You cannot seriously address corporate information requirements without paying careful attention to corporate policies that affect the data and everything else.

Each of these changes in scope has increased the complexity of the problem by at least one order of magnitude.  Yet, in many cases, we are attempting to solve modern systems implementation problems as though nothing had changed.  We still try to use the brute force techniques that more or less worked when we were faced with the very different issue of writing stand alone programs to solve simple logistical problems.  Steve McConnell calls this approach 'code and fix development'.  It used to work sometimes; now it works hardly ever.

Some organizations have recognized the problem and developed more rigorous approaches to dealing with the complexities of systems development.  Typically these are called software development methods rather than systems development ones but they do represent an improvement over the code and fix approach. 

Methodologies
The notion of formal methodologies first arose when we started dealing with 'application systems' as opposed to single programs. The need for something to help manage the growing complexity was evident to those responsible for building systems for quite a while before it was brought to management's attention. The rationale for formal methodologies was supplied by Richard Nolan when he described six phases of data processing maturity based on how an organization managed its information. The first two phases are 'initiation' and 'contagion' which are accurate descriptions of the rush to automation.  The third phase is 'control' which describes the company's reaction to out of control costs and schedules. The idea here is that companies  need to be able to be able to control the costs of software development.

A number of vendors developed packaged methodologies in response to the perceived need for control of the software development process. This started in the late seventies and it is still going on. The early methodologies consisted of large sets of manuals containing numerous detailed procedures, tasks and forms whose avowed purpose was to bring order to the process of software development.  Some of the methodologies proposed that you pick and choose among the options available depending on the perceived complexity of a particular project. Some of the methodologies died, others evolved into computer aided software engineering (CASE) tools and some are still going pretty much as they were initially written.

The Datamorphosis Approach
There have been problems with the packaged methodologies and with the CASE tools.  Most of the problems have to do with resistance, tedium, inaccessibility or all three. Our classes and our consulting address these issues by emphasizing the positive aspects of methods and tools and by providing a path to integrate improved methods into the client's existing development context. In addition, a technique that has worked amazingly well is to publish all specifications from all software tools using Microsoft Word or Adobe Acrobat. This solves the accessibility problem and permits the continued use of various different tools for different kinds of problems. It also helps integrate definitions from all tools into a common shared dictionary.