Data and application integration, in which one makes things work together which were never designed to work together, provides rich opportunities for problems to lurk and miscommunication to thrive. That means that data integration can easily have a higher problem rate than may other things. That is our first “why” – we expect these projects not to go well. That can mean never even trying to fix the root causes of these problems.
Our second “why” is related to thinking of integration as a sequence of projects instead of an entity in its own right. Let’s contrast this with ideas from agile development in which the state of an application or product is moved forward through a sequence of mini projects, where each project results in a successful result. Agile development requires tooling that enables analysis, refactoring, code review and much else. Organization-wide agile data integration needs similar tooling for the overall integration to get better. For instance, integration projects have interrelated dependencies that are often not discovered until late in test cycles, driving up cost and creating delays. If integration is purely project-based, then there is no organizationally acceptable solution, since projects don’t have any incentive to solve future problems.
There are other “whys”, but these three address the very way we think but data and application integration. These three are independent of technology (ESB, cloud, web services, etc.) and methodology (SOA, RESTful services, publish-subscribe, etc.). There are times when what is tacitly known and commonly accepted is an excellent way forward. But there are times when, as Socrates commented, “an unexamined life is not worth living.” Or, at least, a work environment with unexamined assumptions may not be worth living in.