True, SOA (service oriented architecture) builds on the stack of protocols that define Web services, but it is hardly limited to that stack and draws as much on time-honoured notions of business "re-engineering" as it does on XML, SOAP, and WSDL. Simply put, SOA is a broad, standards-based framework in which services are built, deployed, managed, and orchestrated in pursuit of new and much more agile IT infrastructures that respond swiftly to shifting business demands.
The breadth of that vision is what makes SOA seem so maddeningly vague. Nonetheless, the potential benefits of reduced IT costs and greater business agility have spurred many organizations to start down the path to SOA, to the point where most large enterprises now have some sort of SOA initiative under way. One reason for that extraordinary traction: SOA may ultimately have a transformative effect on the entire enterprise, but in contrast to other "big bang" endeavours, most of the applications and infrastructure you've already deployed can remain in place.
Throughout the past two years, -- we have interviewed countless enterprise architects, developers, and officers who are guiding their organizations toward SOA deployment -- and who are learning hard lessons, gaining insight, and encountering infuriating technology gaps along the way. Many are already enjoying SOA's early benefits of easy integration and code reusability. Based on their experiences, and the advice of industry technologists and analysts, we offer this step-by-step guide to planning, building, deploying, and managing an SOA.
As you'll see, SOA provokes many of the same questions that dog most grand IT schemes. Should you buy and deploy SOA-related technology from a single vendor with which you already have a close relationship, or should you mix and match best-of-breed solutions? And, as with any standards-based initiative, what do you do when many of the standards necessary to achieve the real benefits aren't fully cooked yet?
Such questions lack easy answers, and missing pieces of technology, industry disagreements, and vendor lock-in all threaten to dampen SOA's much-ballyhooed benefit of hyper agility. Nonetheless, you'll find most of the key concepts underlying SOA, a number of which may be familiar, right here -- although not necessarily in exactly the right order for you. Just as with SOA itself, how you put it all together depends on what you've got and where you want to go.
1: Think big, start small
SOA starts with a business promise: to enable enterprises to re-engineer themselves on the fly. From the outset, look for opportunities for agility. The more dynamic the business, the more it will benefit from a well-implemented SOA. And the more allies you have who share the SOA vision, the better. In particular, it helps to have powerful partners in your company's business management who understand the ultimate payoffs of cost reduction and accelerated response to change across the organization.
"We're actually kind of fortunate in that we don't have to sell SOA," says Ben Moreland, assistant director of foundation services at The Hartford, which got its SOA initiative rolling a few years ago. "Our senior vice president, John Chu, recognized the benefits and the value of SOA."
This shared vision may be vast, but it pays to start small. "Don't try to do 'boil-the-ocean'-type projects," advises Ed Horst, vice president of marketing at AmberPoint, who has watched overly ambitious initiatives falter. "I think the most successful initial projects we've seen are those that are small in size -- about six to 10 services that integrate two or three things and take around six months to complete."
Many organizations start by provisioning a few mission-critical legacy applications as services, providing access to important data and functionality to other applications for the first time. Or they use shared services to eliminate redundancy among several difficult-to-maintain stovepipe applications that overlap in functionality.
Such projects may yield significant benefits, but SOA delivers the most value -- and will scale far better in the future -- when you begin by drawing a box around a set of related business processes that need streamlining, rather than attacking technology problems first. Scott Thompson, senior architect at H&R Block, puts it this way: "We had to switch our mentality from just rendering data and just making a service out of it because we could, to asking: what's the business problem that we're trying to solve, and what applicability does that business problem have to other areas of the organization?"
Jean-Michel Van Lippevelde, business architect at Accelior Consulting, has reached the same conclusion. "Take a top-down approach from a business-process perspective," he advises. The results can be highly visible, as they were in Accelior's engagement with ING Lease Belgium, which targeted a request-for-quote process that included automatically generating contracts. Before, the process typically took days. But after streamlining the process, provisioning services, and automating formerly manual steps, the wait time was reduced to minutes.
2: Go to the whiteboard
You can't expect to dissect business processes and see what makes them tick all by yourself. In collaboration with business stakeholders, review and rationalize the processes in the domain you've identified. Often, much of the heavy lifting will be done by the business guys, as they scratch their heads and figure out how to restructure processes -- and you determine the breakdown collaboratively and decide where new automation can make a difference.
"We started our process mapping by meeting with each agency individually," says Dan Thomas, director of the DC Stat program at the Office of the Chief Technology Officer in the District of Columbia. Thomas' ambitious program is tying together the data repositories of 65 separate government agencies to give senior officials transparent, up-to-the-minute information with which to make policy decisions.
Thomas' team met with each agency to map out its data-gathering policies and to find out how that data was distilled for presentation. "It's a time-consuming process," he says, "but we didn't try for all 65 agencies out of the gate. We pared it down to a manageable number. Now they're getting in line for our attention."
Timothy Vibbert, senior systems engineer at Lockheed Martin, takes a like-minded approach. In providing professional services to government agencies, he's logged an impressive amount of whiteboard time. "We go through full domain decomposition before we even think about services," he says. At the same time, "you also start looking at what is out there that you might be able to reuse. And then you go into mission or process identification, getting down to a specific thing you want to tackle."
After the scope is defined, Vibbert says, it's important to determine who the participants will be. "And once you define those, you start building use cases. Then you start decomposing the use cases. And then you get into resource allocation and data allocation, start naming your services at a high level, and also talk about the data that flows," he says, emphasizing that these steps are iterative, with multiple passes required to create the right plan of attack.
Every SOA initiative is unique, so the duration of the planning process varies wildly, but Vibbert provides a hint of how long it might take: "If you already have some pieces of an SOA in place, you can trim down the time frame relatively quickly. An SOA from scratch -- it could take months if not longer."
3: Survey your surroundings
Here's where all that process work starts to meet technological reality. Before you implement, look carefully at what you have in place to leverage. A basic tenet of SOA, particularly in its early phase, is to work with what you've got when possible but to avoid locking yourself into practices or technologies that will stymie future interoperability or expansion.
Taking inventory is a multistage process. First, you need to document the data sources and existing applications that will be involved in your initial deployment -- remembering to identify partner services outside the firewall that you may need to connect with and to catalogue those services as carefully as you do internal ones. Second, take stock of technology you have on hand that will play a role in your SOA. Yes, this is a big job, and no, it's not necessary to complete it before moving towards an initial project. But neither can it be ignored if SOA, rather than a limited project, is your goal. An SOA involves a sprawling set of technologies. The short list: tools to build or provision those services; a registry in which to expose them; a messaging infrastructure over which services and applications will communicate; a means of orchestrating services; and some sort of services management involving intermediaries. Application-layer networking may also play a role, and down the road, so may BPM (business process management) and BAM (business activity monitoring) applications. You'll also want to take a hard look at the Web services interfaces of your commercial enterprise apps.
That's quite a stack of stuff, but you don't need to make sweat-inducing technology decisions about what you'll change, add, or keep quite yet. You'll be busy enough figuring out how to map and normalize data among the systems involved. As Timothy Vibbert of Lockheed notes, data among various systems can be "defined 15 different ways, 15 different times for the same data element". Reconciling that metadata is hard, tedious work. If you're not an SOA expert and are leery of hiring a consultant, don't despair. There's no need to run to the learning annex for a crash course. Get as far as you can. If your enterprise consists of little custom code and mostly off-the-shelf software, contact your software vendors one at a time. Ask about their SOA plans and capabilities. Often, you'll be pleasantly surprised by their direction -- and you may obtain valuable information that will affect project scheduling and future platform choices.
"We moved our product portfolio towards SOA specifically because our customers asked us to," says Dwain Kinghorn, CTO of Altiris, a large manufacturer of asset, network, and security management platforms. "It allows our customers to free themselves from our management consoles. They can now grab specific pieces of management data and incorporate those into any SOA-based management dashboards they may have developed on their own."