Vendors from both the integration world and the application development world are tackling the collaborative applications challenge, building new middleware layers they hope will become platforms for the development of these next-generation enterprise applications. Their efforts will ultimately move the enterprise beyond point-to-point integration using proprietary APIs and toward a standards-based, loosely coupled, business-rules-driven software layer that enables enterprises to stitch together existing systems components into collaborative applications.
But several thorny problems need to be solved to make collaborative applications work, including data transformation, business process coordination, and transactionality. Current efforts, based largely on existing open-system interfaces such as XML, are a transitional step toward a more services-oriented approach once Web services interfaces and protocols such as SOAP (Simple Object Access Protocol) and WSDL (Web Services Description Language) become widespread. Here's a rundown of some of the key challenges.
Data transformation and synchronization
Although many legacy systems and packaged applications speak XML, their data models and schemas are frustratingly different. The definition of an "order" or a "customer" may vary greatly, for example, among an SAP AG system, a homegrown order management system, and an SCM (supply-chain management) system. Moreover, many enterprises don't maintain comprehensive data dictionaries except in the heads of the people who originally wrote the systems' code. So business vocabulary management -- the mapping and transforming of data from multiple systems into a single consistent schema -- is one of the tougher problems facing collaborative application platform developers.
New software approaches for rules-driven data cleansing and mapping, as well as dictionary definition, are under development. Vitria Technology Inc., for example, is developing a vocabulary-based transformation approach with a correlation engine that can scan through thousands of documents, each containing thousands of terms, and draw rules-based mapping inferences that can then be approved or rejected by a human analyst.
A related issue is the management of data scattered over multiple origin systems. How, for example, should a company consolidate data into one logical view with a unified architecture, thereby enabling data-source independence? Because application data continues to live and change in the origin systems, the new software layer must be able to retrieve origin data on the fly and also propagate changes back to the origin systems.
"Our intellectual property is all around dynamically aggregating data under different circumstances," explains Rob Beauchamp, CTO of Austin, Texas-based Journee Software Corp., which is developing collaborative middleware focused on CRM apps. Journee provides tools to allow enterprises to define the logical data model of their middle-tier software layer. "It's acting as if it is a database, but it's really not. ... It's really retrieving data from the underlying data stores at run time," he says.
Other vendors, such as Asera Inc. and WRQ Inc., are using structured XML schemas based on the World Wide Web Consortium (W3c) XML Schema standard to provide this unified view, exposing a common aggregate interface to the collaborative application and moving around XML documents with defined semantics, such as validation rules, rather than the underlying data itself.
Although raw XML lacks the more abstracted services-oriented interfaces that will come with SOAP, and thus requires more custom development, it reduces a company's dependence on scarce Java programming talent. "It changes how people do development," says Steve Benfield, CTO of Billerica, Mass.-based SilverStream Software Inc. "It doesn't require a severe technical skill set, and I don't have to think about how I'm going to move data around. ... The tools around XML do most of the work."
Web services protocols would make collaborative application development even easier, providing the conventions and formats to define interfaces, messages, data types, and structures; to map messages onto different transports; and to format and store metadata. But in the meantime, enterprises and their vendors can roll their own using XML.
Managing process flows
Another obstacle to collaborative app development is the lack of standards for describing and orchestrating business process flows across multiple systems. "You're basically stringing a whole bunch of applications together now instead of writing them in a more monolithic way," so issues of modeling and process flow become even more important, Benfield explains. "How do I handle retries and time-outs, and what's my quality-of-service agreement?" he asks, noting that in the absence of standards, most enterprises are just winging it, especially for collaborative applications that reach outside the firewall.
Furthermore, collaborative apps are likely to be long-running; that is, they have several steps that play out over time and potentially include human intervention as part of their workflow process. "Everybody has their own way of defining their own processes and interfaces," says Anil Nori, CTO of Asera in Belmont, Calif. "There is really no workflow standard right now."
Developing collaborative applications still involves a lot of application-specific human engineering: talking directly to the participating applications' developers and finding out what the app can do, what kind of data it can act on, what its security and authorization model looks like, and so on. Only then can common rules and business processes be built atop participating systems.
A raft of proposed standards is in the works to solve this problem, including a host of vertical-market efforts. Microsoft Corp.'s XLANG, IBM Corp.'s WSFL (Web Services Flow Language), and BPMI from the Business Process Management Initiative are three closely watched contenders. Similarly, ebXML (e-business XML) has a business process component, but it is based on an object model and thus lacks abstractions for loosely coupled processes. "I'm just waiting for something very simple, like XLANG, that also makes sense conceptually, like WSFL," says Eric Newcomer, CTO of Iona Technologies PLC.
Until an overarching standard emerges, the tools for building collaborative apps will be targeted at developers rather than the businesses users who best understand the business processes. "Now we're going from programming languages to markup languages to representing the way people think about the world," explains Newcomer, saying that a key to collaborative application platforms will be their capability of allowing users to debug processes at the visual abstraction level rather than the XML/script level.
A third challenge will be providing support for transactions across multiple back-end systems. "The hard part is getting a transaction model wrapped around those back-end systems; so if it didn't work in the third system down, it was able to roll back in those first two systems," Journee's Beauchamp explains.
Transaction definitions must also be consistent with process flow definitions so that transactions can be rolled back to the appropriate level of granularity in the process, which in turn must be tied to the level of granularity in the security model. One approach is XA, a proposed W3C standard for putting locks on systems while doing transactions across multiple systems to assure transaction integrity. But large software vendors such as Siebel don't support XA yet, claiming that customers are not asking for it.
Another issue with XML-based collaborative applications is transaction performance. CORBA or Java RMI (Remote Method Invocation) may be more efficient than XML-based apps for high-volume transactions because all-text XML documents take up more space. "If you're doing millions of these things, you may want to keep the data in a binary format," SilverStream's Benfield says. "But for most applications, we've found that XML works fine."
Those vendors who would provide a platform for collaborative applications claim that their middleware architectures are open and that their competitors' are ill-suited and proprietary. But all agree that some combination of technologies -- XML-oriented middleware, traditional publish-and-subscribe messaging, and services-oriented Web services architectures -- will be necessary to do the job. Enterprises want to leverage their existing application components, want the business logic to reside in middleware rather than at end points, want middleware that's independent of any specific vendor's control, and don't want to write a lot of expensive custom code for each new app. Now it's up to the vendor community to deliver -- collaboratively.
ISVs take a crack at the collaborative apps puzzleAs enterprises push for more collaborative capabilities and for vendor-neutral integration middleware to leverage their existing systems investments, the leading applications software vendors are starting to respond. Several have recently announced new architectures, such as Siebel Systems Inc.'s Universal Application Network and SAP's xApps Architecture and Exchange Infrastructure, to open their environments to collaboration with other apps.
The devil will be in the details of these new efforts, specifically in how much functionality they actually expose and in how standard the set of interfaces will be. As one of the three sets of players colliding in the battle for the enterprise software high ground -- the other two being application server and EAI (enterprise application integration) companies -- the big ISVs, including Siebel, SAP, Peoplesoft Inc., and Oracle, have every incentive to try to leverage their size and market power.
As in the portal wars, this means controlling the crown jewels -- the "dashboard," or application screens, that manage enterprise-wide business logic and processes, as well as related data. This will be harder to do if the big ISVs provide unbundled access to too much of their own code or services.
Both SAP and Siebel claim they will use XML and Web services protocols as the backbone of their collaboration offerings. Bharath Kadaba, vice president of application networking at Siebel in San Mateo, Calif., says the company plans to leverage existing and proposed standards work such as XSLT (Extensible Stylesheet Transformation) for transformation maps, WSFL (Web Services Flow Language), and self-defining XSD (XML Schema Definition) interfaces in its UAN (Universal Application Network) launch this September.
Both companies have also announced prebuilt business process solutions -- for example, customer creation or quote to order -- that will enable their CRM and ERP applications to work smoothly with competitors' apps as well as homegrown and legacy applications. But how much of their own code will they expose to third-party process management solutions?
"What is relevant to expose as a service is driven by customer requirements," Siebel's Kadaba says, citing a pricing engine as a module that Siebel might expose as a callable Web service. Kadaba adds that most EAI vendors' flow and transformation solutions are currently proprietary. "There's a business process paradigm that's imposed by these guys," he explains.
EAI vendors shoot back that the ISVs' architectures won't give enterprises the full composite view they need. But at the end of the day, customers will have the last word. "[The ISVs] might want to have the front face, but the customer needs to integrate into other processes," says Marie Wieck, vice president of development at IBM in Armonk, N.Y. Enterprise decision makers are looking for strategic installations such as integration infrastructure and middleware to bridge different types of needs, not individual point products, she says.
THE BOTTOM LINE
Executive Summary: Enterprises want to leverage existing systems investments and functionality by developing new collaborative applications to facilitate business processes. In response, vendors are starting to offer XML-based middleware platforms to enable fast and economical construction of these collaborative applications.
Test Center Perspective: Several obstacles must be overcome before the vision of a collaborative apps platform can be realized, including data mapping and synchronization, process flow synchronization, and improved transactionality. Enterprises should look closely at the standards issues in each of these areas before making investments in a particular collaborative applications architecture.