SAN MATEO (07/03/2000) - Consider the modern database: sleek, efficient, and able to retrieve records in the blink of an eye. Data representation, management, and storage have risen to heights we dared not dream of only 10 years ago. But ironically, despite these achievements, the hippest, most cutting-edge data management technology today is (drumroll, please ...) delimited text.
Text files have always been the surest and simplest way to represent data. With binary files -- the text file's evil opposite -- you have to worry about details such as byte order, precision, and sign bits. The 32-bit integer your computer stores will get cut in half by another system that believes integers are 16 bits long, and a program looking for a calendar date will be stymied by a packed integer such as 6212000. And that's just the tip of the iceberg. Y2K developers who went combing through legacy data found innumerable horrors, all committed in the name of efficient data storage and transmission.
Thankfully, the emergence of text files has changed the way we manage and exchange data. XML and its supporting technologies have been right at the vanguard of the revolution. With XML, you can structure complex data in a simple, elegant, and easily understandable way. XML is extensible, which means it can easily incorporate new features related to document formatting and automatic processing. Best of all, XML is freely available, which is part of the reason why few companies undertake information management projects without considering the role of XML.
Here to stay
We've seen lots of computing fads come and go, and each one's downfall has brought costly consequences. But XML is here for the long haul, partly because it's a proven, widely adopted technology and partly because its inherent flexibility has led to scores of new technologies and extensions that support and rely on parent XML.
But in many ways, these technology offshoots contribute to XML's biggest problem: complexity. There's at least one XML parser for every popular programming language and operating system. To ensure consistency and compliance with the XML specification, the World Wide Web Consortium (W3C) issues standards (or "recommendations") that govern the structure of XML files and the programming interfaces used to manipulate them. The W3C also evaluates proposed extensions to XML, ensuring that new methods of parsing XML data are consistent before they become standards.
But developers and vendors still have plenty of room to improvise, because the W3C -- best known for its standardization of HTML -- is rather liberal as standards bodies go. It does not enforce its standards, and there is no W3C-compatibility certification program. As a result, the W3C has created a cooperative community not unlike that of the Linux set: Some of the most crucial additions to the XML specification have resulted from a programmer's off-standard riffing.
This means is that if you're planning an XML development project or data management architecture, you'd be well-advised to look beyond the standard XML developer's kit. Most of the truly interesting work is happening in the supporting technologies. Over the years, an impressive body of XML-related work has been assembled that can significantly cut your development costs and time to market. Without extensions, you'd have to write custom code for common operations such as searching and linking. As more vendors implement standards-based XML extensions, you're freed to spend more development time working on the applications.
In this Test Center Analysis, we examine many key XML technologies. Some are already W3C recommendations, others are on the standards track, and still others are just good ideas that some developers have embraced. It's not a comprehensive list, as there are literally hundreds of early-stage extensions to XML, but this core group is worth your immediate attention.
We gave each technology a green, yellow, or red rating on its relevance, acceptance among XML tools vendors, and prospect of becoming a W3C standard.
Any technology with mostly green scores is a necessary part of an XML strategy, whereas lots of red indicates a degree of risk that must be weighed against the technology's value. To read more about them, visit the W3C's Web site at www.w3c.org.
The W3C has already reviewed and recommended DOM (Document Object Model). It is the most widely implemented XML parser interface today. DOM reproduces an XML document's data hierarchy in a programming language's native object format, giving programmers an easy and familiar way of working with the data in the document. Developers can iterate through the document's data elements and even change the document's content programmatically.
However, the DOM recommendation does not cover searching or file input/output (loading and saving XML documents). The DOM API loads the entire XML document into memory, favoring repetitive operations performed on short documents. For lengthy documents, the SAX (Simple API for XML) API is a better choice.
DOM Level 2
This W3C recommendation extends the basic data representation capability of the DOM API to other programming concepts such as custom namespaces, style sheets, events, iterators, filters, and range functions. This gives developers a standardized way of expressing functions that they previously had to create on their own. For example, if you want your document to call a particular function whenever it encounters a specific event in the data, DOM Level 2 provides a simple and standard way of doing so.
Most commercial XML processing software include support only for namespaces and style sheets, so using a product that supports DOM Level 2 will give developers a greater range of flexibility. However, because the other types of triggers are not widely supported, you could run into compatibility problems that negatively affect the portability of your project.
SAX is an open-source project that lets applications manage large XML documents. Instead of loading an entire document into memory as DOM will do, SAX opens the document file and, as it reads the document, fires events related to the data it encounters. By "listening" for specific events, an application can skip to the relevant portion of the document. SAX effectively performs a search of the data, reading the elements and discarding them until it finds one that meets the specified criteria. In this way, SAX is much more efficient than is DOM Level 2, but unlike DOM, it can't make multiple or backward passes through the data. Thus, the SAX API is good for top-down, single-pass processing of large documents or large numbers of documents. SAX 2.0, released in May 2000, adds several enhancements, including support for namespaces.
XML document linking
XBase permits the use of relative URI (Uniform Resource Identifiers) paths in XML documents. URI paths make it easier to code XML documents that have multiple referrals to outside URLs. In these cases you supply XBase with the path prefix (such as http://www.infoworld.com/myxml/) common to those external files. Your references can then be shortened to include only the file names.
This simplifies coding and makes it easier to adapt to document location changes.
XLink gives developers a standard method for specifying navigational links between documents. The mechanism is similar to that of HTML hyperlinks, except that XLink lets you enforce the navigation order through a series of linked documents. XLink's capability of enforcing specific paths suggests applicability for training documents and other materials that should be scanned sequentially. Because XLink is a standard only for representing links -- as opposed to XInclude (below), which fetches and stores the entire referred document -- retrieving and presenting linked documents is the developer's responsibility.
XInclude copies entire XML documents, or selected portions thereof, into the current document. Using XInclude would eliminate the need to perform tedious cutting and pasting among documents during development. Right now, XInclude is in the early stages of the standards review process, but it is a promising candidate for approval; and the technology certainly would fill a gap in present XML implementations. It is key to the development of reusable document fragments, which are blocks of XML used to build other documents. XInclude uses XPointer and XPath to specify which document elements to include.
Schema and structure
The DTD (Document Type Definition) is the original XML schema representation.
(An XML schema defines the structural rules for a class of XML documents. It can mandate the names and order of data elements, place limits on parent/child relationships, and distinguish required elements from optional ones.) But the DTD, which is not expressed in XML, is infamous for being the ugliest, most difficult aspect of most XML projects. There are more stylish replacements, as described below, but only the DTD is universally and consistently implemented.
XML Schema serves the same purpose as the DTD, setting out a standardized structure for a class of XML documents. Unlike DTD, however, XML Schema is itself coded in XML, which means it can be parsed and edited using widely available software. Aside from its simpler syntax, XML Schema specifies data types such as "date" and "string." This eases the import and export of database tables in XML form.
The W3C recently advanced XML Schema to the working draft stage, so we'll have to wait at least a few months before it becomes a proper recommendation.
Because of this, very few parsers currently implement XML Schema. Until the W3C recommendation is released, the XML Schema specification is likely to change, so until it's ratified, it's best to stick with DTD.
When a large number of HTML and XML documents share a location, managing them can be a challenge. The W3C developed the RDF (Resource Description Framework) specification to aid in this process. RDF embeds document meta data, such as search keywords and copyright information, into a document. An RDF schema defines the structure of the meta data so that all documents in a collection can share a common meta data layout. If nothing else, RDF serves as a reminder that all documents should contain meta data that eases their management. As yet, RDF is not widely implemented.
XML Namespaces distinguish application-defined XML data from the special instructions and information used by XML extensions. For example, a document might define a namespace named "RDF" to set embedded meta data apart from the core document. Namespaces are key to the integration of XML extensions and are therefore a required element in commercial parsers.
Searching and transformation
XPath defines a complex pattern-matching syntax used by several XML extensions, including XSL (Extensible Stylesheet Language), XSLT (Extensible Stylesheet Language Transformations), and XPointer. It is essential to master XPath because it is the only standardized method for selecting data in an XML document. For example, applications unfamiliar with a document's structure can perform blind matches such as "the third element from the top of the document."
If your application knows how the document is laid out, XPath permits specific matches such as "every 14-karat gold solitaire engagement ring." The XPath syntax is terse and tough to learn, but there are plenty of good examples of its use in XML books and on the Web.
XPointer uses XPath to identify a portion of an XML document, exposing the selected data for processing. The tremendous size of some XML documents makes them too unwieldy to work with in their entirety. The ability to pull out a subset of data based on user-defined criteria, such as jobs in the accounting department, makes the developer's job far more manageable. XPointer permits string searches and range specifiers, letting you select a single data element, a group of elements matching the XPath criteria, or an arbitrary range of data.
XSL and XSLT
XSL extends HTML's concept of style sheets -- standard formatting rules -- to XML. It lets you encode style rules, such as "all part numbers set in bold type" or "all subparts indented by two tab stops." An XSL style sheet ensures consistent rendering of all XML documents of the same type. If you decide to change the appearance of your invoices, you don't need to modify your invoice XML documents. Just change the style sheet associated with that type of document.
XSLT applies XSL formatting rules (style sheets) to produce either a restructured XML document or an HTML page. Internet Explorer has a built-in XSL style sheet that formats XML files for display. XSLT is extremely powerful and can be used to translate documents from one XML layout to another. Overall, XSL and XSLT are two of the most successful XML spin-offs, so they should be considered essential components of any XML toolkit.
At present, the awkward XPath pattern-matching syntax is the only way to search an XML document. But the W3C is currently embarking on an XML Query specification with the intent of bringing database-style queries to XML documents. Such a facility is sorely needed, so it seems clear that the W3C will eventually ratify some type of query extension. But it's too early to guess at what the W3C will come up with. Until then, impatient vendors will undoubtedly beat W3C to the punch and implement their own query facilities.
Remote object access
The proposed SOAP (Simple Object Access Protocol) specification lets applications invoke object methods, or functions residing on remote servers. A SOAP application creates a request block in XML, supplying the data needed by the remote method as well as the location of the remote object itself. A SOAP server transmits the request to the object, typically using HTTP as the transport. After the remote execution finishes, the method's response is formatted as XML and returned to the application.
SOAP makes remote object access delightfully simple. Early SOAP efforts by IBM Corp. and Microsoft Corp. show promise, but be wary: The SOAP specification leaves implementers plenty of room to diverge -- possibly creating incompatible implementations. For more on SOAP, see next week's Enterprise Toolbox column.
Write to senior analyst Tom Yager at firstname.lastname@example.org.