Developers working with three tiered software architectures -- divided into presentation, business logic and data tiers -- cannot efficiently handle temporary application data that must be sharable between servers to provide high application availability and seamless scalability. But technologies, many of them open source, have emerged that deliver an important piece of infrastructure to manage this work-in-progress data.

To understand this, it is valuable to step back and think in terms of the classes of data we need to manage in tiered applications and the best mechanism to manage each class based on properties such as rate of change, life cycle and need for long-term persistence.

Developers will concede that some types of data don't need to be stored for long, so in most cases such data does not belong in a relational database. This class of data is essentially work-in-progress data for incomplete business workflows -- data that is no longer needed when a workflow is complete, but essential for reliably reaching the end of a workflow.

Examples of such work-in-progress data include user session data regarding the stage of a workflow, user authentication and authorization information, content that a user creates or edits, or customer-service chat messages and related conversation logs. Other cases involve in-memory temporary data-sets used by batch processes for intermediate computations as part of a larger work effort. These might occur in trading systems or applications that perform complex pricing, cost allocation or matching algorithms.

Consider this analogy. We all remember taking multiple-choice tests that used computer-graded forms in which you had to fill in little circles. The paper was a great way to capture the final answers but was not useful in helping us arrive at the answers, especially in math or science, where interim calculations were needed. For these calculations, we often used a scratch pad.

The scratch pad helps us organize data in our brains, perhaps by drawing a diagram, or by using special symbols and tools, or by allowing us to break a problem down into smaller pieces. Working on problems using a scratch pad also affords us the ability to stop work midstream and later pick it up where we left off. We can even efficiently retrace our steps and check for inaccuracies.

When developing three-tier applications today, most developers write an application's incomplete information, or state data, to databases (often through OR mappers), message queues and API-driven caches, whether homemade or from a vendor.

Doing all of this to provide reliability and seamless application restarting amounts to using the wrong tool for the job. If you think about it, storing scratch data in the database is like using Microsoft Word to write down the interim steps and symbols used to answer the questions on a multiple-choice math test. You are moving away from the data's natural form in order to record it temporarily, which is not only burdensome and frustrating, but expensive and slow.

In short, the current developer frameworks, such as OR mappers, used to make three-tier applications, are missing a scratch pad, a system that would organize, distribute and make the state data -- the work in progress of the application tier -- more resilient without reliance on the database. This scratch pad would allow application data to remain in a form that application developers could understand -- the object-oriented form, which is the easiest way for the app guy to understand the application.