Sun just announced its acquisition of MySQL, but for several months the open-source database has been getting the performance tracing integration with the Solaris platform that it needs to follow through on Sun CEO Jonathan Schwartz's committment to performance improvements.
Schwartz wrote on his blog, "MySQL is already the performance leader on a variety of benchmarks - we'll make performance leadership the default for every application we can find (and on every vendor's hardware platforms, not just Sun's - and on Linux, Solaris, Windows, all)."
But a key tool for enabling those speedups so far has no equivalent on Linux and remains a Solaris competitive advantage. DTrace, released in 2005, makes every level of software running in the Solaris environment traceable, and both developers and administrators commonly use it to track performance issues without building special instrumented versions of software. Under Solaris, not only the operating system itself, but also DTrace-enabled applications, can provide data for an administrator running DTrace.
Linux developer Jonathan Corbet, author of Linux Device Drivers and the Linux Foundation's "Linux Weather Forecast," says that Linux doesn't have that kind of top-to-bottom tracing yet.
"The kprobes mechanism used by systemtap supports user-space probes, so the low-level stuff is there," he says in an email interview, and adds, "I don't understand why the relevant companies aren't putting a lot more resources into getting tracing right, quickly - it's mostly just a matter of assembling the pieces at this point."
Corbet points out that a new release of Linux recently added a kernel markers feature for creating DTrace-like probe points within the kernel, and that another project, kprobes, offers tracing both for the kernel and for applications. However, Linux lacks an integrated DTrace-like tool that's convenient for system administrators to use.
For example, a single DTrace session can track function calls within an application program, the system calls that the application makes to request services from the operating system, and services such as sending network traffic that the operating system kernel performs to act on those requests. In September, MySQL developer Brian Aker wrote that he began adding DTrace support to MySQL, making it possible for MySQL administrators on the Solaris platform to do that kind of tracing.
Sun developer and blogger Roman Shaposhnik even suggests using DTrace as a substitute for the "assert" function in C code. An ordinary "assert" is a debugging test that can be built into the final program at compile time, or left out. With DTrace, however, a separate debugging build is not required. "The dynamic nature of DTrace lets you turn any subset of them on when and only when you really need them," he writes.
A race with Solaris, though, could be just what Linux developers need. In 2002, Linux lagged Solaris in performance for applications with large numbers of threads. Within a short time, Red Hat's Ulrich Drepper and Ingo Molnar rearchitected Linux's thread support and dramatically increased performance.