Gregory Golberg - SourceForge



Gregory Golberg

grisha@alum.mit.edu

August, 2005 -- August, 2006

Dbdb – a JDI-based Platform for Cross-language Debugging

Table of contents

Dbdb – a JDI-based Platform for Cross-language Debugging 1

Introduction 4

About this document 4

More up-to-date information 5

Tools 5

Related work and the value of Dbdb 5

Why JDI? 7

What is JDI? 7

Proof of concept 8

Architectural overview 9

Infrastructure 10

Algorithm 12

Mediators 14

Using Dbdb as a framework 17

Registry reform 17

Mediator policy 18

Use cases 18

Framework or tool? 20

Post-mortem 20

Leaky abstractons 20

Speaking too soon: false starts and misdirections 20

Further work 21

Other language types 22

Parallel drill-down 22

Other features 23

Conclusion 24

Appendix A. Code listings 25

Listing 1. org.hrum.dbdb.OracleExample1 25

Listing 2. PL/SQL stored procedures 26

Appendix B. Screenshots 27

Appendix C. Criticism of BEA’s Patent Application 28

Appendix D. Other use cases for single-stack debugging 29

Appendix E. Why not JDI? 30

JDWP (Java Debug Wire Protocol) 30

Eclipse Debug Framework 31

Pro EDF 31

Pro JPDA 32

Why not, indeed? 32

Appendix F. References 33

Introduction

Source-level debuggers, and GUI for them, have been around for a long time and are indispensable tools for many programmers. As most real-world programs involve more than one language, modern IDEs present a consistent graphical front-end to different debuggers, for instance, the Eclipse IDE (for Java, C, C++, Python, Ruby, etc.), Oracle’s JDeveloper (for Java and PL/SQL debuggers), the ubiquitous Emacs (for everything but the kitchen sink), etc. However, while ability to debug multiple languages in a single IDE is no doubt useful, integration between the different debuggers (of importance to projects consisting of modules spanning several languages) still presents a problem. A notable enhancement is presented by tools such as SCORE, a debugger supporting Ada and C and allowing to “seamlessly transition between languages”[1].

A still further improvement would be to allow for single-stack debugging of mixed-language programs. In other words, as program in language A calls a program in language B, the programmer sees a single call stack from A to B, rather than two separate ones. In many modern-day enterprise applications, there are countless instances of code in, say, Java, calling stored procedures in PL/SQL. As one who encounters this situation daily, I believe that a seamless transition of debugging from one language to another and the ability to see the call in a single stack would be of immense value to the multi-language developer. (Other useful applications of this technique are discussed below, in Related work and the value of Dbdb and Other language types chapters.)

To this end, I propose an implementation of a framework for single-stack debugging of multi-language applications based on the JDI (Java Debug Interface), an API layer of JPDA[2] (Java Platform Debugging Architecture).

About this document

In this paper I will present and discuss:

1. An overview of related work in this area, serving as proof that the proposed technique is not merely a whim, but offers a tangible benefit for real-world development

2. The benefits of the proposed solution over the existing related work

3. An overview of the JPDA in general, JDI in particular and the reasons for choosing it as a foundation of the proposed framework, along with a discussion of alternative choices.

4. The working proof of concept, including design, documentation, working code and demo. Here, I will also discuss:

a. the generalization of the proof of concept into a framework

b. directions for further extending the framework, and

c. use cases for plugging into the framework

5. Post-mortem of the project

6. Conclusion and thoughts on further work in this area

More up-to-date information

This project has been released as Open Source, as more than a few colleagues expressed interest in its ultimate value, and in working on it to achieve such. Thus, a reader is referred to the following up-to-date online maintained sources:

1. Homepage at [1]

2. Project [b]log at

3. Project at , at

4. DefectBug tracking at

5. Demo at

Tools

This project was implemented and tested with the Eclipse 3.2 IDE (), Java Standard Edition 6 “Mustang” (), and Oracle 10g.

As for the creation of this document, free trial of EndNote () was very useful for managing references in this document. .

Related work and the value of Dbdb

The value of seamless, single-stack cross-language debugging is recognized in the industry. For example:

1. Stylus Studio features an “XSL debugger features include the ability to seamlessly switch contexts and step into Java XSLT extension functions using Stylus Studio's integrated Java IDE, then returning back to calling the XSLT stylesheet.”[3] (See also Other language types chapter below).

2. According to Abdul Al-Azzawe, DB2 Development Tools Architect, IBM is investigating “cross-language debugging support in the same call stack, such as mixing of Java and SQL nested stored procedure calls”[4]. I am not aware of any implementation available to the public to date.

3. According to David Alpern of Oracle, their “combined-VM algorithms

are more general […] and from the early stages of this work it was

clear to us that it would be desirable to also support combined-VM

debugging more generally for cross-tier calls of various sorts.” However, it is unclear whether, if at all, this functionality (if it exists) will be available, as he notes that he is “not at this time allowed to say when cross-tier debugging support might actually appear in our released products.”[5]

4. Engineers at BEA Systems also thought about this problem, and submitted a patent application, for a method that provides, in part, that “if more that one language appears on a stack, a developer can see the frames for each language, as well as be able to inspect variables for each language.”[6] This particular offering is examined in more detail in Appendix C. Criticism of BEA’s Patent Application below.

Other possible uses of single-stack cross-language debugging includes stepping into C implementations of native Java methods, for example. In describing a technique debugging these kinds of mixed-language applications, Matthew White of IBM laments, “do you effectively debug the Java/C hybrid programming since there is no debugger available that can check on this software chimera?”[7]. Almost answering his plea are Eclipse developers, listing “investigat[ing] debugging from Java into natives and back” as “potential debugger work items and areas of investigation for 2.1 and future releases of Eclipse”[8].

Additionally, Wrapped Application Debugger (WAD)[9, 10] attempts to address integration of C/C++ code with scripting language (Perl, Python, Tcl) extensions by “embedding a debugger into the application”. The authors’ complaint that “little work has been done on the area of debugging in mixed-language environment” can perhaps be assuaged, at least in part, by considerations presented here.

A single-stack view may also be of interest in debugging distributed applications using remote calls (RPC, RMI, EJB, etc.) Lovas et al. discuss a framework (using JPDA) usage for a “step-into mechanism for remote method invocations (RMI) that unifies the debugging mechanism for local and remote calls”[11].

Further, a casual perusal of various discussion groups reveals the desire for the single-stack debugging popping up often enough; some of these is summarized in Appendix D. Other use cases for single-stack debugging

And, finally, upon taking “a quick look” at , Darin Wright of Eclipse Debug team had this to say: “it certainly looks interesting. I know that people are interested in cross language debugging in general. […] I wonder if it could be applied in a more general way to address cross-language debugging, or if it is specific to stored procedures/SQL and Java?”[12]

Why JDI?

First, of course, in order to have a multi-language debugger (never mind combined call stacks yet), one needs a common framework that provides access to the needed debuggee’s information. I propose using the JDI layer of JPDA (Java Platform Debugger Architecture) as such a framework. As I will show, JDI is really quite language-independent (“J” for Java notwithstanding)[2].

What is JDI?

JPDA consists of three layers of APIs; here, I propose using one of them, JDI (Java Debug Interface). For the discussion of the choice of this layer vs. other JPDA layers, or other similar approaches, see Appendix E.

JDI is a set of Java interfaces, the implementations of which are to be provided by a debuggee – that is, the target program’s execution environment. At the root of it is the VirtualMachine[3]interface which represents the debuggee, that is, the target execution environment. While nominally the debuggee is meant to be a Java Virtual Machine, it does not have to be one. For example, it could be an interpreter, an intermediary program driving yet another debugger (say, gdb), or a native executable specially compiled[4]. As long as the appropriate interfaces are implemented, any JPDA-compliant debugger can control any debuggee. By manipulating these implementations, the debugger is able to query and control the debuggee’s execution: suspend and resume threads, set breakpoints, evaluate variables and methods, etc. As we will see, these interfaces are generic enough to represent the execution of a program in any imperative programming language.

Two of the methods of VirtualMachine interface, eventRequestManager() and eventQueue(), return implementations of EventRequestManager and EventQueue, respectively. These two objects are the primary channels of communication between the debugger and debuggee. EventRequestManager is used by the debugger to request to be notified when a certain debuggable event occurs. (Such a request can also contain a directive to suspend the thread in which the event occurred). The debugger then polls EventQueue for the events it is interested in. Examples of such events are MethodEntryEvent, ThreadStartEvent, StepEvent, ExceptionEvent, etc.

Each event contains references to other artifacts of the debuggee. For example, as can be expected, a MethodEntryEvent contains a reference to the method entered, and to a thread in which this entry occurred. The thread is represented by ThreadReference interface, which further contains a reference to the stack of StackFrames in this thread. In turn, a StackFrame, which represents the state of one method invocation on a thread's call stack, has references to the variables visible in this frame’s scope.

Notice the lack of anything Java-specific[5]. JDI, in fact, is so well designed that (especially with the adoption of JSR-45[14]) the J for “Java” in JPDA is an unnecessary qualifier. Its set of abstractions is similar to that of Eclipse’s language-neutral debugging framework[6] (centering on threads, stack frames, breakpoints and stepping into/over/out).

Finally, JDI is also used very creatively for such purposes as: dynamic analysis of running applications[15], self-healing applications[16], load-time transformation of compiled Java programs for the purposes of Aspect-Oriented Programming[17], dynamic creation of sequence and dependency diagrams from running programs[18], etc.

A rough block diagram of JPDA-debugger integration is shown in Figure 1. The JVMDI and JDWP components of JPDA are discussed in Appendix E. Why not JDI?.

[pic]

Figure 1. JPDA - debugger block diagram.

Proof of concept

For a proof of concept, I will consider a simple Java program that calls a simple PL/SQL stored procedure. The Java program is org.hrum.dbdb.example.OracleExample1. We are interested in the fragment shown in Listing 1. org.hrum.dbdb.OracleExample1 in Appendix A. The stored PL/SQL function FUNC, called from line 25 (and introduced in lines 20-21), and the function it calls in turn (FUNC2) are presented in Listing 2. PL/SQL stored procedures.

Architectural overview

Very roughly, a debugger capable of multiple language support (such as Eclipse) can be described by a block diagram in Figure 2. (Oracle prior to 10 is specified to provide yet another way of interfacing with the actual debugger; 10 and after support JDWP[19]).

[pic]

Figure 2. Multi-language debugger

The proposed solution would look as shown in Figure 3. The Dbdb JDI implementation will interface to the debugger in the same way as the Java debugging module did. Everything else is hidden from the IDE. All communications between the IDE and the multiple debuggee targets is mediated by the Dbdb layer, which provides a single stack per thread executed, even though different frames of the stack may represent different languages.

[pic]

Figure 3. Combined call-stack multi-language debugger

A sample sequence diagram of a debugging session with this approach is shown in Figure 4. (Here, Program 1 calls Program 2).

[pic]

Figure 4. Sequence diagram of combined call-stack debugging.

The actual implementation is more involved than that, such as:

1. Since Program 1 and Dbdb JDI Implementation run asynchronously, the sequence of setting a breakpoint right after the method in Program 1 that calls Program 2 is reached is done by other means. For example, if Program 1 is a JPDA-enabled target, this is done by creating a MethodEntryRequest for the appropriate method that will suspend the thread as soon as the method is entered.

2. The “modification of arguments” step may not be necessary if a debugger can be launched separately and can then be attached to the debuggee process.

Infrastructure

The following are the basic building blocks for a JPDA-based combined call stack debugger.

1. DbdbVirtualMachine[7]

Implementing VirtualMachine, this was originally intended to hold the virtual machines encountered throughout the debugging and switch among them. This is not the case (see below). The primarily function of this class is to present itself to the debugger and return DbdbEventQueue and DbdbRequestManagerInvocationHandler. Everything else this class does is there for a reason but is not really important.

2. DbdbRequestManagerInvocationHandler

Serves as an implementation of RequestManager for the virtual machines managed by DbdbVirtualMachine. The client (debugger’s) requests are routed through this class, but components of Dbdb itself call the appropriate RequestManagers of managed virtual machines directly. More on this below, under Mediators.

3. DbdbVirtualMachine.Registry

This is merely a collection of Maps, that holds mappings such as:

1. ThreadReference with which a DelegatingThreadReference was constructed to the DelegatingThreadReference that contains it

2. ObjectReference for a java.sql.Statement to an ObjectReference for a java.sql.Connection which created this Statement

3. A from2To Map, which associates a ThreadReference in one VM to the one in the VM that will be called from. For example, it will associate a ThreadReference that calls Statement.execute() in Java VM to the ThreadReference of the actual stored procedure that this Statement.execute() invokes in the Oracle VM.

NOTE: This one is specifically mentioned here, because it will be referred to heavily in the Algorithm chapter below.

4. Etc. (See also Registry reform below).

4. EventListener hierarchy

Classes implementing this interface are called by the DbdbEventQueue to handle MethodEntryEvent and MethodExitEvent appropriately in their process() method. They are expected to be self-registering (registered upon instantiation). More defails are given below.

5. DbdbEventQueue

Implementation of EventQueue, that is responsible for:

- polling EventQueues of all Virtual Machines being debugged

- calling all EventListeners

- deciding which events to return to the caller and which to consume (based on the return value from EventListener.process() method)

- doing more things than it, perhaps, should upon encountering a StepEvent. This piece of logic should be removed from this class in order for Dbdb to be a more flexible framework. As it stands now, and as will be seen in the Algorithm chapter below, this class is responsible for recognizing

More on this below, under Mediators.

6. DelegatingThreadReference

An implementation of ThreadReference that is responsible for exposing a unified call stack to the user. Internally, it holds a stack of ThreadReferences from managed virtual machines. The Dbdb framework (normally, various EventListeners) will push to and pop this stack as needed.

7. FakeStepEvent, DelegatingStackFrame, DbdbEventSet

Further infrastructure for supporting the DelegatingThreadReference.

8. org.hrum.dbdb.plugin package

This package contains the Eclipse plug-in for the Dbdb functionality. It includes code for dynamically looking up code for the stored procedures and displaying it in an editor.

Algorithm

The algorithm employed is as follows (we will use the OracleExample1 program for illustration):

1. When a debuggee is launched, the DbdbVirtualMachine is initialized with:

a. VirtualMachine object gotten from the appropriate Connector

b. Currently, the following EventListeners (others can be added, of course):

i. __java_sql_DriverManagerListener

ii. __oracle_jdbc_driver_PhysicalConnectionListener

iii. __java_lang_RuntimeListener

These are explained below.

2. On receiving MethodExitEvent from DriverManager.getConnection() method (line 15), __java_sql_DriverManagerListener:

a. Obtains the reference to the java.sql.Connection object as the MethodExitEvent’s returnValue().

b. Starts a thread with a ListeningConnector (com.sun.jdi.SocketListen, or, for Eclipse, org.eclipse.jdi.internal.connect.SocketListeningConnectorImpl, listening on a port.

c. Starts a thread that will execute DBMS_DEBUG_JDWP.CONNECT_TCP[19] on the connection obtained above. This will establish a debugging session within Oracle.

d. Waits until the connection is made, and obtains the Oracle’s virtual machine from the ListeningConnector.

e. Instantiates a of OracleDbmsMethodEntryEventListener, which will listen for MethodEntryEvents from the Oracle’s virtual machine. This listener, during construction, associates (in Registry.from2To, see 3 above) the Java thread[8] it the connection was created in with a DelegatingThreadReference.WAITING_FOR_THREAD_REFERENCE constant.

3. On receiving a MethodExitEvent from Connection.prepareCall() (line 20-21), __oracle_jdbc_driver_PhysicalConnectionListener:

a. Loads the source code for the stored procedure being prepared from Oracle (a little earlier than necessary, but why not…)

b. Associates reference to the returned java.sql.Statement from the MethodExitEvent’s returnValue() to the reference to java.sql.Connection which created the statement.

4. On receiving a MethodEntryEvent from an Oracle stored procedure, OracleDbmsMethodEntryEventListener:

a. Finds the ThreadReference for the Java thread created in its constructor (see 2.e above), creates a DelegatingThreadReference for it, and pushes onto it the ThreadReference from the Oracle stored procedure.

This step also associates the found ThreadReference with the DelegatingThreadReference in the Registry (instead of earlier WAITING_FOR_THREAD_REFERENCE created in 2.e above)

b. Instantiates an OracleDbmsMethodExitEventListener

c. Returns, for the debugger, a FakeStepEvent, with this DelegatingThreadReference.

5. On receiving a MethodExitEvent from an Oracle stored procedure, OracleDbmsMethodExitEventListener (see 4.b above):

a. Executes a popThreadReference() on the DelegatingThreadReference instance it is constructed with

b. Resumes the thread[9] that originally called the popped thread.

Mediators[10]

All the while, Dbdb’s implementations of EventRequestManager. and EventQueue mediate the interaction between the debugger and the debuggee as follows:

6. DbdbRequestManagerInvocationHandler, merely, forwards all requests to EventRequestManagers of all managed VMs. The only special case is a call to createStepRequest(). If this is received, the Registry.from2To map (see 3 above)is checked. If the ThreadReference on which the step is to be created maps to a “to” ThreadReference object which is not a WAITING_FOR_THREAD_REFERENCE constant, the request is forwarded, of course, to the VM of the “to” ThreadReference.

7. DbdbEventQueue, every time the debugger wishes to retrieve an EventSet from the VM it thinks it queries, does the following:

a. Polls EventQueues of all managed VMs. All obtained EventSets are added to a local q2 variable.

b. If q2 is empty, just return null[11].

c. Create a new instance of DbdbEventSet, called toReturn. This will be the EventSet that will be returned to the debugger; its VM is DbdbVirtualMachine.

d. Removes the EventSet from the local q2 variable, and, for each Event evt, executes the algorithm shown below, in Figure 4. DbdbEventQueue algorithm, part 1 .

e. Finally, if toReturn set is not empty, it is returned. If it is:

i. If resumed is false, a resume is called on it.

ii. A null is returned[12].

[pic]

Figure 5. DbdbEventQueue algorithm, part 1

[pic]

Figure 6. DbdbEventQueue algorithm, part 2.

The resulting combined call stack is shown in Appendix B. Screenshots .

Using Dbdb as a framework

While this may not be a full-fledged framework at this point, let us consider two use cases of a developer wishing to extend this approach.

Registry reform

First and foremost, a better solution should be found for DbdbVirtualMachine.Registry, for this to be truly extensible. But this should, perhaps, wait, until the Dbdb applications has been extended further.

Some thought may be given of implementing a simple relational-database-like data structure in a programming language. In Java, for instance, Map is nice, but what if you could have an easy SymmetricMap, or, going further, something resembling an RDBMS table structure (without the complexity of triggers et al.)

Mediator policy

Secondly, then, a policy of using Dbdb implementations of com.sun.jdi.* interfaces (such as VirtualMachine, Event, EventManager, EventRequest, etc.) vs. managed VMs implementations needs to be clarified. In other words, when is a Dbdb implementation returned to the debugger, and when is it an implementation for the managed VM that the client is currently interested in? Currently it is not explicitly defined. It is my opinion that a few more extensions of Dbdb are needed (such as are outlined in use cases below) in order to formulate this policy better.

Use cases

Now, to the more general things…

Actions required of a developer using Dbdb in the use cases listed below could be generalized; in other words, made simpler or pluggable. The comments on how to do that are in corresponding footnote for the item.

Use case 1: Repeat the proof-of-concept on another JDBC-compliant RDBMS

1. Find out the appropriate implementation of java.sql.Statement for this RDBMS, and create an EventListener for it, using __oracle_jdbc_driver_PhysicalConnectionListener as a template. In particular:

a. It is advised that this EventListener extend the MethodXEventListener, which would require for it to be named in the following pattern (as should be obvious):

i. The name starts with __ (two underscores).

ii. The rest of the name is the fully qualified name of the class implementing java.sql.Statement for this JDBC vendor, with periods replaced by underscores, and a suffix Listener added at the end.

iii. Register this EventListener with the DbdbVirtualMachine[13].

b. Modify __java_sql_DriverManagerListener to:

i. Actually parse the arguments to getConnection() method(s[14]) to see whether the JDBC URL provided is the one that is handled by the target RDBMS[15].

ii. If so, get the appropriate com.sun.jdi.VirtualMachine implementation for the debuggee. (If, unlike Oracle, no direct JPDA-compliant VirtualMachine implementation is available), see Use case 2. Implement cross-language debugging of other languages below.

c. Implement the equivalents of OracleDbmsMethodEntryEventListener and OracleDbmsMethodExitEventListener[16], using the above classes as guidelines.

Use case 2. Implement cross-language debugging of other languages supporting JPDA

For the sake of example, let us consider a made-up need to debug another Java program called from our Java program. All that needs to be done is an implementation of DbdbProcessDebugger has to be created (the beginnings can be seen in JavaProcessDebugger class). The fully qualified name of this class has to be registered in dbdb_debuggers.properties file which must be on the classpath[17].

This is how it works.

Use case 3. Implement cross-language debugging of other languages

For the sake of example, let us consider a made-up need to debug a Perl script called from Java. In this case, in addition to performing the steps from the above use cases, a developer would need to provide the adapters from the target debugging framework[20] to implementations of com.sun.jdi.VirtualMachine and all related artifacts.

Framework or tool?

As can be seen above, while this not a defined framework/API, it is certainly poised to evolve into one.

Post-mortem[18]

A postmortem is a useful exercise. Not that I believe that this project is dead, but this concept can and should be applied even to a release. Thus, for this proof-of-concept, here is a post-mortem. It explains some pitfalls, false leads, and reasons for this project taking more than it should.

Leaky abstractons

The following “Leaky abstractions”[21] certainly took away time and effort:

1. For all their talk about good OO design and interfaces, Sun shows that they are sinners[19] as well. It is nice to learn that Sun's implementation when manipulating ostensibly insterfaces, in fact, expects them to be implementation classes inside. For example, com.sun.jdi.MirrorImpl.validateMirrorOrNull() inside casts things to com.sun.jdi.MirrorImpl rather than com.sun.jdi.Mirror. Ditto for ThreadReference vs ThreadReferenceImpl.

2. This is not a big deal, but a caveat nonetheless for those attempting to integrate Dbdb with exiting debuggers. What is one to do with a null EventSet? What about an empty one? Javadt does not like a null EventSet -- it just does not check for nulls (which is ok, for a throwaway reference implementation). But an empty one is not good for Eclipse, because it indiscriminately calls a resume() on it, which is not what we want. Back to returning null then. But a thought: how many of such little things would render this "framework" not really a framework (still, certainly, usable, but just a nagging feeling…)? Or should this all be configurable?

Speaking too soon: false starts and misdirections

1. Because of my general fear of having to spend time on learning Eclipse’s plug-in architecture, I proposed putting off integration with this very popular IDE in favor of using simpler tools. But indeed – “Quis emendabit ipsos emendatra”[20]. Eclipse should have been used from the very beginning. This would have saved quite a lot of time and I would never learn about the leaky abstraction 2, above…

On a related note, see the chapter Why JDI? above for the discussion of choosing JPDA over Eclipse Debug Framework. Had I started again, I would probably had used the latter…

The idea of using JDWP was judged to be an overkill, and ultimately abandoned[21] (but see also Appendix E. Why not JDI?). The discovery of Oracle’s support of JPDA-based debugging would have sped things up. But it happened after I spent inordinate amount of time on task that ultimately is not needed – researching ways of using existing JDWP implementations. JSR-45 is not needed as well[22].

2. The idea of loading debugged classes with an alternate class loader, which would substitute “real” classes (in this case, JDBC drivers and their subsequent artifacts) for their equivalents, and attendant ideas (such as using bytecode modification) did not find its use, despite much time spent on this research. This was due to:

a. Mouthing off about this in proposal without really knowing that much about JPDA

b. It is hard to debug (but see Parallel drill-down chapter below).

However, this may yet find its use. I just have not thought about it again…

3. I have labored for some time believing that returnValue() feature exists in JPDA. When I realized that it does not, I was forced to produce an elaborate workaround, which was quite complicated and not extremely robust. It is only after the workaround has been created that I found that this feature indeed exists in JPDA starting with Java 6 – so add more time to clean up the ugly workaround, to use more elegant solution with returnValue()[23].

Further work

The project is released to open-source, and will continue to evolve[24] in its current incarnation as an Eclipse plug-in primarily for stored procedure debugging. In this chapter I will outline some other interesting possibilities for this project.

Other language types

It would be interesting to develop a use for this framework for debugging not just procedural/imperative[25] languages for which it is created, but also for:

1. Logical/constraint languages (such as Prolog, which has been adapted for Eclipse[22]).

Perhaps as a subset of this functionality, or of an application in its own right, could be adaptation of this technique for debugging parsers (such as those based on grammars – ANTLR, yacc, etc.)

2. Other declarative languages, such as XSLT.

3. Cross-cutting concerns/Aspects (I have not even begun to appreciate this, but I thought I’d throw it into the mix[26]).

As with Prolog implementation, the three areas above may involve some creative redefinition of semantics of certain debugging operations, such as stepping. These problems, as applied to declarative, and an Eclipse-based solution, are presented in detail by Hui Wu et al. in “Grammar-Driven Generation of Domain-Specific Language Testing Tools”[23].

4. Stepping into, C/C++ implementation of Java native methods, or, more generally, into assembly/machine code/byte code (but this is, perhaps, more of a topic for the “Parallel drill-down” approach, outlined below).

Parallel drill-down

Often, the call stack actually executed is different from the “abstract” call stack though of by a developer (the latter being the one depicted as a sequence diagram). In this example, it is useful for a developer to think of a call to java.sql.Statement.execute() as logically leading directly into the stored procedure.

For example, from a standpoint of an application developer, it would be good do away with the Oracle implementation, which we currently include in a combined stack shown in Appendix B. Screenshots . In other words, in this particular screenshot, your average application developer is only interested in first (the Java frame containing a Statement.execute() call) and last (in this screenshot, the first and only, but in general, the first PL/SQL frame) stack frames of the shown call stack. On the other hand, for a developer of Oracle’s JDBC driver, both may be useful.

For a lack of a better term, I’d like to coin one: “parallel drill-down” (OK, so it’s not a cool term). This would mean creating a framework for debugging where at a point where “logical” and “implementation” scenarios diverge, the developer is presented with multiple parallel call stacks to step through. Any path can be selected, and the developer can switch between paths at will; a step in one path must be synchronized with other possible paths. For all my criticism of the BEA patent application (see Appendix C. Criticism of BEA’s Patent Application), the authors correctly identified this particular problem when they wrote: “For example, it is not uncommon for a developer to see stack information not directly related to the software being debugged when encountering a stack frame for one language, when using a debugger intended for another language. As another example, when using a debugger intended for the Java language, a Java stack will not include the XScript […] stack, and can sometimes show the set of Java classes that implement the XScript engine (these are part of the environment, but not the software the developer is working on).”[6]

Obviously, there is no limit to splitting these, however, common scenarios can be a well-defined and be a part of the framework, while also providing the developer with a way to define her own. Here, SMAP mechanism of JSR-45 can probably be leveraged.

Another example is a call to by reflection or related things. For instance, debugging things using java.lang.reflect.Proxy mechanism (such as Dbdb’s own) is notoriously painful. We usually want “logical” debugging – the developer may not want to see internal JDK infrastructure, but sometimes we may want “implementation” one. Since reflection is quite common when using generic framework, this may be a nice-to-have feature.

As an example of the usefulness of this feature, consider the screenshot in Appendix B. Screenshots From the point of view of the application developer, the stack frames between the application code calling Statement.execute() and the stored procedure code are extraneous “plumbing”.

If this feature is implemented, providers of generic frameworks can provide the “split” scenarios as part of the product, allowing the user of the framework an easier way to separate the debugging of her logic vs. the compound logic of her components and the underlying framework.

This approach would be even more useful to programs that are worse than those using reflection – those using byte code manipulation.

Other features

Other features that are not implemented in proof of concept, but would be useful, include

“drop-to-frame” functionality, allowing “popping a selected stack frame (and all frames above it) from the execution stack and then stepping back into the frame.”[24].

Conclusion

Dixi.

Appendix A. Code listings

Listing 1. org.hrum.dbdb.OracleExample1

01: package org.hrum.dbdb.example;

02:

03: import java.sql.CallableStatement;

04: import java.sql.Connection;

05: import java.sql.DriverManager;

06: import java.sql.Types;

07:

08: public class OracleExample1 {

10:

11: public static void main(String[] args) {

12: try {

13: Class.forName("oracle.jdbc.driver.OracleDriver");

14: System.out.println("DriverManager.getConnection()");

15: Connection con = DriverManager.getConnection(

16: "jdbc:oracle:thin:@localhost:1521:dbdb",

17: "sys",

18: "password");

19: System.out.println("con.prepareCall()");

20: CallableStatement st =

21: con.prepareCall("begin ?:=func(?); end;");

22: st.setInt(2, 1);

23: st.registerOutParameter(1, Types.INTEGER);

24: System.out.println("Statement.execute()");

25: st.execute();

26: System.out.println(st.getObject(1));

27: } catch (Exception e) {

28: e.printStackTrace(System.out);

29: System.exit(-1);

30: }

31: }

32: }

Listing 2. PL/SQL stored procedures

The PL/SQL stored procedures are created with the following code[27]:

create or replace function func2(num number) return number

is

begin

return num * 3;

end;

/

create or replace function func(ctr number) return number

is

ret number:=0;

begin

for i in 1..ctr loop

ret := ret + func2(i);

ret := ret + 3;

ret := ret - 3;

end loop;

return ret;

end;

/

alter function func compile debug;

alter function func2 compile debug; [28]

Appendix B. Screenshots

Appendix C. Criticism of BEA’s Patent Application

I’d like to expound more on this particular proposition, as it is a claim for a patent, especially in light of claims that Oracle has a similar patent filing[5]. No implementation has been made available to my knowledge; and the approach itself, as described in the application, does not seem to be “novel and non-obvious” in November of 2004. At this time both JPDA and Eclipse Debug Framework (discussed in Why JDI? chapter below) were already available, and given that, a remark that “creating debugging tools that can be applied to software applied to more than one programming language, and running in the same environment, has proved to be extremely difficult” seems particularly disingenuous. But at least they saw a need, and briefly identified (but not addressed) some problems I also discuss below, in the Parallel drill-down chapter below.

Further, consider this pronouncement “One multi-language debugger, JSR 45, can only be used to debug languages that are easily transformed into Java and then compiled”. Such a statement makes one wonder whether this was merely lost in translation to the patent attorney, or the claimants misunderstand or, for the purposes of patent claim, misrepresent, JSR 45. First, it is not a debugger, but a specification. Second, JSR 45’s stated goal is to establish “[a] mechanism […] by which programs executed under the Java virtual machine but written in languages other than the Java programming language, can be debugged with references to the original source”[14] (Emphasis mine). This renders their next point a non-sequitur: “This and most other multi-language debuggers won't work with languages such as XScript that where [sic] the language will be run by an interpreter or the language can not be mapped directly to Java because, for example, the language has a different data structure.” Perhaps what they are trying to say is that this is the case when a language cannot be easily mapped to a procedural/imperative model (like XScript, perhaps), which indeed seems to be the case with JPDA. However, they provide no proof that their model does...

Appendix D. Other use cases for single-stack debugging

I'm wondering: do you have ideas on how to support cross-language debugging for RDT, ie. a (J)Ruby methods that call Java methods (and the other way around), ie. the StackTrace would contain both Ruby and Java stack frames. The question is: if the JRuby process is launched with the Debugger, you'll have a JDT DebugModel, but you'll also want a Ruby DebugModel. I'm not sure if there's an easy way to combing StackTraces from both DebugModels (maybe some kind of Delegating DebugModel that collects StackTraces from JDT and Ruby DebugModels).



Is there an easy way to set up integrated debugging between a Java application and PL/SQL?

A reply to a tutorial on using DBMS_DEBUG, Oracle’s debugging package at

In my private fantasy land, I'd be able to run emacs and somehow invoke pdb with pdbtrack to do source-level debugging of my python code, then automagically step into gdb when the python calls out to C++ code I've written via boost::python.



Appendix E. Why not JDI?

The choice of JDI was a bit opportunistic. It did, indeed, strike me as a fairly generic framework; however, its choice was also influenced by my greater familiarity with it (versus Eclipse Debug Framework) and by relative ease of implementation of the proof-of-concept for a JPDA-compliant debugger (versus JDWP, see below). This chapter attempts to examine alternatives to JDI for the implementation

JDWP (Java Debug Wire Protocol)

The two API layers left unexamined in here are JDWP (Java Debug Wire Protocol), which “which defines the format of information and requests transferred between the debugging process and the debugger front end” [2] and JVMTI (JVM Tools Interface), which “a programming interface [that] provides both a way to inspect the state and to control the execution of applications running in the Java virtual machine”[25]. The latter is outside the scope of this paper[29]. But JDWP deserves further examination here.

JDWP provides a packet-based, stateless protocol for communication between a debugger and a debuggee. It can be said that it defines a serialization model for the JDI objects discussed in chapter What is JDI?, above. Another way of saying it is that the Sun’s JDI API[30] is merely a reference implementation, in Java, of JDWP. In this way, it is completely agnostic as to the implementation of a debugger and can be used as a universal debugging protocol. As the JPDA FAQ points out, “[t]heoretically JPDA could have only one interface, the Java Debug Wire Protocol (JDWP)”[26]. Indeed, there already is a Common LISP[27] and a Ruby[28, 29] implementation of JDWP.

At this time, despite the two above-mentioned (and encouraging) cases, pretty much all JDWP-aware debuggers are those that are fully JPDA-aware; that is, these are Java debuggers, which already use JDI API. If it is recognized that JDWP can be used not just as a Java debugging protocol, but as a generic one, and other debuggers are aware of it, the single-stack implementation could perhaps be pushed down to the layer that is responsible for reading/writing JDWP packets and creating the data structures, rather than on top of the data structures themselves. In fact, as we saw earlier (and made use of in proof of concept), Oracle, as of version 9i, provides for JDWP-based remote debugging of its stored procedures – not only Java, but PL/SQL[19]. This is a real-world example of JPDA usage for non-Java languages.

It is true that “[w]riting directly to JDWP however is painstaking work, information sent across the wire must be read and written precisely”. But implementations for doing that already exist in Java (GNU Classpath[30]), C (Sun’s own implementation of JDWP), and, as we have seen above, in Common LISP and Ruby. As it is a well-designed protocol, perhaps the next iteration of this project should be rewritten with JDWP in mind.

Both JDWP and JDI approaches can be used, out of the box, for remote debugging, which is very useful.

And, one a final thought. As I mentioned in chapter What is JDI?, when debugging a native executable, it may be simple enough to create Java mediator programs that would drive the third-party debugger (e.g., gdb) and translate their interactions with this debugger into the objects implementing JDI API. Driving the third-party debugger through, perhaps, a command-line interface (CLI) seems straightforward. However, in case of the gdb debugger, it was found that CLI “has proven to be highly unreliable” and Eclipse has since switched to use machine-oriented text interface[13] which “is quickly becoming a de facto standard for integrating debuggers into a variety of environments” [31]. But should using CLI with another third-party debugger present a problem, and there is no MI-like solution, perhaps it could be more efficient to devise a way to enhance a compiler so that the executable itself is JDWP-compliant.

Eclipse Debug Framework

Obviously, well-designed IDEs that provide multi-language debugging have also implemented language-independent debugging frameworks. Perhaps one of the most well-adopted and best-designed of these[31] is Eclipse Debug Framework[32] (hereafter, EDF). Because it is language-neutral, one may think it a naturally better candidate for this project. Here, I will examine this proposition. Full disclosure: large portion of the work was already completed using JDI by the time I have more thoroughly familiarized myself with this alternative, which may make me seem biased against it…

Pro EDF

1. It is intended to be language-independent. Its design, centered around the notion of threads, stack frames, breakpoints and stepping seems similar to JPDA and so seemingly not well-suited for languages that are not procedural/imperative languages (see the Other language types chapter below). However, even those can be accommodated: for example, a working Prolog debugger[22, 33] utilizing EDF creatively redefines the semantics of stepping into and out[33].

2. For Eclipse-based debuggers, implementing single-stack debugging in this layer (in a way probably very similar to the one described in this paper) is easier, because no adaptation is needed for a new language to support JPDA.

Pro JPDA

1. EDF is indeed designed for “[m]odeling a set of artifacts and actions common to many debuggers, known as the debug model. For example, some common debug artifacts are threads, stack frames, variables and breakpoints; and some common actions are suspending, stepping, resuming and terminating.”[32] However, as we saw earlier, a functionally equivalent set artifacts and actions is available via JPDA. “The Pragmatics of Java Debugging” article also hints at the cross-language potential of this architecture[34].

2. For JPDA-compliant non-Eclipse-based debuggers (and there are a number of those, the obvious advantage is that EDF does not need to be re-implemented. Further, because of the definition of JDWP, and existing implementations of it in a number of languages (Java, C, Ruby and LISP), it can be easier integrated with debuggers not written in Java, and allow for remote debugging as an added bonus.

Why not, indeed?

As we have seen above, the choice of JDI was largely expedience and opportunism. This implementation certainly does the job, and JDI can be used in similar applications to great success. But I am not going to rationalize my choices to no end. I believe that the best implementation is JDWP, as it is closer to a universal debugging protocol.

Appendix F. References

1. SCORE: Multi-Language Debugger.

2. Java Platform Debugger Architecture.

3. Stylus Studio's XSL Debugger

4. Al-Azzawe, A. (2004) Abdul Al-Azzawe on development enhancements in DB2 Universal Database V8.2. IBM developerWorks.

5. Alpern, D., Personal communication (e-mail). 2005.

6. Pugh, W.A. and J.M. Eckels. Patent Application: System for multi-language debugging, Provisional application No. 60/450,014. 2003

7. White, M. (2001) Debugging integrated Java and C/C++ code: Two approaches using JNI. IBM developerWorks.

8. 2.0.1 Candidate Fixes: Possible Work Items.

9. Beazley, D.M. An Embedded Error Recovery and Debugging Mechanism for Scripting Language Extensions in USENIX. 2001

10. Cao, J. and D.M. Beazley (2005) Embedded Debugging of C/C++ Plugins and Extension Modules. Technical report TR-2005-07, Department of Computer Science, University of Chicago.

11. Lovas, R. and V. Sunderam. Debugging of metacomputing applications. in International Parallel and Distributed Processing Symposium. 2002. Ft.Lauterdale, Fl

12. Wright, D., Personal communication (e-mail). 2006.

13. Leszek, P. C/C++ development with the Eclipse Platform: How to use the C/C++ Development Toolkit (CDT). IBM developerWorks.

14. Bracha, G. and A. Ryman. JSR (Java Specification Request) 45: Debugging Support for Other Languages. 2003

15. Gueheneuc, Y.-G., R. Douence, and N. Jussien (2002) No Java without Caffeine: A Tool for Dynamic Analysis of Java Programs. Ecole des Mines de Nantes.

16. Savarese, D.F. (2002) Application, Heal Thyself. JavaPro. (login required)

17. Kniesel, G., P. Costanza, and M. Austermann, JMangler - A Powerful Back-end for Aspect-oriented Programming, in Aspect-oriented Software Development, R.Filman, T.Elrod, and S.Clarke, Editors. 2004, Prentice Hall

18. Loton, T. (2001) Using The Java Platform Debugger Architecture. Java Developer's Journal.

19. Antognini, C. Debugging PL/SQL and Java Stored Programs with JPDA.

20. Foy, B.D. (2001) Using the Perl Debugger. Dr.Dobb's Journal.

21. Spolsky, J. (2002) The Law of Leaky Abstractions.

22. Kroening, M. (2005) Eclipse: Adapting and Updating an IDE. Dr.Dobb's Journal.

23. Wu, H., J. Cray, and M. Mernik. Grammar-Driven Generation of Domain-Specific Language Testing Tools. in OOPSLA. 2005. San Diego

24. Eclipse 3.1, documentation of org.eclipse.debug.core.model.IDropToFrame interface (Javadoc).

25. JVM Tool Interface

26. Java Platform Debugger Architecture FAQ.

27. Lichteblau, D. CL-JDI (a Common Lisp implementation of Java Debug Protocol).

28. Kilmer, R., Ruby and the Java Debug Wire Protocol: What were we thinking?, in RubyConf 2003. 2003.

29. Fowler, C. and R. Kilmer. Ruby implementation of the Java Debug Wire Protocol.

30. GNU Classpath.

31. Roberts, N., B. Rossi, and E. Zaretskii. Debugger Machine Interface.

32. Wright, D. and B. Freeman-Benson (2004) How to write an Eclipse debugger.

33. Kroening, M. Amzi! Prolog: Source Code Debugger.

34. Winchester, J. and A. Ryman (2001) The Pragmatics Of Java Debugging. Sys-Con India.

-----------------------

[1] While Javadoc is available, this document can complement that.

[2] It is language-independent (or, rather, language-agnostic) in terms of the debuggees it can handle. Of course, being a Java API, it is most suited for use with debuggers written in Java (such as Eclipse). For simplicity of implementation, this will suffice, as I was interested in creating a proof of concept rapidly. It is with this in mind that this chapter should be understood. A truly independent implementation, agnostic both to the debugger and the debuggee, is discussed in Appendix E.

[3] All JDI interfaces are located in com.sun.jdi package or its subpackages. For more information, see .

[4] This is indeed how Eclipse Debug Framework does it. Their tutorial features a rudimentary implementation (in Perl) of a debugger for a made-up assembly language which exposes a TCP interface; the existing Eclipse C debugger is driving GDB behind the scenes13. Leszek, P. C/C++ development with the Eclipse Platform: How to use the C/C++ Development Toolkit (CDT). IBM developerWorks. . As we show here, and in Appendix E. that the JDI approach is equivalent to it. This is also discussed further in Use cases.

[5] One could make a case that data types are indeed those that Java uses. However, these can be used with other languages, especially since an Object data type can really simulate anything else – say, C’s struct. (I will gloss over the intricacies of C structures such as pointer/array model, or unions, as this is outside the scope).

[6] Further discussed in Appendix E. Why not JDI?

[7] All names of Dbdb classes mentioned here are in org.hrum.dbdb package, unless a fully qualified name is explicitly specified

[8] See issue #1514829 at .

[9] This is a bit crude – assuming this thread was originally suspended by the STEP_INTO request of a debugger. But if it wasn’t, no harm – no foul.

[10] Who cares if GoF and their flock differ with us on minute nomenclature?

[11] See also .

[12] See item 2 in Leaky abstractons below.

[13] Building a framework: this should be automatically picked up by DbdbVirtualMachine from a properties file, for example, rather than requiring to modify DbdbVirtualMachine class.

[14] Building a framework: There is currently only one – taking 3 arguments, URL, username and password. But all the overloaded methods should be taken care of, it’s just a matter of typing.

[15] Building a framework: Of course, a mechanism to do this based on some sort of property files, rather than modifying __java_sql_DriverManagerListener every time, is a step that should be done prior to that. The more generic __java_sql_DriverManagerListener should thus be extended to dynamically (again, based on property files or similar mechanism) use strategy design pattern for JDBC URL recognition, getting a VirtualMachine object from the debuggee connection, etc.

[16] Building a framework: Perhaps this one is not even needed, with the right reworking of the framework.

[17] Since a sample dbdb_debuggers.properties file is included with Dbdb, the developer-modified one must be earlier on the classpath than Dbdb.

[18] Lots more on this on the project’s blog: .

[19] It’s nice to see this (very Shakespearean) comment in com.sun.tools.example.debug.bdi.JDIEventSource:

//### Gross foul hackery!

[20] Or something. This is described in , and .

[21] More detail on this at .

[22] More detail on this at .

[23] Which, in turn, required another little workaround, described at .

[24] A “to-do” list, masquerading as a bug list, is at .

[25] The Object-Oriented part is incidental to this debugging architecture. Because of models such as StackFrame, and things such as MethodEntryEvent, JPDA is based on a procedural/imperative model.

[26] Pun intended.

[27] Don’t forget to commit. Duh!

[28] See Defect #1497650 ().

[29] It can, though, be used for a special case of mixed-mode debugging; per the JPDA FAQ:

Can JPDA be used to write a mixed mode (Java and C/C++) debugger?

Yes, however this is a case where you would probably need to use JVMDI (see Which interface layer should I use?). We know of one product, not released but working quite well, that uses a combination of JVMDI and native debugging functionality to provide mixed mode debugging.

[30] Including in this API, of course, the native C code library used to actually read and write JDWP bits

[31] Considering only open-source variants, of course.

-----------------------

[pic]

Figure 7. Dbdb as Eclipse plug-in.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download