University of Maryland The New Dyninst Event Model James Waskiewicz

University of Maryland Motivation: the series Several years in the making: Paradyn separation requires… DyninstAPI support for threads –Among other things Supporting threads (a la Paradyn) requires… –Asynchronous event handling, which requires… Solving the multi-event-source problem –Eg. Listen for and handle events coming from Socket / file descriptor System call ( wait() ) –Solution: use threads Just a can of worms … Or Pandora’s box?

University of Maryland Overall Design Concerns Keep thread stuff out of the way –Don’t want to “pollute” dyninst code with synchronization primitives –Dyninst is already hard to fix when broken Keep threading issues modular Try to keep existing code thread- context free –In other words, most dyninst code should be able to run successfully no matter what thread executes it. –There can be some tricky situations here (inferior RPCs for example) –Obvious exceptions in signal handling

University of Maryland The Plan: The 3 thread model 3 threads –(1) User Interface Thread API user’s mutator –(1) Synchronous Event Thread Eg. Loop on waitpid() –(1) Asynchronous Event Thread Eg. Loop on select() And a mailbox –Hide internal threads by forcing non UI threads to “mail” events to the UI thread. Eg. ensure that user-supplied callbacks are only executed on the UI thread.

University of Maryland The Plan: The 3 thread model UI ThreadMutatee 1 Mailbox Synchronous Event Thread waitpid() poll() Asynchronous Event Thread select() Mutatee 2 Mutatee N

University of Maryland But there’s a problem (linux) Ptrace() may only be called by one thread, period. –But we really want all threads to be able to issue ptrace() commands Process::pause() Write/readDataSpace() Solution: Add another thread –Solely responsible for accepting and handling requests for calling ptrace() –And sometimes waitpid() and fork() …

University of Maryland The Plan: The 4 thread model UI ThreadMutatee 1 Mailbox Synchronous Event Thread waitpid() poll() Asynchronous Event Thread select() Mutatee 2 Mutatee N Ptracer (DBI) Thread

University of Maryland But there’s another problem (windows) Restrictive OS level debugging API –The thread that starts the mutatee process is the only thread that is allowed to call WaitNextDebugEvent() Solution: add N threads ! –N = number of mutatees This is actually a good idea –One “listener” thread per process Elegant in terms of code modularity A nice simple concept: a thread that starts a process and receives events from it until termination/detach Handles events as they occur.

University of Maryland The Plan: The (n+3) thread model UI Thread Mailbox Synchronous Event Thread 1 waitpid() Poll() Asynchronous Event Thread select() Mutatee 2 Ptracer (DBI) Thread Mutatee 1 Mutatee N Synchronous Event Thread 2 waitpid() Poll() Synchronous Event Thread N waitpid() Poll()

University of Maryland But there’s another problem: recursion Dyninst is full of recursive event patterns –ie. An event that causes another event, and needs to wait for its completion Best solution will involve encapsulating this behavior into a “natural” structure –Either that or re-architect many functional elements of dyninst –And don’t forget: add more threads! Solution: decouple event handling concept into: –Event generation One dedicated thread (per process) –Event handling Multiple possible threads (thread pool) Number of threads depends on recursion depth This is a bit nasty –Could probably be done using the mailbox’s recursive callback features, but… They’re complicated too, and need more thought

University of Maryland The Plan: The (2n+3+x) thread model UI Thread Mailbox Signal Generator Thread 1 Asynchronous Event Thread select() Mutatee 2 Ptracer (DBI) Thread Mutatee 1 Mutatee N SH Signal Generator Thread 1 SH Signal Generator Thread 1 SH

University of Maryland A simple example: full stop UI Thread issues stopExecution () –Sends SIGSTOP to mutatee –Blocks inside waitForStop() UI Thread kill (SIGSTOP) Mutatee Signal Generator Thread Signal Handler Thread Event Gate Wait for event: evtProcessStop block

University of Maryland A simple example: full stop Mutatee Stops –Notifes parent process of SIGSTOP –This is received by SignalGenerator, which is listening for mutatee events UI Thread kill (SIGSTOP) Mutatee Signal Generator Thread Signal Handler Thread SIGSTOP Event Gate Wait for event: evtProcessStop block

University of Maryland A simple example: full stop Signal Generator decodes SIGSTOP and assigns event –Creates an Event with platform independent type evtProcessStop –Assigns Event to available SignalHandler Thread UI Thread kill (SIGSTOP) Mutatee Signal Generator Thread Signal Handler Thread SIGSTOP evtProcessStop Event Gate Wait for event: evtProcessStop block

University of Maryland A simple example: full stop Signal Hander handles evtProcessStop –Platform independent code for handling evtProcessStop –Signal waiting EventGates that evtProcessStop happened UI Thread kill (SIGSTOP) Mutatee Signal Generator Thread Signal Handler Thread SIGSTOP evtProcessStop Event Gate Wait for event: evtProcessStop Signal event: evtProcessStop block

University of Maryland A simple example: full stop Event Gate receives target event –Release UI Thread from block UI Thread kill (SIGSTOP) Mutatee Signal Generator Thread Signal Handler Thread SIGSTOP evtProcessStop Event Gate Wait for event: evtProcessStop Signal event: evtProcessStop resume block

University of Maryland A (slightly) more complex example User registers a DynLibrary Callback –When a library is loaded, insert some instrumentation UI Thread Mutatee Signal Generator Thread SH - 0 CB Manager register callback Mailbox SH - 1

University of Maryland A (slightly) more complex example Continue mutatee execution –Wait for it to exit –Mutatee runs UI Thread Mutatee Signal Generator Thread SH - 0 Block until exit event CB Manager register callback Continue execution Mailbox kill (SIGCONT) SH - 1 Event Gate

University of Maryland A (slightly) more complex example Eventually, mutatee executes dlopen() –Stops, sending SIGTRAP to mutator UI Thread Mutatee Signal Generator Thread SH - 0 SIGTRAP Block until exit event CB Manager register callback Continue execution Mailbox kill (SIGCONT) dlopen() SH - 1 Event Gate

University of Maryland A (slightly) more complex example SignalGenerator Decodes SIGTRAP –Analyzes runtime link maps, finds new library –Sends an evtLoadLibrary Event to a Signal Handler UI Thread Mutatee Signal Generator Thread SH - 0 SIGTRAP evtLoadLibrary Block until exit event CB Manager register callback Continue execution Mailbox kill (SIGCONT) dlopen() SH - 1 Event Gate

University of Maryland A (slightly) more complex example SignalHandler handles evtLoadLibrary –Adjusts Dyninst Internal State (platform indep) –Obtains relevant Callback from CBManager –“Executes” loadLibrary callback (zoom in to mailbox) UI Thread Mutatee Signal Generator Thread SH - 0 SIGTRAP evtLoadLibrary Block until exit event CB Manager register callback Continue execution Mailbox dlopen() SH - 1 Event Gate

University of Maryland A (slightly) more complex example Callback Execution (zoom in): –Registers instance of callback with Mailbox (target = UI Thread) –Signals Event Gates that evtLoadLibrary happened –Blocks until completion of callback UI Thread SH - 0 Block until exit event Mailbox Event Gate CB Block until CB complete Signal evtLoadLibrary Register CB

University of Maryland A (slightly) more complex example UI Thread, signalled, wakes up –Checks mailbox and finds callback with target_thread == UIThread –But the Event does not match target –Executes callback (zoom back out… ) UI Thread SH - 0 Block until exit event Mailbox Event Gate CB Block until CB complete Signal evtLoadLibrary Register CB Wake up, Execute Callbacks CB

University of Maryland A (slightly) more complex example But the callback triggers another event –InsertSnippet needs to allocate memory before insertion –Issues inferiorMalloc RPC and blocks for its completion UI Thread Mutatee Signal Generator Thread SH - 0 Block until exit event CB Manager register callback Continue execution Mailbox dlopen() SH - 1 Inferior Malloc RPC Insert: resume block Event Gate block

University of Maryland A (slightly) more complex example Mutatee issues malloc() and traps SignalGenerator receives SIGTRAP –Discovers that we just completed a RPC UI Thread Mutatee Signal Generator Thread SH - 0 Block until exit event CB Manager register callback Continue execution Mailbox dlopen() SH - 1 Inferior Malloc RPC Insert: resume block Event Gate malloc() block SIGTRAP

University of Maryland A (slightly) more complex example SignalGenerator sees that SH-0 is busy (blocked) –Assigns evtRPCSignal event to SH-1 UI Thread Mutatee Signal Generator Thread SH - 0 Block until exit event CB Manager register callback Continue execution Mailbox dlopen() SH - 1 Inferior Malloc RPC Insert: resume block Event Gate malloc() block SIGTRAP evtRPCSignal

University of Maryland A (slightly) more complex example SH -1 handles RPC completion event –Manages Dyninst internal state (RPC Manager) –Signals Event Gates that evtRPCSignal happened UI Thread Mutatee Signal Generator Thread SH - 0 Block until exit event CB Manager register callback Continue execution Mailbox dlopen() SH - 1 Inferior Malloc RPC Insert: resume block Event Gate malloc() block SIGTRAP evtRPCSignal

University of Maryland A (slightly) more complex example EventGate waiting for evtRPCSignal wakes up –Releases UI thread from RPC block UI Thread Mutatee Signal Generator Thread SH - 0 Block until exit event CB Manager register callback Continue execution Mailbox dlopen() SH - 1 Inferior Malloc RPC Insert: resume block Event Gate malloc() block SIGTRAP evtRPCSignal

University of Maryland A (slightly) more complex example Callback finishes executing –Inserts instrumentation in newly malloc()’d space –SH-0 is released from blocking state UI Thread Mutatee Signal Generator Thread SH - 0 Block until exit event CB Manager register callback Continue execution Mailbox dlopen() SH - 1 Inferior Malloc RPC Insert: resume block Event Gate malloc() resume SIGTRAP evtRPCSignal

University of Maryland A (slightly) more complex example UI Thread resumes blocking for process exit –Inserts instrumentation in newly malloc()’d space –SH-0 is released from blocking state UI Thread Mutatee Signal Generator Thread SH - 0 Block until exit event CB Manager register callback Continue execution Mailbox dlopen() SH - 1 Inferior Malloc RPC Insert: resume block Event Gate malloc()

University of Maryland A (slightly) more complex example Finally, Mutatee Exits –evtProcessExit propagates through Event Handling –UI Thread is released from block UI Thread Mutatee Signal Generator Thread SH - 0 resume Block until exit event CB Manager register callback Continue execution Mailbox dlopen() exit() SH - 1 Event Gate malloc() evtProcessExit Signal event: evtProcessExit

University of Maryland Thread Inflation aside… Good things have emerged too… Unified Dyninst event concept –All event oriented subsystems operate on the same generic event concept Well, its really just a new c++ class Unified Dyninst event handling –Event handling framework is now platform independent –Dyninst takes uniform actions when specific events occur, across platforms More or less But more than before

University of Maryland Implications (future features) Finer grained Callback APIs –We currently have callbacks for events like LoadLibrary Process Exit Thread Creation –Now possible (and easy) to expose generic event callback to user Eg. Execute func() when mutatee receives signal S Eg. Execte func() when mutatee issues dlclose()

University of Maryland Implications (future features) Spin-off “Debugger API” library –Separate out code for process control, signal handling, other debugger-like operations into shared library –Requires solving some complex problems: Eg. How to deal with RPCs How much gets spun off? –This is a hard question to answer –Related RPC and shared-object subsytems are heavily intertwined. –Where to draw the line?