1. Ritu Varma Roshanak Roshandel Manu Prasanna
Toward Target Level Testing and Debugging for Embedded Software Koehnemann, Lindquist
Ritu Varma
Roshanak Roshandel
Manu Prasanna

2. Introduction
Identifies problems with the test case execution for embedded systems and propose solutions
Testing is done to reveal errors whereas debugging is done to locate and correct the cause of error.
Embedded systems are usually constrained by
Concurrent design
Real time constraints
Distributed hardware architecture
Device control dependencies
Execution visibility and control severely restricted
Why testing in Embedded Systems necessary?

3. Software Testing Testing for embedded systems has four stages:
Module level Testing
Integration Testing
System Testing
Hardware/Software Integration Testing
Testing Methods which address the problem identified by the embedded systems
Concurrency
Real-time constraints
Embedded Environment
are of interest to this paper.

4. Testing Concurrent Systems
Concurrency increases testing difficulties:
Large set of execution sequences
Subsequent Execution with the same input  different results
Valid result as well as how the program arrived at that result
Effect of the environment on testing and its results
Acceptable output more important than correct output- Control over program execution
Research:
Static analysis: must examine a large set of state  not scalable
Dynamic testing: passive watch of execution and replaying the execution in case of failure
Runtime control of concurrent program
Static analysis- determine all parallel actions and states that could block a task from executing.
Run time control –regulates the sequence of events – assign each concurrent event in the test program a unique time value. test clock
The event would execute if its time value > test clock

5. Non-intrusive Testing
Control and testing techniques must not interfere with behavior of test program, timing issues
Why not instrumentation?
Insert probes to watch execution, link users object code with the rest of debugging system, Code interferes with execution behavior of the program
Intrusive, voids execution of system
Non-intrusive debugger for Ada
Separate processor executes testing system and communicates with the target processor through specialized hardware
Good start but don’t deal with the high level activities (process scheduling ,fault handling) only low level ( monitoring bus activity)
Other Real time embedded tools:
ROM monitors, Emulators, Bus Monitors

6. Impact of Underlying System
Problem: dealing with abstraction
Concurrency, IPC at high level and implementation at complier level (on run-time system)
In Development phase abstraction is fine but in Testing phase?
Complication
Without sufficient control over program , there is no way to ensure that a test case is testing the code.
Implementation difference in the host and the target environment.
Compilers provide runtime system operation to bridge the gap between language construct and hardware.

7. Current State of Embedded Testing
Concurrency causes Problems for testing and debugging
H/S integration testing reveal problems such as:
Incorrect handling of interrupts, Distributed Communication problems, Incorrect order of concurrent events, etc.
Goal: reveal errors early as possible Problem: Target level testing tools don’t exist
Two environments-
Host : supports s/w development and tools which give user control over program execution (high level interface)
Target: no support for software development and tools as minimum cost and space are the goals of the target.
Target level testing occurs late in the development cycle and a
small part is allocated for H/S integration testing.
But,Target is the only location where errors can be revealed.

8. Current Solutions Hardware solutions Software solutions
Attempts at gaining execution visibility and program control
Bus monitors, ROM monitors and in-circuit emulator
Minimal effectiveness for software development - low level machine data gathered only
Mapping b/w low level events and entities in the program
Software solutions
Attempts to reduce the cost and time spent testing on target
Determining factors in software testing:
Level of Criticality of software module
Test platform availability
Test Classification
Criteria's that assign program modules to test platforms
Type of software, Hardware requirements, Test classification, Platform availability, Certification Requirements
Current solution attempts to reduce the amount of target testing

9. Are these Techniques Effective?
Expensive
Low level of functionality
Available too late
Poor test selection criteria
What about new architectures?

10. Expensive Custom validation facilities Little or no reuse
Hardware is late and erroneous
Software must be malleable

11. Testing Problems Host: high-level interface, language
Target: machine instruction and physical address
Translation: Tedious and error prone
Migration to target
Hardware’s errors have to be mitigated by software
Late in the development process – costs

12. More Problems
Test cases and scenarios depend on tools and platforms instead of theoretical test criteria
Guidelines not being followed
Hardware architecture changes rapidly
Solution:
Adding functionality to the underlying system to support debugging
Hardware and run-time system
Specialized Middleware for development
and testing of embedded systems?

13. Model Debugging System Add functionality to target
Ada Semantic Interface Spec toolkit
Two communicating processor:
Running the test program
Running the debugger
Target processor can:
Execute and debug code only at a break point
Run the debugger as a separate processor
Provide a separate execution unit to execute the debugger
Interface between internal and external debugger

14. Changes in HW Architecture
Hardware partitioning of memory
Computational facilities for debugger
Hardware break points
Dedicated bus

15. Memory Partitioning Software is infinitely malleable
Any change requires several tests and retests
Late changes is expensive
Hardware should bound software
SW is usually partitioned based on critical level instead of design factors
 Hardware partitioning – protected address space for processes – restriction on access

16. Computational Facilities for Debugger
External debugger vs. Internal debugger
Internal debugger: located on target, executes from the target without interfering with the application
Runs as a regular process on the processor
Low-level or periodic high level process
Architecture provides separate facility for debugger execution
Target processor provides computational facility
Computation Facility utilize architecture to dump some representation of instruction being executed
Info must be parsed to create history. Implement state machines for messages sent by HW.
Program the (stateful) functional unit within the processor. At boot time, the internal debugger code gets loaded and reside there. Messages are passed between internal and external debuggers.

17. Hardware Breakpoints
Software and conditional breakpoints are not feasible for real time systems
Breakpoint registers – data and instruction types
State of a processor – all internal registers, pipeline and cache information must be saved when a break is encountered
Conditional breakpoints – how are they dealt with?

18. Architecture Support for Abstractions
Utilities must be built into architecture to provide basis for emulation, testing and program visibility
A new level of abstraction is needed:
Basic type (int, float)
Process, memory management, semaphore, etc
Hardware should be more aware of programming elements (software)
Awareness of the programming environment is important

19. Dedicated Bus
An interface between the processor and the external world
Protocol used across the connection
data rate, volume, direction, etc.
Master-slave relationship
Internal and external debuggers
Bus and debugger
Determination of active element within the processor
Processor is active – sends messages to debugger
Use special debugger portion of RTS – both debugger and RTS are active
Perhaps a combination of the two

20. Runtime System Additions
Goal: to minimize data and computational requirements of the internal debugger along with communication with the external debugger
Standardize services that provide visibility into RTS abstractions
Processes: concurrency is a common abstraction
Need to support modifying task state, communication and synchronization, and process scheduling
Interrupt management, Time management, Memory management, Exception Handling

21. Summary Embedded system testing techniques are ill-equipped
Architectural and RTS changes have to be made
Evaluate the changes and check feasibility
Definition of Embedded Systems??
What is the border between HW/SW?
Software techniques more suitable for less resource constraint system
Need for classification of embedded system??
Need for specialized middleware for both design and testing of embedded systems
Need for software development techniques although not discussed in the paper