1. Embedded Software for Video Wayne Wolf Princeton University and MediaWorks Technology

2. Outline zInitial observations. zSample application: smart cameras. zLevels of abstraction in software: yprocess; ydata; yinstructions. zLessons.

3. Characteristics of embedded computing zFunctional and non-functional requirements: performance, power, cost. zSome aspects of design methodology are similar to hardware design. zEmbedded software delivers more complex functions than HW models. zWill often be delivered in chips---that’s where the money is.

4. Ozer et al: human activity recognition JPEG IMAGE/ MPEG VIDEO Uncompressed IMAGE/VIDEO EIGENSPACE MATCHING PRINCIPAL COMPONENTS RELATIONAL GRAPH MATCHING Motion Vectors DCT Extracted Area Pixel Values OBJECT DETECTION

5. Previous work zWatlington et al.: Dataflow architecture for real-time video processing. zWandell et al.: Programmable sensor array. zFoote et al.: Panaromic camera system. zNicolescu et al.: Algorithms to pan, tilt, zoom array of cameras. zMIT Media Lab.: Smart rooms for several applications. zKeck Lab.: Multiple cameras on networked PCs.

6. Our hardware platform TriMedia Board TM32 (VLIW) PCI Bus Host PC Camera NTSC Shared Memory TriMedia Board TM32 (VLIW) Shared Memory Camera NTSC

7. Background elimination Skin color detection contour following Superellipse fitting graph matching Background elimination Skin color detection Contour following Superellipse fitting Graph matching

8. Skin color detectionContour followingSuperellipse fitting time

9. Skin detection CPU time

10. Contour following CPU time

11. Superellipse fitting CPU time

12. Graph matching CPU time

13. Characteristics of current video computing zStreaming data (with control). zA wide range of types of algorithms. Data volume Algorithm complexity

14. Challenges in embedded video software zVideo is more than pushing pixels. zPrograms run a long time---harder to verify. zMust derive programs through many stages from functional to real-time. zMust effectively use a deep memory hierarchy: registers through off-chip.

15. Characteristics of next- generation video systems zDistributed computation and networking: yDistributed analysis of video. yData fusion.

16. Design methodology for this system zAlgorithm design: Matlab. zC implementation on existing platform. zSoftware optimization. zDesign of custom platform and software.

17. Memory hierarchy management On-Chip Memory Off-Chip Memory Cache Buffers Registers

18. Levels of abstraction and optimization zInstruction: opcode selection, register allocation. zData: loop transformations, data layout, buffer optimization. zModule: inlining, outlining. zProcess: decomposition, communication, etc.

19. Questions from smart camera zInstructions: yWhich loops are predictable and which are not? zData: yHow does data layout change at each stage? yHow do we buffer and lay out data? zProcess: yHow do we manage multiple processes?

20. View from a start-up zStandards body software sucks. zConfigurable CPUs: offer some advantages, but harder to characterize. zEmbedded OS verification: WinCE, Linux. zMust manage multiple versions of software. zCan’t buy many useful components. zVerification, verification, verification.

21. ACM TECS zLook for the TECS Web site soon at www.ee.princeton.edu/~tecs