MicroTESK: Automation of Test Program Generation for Microprocessors Alexander Kamkin Institute for System Programming of the Russian Academy of Sciences (ISPRAS)

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 2 of 23 What does “MicroTESK” stand for? Micro = Microprocessor  Microprocessor = programmable device Programmable = program-controlled device TESK = TEsting and Specification ToolKit  Testing = simulation-based verification  Specification = specification-driven approach  Toolkit = computer-aided automation MicroTESK is a specification-driven (model-based) approach that automates simulation-based verification of programmable devices by means of test programs

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 3 of 23 Levels of microprocessor verification lui s1, 0xdead ori s1, s1, 0x0 lui s3, 0xbeef ori s3, s3, 0xf add v0, a0, a2 sub t1, t3, t5 add t7, s1, s3 Core-level verificationUnit-level verification Model of a microprocessor is tested as a whole with the help of test programs Model of a particular unit is tested via inputs and outputs signals

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 4 of 23 Test programs-based verification Target Design (RTL, FPGA) lui s1, 0xdead ori s1, s1, 0x0 lui s3, 0xbeef ori s3, s3, 0xf add v0, a0, a2 sub t1, t3, t5 add t7, s1, s3 Test programs (assembler, C) Reference simulator (C/C++, ADL) Execution traces (formatted text) 0x2000: lui... 0x2004: ori... 0x2008: ori... 0x200c: lui... 0x2010: add... 0x2014: sub... 0x2018: add... Trace comparator (Perl, C/C++) 0x2000: lui... 0x2004: ori... 0x2008: ori... 0x200c: lui... 0x2010: add... 0x2014: sub... 0x2018: add...

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 5 of 23 Approaches to test program construction IF IDEXWB IFIDEXWB Requirements Design Unformalized knowledge Formalized knowledge Test programs Manual Development Random Generation Template-Based Generation Model-Based Generation ?  lui s1, 0x2779 ori s1, s1, 0x0 lui s3, 0x4ee ori s3, s3, 0xf add v0, a0, a2 sub t1, t3, t5 add t7, s1, s3

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 6 of 23 Advantages of model-based testing Formalization helps to find problems in a design  Deeper comprehension of requirements  Formal checking of model consistency Models automate verification of a design  Correctness estimation  Test sequence generation  Testing adequacy criterion Models simplify tests maintenance  One model, many tests  Reuse of models and testing knowledge

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 7 of 23 Approaches to design modeling ADD x, y, z ≥ 0 LD u, x, Cache_Hit DIV y, u, Division_By_Zero... DIV.S f, g, Hard_To_Round ADD.S i, f, Inexact MUL.S g, i, Invalid Instruction-level modelsCycle-accurate models Test templates are used for verificationTo generate tests, FSM models are used ABS.fmt (fd:FPR, fs:FPR) {...} ADD (rd:GPR, rs:GPR, rt:GPR) {...} ADD.fmt (fd:FPR, fs:FPR, ft:FPR) {...}... XOR (rd:GPR, rs:GPR, rt:GPR) {...} XORI (rd:GPR, rs:GPR, rt:IMM) {...}

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 8 of 23 Combinatorial test program generation Tests are constructed by combining  Sequences of instructions  Test situations for instructions  Dependencies between instructions  Execution traces, if branches are used Volume of tests is reduced by heuristics  Equivalence of instructions  Length of sequences  Depth of dependencies  Length of execution traces

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 9 of 23 Structure of test programs initialization test case … test action … load a, (b) add c, a, d load e, (f) move c, 1 store t, (b) move t, 0 move b, 0x4 move f, 0x4 store t, (f) move t, 2 move g, 0xd... bne c, g, error test oracle address dependency register dependency test situation

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 10 of 23 Instruction specification Instruction interface  Instruction operands  Instruction precondition Execution function  Calculates outputs  Updates a model state Assembler format  Textual representation temp  (GPR[rs] 31 ||GPR[rs] ) + (GPR[rt] 31 ||GPR[rt] ) if temp 32  temp 31 then SignalException(IntegerOverflow) else GPR[rd]  sign_extend(temp ) endif if NotWordValue(GPR[rs]) or NotWordValue(GPR[rt]) then UNPREDICTABLE endif output rd: GPR, Word input rs: GPR, Word input rt: GPR, Word ADD rd, rs, rt instruction ADD(rd, rs, rt)

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 11 of 23 Test situations and dependencies Parameters  Specifies family of test situations / dependencies Constructor / constraint  Constructs / constrains values of the operands Preparator  Creates initialization program of the instruction if NotWordValue(GPR[rs]) or NotWordValue(GPR[rt]) then UNPREDICTABLE endif temp  (GPR[rs] 31 ||GPR[rs] ) + (GPR[rt] 31 ||GPR[rt] ) if temp 32  temp 31 then SignalException(IntegerOverflow) else GPR[rd]  sign_extend(temp ) endif 0x0af1036f0x1 0xf47298b90x3 0xffff0a31 0x0 0xdeadbeef0xd datatag 0x19d02c340xa 0x7420ba9c0x2... Cache (L1/L2)TLB 0xa0000x0000 0xb0200x xa0100x003e 0xb0000x003f PFNVPN

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 12 of 23 load ?, (?) add ?, ?, ? load ?, (?) Test program generation scheme load a, (b) add c, a, d load e, (f) cache miss cache hit normal move c, 1 store t, (b) move t, 0 move b, 0x4 move f, 0x4 store t, (b) move t, 2 Test program add... sub store load cache hit cache miss normal overflow register dependency address dependency register dependency   generation parameters lui s1, 0xdead ori s1, s1, 0x0 lui s3, 0xbeef ori s3, s3, 0xf add v0, a0, a2 sub t1, t3, t5 add t7, s1, s3

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 13 of 23 MicroTESK tool support The approach is supported by the MicroTESK test program generator  Test programs are generated according to the specified goals  Technology and tool provide high level of automation  Test programs might include checks of microprocessor state  Specifications are developed in Java language  There are ready-to-use test data generators  Generator has graphical user interface

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 14 of 23 Generator development process

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 15 of 23 How a MicroTESK generator looks like

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 16 of 23 Case studies Memory subsystem MIPS64 microprocessor DSP co-processor Floating-point co-processors DMA controller Branch prediction unit

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 17 of 23 Some statistics CharacteristicMMUMIPS64DSPCP1CP2 Number of instructions Efforts, man-months Volume of specifications and tests, lines of code Number of found bugs in a reference simulator 06> 10 Number of found bugs in an RTL model 19> 10 Many critical bugs were found in the designs, which had been verified by hand-written, random-generated, and template-based test programs

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 18 of 23 Future directions Automatic extraction of test coverage Advanced test template generation Integration with ADL (ISE, nML, LISA, etc.)

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 19 of 23 Extraction of test situations L1 TLB Hit, ValidHit, Invalid Exception: TLBInvalid HitMiss Exception: TLBRefill Miss Normal1Normal2

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 20 of 23 Extraction of dependencies associative buffer TLB { VIRTUAL_PAGE_NUMBER tag( VIRTUAL_ADDRESS va) {... }... INTEGER index(VIRTUAL_ADDRESS va) {... }... } structure TLB_ENTRY { VIRTUAL_PAGE_NUMBER vpn; PHYSICAL_PAGE_NUMBER pfn;... }

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 21 of 23 MicroTESK project Download the MicroTESK demo

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 22 of 23 Contacts Institute for System Programming of RAS (ISPRAS) Hardware Verification ISPRAS Alexander Kamkin

IEEE East-West Design and Test Symposium - September 18-21, Moscow, Russia 23 of 23 Thank You! Questions?