1 © Copyright 2008 EMC Corporation. All rights reserved. Processor-Controlled Test Updates and DFT Requirements Jeffrey A Moore EMC Corp.

2 © Copyright 2010 EMC Corporation. All rights reserved. A Typical Storage Processor (SP) Core CPU DIMMs PCT Pod DO AS YOU’RE TOLD Storage Processor Core ICH IOH LAN SIO PCIe NVRAM FLASH POST BIOS COMM2COMM1 LAN 1LAN 0  PCT Software controls CPU through PCT POD & Adapter  CPU registers in turn initialize and operate busses at normal speeds  Tests progress outward from CPU one chip at a time until all chips are tested  No need for DRAM, or BIOS, or Flash memory to be working or loaded with data  Libraries of chip models make init and operations easy to program

3 © Copyright 2010 EMC Corporation. All rights reserved.  Replace Custom Functional Test (CFT) w/ Basic Functional Test (BFT) Test Stage Replacement AOI AXI ICTESS (FRU Point) CFTCITFC AOI AXI ICTESS (FRU Point) BFTCITFC Test Suite to Contain: CMD Power Tests Boundary Scan (BScan) PCT Functional Test Code Load, Boot-up Custom Diag, POST, and Resume Write

4 © Copyright 2010 EMC Corporation. All rights reserved. Basic Functional Test (BFT) Improvements  BFT now combines 6 test steps into one –All steps using same test station, so no added tester costs or test stage –CMD Power test Checks voltages and margin status –BSCAN test checks shorts/opens and some memory tests –PCT test checks functionality of entire board –PCT / Embedded SAS Tests guarantee operation of 25 drive slots –Code load programs the CMD, MCU, BIOS, POST ROMs –Boot-up test allows board to boot and run POST to verify readiness for next stage CMDBSCANPCTSAS LOOP CODE LOAD BOOT-UP

5 © Copyright 2010 EMC Corporation. All rights reserved. Advantages of Combining all tests  CMD / BScan / PCT / Code Load merger into One Tester –Improved coverage significantly –Provides better diagnosis software at the CM for functional test –CMD provides quick debug of power problems –PCT reduces debug repair costs & eases lack of ability to use scope probes –PCT Diagnoses “No-Boot” problems to boost CFT, ESS, or CIT yield –SAS controller and SAS/Drive loop back bus tests provide quicker debug and better coverage

6 © Copyright 2010 EMC Corporation. All rights reserved. BFT CMD Test Capabilities  CMD Power Sequencer Hardware (Control & Monitoring Device) –All CPU & I/O boards have a CMD chip (DSP w/many A/D and PCM ports) –CMD sequences, monitors & adjusts all DC-DC power circuits –If a power-up fault occurs, sequencer stores values of supplies before shutting down –Communicates by TWI bus to test card’s USB-to-TWI pod  CMD Sequencer Test Software Features –Check CMD status –Change margins (normal/high/low) –User configuration data (Set margin limits) –Restore p-good limits to factory settings –Display Power down log (Useful for boards that won’t power up) –Reads the POL and Vin voltages and compare them against the PTR limit

7 © Copyright 2010 EMC Corporation. All rights reserved. Current SAS Testing at BFT  SAS Testing Carried out at BFT –Check for access & training to the SAS Controller (read Device/Vendor ID) –Individually test 4 SAS Controller PHYs that connect to the SAS Expander –Test 4 SAS Controller PHYs connected to SAS Front Panel Expansion Port –Test access to the SAS Controller EEPROM and Flash –PHYs that connect the SAS Expander to the SAS Controller are tested (4 Direct and 4 via the SAS Cable.)  Command sent to each Drive PHY to test device presence and speed  The SAS Address of the Attached Device is read and the Device is registered.  Buffer test is carried out on the Drive Buffer

8 © Copyright 2010 EMC Corporation. All rights reserved. Cost Savings Realized at BFT SAS Test  Removed SAS Controller Flash Prom. –Saved $2 per board - $1 for Flash part & $1 for Pre-programming it. –This eliminated ability to run SAS tests in BFT –Developed a method to store Flash data in memory and then have SAS Controller download it from there over PCIe bus. –Now SAS test is running OK again.  Embedded Test Techniques Save Tester Cost –Removed expensive 6GB SAS drives (Qty. 25) –Replaced drives with loop-back midplane –Developed use of chip vendors’ internal HS serial loop back tests –Protocol agnostic – vendor supports loopback test –Requires some DFT in design and vendor supplied firmware –Reduced tester cost by $7500 and saved headaches from having to debug drives through the fixture as they fail. –Able to show significant changes in the PMC DFE 0 and Gain values when a cap was removed

9 © Copyright 2010 EMC Corporation. All rights reserved. SP (Storage Processor) w/ Custom 25-Drive Midplane

10 © Copyright 2010 EMC Corporation. All rights reserved. Typical BFT Hardware

11 © Copyright 2010 EMC Corporation. All rights reserved.  Tests controlled and results displayed via custom “Block Diagram” GUI –GUI displays each chip tested and lights each one Green or Red as it is tested –Techs can use script language to flip bits, exercise buses, read back data, etc. –GUI written by Cork, Ireland Test Engineers using PCT vendor’s standard API  Test Time –Total BFT test time of UUT with 2 I/O cards is 13 minutes/unit –Code Load + Boot + POST is about 47 minutes – ouch!  Coverage –Was 51% coverage on robotic fault insertion vs 35-40% on CFT –Now have 87% coverage for BFT Tests –90% diagnostic accuracy vs ~70% on CFT –In actual run of boards, of 11 faults found by PCT, 10 were predicted correctly to a faulty device/device group  Yields –Prototype BFT yield was 83.5% (Boards had no ICT tests yet) –Subsequent CFT test had 92.8% yield –PCT test was improved to catch more faults, so final CFT yield is now 97.6% EMC BFT Test Features

12 © Copyright 2010 EMC Corporation. All rights reserved. DFT Needed for PCT Test  XDP connector too expensive and weak  Route signals through front panel or midplane connector  TCK, TMS, TDI, TRST, TDO routed to Processor  Main board 3.3V reset & 1.1V Reset from Processor  2 Handshake signals from Processor (PREQ & PRDY)  Most are 1.1V levels so pay attention to signal quality  Need to keep round-trip-time short from test card – cable – CPU  EMC BFT runs at 10 MHz TCK  PCT can run up to 20MHz  Need to use mux to switch CPU signals between XDP & External Conn  Test card uses mux to switch BSCAN / PCT/ MCU signals to JTAG bus

13 © Copyright 2010 EMC Corporation. All rights reserved. Conclusions / Future Needs  Conclusions –PCT is now integral part of BFT test suite –BFT is now integral part of our test process –Achieved replacement of CFT stage on latest products –PCT Improves Diagnosability of BFT tests –Use of Embedded SAS tests reduced cost of tester –Decreases Time To Market for new products –Enables use of less experienced techs at CMs –Enables development of more embedded tests and instruments  Future Needs –Looking for ways to reduce Code Load time –Need to develop multiple test sites on one tester like CFT –Investigating use of many PCT pods over Ethernet to one tester