M Achutha KiranKumar V Bindumadhava S S Abhijith A Bharadwaj

Slides:



Advertisements
Similar presentations
Using Formal Verification to Replace Mainstream Simulation Erik Seligman Intel Brandon Smith Intel
Advertisements

April 30, A New Tool for Designer-Level Verification: From Concept to Reality April 30, 2014 Ziv Nevo IBM Haifa Research Lab.
Putting It All Together: Using Formal Verification In Real Life Erik Seligman CS 510, Lecture 19, March 2009.
- Verifying “Golden” reused IPs The Evil’s in the Edits William C Wallace Texas Instruments Nitin Jayaram Texas Instruments Nitin Mhaske Atrenta Inc Vijay.
The Future of Formal: Academic, IC, EDA, and Software Perspectives Ziyad Hanna VP of Research and Chief Architect Jasper Design Automation Ziyad Hanna.
CS351 © 2003 Ray S. Babcock Software Testing What is it?
SE 450 Software Processes & Product Metrics Reliability: An Introduction.
CMMI PMC Group Members Inam ul Haq Sajjad Raza Nabeel Azam
16/27/2015 3:38 AM6/27/2015 3:38 AM6/27/2015 3:38 AMTesting and Debugging Testing The process of verifying the software performs to the specifications.
EE694v-Verification-Lect5-1- Lecture 5 - Verification Tools Automation improves the efficiency and reliability of the verification process Some tools,
Swami NatarajanJuly 14, 2015 RIT Software Engineering Reliability: Introduction.
Software Testing Prasad G.
Stoimen Stoimenov QA Engineer QA Engineer SitefinityLeads,SitefinityTeam6 Telerik QA Academy Telerik QA Academy.
Introduction to Software Testing
Handouts Software Testing and Quality Assurance Theory and Practice Chapter 11 System Test Design
University of Palestine software engineering department Testing of Software Systems Fundamentals of testing instructor: Tasneem Darwish.
© BJSS Limited Going Agile UK TMF - April 2011 Mark Crowther, Test Consultant.
Project Tracking. Questions... Why should we track a project that is underway? What aspects of a project need tracking?
The First in GPON Verification Classic Mistakes Verification Leadership Seminar Racheli Ganot FlexLight Networks.
1 Hybrid-Formal Coverage Convergence Dan Benua Synopsys Verification Group January 18, 2010.
From Quality Control to Quality Assurance…and Beyond Alan Page Microsoft.
Confidential Continuous Integration Framework (CIF) 5/18/2004.
Introduction to Hardware Verification ECE 598 SV Prof. Shobha Vasudevan.
Software Test Plan Why do you need a test plan? –Provides a road map –Provides a feasibility check of: Resources/Cost Schedule Goal What is a test plan?
Cookie-cutter properties to assist non Formal experts Bin Xue.
Grid as a Service. Agenda Targets Overview and awareness of the obtained material which determines the needs for defining Grid as a service and suggest.
Practical Methods in Coverage- Oriented Verification of the Merom Microprocessor Alon Gluska.
Introduction to CSCI 1311 Dr. Mark C. Lewis
Tool Support for Testing
Applying Combinatorial Testing to Data Mining Algorithms
Architecture Review 10/11/2004
Items to consider before automating an application
MASS Java Documentation, Verification, and Testing
Agile Metrics that Matter
TEST AUTOMATION IN BDD WAY
Preparing for Automation: Expanding Your Coverage
A scalable approach for Test Automation in Vector CAST/Manage with
Platform Overview Provide your marketing and sales groups with a single, integrated, web based on-demand platform that allows them to easily automate and.
Test Around the Clock Testing Revolutionized
Software Verification and Validation
421 Review Questions Does software engineering add documentation that slows down the project? Is there one software process that is better than the others.
Continuous Integration and Testing
Test Driven Development 1 November Agenda  What is TDD ?  Steps to start  Refactoring  TDD terminology  Benefits  JUnit  Mocktio  Continuous.
T Project Review Group: pdm I2 Iteration
Software engineering – 1
Formal Methods (i.e. mathematical, algorithmic) for Software and Hardware Designs and, more generally, Design Tools and Technologies
Balanced Scorecard Designer
Effective way to build test Automation strategy in Agile
Raytheon Parts Management
Alon Flaisher Alon Gluska Eli Singerman Intel Corporation
Product Development Scenario Overview
Introduction to Software Testing
YeahMobi CD Practice based on Container -- openstack meetup
Thursday’s Lecture Chemistry Building Musspratt Lecture Theatre,
Automation Of Software Test
Template Name: ENGAGE™ Business Readiness Checkpoint Report
Using Formal Coverage Analyzer for Code Coverage improvement
CSE 303 Concepts and Tools for Software Development
1.2 System Design Basics.
Baisc Of Software Testing
Test Case Test case Describes an input Description and an expected output Description. Test case ID Section 1: Before execution Section 2: After execution.
Scriptless Test Automation through Graphical User Interface
Automated test.
HP Quality Center 10.0 The Test Plan Module
Overview of Workflows: Why Use Them?
Agile Development – a new way of software development?
SOFTWARE DEVELOPMENT LIFE CYCLE
CSE 1020:Software Development
Executive Project Kickoff
Building your Webex adoption dream team
Presentation transcript:

M Achutha KiranKumar V Bindumadhava S S Abhijith A Bharadwaj Making Formal Property Verification Mainstream: An Intel® Graphics Experience M Achutha KiranKumar V Bindumadhava S S Abhijith A Bharadwaj © Accellera Systems Initiative

© Accellera Systems Initiative AGENDA Why? Need for FV Deployment Drive How? VPG FV Deployment Planning and Execution What was Achieved? Results of FV Deployment © Accellera Systems Initiative

Need of FV Deployment Drive Formal needs no introduction: Established 100% state space coverage NO/Minimal Bug Escapes High Verification Confidence Testbench Free Verification Various dedicated Apps for Specific problems: RTL Equivalence Sequential Check, Connectivity, X verification, Security, Low power, etc Still… Why are the FV adoption rates are still low? Why was the dedicated FV deployment drive in VPG needed? © Accellera Systems Initiative

Need of FV Deployment Drive Common FV Roadblocks: FEAR: MYTH: FV is only meant for PhDs MYTH: Will drain my time/resources No Progress Metrics No intermediate check-points ROI: TRUTH: What eventually matters?: ROI How to compare with DV results? © Accellera Systems Initiative

© Accellera Systems Initiative AGENDA Why? Need for FV Deployment Drive How? VPG FV Deployment Planning and Execution What was Achieved? Results of FV Deployment © Accellera Systems Initiative

VPG FV Deployment Drive © Accellera Systems Initiative

VPG FV Deployment Drive © Accellera Systems Initiative

VPG FV Deployment Drive: Survey Immediate needs for most of the users: Automated setup/property_coding Assertion Libraries for common design components Dedicated Trainings Local Contact Points © Accellera Systems Initiative

VPG FV Deployment Drive: Preparation Phase GUI creation Provided a quick setup interface to new users One stop shop for all formal flows Easy way of generating all collaterals for formal © Accellera Systems Initiative

VPG FV Deployment Drive: Preparation Phase Cookie-cutter Property Templates Automated properties © Accellera Systems Initiative

VPG FV Deployment Drive: Preparation Phase Cookie-cutter Property Templates Automated properties Standard Interfaces © Accellera Systems Initiative

VPG FV Deployment Drive: Preparation Phase Cookie-cutter Property Templates Automated properties Standard Interfaces Arbiter Properties © Accellera Systems Initiative

VPG FV Deployment Drive © Accellera Systems Initiative

VPG FV Deployment Drive: Training Phase Identifying Local Experts FV champion from each cluster Immediate support Setting-up Self Help Materials Dojos FAQs Training Dungeons 2-week rigorous training © Accellera Systems Initiative

VPG FV Deployment Drive © Accellera Systems Initiative

VPG FV Deployment Drive: Test Planning Phase Clear Goal Definitions Assigning FV problem to correct category FV Problem Full Proof Only Bug Hunting Equivalence Check © Accellera Systems Initiative

VPG FV Deployment Drive: Test Planning Phase Progress Continuum Defining measurable sub-goals Allows progress to be graded w.r.t. time Decided % completions to be achieved for each Val Milestone © Accellera Systems Initiative

VPG FV Deployment Drive: Test Planning Phase Activity Ownership Designer needs to monitor FV completion for his unit © Accellera Systems Initiative

VPG FV Deployment Drive © Accellera Systems Initiative

VPG FV Deployment Drive: Execution & Closure Execution and Tracking Round-the-clock support extended to FV owners after dungeons Weekly FV progress tracked and published Deep Bug Exploration Abstractions, different engine settings Lots of Covers added Coverage and Closure Line and branch coverage obtained Properties added for holes © Accellera Systems Initiative

VPG FV Deployment Drive © Accellera Systems Initiative

VPG FV Deployment Drive: Communicate Results ROI calculation standard was established ROI numbers calculated for activity and compared against DV Parameter Depiction Number of engineers worked on the problem NE_F Average bandwidth/engineer BW_F Number of weeks spent on effort Ww_F Total time spent in hours Ts_F = Bw_F a Ww_Faa 40b Project schedule in weeks Pw_F Scaled engineering costs CE_F = Ts_F/(Pw_F a 40) Machines used for computations Mc_F Runs/week Rw_F Number of days of run Dw_F Total machine-days/week Tm_F = Mc_F a Rw_F a Dw_F Project machine-days/week Pm_F Scaled machine costs Cm_F = Tm_F/Pm_F Coverage (% cover points hit) Covtot_F Bugs found by the method NB_F aAssuming eight hours/day for a five-day work week = 40 hours/week. bMachine runs are not limited to a working day but for the whole week ROI Parameter Formal Dynamic Relative Bugs% found by FV NB_F/(NB_F+NB_D) NB_D/(NB_F+NB_D) NB_F/NB_D Total coverage achieved Covtot_F Covtot_D   Bug - engineering cost ROI BEC_F = NB_F/CE_F BEC_D = NB_D/CE_D BEC_F/BEC_D Bug - machine cost ROI BMC_F = NB_F/Cm_F BMC_D = NB_D/Cm_D BMC_F/BMC_D Coverage - engineering cost ROI CEC_F = Covtot_F/CE_F CEC_D = Covtot_D/CE_D CEC_F/CEC_D Coverage - machine cost ROI CmC_F = Covtot_F/Cm_F CmC_D = Covtot_D/Cm_D CmC_F/CmC_D © Accellera Systems Initiative

VPG FV Deployment Drive © Accellera Systems Initiative

VPG FV Deployment Drive: Check-In & Regress FV cannot be done once & forgotten RTL in constant churn Regressions to be enabled to find bugs due to changes © Accellera Systems Initiative

© Accellera Systems Initiative AGENDA Why? Need for FV Deployment Drive How? VPG FV Deployment Planning and Execution What was Achieved? Results of FV Deployment © Accellera Systems Initiative

Results of VPG FV Deployment Quality of reception: 80+ engineers became FV experts 15 bugs caught within 2 weeks of activity It uncovered many scenarios in the beginning for which the Fulsim tests came very late. it has given very high confidence that new RTL has achieved the original health. FV is awesome; I write code, run FV before even committing it to my local repo, and get it clean. If it wasn't for this tool, I'd be building micro-testbenches before the ULT is ready. FV is the prime mode of validation for one of my unit. While DV gave me a bill of 6-8 weeks, we verified all the DCN changes in less than 2 weeks using FV Found couple of dozen bugs even before DV was up and helped clean the RTL On a FV Quality bug hit: Hitting this bug in functional random env would have taken a long time and the bug is fatal enough to cripple the feature. © Accellera Systems Initiative 11/27/2018 Achutha KiranKumar V M, Intel Corp. 26

Results of VPG FV Deployment Quality of Bugs: 85+ bugs caught in 12 weeks (activity was started after half way through DV drive) Most of the bugs caught fell in corner-case category 32-bit Internal Counter (CI) A B If CI = 32’h8001_000; ABBB seq at I/P received |-> CO = (CI – 2) CO Expected Value = 32’h8000_FFFE Actual Value = 32’h8001_FFFE © Accellera Systems Initiative

Results of VPG FV Deployment ROI: Weighted ROI of 220x compared to DV Some activities gave as much as 138x w.r.t. engg costs; 3692x w.r.t. machine costs © Accellera Systems Initiative

© Accellera Systems Initiative CONCLUSION FORMAL VERIFICATION IS NOW FORMALLY IN THE PROJECT!! © Accellera Systems Initiative

© Accellera Systems Initiative Questions © Accellera Systems Initiative