11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center

22 Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALE AND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTY RIGHT. Intel may make changes to specifications, product descriptions, and plans at any time, without notice. Intel may make changes to specifications, product descriptions, and plans at any time, without notice. All dates provided are subject to change without notice. Intel is a trademark of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Copyright © , Intel Corporation. All rights are protected.

33 Agenda Overview Architecture Auto Nightly Test Infrastructure Auto Nightly Analysis Plan Reference Catch and analyze every regression!

44 (commits)

55 Background Integrated Test suites: –Intel-gpu-tools (11 cases) –Rendercheck (10 cases) –Piglit (7500+ cases) –Cairo_test (1272 cases) –OpenGL conformance (4565 cases) –OpenGL ES 2.0 conformance (1198 cases) –Webglc (5685 cases)

66 Architecture DATABASE Auto Build Auto Test Auto Analysis Test Result Page gfx-ab-at infrastructure auto_bisect

77 Architecture

88 Information required for analysis Hardware information –GPU, CPU, Memory Build information –Software version Test information –Test case version –Test case result

99 Auto Nightly Build Build & Distribute Xorg components Build & Distribute Kernel package Build & Distribute Test suites Archives –X11R7.*.tgz –Kernel.* package Build info -> DATABASE –Component branch/commit –Infrastructure branch/commit –Test suite branch/commit

10 Auto Nightly Test Run test cases one by one Test log collection –Run time –Output –Dmesg per case test result ->DATABASE Test status watchdog –TIMEOUT -> killed –GPU hang -> reboot -> resume to next case –System hang -> reboot -> resume to next case Pre/post-condition check

11 Auto Nightly Test Process Target Test machine GPU hang check TIMEOUT control Running test case Build/control machine System hang check TIMEOUT control Resume after reboot Auto Test Control Init & control Finish & collect logs

12 Auto Nightly Analysis Track result change –Regression –Improvement Locate culprit component –Xorg component –Infrastructure –Testsuite Locate culprit commit –Xorg component –Infrastructure –Testsuite

13 Auto Nightly Analysis Process Test suite regression analysis Infrastructure regression analysis Kernel regression analysis Auto bisect Component regression analysis package regression analysis

14 Component regression analysis Purpose –Locating culprit component –uncommon component break difficulties –Too many components –Library dependency

15 Locate culprit component

16 Locate culprit component Reduce analysis effort –Only analyze libraries in use –Only analyze changed libraries Reduce dependency –Group related library

17 Locate culprit commit

18 bad branch first algorithm If A is bad commit, and C fixed it, and then F is bad again, A -> B -> C -> D -> E -> F -> G -> H (master) \ \ / O -> P... Q -> R -> S->T (feature 1) Start with H as bad commit, and D as good commit, it's possible git-bisect would jump to Q, and it will lead to wrong direction

19 Bad branch first algorithm EXAPMLE 2: Given this condition: A -> B -> C -> D -> E -> F -> G -> H (master) \ O -> P -> Q -> R -> S / (feature 1) \ X -> Y -> Z/ (feature 2) start with H as bad commit, and A as good commit, if y is the target bad commit.

20

21 Auto Nightly Analysis Arch

22 Auto Nightly Analysis Build machine Test machi ne Regression analysis Engineer s Build&d ist Case 1 Analyze case 1 Analysis report reques t Case 2 All case complete Test result Test report summary Case n Analyze case n Analysis report reques t regression

23 Nightly Test Result Page

24 Analysis enhancement Analysis accuracy –Verification Make sure all parents commit are good Analysis acceleration –Archive built binaries –Analyze regression in batch –Archive testing result

25 Some results Regression auto bisected Improvement auto bisected 2011-Aug Jul Jun May Apr Mar Feb Jan1933

26 Risk Unstable –System may hang –GPU hang –Test case unstable

27 Plan triggered patch test system PRTS (Patch Regression Test Suite) –Select test case set based on code modification Nightly Auto Test –Capture more info when case crashed Auto bisect for kernel/GPU hang Nightly test result page –Single case test result query –Quality trend indicators Pass rate Regression rate …

28 Q&A