1. Testing framework for BlackFin on uClinux Grace Pan

2. Testing on Blackfin  Toolchain testing: Use Dejagnu to ease/automate 60,000+ Toolchain tests on simulator, hardware, and hardware with fdpic format. All features of the toolchain (C, C++, gdb, gas, binutils, etc.) are ‘exercised’  Kernel testing All tests run on different targets: BF533-STAMP, BF537-STAMP, BF561- EZKIT, BF533-EZKIT, even custom boards. Run Linux Test Project (LTP)(about 600 cases), which is a collection of test suites to validate the reliability, robustness, and stability of Linux. Blackfin/uClinux is the first architecture to run LTP on uClinux/uClibc/nommu architecture. Automated stress tests on kernel and device drivers using expect scripts.  BootLoader Tested on BF533-STAMP and BF537-STAMP of uboot commands and features. All tests are Well documented on docs.blackfin.uclinux.org and can be reproduced

3. Uclinux-dist testsuites  under uclinux-dist source tree testsuites: kernel_test : a line that connects all the cases under this folder to run one by one. kernel_config.exp: the file that sets the common used params. xxx/build_xxx_kernel.exp: to build a kernel for a certain xxx case. xxx/xxx_test.exp : auto script for xxx testing. test_scripts/run_kernel_test: run kernel_test and call summary and compare scripts. test_scripts/get_test_summary: do summary for each round of testing. test_scripts/compare_kernel_results: compare test results of two rounds of testing. test_scripts/test_runtime_control: kill the case that runs abnormal and lasts too long.  Run a certain case, rtc,for example: cd testsuites/rtc. Run./build_rtc_kernel.exp BF537-STAMP Run./rtc_test.exp. Log files are generated as build_rtc_kernel_log and rtc_test_log.  Run the testcases on regression machine: cd /home/test/test_scripts/uclinux-dist cp run_kernel_test to this folder and set the particular value for the testing params. Since this file is different for each testing bench, so it is not updated and copied out when setting up the testing, no much modification. do./run_kernel_test.

4.  Toolchain Testing machine: 10.99.22.40 - toolchain 3.4 BF537, host Suse 10.99.22.41 - toolchain 4.1 BF537, host Suse 10.99.22.42 - toolchain 4.1 BF533, host Suse 10.99.22.43 - toolchain 4.1 BF561, host Suse 10.99.22.146- toolchain 4.1 BF537, host Mac  Uboot Testing machine: 10.99.22.58 - BF533, host Suse 10.99.22.59 - BF537, host Suse

5. Uclinux-dist Testing machine: 10.99.22.50 - BF537-STAMP-CF, FLAT 10.99.22.51 - BF537-STAMP-STD, FLAT 10.99.22.52 - BF537-STAMP-AD73311, ELF 10.99.22.54 - BF537-STAMP-ADV7393, FLAT 10.99.22.55 - BF533-STAMP-STD, FLAT 10.99.22.56 - BF533-STAMP-STD, ELF 10.99.22.57 - BF537-DIGIUM-PBX, FLAT 10.99.22.46 - BF533-EZKIT-STD, FLAT 10.99.22.47 - BF561-EZKIT-STD, FLAT

6. CruiseControl Overview  CruiseControl is a framework for a continuous build process. It includes, but is not limited to, plugins for email notification, Ant, and SVN, etc.  CruiseControl 2 main modules: the build loop: core of the system, it triggers build cycles then notifies various listeners (users) using various publishing techniques. The trigger can be internal (scheduled or upon changes in a SCM) or external. It is configured in a xml file which maps the build cycles to certain tasks, thanks to a system of plugins. Depending on configuration, it may produce build artifacts. the build loopconfigured in a xml fileplugins the legacy reporting allows the users to browse the results of the builds and access the artifacts the legacy reporting  Cruisecontrol finds bugs during development process, not at the last stage of testing.  Main page: http://cruisecontrol.sourceforge.net/index.htmlhttp://cruisecontrol.sourceforge.net/index.html

7. Set up CruiseControl  Download cruisecontrol-xxx-2.x.zip from the website.cruisecontrol-xxx-2.x.zip  cd /home/test/work, unzip the zip package.  Follow the instructions in page: http://cruisecontrol.sourceforge.net/gettingstarted.html to build and install cruisecontrol.http://cruisecontrol.sourceforge.net/gettingstarted.html  cd /home/test/work/cruise/checkouts/. Check out the project svn source tree here, for example uclinux-dist project.  cd /home/test/work/cruise/, edit config.xml, build-uclinux-dist.xml, checkouts/uclinux- dist/build.xml to fit into this testing.  mkdir /home/test/work/cruise/test_scripts/uclinux-dist/, copy run_kernel_test over as the startup script from uclinux-dist source tree subfolder testsuites/test_scripts.  cd /home/test/work/cruise/, run./main/bin/cruisecontrol.sh. It would use the config.xml in that folder to read the config params.  The log files would be saved at test_scripts/uclinux-dist/logs and part of it would be copied to logs/uclinux-dist/.  Add gforge jars if the result need to be reported to gfore website.

8. Cruisecontrol config.xml <plugin name="gforge" classname="org.gforge.cruisecontrol.publishers.GForgeSOAPExternalPublisher" host="blackfin.uclinux.org/" ssl="false" /> … …

9. <htmlemail mailhost="10.99.22.20" returnaddress="grace.pan@analog.com" buildresultsurl="http://10.99.22.164:8080/cruisecontrol/buildresults/${project.name}" skipusers="true" spamwhilebroken="true" css="/home/test/work/cruise/reporting/jsp/webcontent/css/cruisecontrol.css" xsldir="/home/test/work/cruise/reporting/jsp/webcontent/xsl" logdir="logs/${project.name}"> <ftppublisher destDir="cruisecontrol/${project.name}" srcDir="logs/${project.name}" targetHost="blackfin.uclinux.org" targetUser="test" targetPasswd=""/> <gforge projectId="17" cruisecontrolId="5" hash="JzL5PCUxfHCEBzNS6FLkAG9Rp0k=" />

10.  build-uclinux-dist.xml <project name="build-uclinux-dist" default="build" basedir="checkouts/uclinux-dist">  build.xml

11.  Execution sequence:./main/bin/cruisecontrol.sh ↓ config.xml ↓ build-uclinux-dist.xml ↓ checkouts/uclinux-dist/build.xml ↓ run_kernel_test ↓ kernel_test

12. Cruisecontrol starting up./main/bin/cruisecontrol.sh Using Cruise Control at /home/test/work/cruise/main /usr/lib/jvm/java/bin/java -cp /usr/lib/jvm/java/lib/tools.jar:/home/test/work/cruise/main/dist/cruisecontrol.jar:/home/test/work/cruise/main/lib/log4j.jar:/ho me/test/work/cruise/main/lib/jdom.jar:/home/test/work/cruise/main/lib/ant/ant.jar:/home/test/work/cruise/main/lib/ant/ant- launcher.jar:/home/test/work/cruise/main/lib/xercesImpl-2.8.0.jar:/home/test/work/cruise/main/lib/xml-apis- 2.8.0.jar:/home/test/work/cruise/main/lib/xmlrpc-2.0.1.jar:/home/test/work/cruise/main/lib/xalan- 2.7.0.jar:/home/test/work/cruise/main/lib/serializer-2.7.0.jar:/home/test/work/cruise/main/lib/jakarta-oro- 2.0.3.jar:/home/test/work/cruise/main/lib/mail.jar:/home/test/work/cruise/main/lib/junit.jar:/home/test/work/cruise/main/lib/a ctivation.jar:/home/test/work/cruise/main/lib/commons-net-1.1.0.jar:/home/test/work/cruise/main/lib/starteam- sdk.jar:/home/test/work/cruise/main/lib/mx4j.jar:/home/test/work/cruise/main/lib/mx4j- tools.jar:/home/test/work/cruise/main/lib/mx4j- remote.jar:/home/test/work/cruise/main/lib/smack.jar:/home/test/work/cruise/main/lib/comm.jar:/home/test/work/cruise/mai n/lib/commons-logging.jar:/home/test/work/cruise/main/lib/x10.jar:/home/test/work/cruise/main/lib/fast- md5.jar:/home/test/work/cruise/main/lib/maven-embedder-2.0.3- dep.jar:/home/test/work/cruise/main/lib/axis.jar:/home/test/work/cruise/main/lib/jaxrpc.jar:/home/test/work/cruise/main/lib/ wsdl4j-1.5.1.jar:/home/test/work/cruise/main/lib/commons-codec-1.3.jar:/home/test/work/cruise/main/lib/commons- discovery-0.2.jar:/home/test/work/cruise/main/lib/commons-httpclient-3.0- rc3.jar:/home/test/work/cruise/main/lib/crimson.jar:/home/test/work/cruise/main/lib/saaj.jar:/home/test/work/cruise/main/lib/ gforge-external-publisher.jar:. -Djavax.management.builder.initial=mx4j.server.MX4JMBeanServerBuilder CruiseControl [cc]Jun-21 17:48:59 Main - CruiseControl Version 2.5 Compiled on June 1 2007 1205 [cc]Jun-21 17:48:59 LConfigManager- reading settings from config file [/home/test/work/cruise/config.xml] [cc]Jun-21 17:48:59 trolController- projectName = [uclinux-dist] [cc]Jun-21 17:48:59 trolController- No previously serialized project found [/home/test/work/cruise/uclinux-dist.ser], forcing a build. [cc]Jun-21 17:48:59 LConfigManager- using settings from config file [/home/test/work/cruise/config.xml] [cc]Jun-21 17:48:59 Project - Project uclinux-dist starting [cc]Jun-21 17:48:59 Project - Project uclinux-dist: idle [cc]Jun-21 17:48:59 Project - Project uclinux-dist started [cc]Jun-21 17:48:59 Project - Project uclinux-dist: in build queue [cc]Jun-21 17:48:59 BuildQueue - BuildQueue started

13. Log20070615145327.xml: 5394 /trunk/lib/ffmpeg/ffmpeg-svn-8891/libavutil/common.h 06/12/2007 19:26:33 mmh 5394 …. …… ……

14. Changes in CruiseControl  Add test result judgement in cruise/main/src/net/sourceforge/cruisecontrol/util/XMLLogHelper.java: public boolean isBuildSuccessful() { String filename = "/home/test/work/cruise/test_scripts/uclinux-dist/logs/thisrun/test_summary/test_results"; FileReader fr = null; try { fr = new FileReader(filename); BufferedReader br = new BufferedReader(fr); String line = null; String result = null; line = br.readLine(); while (line != null) { result = line.trim(); if (result.equals("success")) { return true; } else if (result.equals("failed")) { return false; } br.close(); fr.close(); } catch (IOException e) { e.printStackTrace(); } return (log.getChild("build").getAttribute("error") == null); }

15.  Add detailed log in html mail. The xsl scripts to translate the xml log into html is under cruise/reporting/jsp/webcontent/xsl. Add in buildresults.xsl : … Create build.xsl according to the template of errors.xsl.

16. Example of cruisecontrol log mail [Uclinux-dist-test-notify] uclinux-dist Test Failed BUILD COMPLETE - build.3 Date of build:06/19/2007 12:07:28 Time to build:1,560 minutes 45 seconds Last changed:06/19/2007 09:10:31 Last log entry:[#2443] Update test scripts. Log: Testing on 10.99.22.50 with BF537-STAMP-CF (BF537) flat slab(p2) ICACHE(on) DCACHE(on) write_through dma Wed Jun 20 05:07:57 CST 2007. Process: ltp_auto_test.e, ID: 21457 to be killed. Here is the difference between lastrun and thisrun test results: Building kernel that now fail, but passed before: build_pbx_kernel_log build_pound_kernel_log #########Test Summary ############# ######The following build kernel fails:####### build_linphone_kernel_log ############ Pass Num is: 27.############# DLL TEST DPMC TEST … Modifications since last successful build: (70) modifiedvapier/trunk/include/asm-blackfin/unistd.h06/18/2007 23:23:04move cond_syscall() behind __KERNEL__ like all other architectures modifiedvapier/trunk/scripts/checkstack.pl06/18/2007 23:18:45fill in the Blackfin specific pieces in the check stack script modifiedadamliyi/trunk/user/blkfin-apps/linphone/linphone-1.6.0/console/linphonec.c06/16/2007 06:33:43for task [#2386], add video display in linphone …

17. Web page Publisher  If you want to see it from the web pages,like : http://10.99.22.164:8080/cruisecontrol/buildresults/uclinux- dist?log=log20070215173903Lbuild.1 (This link is configured in config.xml), you need to set the write permission of cruise/logs/uclinux-dist, since it dynamically convert xml to html and put them under this folder name _cache by default.http://10.99.22.164:8080/cruisecontrol/buildresults/uclinux- dist?log=log20070215173903Lbuild.1  Also you need set up tomcat to hold these jsp pages.

18. CruiseControl Merits:  Allows you to see what is happening in the source tree. It shows you: who checked in what (by showing modifications); Test of what config has built and test successfully; what platforms are broken and how they are broken (the build logs);  With this - the system can automatically determine who broke the build, so fixes can occur quickly

19. 19 Thank You.