1. Fuzzy Plans and Other Test Integrations
Renfei Wang
Adam Farley

2. Future is Fuzzy Fuzzy plans Transformation
Collaboration and innovation

3. Who We Are Renfei Wang Adam Farley Software developer QA engineer
Test framework design
Adam Farley
AdoptOpenJDK Contributor
OpenJDK Contributor

4. Goals
Open and transparent testing of JDK implementations for the community (confidence in quality of binaries)
Find defects early, report quickly to upstream projects
Empower developers (easily run all tests)
Quick integration of new test material

5. The Scope Projects: Eclipse OMR, Eclipse OpenJ9, AdoptOpenJDK
Ensuring Free and Verified Java™ for the Community
6+ Jenkins servers
AdoptOpenJDK

6. Degrees of Freedom
implementations:
Versions:
Platforms:
Tests:

7. Testing Scope eclipse/omr: internal:
functional, system, external, openjdk, perf
eclipse/omr:
(featuring language agnostic, component tests, opcode tests, IL injection)
eclipse/openj9:
Functional, system
adoptopenjdk:
openjdk, system, external, perf
adoptopenjdk:
openjdk, system, external, perf, jck

8. Gather Great Tests 20,000 tests ~25,000 tests
functional
openjdk
perf
jck
external
system
20,000 tests
~25,000 tests
8+ apps, 3 microprofile TCKs
~50,000 container tests
2900 tests
6600 per release
~ tests
(select release builds)
6+ benchmarks (micros & full suites)

9. Gather Great Tests openjdk perf jck external system functional
testNG, cmdlinetester
STF
junit & others
javatest
Assorted benchmarks
Jtreg, testNG
$(JAVA_BIN)/java -cp GeneralTest.jar org.testng.TestNG -d $(REPORTDIR) testng.xml -testnames floatSanityTests -groups sanity
perl stf.pl -test-root=openjdk-systemtest_root -systemtest-prereqs=systemtest_prereqs -java-args=-Xjit -results-root=reportdir -test=ConcurrentLoadTest
$(JAVA_BIN)/java -Xmx512m -jar jtreg.jar -vmoptions:"-Xmx512m" -w work -r report -jdk:$(JDK_HOME) -exclude:ProblemList.txt jdk_math
$(JAVA_BIN)/java -jar javatest.jar -verbose:max -config
${ANT_HOME}/bin/ant -Divy_install_path=${ANT_HOME}/lib -lib ${ANT_HOME}/lib -f ${LUCENE_SOLR_HOME}/lucene-solr/build.xml -Duser.home=${LUCENE_SOLR_HOME} -Dcommon.dir=${LUCENE_SOLR_HOME}/lucene-solr/lucene test
sbt "partest $(TEST_SUITE)”
$(JAVA_BIN)/java -DBumbleBench.listOptions -jar BumbleBench.jar [Benchmark name]

10. Curator’s Requirements
Latitude
Flexible
Common
Standardized
Simple

11. Consolidate and Curate
functional
openjdk
perf
jck
external
system
testkitgen
testNG, cmdlinetester
STF
junit & others
javatest
Assorted benchmarks
Jtreg, testNG
make _functional
make _openjdk
make _system
make _external
make _jck
make _perf

12. TestKitGen 😃

13. TestKitGen 😃
Test definition
- Test case name
- Test command

14. TestKitGen 😃
Testing options:
- JVM options
- Platform Requirements

15. TestKitGen 😃
JDK info:
Implementation
JDK version

16. TestKitGen 😃
- levels and groups

17. Functional Test Repo

18. Test Structure

19. External Test Repo

20. Test Structure

21. Demo
Manual testing
Jenkins Grinder

22. Demo review clone test material and testKitGen
set environment variable
configure - generating test targets
compile
execute the targets
view the results

23. What Tests to Integrate: Near-Term
Microprofile TCKs ✅
Security
Performance micro benchmark

24. How?
+ Build.xml
+ Playlist.xml

25. Integration Example: micro benchmark

26. Future Forward Plans, improve?
Test smarter
Easier reporting
Tools for triage
Easier on-boarding

27. Tests Results: TRSS & Triage
Monitor different Jenkins servers

28. Tests Results: TRSS & Triage
Failed: 0 / Passed:52 / Executed: 52

29. Tests Results: TRSS & Triage
All Platforms | Deep History

30. Tests Results: TRSS & Triage
Found 3 “floatSanityTests”

31. Are our Plans Really Fuzzy?
Our plans are clear. The future is fuzzy.
AI-driven fuzz testing in collaboration with university of Edinburgh
Center of excellence and innovation for testing
innovation happens through open collaboration

32. Connect & Collaborate! Website Github Twitter Upcoming Talk:
AdoptOpenJDK
adoptopenjdk.net AdoptOpenJDK/openjdk-tests @adoptopenjdk
eclipse.org/openj eclipse/openj @openj9
eclipse.org/omr eclipse.org/omr @eclipseomr
Upcoming Talk:
AdoptOpenJDK: Lessons Learned from the New Build Farm for Java Itself