Thoughts about Archiving Experimental Computer Science Artifacts (including Experiments) Jack W. Davidson Department of Computer Science University of Virginia

Platforms Good News Fewer architectures in the desktop high- performance area X86, PowerPC Hardware performance counters Fewer operating systems Linux, Windows Accessible software infrastructures gcc, llvm, jikes, openjdk, etc. Available benchmark suites SPEC, DaCapo, Parsec, etc.

Platforms Bad News Rapid changes in microarchitecture Same ISA, but very different microarchitecture Hardware performance counters are often not documented correctly GPUs becoming important Still lots of embedded architectures— although ARM seems to be becoming dominant All Linuxes are not the same

Platforms Bad News Existing software infrastructures are sometimes impediments to experimentation Cloud computing and public queue systems make controlling the environment difficult Push to lock-down configurations because of security concerns Embedded benchmarks, especially hard real- time benchmarks, are difficult to obtain because of proprietary concerns

Virtual Machines Good News Virtual machines allow entire platforms (OS, software tools, etc) to be packaged and delivered Very useful in security research where exploits are very fragile Bad News Hides machine details (not working on bare metal) Longevity of VM images an issue Licensing issues with Windows

Final Thoughts Education Most graduate students don’t know how to do experimental computer science Hard to teach—only learn by doing Can we create a course that teaches fundamental concepts? Incentives Need good incentives for archiving Can be in conflict with faculty entrepreneurs