Alexey Pakhunov /XCG, Microsoft Research/ March 30 th, 2011

 Black Cloud OS:  A fork of Singularity OS  Our playground for experimenting with message passing in non-cache coherent environment  This presentation covers only our development experiences on the SCC  Submission of the paper is on its way 2

 A quote from Singularity home page: “A research operating system prototype, extending programming languages, and developing new techniques and tools for specifying and verifying program behavior”  Written in managed code  Some Assembler and C++ in the boot loader and kernel  IPC and inter-component communications are based on passing messages 3

Tile R R R R R R R R R R R R R R R R R R R R R R R R DDR3 MC VRC System Interface PCI-E Management Console (Linux) sccTcpServer/mceGui Management Console (Linux) sccTcpServer/mceGui Desktop PC (Windows) RcLoader.Net, KdProxy, WinDbg, etc. Desktop PC (Windows) RcLoader.Net, KdProxy, WinDbg, etc. TCP/IP 4

 Configuration  Generates the system memory map  Configures the SCC registers  Uploads the boot loader and OS images  Supports manual editing of the SCC configuration  Debugging  Allows inspecting the memory and configuration registers 5

6

 No serial port or console  Memory at 0xb8000 is the console buffer  I/O redirection doesn’t work as expected  Execution of IN or OUT instruction effectively halts the core and sccTcpServer  Serial KD transport is emulated  A couple of ring buffers on the SCC side  KdProxy.exe exposes a named pipe interface for the debugger 7

 No BIOS  The system memory map is patched directly in the boot loader  No standard devices  Local APIC is used instead of i8254 timer and PIC  No RTC clock  No modern instruction supported  Context handling code was updated due to lack of MMX ▪ 32bit flavor of Singularity uses only x87 for floating point calculations  Bartok compiler was patched due to lack of CMOV instructions 8

 Turning on MPB bypass bit causes a race causing memory corruptions  Minus three days of debugging :-)  We couldn’t take advantage of fast MPB access  Large pages cannot be used together with MPB  Singularity uses large pages to create the identity mapping spanning 4GB 9

 A telnet connection to each core  The same serial transport emulation via KdProxy.exe was used 10

 A read-only OS image is shared among all cores  Message passing code uses MPB-mapped buffers and CL1FLUSH-aware memcpy()  Large shared memory storage is accessible via dynamically remapped LUTs  R/W access is possible with proper cache flushing and/or caching settings in PTEs 11

 Core’s memory interface bandwidth is limited  One outstanding memory operation 12 Frequency (MHz)A cache line writing latency (cycles) Maximum bandwidth (MB/s) (cached) (uncached)124.2

 Memory controller bandwidth is limited 13

 The SCC is an experimental platform tailored for message passing  Lack of cache coherency makes us think hard how about message passing  The chip has enough cores to play with scalability  Compare apples to apples  The cache and memory subsystems are significantly different  The SCC is super parallel, not super fast 14

15