1. RCE Platform Technology (RPT) Sergio Maldonado (smaldona@slac.stanford.edu) Software Architecture Development Environment

2. 2 RCE Platform Technology – Software Overview Topics Covered: Embedded System Software - Runtime Environment: The Core - Service Libraries - System Configuration Software Development - SDK - Development Model Protocol Plugin Software - Socket Abstraction Services (SAS)

3. 3 RCE Embedded System Software The RCE system software consists of: Bootloaders - Xilinx FSBL - U-Boot RTEMS operating system - Zynq board support package - OS primitives: semaphores, message queues, timers, tasks, etc - POSIX support, FAT filesystem Kernel extensions and enhancements - Provided by the RPT group Linux Operating system - Filesystem administration and system maintenance Core Services Applications Bootloaders

4. 4 RCE Embedded System – Core and Services Applications Boot Loaders Services Core NFS Telnet Shell RTS DSL ATCA Network 1/10 GE drivers SAS Kernel SD & UART drivers Memory & task mgmt Dynamic linker/loader SVT Console

5. 5 RCE Runtime Core The Core is the set of software that provides the base level of runtime operation RTEMS kernel C runtime libraries RPT additions to kernel functionality - SD device driver - UART device driver - Dynamic linker/loader - Memory management utilities - Task utilities - Configuration using Symbol/Value tables - Filesystem abstraction: Namespaces

6. 6 RCE Runtime Services A set of services are built on top of the Core to provide: Embedded console Shell and extensions 1-GE or 10-GE network and TCP/IP stack C++ runtime library Telnet NFS Distributed Services Lookup (DSL) ATCA Services Application Initialization

7. Shareable Images

8. 8 The RCE runtime environment operates by executing code located in ELF formatted, dynamically loaded/linked shareable objects. The internal format of these ELF objects has been crafted specifically for the RCE runtime environment. These shareable images are separated into three categories. Shared Library - implementation Executable - tasks Symbol/Value Table - configuration

9. 9 Shareable Images - Shared Library Implementations for general consumption by other shared images Not associated with a specific task Define an initialization entry point - Executed by linker prior to memory fixup - Used for initialization of library prior to use by other shared images. Can be installed as memory resident - Memory resident libraries are loaded once but may be linked against by many other shareable images. Can be linked against any other shared library Use the.so filename extension

10. 10 Shareable Images - Executable Defined as a pure RTEMS task along with its corresponding dynamically loaded/linked executable code. Must satisfy certain Core requirements - Task entry hook – Task_Start - Task exit hook – Task_Rundown - Task specifications (name, priority, etc) are defined in a data structure, which may be located in SVT tables. – Task_Attributes Can link against any shared library and may utilize any installed SVT. Use the.exe filename extension

11. 11 Shareable Images - Symbol Value Table Data only files containing values (data structures) mapped to symbol names (character string). Storage of system and application configuration data structures Do not link against shared libraries Use the.svt filename extension

12. 12 Shareable Images – Linker and Loader The dynamic linker/loader is a fundamental component of the Core. Locates image files on the target filesystem Loads image segments into appropriate memory regions Relocates and fixes up symbols Dynamically resolves library dependencies - Recursively loads and links all required images

13. 13 Shareable Images - Benefits The usage of shareable images provides the maximum flexibility for development and maintenance of both system and application software. Code can be modularized to encapsulate common functionality Software interfaces can be extended and deployed without requiring a full recompilation of code bases The Core and all system service libraries can be developed and maintained independent of deployed application code Development of applications can be more easily distributed amongst several functional groups.

14. 14 Shareable Images – Namespaces (I) The dynamic loader locates images by employing the binding of a logical name, or namespace, to a specific directory on the target filesystem. A namespace is a string identifier used to associate a set of target images with some user defined category or organizational structure The namespace string must - be unique over the set of all assigned namespaces in a running system. - not contain the reserved character ":” The namespace gets mapped at runtime to a path on the target filesystem. - This mapping may be stored in a configuration file The set of images associated with a namespace must all reside in the same directory on the target filesystem All dynamic images must specify a namespace and filename at link time using the -soname linker flag The format of the argument to the -soname linker flag must be: - " : ”

15. 15 Shareable Images – Namespaces (II) The benefits of using namespaces include: Separation of software images into logical functional groups. Image portability throughout the filesystem Execution of images over a network filesystem (NFS) Deployment of images to the target filesystem need not take place until the test and debug cycle is complete.

16. 16 Shareable Images – Namespaces (III) User-Defined Namespace(s) Config Namespace System Namespace /rtemsapp. user.so. user.exe. user.svt..../config. /rtems sys.svt app.svt rtems.so dsl.so rts.so xaui.so shell.so sas.so These files have shared ownership by Applications and System These files are System owned These types of files are owned by Applications

17. Core Utilities

18. 18 Core Utilities - Memory Management The Core provides the following memory management features Separate memory regions each with a unique set of MMU attributes - Read/write/execute - 4k or 1Mbyte page size - Cacheability Allocations from managed regions Region address lookup by name Cache operations - Flush - Invalidate Atomic Resource Management - Allocate memory resources reentrantly

19. 19 Core Utilities - Task Management A Core Task is defined as a pure RTEMS task along with its corresponding dynamically loaded/linked executable code. The Core provides the infrastructure to dynamically load and link executable shared images using namespaces - This occurs before its corresponding RTEMS task is started Task Control - Start - Stop

20. 20 Core Utilities - Symbol Value Tables Loading and installation of SVT shared images Symbol translation - Resolves the object pointer corresponding to the symbol System and application SVTs are loaded at Core initialization - These SVTs are available for immediate use by applications and do not need to be explicitly installed.

21. 21 Core Utilities - Tools The Core provides additional low level utilities for general use: Linked lists Key value tables Byteswap routines ELF image utilities Hash tables System logging

22. System Libraries

23. 23 System Libraries - Runtime Support The runtime services library, rts.so, provides full C++ support and the following features System message logging classes Exception classes String utilities

24. 24 System Libraries – Distributed Services Lookup Distributed Services Library, or DSL (dsl.so) represents a lightweight client server model Resolves a service name string to a shareable image Lazy deferral of image loading Host side and embedded interfaces Host side client message is delivered over ethernet Server executes entry point in registered service library, with the message as an argument Service processes the message and returns a response to the Server The Server delivers the service specific response to the client

25. 25 System Libraries - ATCA The ATCA library (atca.so) is a registered DSL service. Server side component “looks up” networking information corresponding to an ATCA location - IP address - MAC address Host side client interface allows for building of requests and processing of server side responses Requests are broadcast to all servers on a given subnet. Only the node that has the matching ATCA address will respond to the request.

26. Configuration

27. 27 System Configuration - SVT Components of the Core and system services are configured at startup using the SVT mechanism (sys.svt). The system configuration is owned by the RPT and is suitable for most deployments. System task specifications - Console - DSL Network stack configuration Telnet configuration System DSL definitions (currently only ATCA)

28. 28 Application Configuration - SVT Activation of system services is configured at startup using the SVT mechanism (app.svt). This is considered to be jointly managed by the system and applications. Applications may tailor the service set to its needs, but the svt filename and location is fixed and must be present in order for the system to successfully boot. Services to start at boot - Console - Shell - Network Driver - NFS - DSL Activation of DSL services - ATCA service is required by the system Activation of user application specific image - appinit.so

29. 29 Application Initialization At the completion of starting up system services, application code can then be conditionally loaded. The supported method is to create a dynamic image named appinit.so, install it in the rtemsapp partition, and add it to the services portion of app.svt. The appinit.so image can be tailored to the needs of the applications. Define application specific namespaces Mounting of scratch filesystem Mounting of NFS directories Loading and activation of application executable images Any other required initialization

30. RCE Software Development Kit (SDK)

31. 31 RCE Software Development Kit - SDK The SDK contains host side utilities to interact with a running RCE Core, but its primary focus is to build shareable images for test and deployment in embedded execution environments. Execution environments are the pairing of an operation system with a hardware platform. Here, these are referred to as the target (ARM RTEMS RCE) and host (I86 Linux 32/64). C/C++ code must be cross compiled on the host for execution on the target. C/C++ code may be compiled natively for execution on the host.

32. 32 SDK Development Model Both the SDK and the Core runtime environment contain support for developing software with speed and agility. The phases presented here were employed by the RPT group in the development of RCE system images, and constitute our recommended methodology. Shareable image implementation Test and debug Image deployment and installation

33. 33 RCE Software Development Kit - SDK Host SystemTarget System Filesystem SDK 1)Compile source files against SDK include files (produces *.o) 2)Link object code against SDK libraries (produces *.so, *.exe, or *.svt) Filesystem scp (Linux) or NFS mount.so.exe.svt Runtime Core Runtime Core provides the dynamic linker/loader to activate shareable images from the target filesystem

34. 34 SDK Development Model – Implementation Develop source code, compile, and link shareable images. Takes place on a supported Linux host machine Software can first be prototyped using native host compilers Allows images to be partially tested and debugged prior to cross compilation (algorithmic routines, data structures). Creation of application specific.exe.so and.svt images Images can be examined for size constraints and machine code optimization using standard ELF utilities.

35. 35 SDK Development Model – Test and Debug Cross compiled images are executed on the target using NFS mounted directories mapped to namespaces Library loading and executable activation using shell utilities Debugging using the console or telnet shell Application images are not required to reside on the target filesystem Does not affect deployed software.

36. 36 SDK Development Model – Deployment At this phase, images are fully tested and ready to be deployed to the target filesystem. Configuration and initialization components on the deployed system are updated to support activation of new images from the target resident filesystem - SVT configuration files and/or application specific initialization libraries must be updated to map namespaces, task start directives, and any explicit library loads required for the new images. Images are copied to the appropriate directories on the target filesystem Images are activated at startup, after system services are available.

37. 37 RCE Software Development Kit - Contents RTEMS SDK - RTEMS cross compilers - Core and service libraries to link code against - Header files - Bootloaders - Firmware bitfiles I86 Host Linux SDK - Host development header files and libraries - Administration and maintenance tools ArchLinux SDK - Utilizes Xilinx tools - Cross-development for execution on an embedded linux platform - Work in progress

38. 38 RCE Software Development Kit Must be integrated into existing build systems - Compilers and linkers sourced from SDK installation path - Include files - use –I on the compiler command line - Library paths - use –L: or –l linker command line A single SDK installation may be shared by one or more developers

39. 39 RTEMS SDK Compilers and Linkers The SDKs make use of the RTEMS cross-compiler tools and native gcc installations. SDK compilers and linkers are essentially wrappers around these tools. The wrappers contain the command line arguments required to support the dynamic image format used by the Core runtime. User specific arguments are appended to the default SDK arguments. SDK RTEMS Compilers - rtems-gcc, compile C object code - rtems-g++, compile C++ object code SDK RTEMS Linkers - rtems-ld, link object code into shareable images (.so) - rtems-task, link object code into task images (.exe) - rtems-svt, link object code into symbol value tables (.svt)

40. 40 I86 Linux Host SDK Compilers and Linkers Host executables and libraries can be compiled using an SDK. In this case, the SDK commands are merely indirections to the native gnu tools. Host Linux SDK Compilers - linux-gcc - linux-g++ Host Linux SDK Linkers - linux-ld

41. Protocol Plugin Software

42. 42 Protocol Plugin Support – SAS (I) Support for the software side of the protocol plugin socket is handled by the Socket Abstraction Services (SAS). The implementation is provided by the service library sas.so. SAS interacts with plug-in firmware by reading and writing AXI memory mapped I/O addresses. SAS exposes plug-in functionality using mailboxes. A mailbox is a container, implemented in firmware which obeys FIFO (first-in, first- out) discipline. A mailbox contains file descriptors or messages. SAS supports three type of mailboxes. - Inbound mailboxes - Outbound mailboxes - Unnamed mailboxes

43. 43 Protocol Plugin Support – SAS (II) … Task Ob Mbx Ib Mbx Mbx 0 Mbx 31 Session Plug-in Port FIFO SAS

44. 44 Protocol Plugin Support – SAS (III) SAS uses sessions to wait for input from mailboxes. A session is the association of software with a socket. Mailboxes are bound to sessions at runtime. The types of mailbox input are: Arrival of an inbound frame - 32-bit SAS file descriptor Notification of placement of an inbound frame - 32-bit SAS message Notification of placement of an outbound frame - 32-bit SAS message Waiting on input from a session is always done in the context of a task. A session is associated with a task. Processing of inbound frames use two sessions 1. Wait for the arrival of an inbound frame descriptor. - Also used to post a file descriptor to initiate placement of inbound frame payload 2. Wait for the completion of placement of an inbound frame Transmission of an outbound frame uses one session: 1. Wait for the completion of placement of the outbound frame

45. Thank You. Questions?

46. Backup Slides

47. 47 RCE Software – SD Storage All software images reside on an SD flash. The 32 GB (27.9 usable) microSD is formatted with 6 partitions Boot - /boot - Bootloader files boot.bin, fpga.bit, uboot.env - 512 MB FAT32 RTEMS system - /rtems - The Core, service libraries, system executables - 512 MB FAT32 RTEMS application - /rtemsapp - Application specific images - System/application shared configuration - 4GB FAT32 user/application scratch - /scratch - 16 GB FAT32 Linux Kernel - /linux - System maintenance/recovery - 512 MB FAT32 ArchLinux Filesystem - /archlinux - System maintenance/recovery - 8 GB EXT3

48. 48 RCE Bootstrap Process Xilinx first stage bootloader (FSBL) - SOC initialization - CPU register configuration U-Boot - Xilinx distribution includes Zynq hardware support - Firmware bitfile loader - BSI handshake – IPMI ready signal - Networking parameter configuration - Rescue operations Core bootstrap - Board support package, MMU, and kernel initialization - Relocation/fixup of Core elf image - Kernel initialization - RTEMS - Memory and driver initialization - Loading of system and application SVTs - System service load and startup - Application load and startup (optional)

49. 49 Protocol Plugin Support – SAS Configuration SAS configuration for protocol plugins is defined in an SVT file. Inbound/outbound header sizes Inbound/outbound buffer count Plugin name Memory map for plugin register space

50. 50 Protocol Plugin Support – Firmware Registers Most plugin firmware will expose a set of configuration registers. This register block is typically mapped to memory locations in the AXI space. SAS interface to configure the AXI register address mapping SAS interface to retrieve a pointer to the memory mapped register block.

51. 51 Protocol Plugin Support - XAUI Plugin software has been implemented using SAS to support the operation of the XAUI plugin. This plugin is used by system software to enable 10-GE TCP/IP stack. The plugin also supports transmit and receive by other applications that need to take advantage of the high speed link using other protocols. Data receive enabled by registering a protocol callback routine with the XAUI driver. Data transmit is performed by directly posting to the XAUI SAS outbound mailbox.