1.  Wind River Systems, Inc. 1997 Appendix - E Shared Memory Network

2. E-2 Shared Memory Network Overview

3. E-3 Shared Memory Network Implementation Uses global (“shared”) memory accessible to all CPU’s on the backplane. Uses test-and-set atomic operation to mutually exclude access to global data structures. Destination CPU notified of packet arrival via: Mailbox interrupts (most flexible). VMEbus interrupts (limited number). Polling (if interrupts not available).

4. E-4 Terminology Shared Memory Master - CPU 0. Initialize the shared memory network. Once initialized, all boards are peers. Shared Memory - Memory used for passing packets between processors. Can reside on CPU0 or a memory card. Anchor - Structure containing ready value and pointer to shared memory. Must be at a known address. Heartbeat - Integer incremented once per second by CPU0, used to indicate that the shared memory network is alive. Ready Value - Indicates if the shared memory network is initialized.

5. E-5 Choosing a Network Number Shared memory network number options: Distinct networkIn an isolated environment, choose any unused network number. SubnetworkIf the host is already on a subnet, choose an unused subnet. Proxy networkUse the same network number as the host. Easiest to configure. Examples: RelationshipHost networkSM network Distinct90.0.0.0100.0.0.0 Subnet90.1.0.0:ffff000090.2.0.0:ffff0000 Proxy 90.1.0.0:ffff000090.1.0.0:ffff0000

6. E-6 Sequential Addressing The internet address of CPU0 on the SM network: If INCLUDE_PROXY_DEFAULT_ADDR is defined, it is one more than inet on ethernet (e). Otherwise, assigned in the boot parameters. The internet address of other boards on the SM network: If INCLUDE_SM_SEQ_ADDR is defined, will be sequentially ordered (by processor number). Otherwise, assigned in the boot parameters.

7. E-7 Example Configuration Overview 1. Configure the host’s network softwre: Modify the host table. On UNIX, modify.rhosts. (UNIX Host) 2. ConfigAll.h specifies as defaults: shared memory network support sequential addressing proxy client support 3. Modify config.h for CPU0 to specify: proxy server support proxy default addressing. where shared memory resides, how large it is. 4. Calculate the address of the anchor.

8. E-8 Example Configuration

9. E-9 UNIX: Host Configuration Configure host so that targets can boot: Host table Routing (if necessary -- when proxy ARP not used) Permissions Example/etc/hosts file: 90.0.0.49 vxEth0 90.0.0.50 vxSm0 90.0.0.51 vxSm1 Example ~/.rhosts file (if using RSH): vxEth0 vxSm0 vxSm1

10. E-10 Installing the Shared Memory Network These defaults are specified in configAll.h: #define INCLUDE_PROXY_CLEINT... #define INCLUDE_SM_NET #define INCLUDE_SM_SEQ_ADDR... #define SM_TAS_TYPE SM_TAS_HARD #define SM_ANCHOR_ADDR... /* arch. specific */ Modify CPU0’s config.h file to specify: #define INCLUDE_PROXY_SERVER #define INCLUDE_PROXY_DEFAULT_ADDR #define SM_OFF_BOARD FALSE #define SM_MEM_ADRS... #define SM_MEM_SIZE...

11. E-11 Shared Memory Location / Size Example wind/target/config/target/config.h stub: #define SM_OFF_BOARD FALSE #if SM_OFF_BOARD #undef SM_ANCHOR_ADRS #define SM_ANCHOR_ADRS ((char *) 0xfb800000) #define SM_MEM_ADRS SM_ANCHOR_ADRS #define SM_MEM_SIZE 0X80000 /* 512k */ #else #define SM_MEM_ADRS NONE /* NONE = ALLOCATE */ #define SM_MEM_SIZE 0X10000 /* 64k */ #endif /* SM_OFF_BOARD */

12. E-12 CPU0 Boot Parameters boot device : in processor number : 0 host name : ohio file name : wind/target/config/frc40/vxWorks inet on ethernet (e) : 90.0.0.49 inet on backplane (b): host inet (h) : 90.0.0.2 user (u) : marc

13. E-13 CPU0 Booting Example... Attaching network interface ln0... done. Attaching network interface lo0... done. Loading... 393496 + 102988 + 25610 Starting at 0x1000... Attaching network interface ln0... done. Initializing backplane net with anchor at 0x600... done. Backplane anchor at 0x600... Attaching network interface sm0... done. Backplane address: 90.0.0.50 Creating proxy network: 90.0.0.50...

14. E-14 Booting Other CPU’s  Other boards must be able to access the anchor, through the VMEbus, in order to boot.  CPU’s local anchor address (0x600) is not identical to its corresponding VMEbus address.  To attach to the Shared Memory Network, other CPU’s must: 1. Determine the VMEbus address of the anchor (how CPU0 dual-ports its local RAM). 2. Determine the local address, on CPU1, which is used to access the anchor’s VMEbus address (how CPU1 maps its address space).

15. E-15 VMEbus Address Spaces The VMEbus provides several address spaces to accommodate varying needs. Address space is specified by an address modifier, which encodes: Address size32 bit (extended), 24 bit(standard), or 16 bit (short I/O). Privilege level User or supervisor. VxWorks always uses supervisor mode. Access typeData, instruction, or block transfer (DMA). Symbolic constants for address modifiers defined in vme.h (e.g., VME_AM_EXT_SUP_DATA).

16. E-16 Off Board Memory Access

17. E-17 Calculating the VMEbus Address of the Anchor from CPU0 -> busAdrs = 0 new symbol “busAdrs” added to symbol table. value = 0 = 0x0 -> sysLocalToBusAdrs (0xd, 0x600, &busAdrs) value = 0 = 0x0 -> busAdrs busAdrs = 0xffe7d0: value = 16778752 = 0x1000600 value = 0 = 0x0

18. E-18 Local Address of Anchor As Seen from Other Boards -> localAdrs = 0 new symbol “localAdrs” added to symbol table. localAdrs = 0x3fe7d0: value = 0 = 0x0 -> sysBusToLocalAdrs (0xd, 0x1000600, &localAdrs) value = 0 = 0x0 -> localAdrs localAdrs = 0x3fe7d0: value = 16778752 = 0x1000600 Can’t perform these steps from CPU1, since not yet booted. Can perform from CPU0 if boards are the same. Can examine sysLib source code otherwise.

19. E-19 Calculating the Anchor 1. Call sysLocalToBusAdrs( ) on CPU0 to calculate the VMEbus address of the anchor. 2. Call sysBusToLocalAdrs( ) on the other CPU boards to calculate the local address which maps to the anchor’s VMEbus address. May need to examine source code for this routine.

20. E-20 CPU1 Boot Parameters boot device : sm=0x1000600 processor number: 1 host name: ohio file name : wind/target/config/frc40/vxWorks inet on backplane (b): host inet (h) : 90.0.0.2 gateway inet (g) : user (u): marc flags (f) : 0x100

21. E-21 Booting CPU1... Backplane anchor at 0x1000600... Attaching network interface sm0... done. Backplane inet address: 90.0.0.51 registering proxy client: 90.0.0.51.done Attaching network interface lo0... done. Loading... 393496 + 102988 + 25610 Starting at 0x1000......

22. E-22 Other Configurations If not using a proxy network: Must modify the host’s routing table. Slaves must specify gateway inet (g) boot parameter. Can use BOOTP with the proxy network. See the Programmer’s Guide for details.

23. E-23 Avoiding Bus Errors Default bus arbitration favors boards close to first slot. Distant boards may experience bus (timeout) errors while booting. Possible solutions: Stagger the boot countdown for each board by modifying wind/target/config/all/bootConfig.c. Increase bus timeout value. Modify the bus arbitration scheme (software changes are typically made in sysProcNumSet( ) and sysHwInit( )). Software solutions require making new boot ROM’s which is discussed in the Reconfiguration chapter.

24. E-24 A Troubleshooting Tool -> smNetShow Anchor Local Addr: 0x1000600 Sequential addressing enabled. Master address: 90.0.0.50 heartbeat = 6712, header at 0x1fb8144, free pkts = 29 cpuint typearg1 arg2 arg3queued pkts ----- -------- ---- ----- ------ --------------- 0mbox-1r0x2d0x8002 0 0 1mbox-1r0x2d0x8102 0 0 input packets = 109 output packets = 109 input errors = 0 output errors 0 collisions = 0 value = 0 = 0x0

25. E-25 Summary Host configuration: Modify host tables. Modify ~/.rhosts. (UNIX Host) Modify routing table if not using a proxy network. Defaults in configAll.h: INCLUDE_SM_NET and INCLUDE_SM_SEQ_ADDR SM_ANCHOR_ADRS INCLUDE_PROXY_CLIENT CPU0 (the shared memory master) specifies SM_OFF_BOARD SM_MEM_SIZE and SM_MEM_ADRS INCLUDE_PROXY_SERVER INCLUDE_PROXY_DEFAULT_ADDR

26. E-26 Summary Additional boot parameter for CPU0: inet on backplane (b): If using default proxy address, leave blank. Accessing the anchor from other boards: sysLocalToBusAdrs( ) sysBusToLocalAdrs( ) Additional boot parameters for CPU1: boot device: sm=localAdrsOfAnchor inet on backplane (b). If using sequential addresses, leave blank. gateway inet(g). If using proxy ARP, leave blank. register proxy client boot flag (0x100)