1. Building a Reliable Onboard Network with Ethernet: A GSFC Prototype Jane Marquart NASA/GSFC

2. 8 June 2004 2 Fourth Space Internet Workshop Requirements Ethernet/IP Onboard Network Bus Critical real-time data must be delivered reliably onboard the spacecraft (target is 5 milliseconds)

3. 8 June 2004 3 Fourth Space Internet Workshop Protocol Options TCP TCP works but doesn’t support reliability features required by typical flight software UDP UDP is a better fit, but requires reliability be implemented by the user UDP + Reliability Where to implement reliability? Application layer or data link layer?

4. 8 June 2004 4 Fourth Space Internet Workshop Reliability Trade Application layer: –Rx & tx packet latency through the stack is highly variable and non-deterministic (regardless of direction) –Latency up to several milliseconds on a MCP750 @ 233mhz Datalink layer: –avoids IP stack latency, but; –the issue is standardization and portability of a datalink layer solution.

5. 8 June 2004 5 Fourth Space Internet Workshop IEEE 802.3 LLC LLC – “Logical Link Control” offers a IEEE standardized datalink layer reliability protocol, adding a 3 byte header to the ethernet frame Supplies 3 major types of service; –Type 1 (unreliable packet exchange, same as plain ethernet) –Type 2 (reliable, statefull, connection oriented; used by FDDI, Wireless, Token Ring) –Type 3 (reliable, stateless, connectionless) Type 3 selected –Stateless but reliable –Well-defined (and simple) state machine –Suitable for a variety of physical layers –Packet overhead is 3 bytes

6. 8 June 2004 6 Fourth Space Internet Workshop Data-link Layer Reliability: LLC Ack/retry algorithm implemented within the NIC driver. Latency & delay well defined since control is right at the hardware. Widely supported, if not tolerated, by common OS IP stacks, routers, etc. Well defined & supported but ad-hoc mapping onto standard ethernet framing. Reliability protocol only useable between conformant drivers.

7. 8 June 2004 7 Fourth Space Internet Workshop Packet data Ethernet frame headerLLC header PDU Formats ethernet dest (6 bytes) ethernet source (6 bytes ) ethernet type (2 bytes) DSAP (1 byte) SSAP (1 byte) Control (1 byte) user data & fill if req’d Ethernet destination (6 bytes) Ethernet source (6 bytes) Ethernet type (2 bytes) User data &fill if req’d Default datalink pdu Reliable pdu (data packet) Source SAPDestination SAP Reliable pdu (ack packet) ack pdu's swap DSAP & SSAP, along with ethernet source/destination so reply is sent to originating host. Standard ethernet frame w/ EthernetType >= 0x600 EthernetType < 0x600, set to frame length indicating a 802.2 LLC pdu Ethernet frame header Packet dataEthernet frame headerLLC header ethernet dest (6 bytes) ethernet source (6 bytes ) ethernet type (2 bytes) DSAP (1 byte) SSAP (1 byte) Control (1 byte) No user data, Fill to min. valid pkt length

8. 8 June 2004 8 Fourth Space Internet Workshop Ethernet/IP Multi-node Testbed SWITCH Subsys ‘A’ Coldfire MEDIA CONVERTER Ethernet Hub "Comm Card" Dell PC ITOS GSE Router Serial Channel Link Simulator Serial 4 Mbps 50 kbits 150kbits 4 Mbps 1 kbits 200 kbits Flight NIC Net Mgr SB TO CI CFDP NIC Driver IP OS Stack MEDIA CONVERTER CFDP Subsys ‘B’ PC

9. 8 June 2004 9 Fourth Space Internet Workshop NIC Driver Architecture IP stack RX packets TX packets RX interrupt Packet Scheduler Async Pkt Channel Queues SOIS Independent Layer SOIS dependent Data-link layer Packet validation & LLC processing Packet retrieved TX done interrupt Packet submitted to NIC LLC ack/error packets Flight NIC Eth1Eth0 RX packets TX packets

10. 8 June 2004 10 Fourth Space Internet Workshop Use Case Examples nominal Pkt sent with reliable QOS Driver returns ‘ACK’ Receive ACK Done pkt drop Pkt sent with reliable QOS ACK Time-out, Resend pkt Driver returns ‘ACK’ Receive ACK Done ACK drop Pkt sent with reliable QOS Driver returns ‘ACK’ ACK Time-out, Resend pkt, Log error Driver returns ‘ACK’ Discards redundant pkt Logs error Receive ACK Done pkt lost Pkt ‘N’ sent with reliable QOS ACK Time-out, Resend pkt ‘N’ Log error Driver returns ‘ACK’; logs error Receive ACK Done ACK lost Pkt N+1 sent with reliable QOS Driver returns ‘ACK’ ACK Time- out, Resend pkt, Log error Driver returns ‘ACK’ Discards redundant pkt Logs error Receive ACK Done ACK Time-out, Log error Pkt ‘N’ sent with reliable QOS Driver returns ‘ACK’ ACK Time- out, Log error Pkt N+1 sent with reliable QOS Pkt lost Pkt lost Pkt lost ACK lost ACK lost ACK lost

11. 8 June 2004 11 Fourth Space Internet Workshop NIC/OS Transmit Path Measurements User app. opens socket to send packet (UDP Client) Stack Processes Packet Key : = start receive markers Tx_Stack_to_ Staging_Buffer NIC receives pkt. Metrics: 1.) Stack Process Time 2.) Driver copy from stack to Staging buffer 3.) DMA vs. Programmed I/O time 4.) Switch Latency Switch = logic analyzer timestamp = timestamp marker (software) Tx Stack_to_NIC Tx DMA/PIO Tx App_to_NIC NIC-NIC Separate test with SmartBits Analyzer

12. 8 June 2004 12 Fourth Space Internet Workshop NIC/OS Receive Path Measurements Packet Recv’d Interrupt Key : = start receive marker Metrics: 1.) DMA vs. Programmed I/O 2.) Copy time from staging buffer(receive_buffer) to stack = timestamp marker (software) Rx DMA/PIO Rx_NIC_to_Stack App receives pkt. Rx_receive_buffer _to_stack Stack Process Time (TBD)

13. 8 June 2004 13 Fourth Space Internet Workshop Preliminary Performance Results 26.148 20.064159.421 7.740 TX_STACK_TO_NIC* 23.849 17.256144.600 5.100 TX_BUFFER_TO_NIC (PIO or DMA)* 2.523 14.880 1.260 TX_STACK_ PROCESSING** TX_STACK_TO_ STAGING BUFFER* 39.156 27.843299.58118.481 RX_NIC_TO_STACK* 0.481 2.525 3.360 1.500 Rx_BUFF_to_STACK* 39.073 25.264296.40016.560 Rx_NIC_to_BUFF (PIO or DMA)* Std. Dev (us) Avg Width (us) Max Width (us) Min. Width (us) * Numbers are for Programmed I/O (which can be up to ~4 times slower than DMA), using TCP (ftp), MTU sizes vary between 64-1500 bytes ** Numbers are for Programmed I/O, UDP, MTU sizes between 64-1500 TBD TBD Std. Dev (us) Avg Width (us) Max Width (us) Min. Width (us) Without LLC With LLC TBD TBD 2.346

14. 8 June 2004 14 Fourth Space Internet Workshop Status LLC working in lab (2 millisecond timeout) Additional performance testing includes: –LLC Bi-directional benchmark Worst case footprint of LLC from CPU & network throughput cost –Fixed packet size (minimum and maximum) –Linux OS (should be worse due to virtual memory model)

15. 8 June 2004 15 Fourth Space Internet Workshop Conclusion Preliminary tests indicate that: the onboard network architecture, using LLC for reliability, will meet the 5 millisecond requirement for packet transfer

16. 8 June 2004 16 Fourth Space Internet Workshop The Team Software Greg Menke Art Ferrer Freemon Johnson Alan Cudmore Jonathan Wilmot Jane Marquart Hardware Mike Lin Scott Edfors For more information contact: Jane Marquart at jane.marquart@nasa.gov