Building a Reliable Onboard Network with Ethernet: A GSFC Prototype Jane Marquart NASA/GSFC

8 June Fourth Space Internet Workshop Requirements Ethernet/IP Onboard Network Bus Critical real-time data must be delivered reliably onboard the spacecraft (target is 5 milliseconds)

8 June Fourth Space Internet Workshop Protocol Options TCP TCP works but doesn’t support reliability features required by typical flight software UDP UDP is a better fit, but requires reliability be implemented by the user UDP + Reliability Where to implement reliability? Application layer or data link layer?

8 June Fourth Space Internet Workshop Reliability Trade Application layer: –Rx & tx packet latency through the stack is highly variable and non-deterministic (regardless of direction) –Latency up to several milliseconds on a 233mhz Datalink layer: –avoids IP stack latency, but; –the issue is standardization and portability of a datalink layer solution.

8 June Fourth Space Internet Workshop IEEE LLC LLC – “Logical Link Control” offers a IEEE standardized datalink layer reliability protocol, adding a 3 byte header to the ethernet frame Supplies 3 major types of service; –Type 1 (unreliable packet exchange, same as plain ethernet) –Type 2 (reliable, statefull, connection oriented; used by FDDI, Wireless, Token Ring) –Type 3 (reliable, stateless, connectionless) Type 3 selected –Stateless but reliable –Well-defined (and simple) state machine –Suitable for a variety of physical layers –Packet overhead is 3 bytes

8 June Fourth Space Internet Workshop Data-link Layer Reliability: LLC Ack/retry algorithm implemented within the NIC driver. Latency & delay well defined since control is right at the hardware. Widely supported, if not tolerated, by common OS IP stacks, routers, etc. Well defined & supported but ad-hoc mapping onto standard ethernet framing. Reliability protocol only useable between conformant drivers.

8 June Fourth Space Internet Workshop Packet data Ethernet frame headerLLC header PDU Formats ethernet dest (6 bytes) ethernet source (6 bytes ) ethernet type (2 bytes) DSAP (1 byte) SSAP (1 byte) Control (1 byte) user data & fill if req’d Ethernet destination (6 bytes) Ethernet source (6 bytes) Ethernet type (2 bytes) User data &fill if req’d Default datalink pdu Reliable pdu (data packet) Source SAPDestination SAP Reliable pdu (ack packet) ack pdu's swap DSAP & SSAP, along with ethernet source/destination so reply is sent to originating host. Standard ethernet frame w/ EthernetType >= 0x600 EthernetType < 0x600, set to frame length indicating a LLC pdu Ethernet frame header Packet dataEthernet frame headerLLC header ethernet dest (6 bytes) ethernet source (6 bytes ) ethernet type (2 bytes) DSAP (1 byte) SSAP (1 byte) Control (1 byte) No user data, Fill to min. valid pkt length

8 June Fourth Space Internet Workshop Ethernet/IP Multi-node Testbed SWITCH Subsys ‘A’ Coldfire MEDIA CONVERTER Ethernet Hub "Comm Card" Dell PC ITOS GSE Router Serial Channel Link Simulator Serial 4 Mbps 50 kbits 150kbits 4 Mbps 1 kbits 200 kbits Flight NIC Net Mgr SB TO CI CFDP NIC Driver IP OS Stack MEDIA CONVERTER CFDP Subsys ‘B’ PC

8 June Fourth Space Internet Workshop NIC Driver Architecture IP stack RX packets TX packets RX interrupt Packet Scheduler Async Pkt Channel Queues SOIS Independent Layer SOIS dependent Data-link layer Packet validation & LLC processing Packet retrieved TX done interrupt Packet submitted to NIC LLC ack/error packets Flight NIC Eth1Eth0 RX packets TX packets

8 June Fourth Space Internet Workshop Use Case Examples nominal Pkt sent with reliable QOS Driver returns ‘ACK’ Receive ACK Done pkt drop Pkt sent with reliable QOS ACK Time-out, Resend pkt Driver returns ‘ACK’ Receive ACK Done ACK drop Pkt sent with reliable QOS Driver returns ‘ACK’ ACK Time-out, Resend pkt, Log error Driver returns ‘ACK’ Discards redundant pkt Logs error Receive ACK Done pkt lost Pkt ‘N’ sent with reliable QOS ACK Time-out, Resend pkt ‘N’ Log error Driver returns ‘ACK’; logs error Receive ACK Done ACK lost Pkt N+1 sent with reliable QOS Driver returns ‘ACK’ ACK Time- out, Resend pkt, Log error Driver returns ‘ACK’ Discards redundant pkt Logs error Receive ACK Done ACK Time-out, Log error Pkt ‘N’ sent with reliable QOS Driver returns ‘ACK’ ACK Time- out, Log error Pkt N+1 sent with reliable QOS Pkt lost Pkt lost Pkt lost ACK lost ACK lost ACK lost

8 June Fourth Space Internet Workshop NIC/OS Transmit Path Measurements User app. opens socket to send packet (UDP Client) Stack Processes Packet Key : = start receive markers Tx_Stack_to_ Staging_Buffer NIC receives pkt. Metrics: 1.) Stack Process Time 2.) Driver copy from stack to Staging buffer 3.) DMA vs. Programmed I/O time 4.) Switch Latency Switch = logic analyzer timestamp = timestamp marker (software) Tx Stack_to_NIC Tx DMA/PIO Tx App_to_NIC NIC-NIC Separate test with SmartBits Analyzer

8 June Fourth Space Internet Workshop NIC/OS Receive Path Measurements Packet Recv’d Interrupt Key : = start receive marker Metrics: 1.) DMA vs. Programmed I/O 2.) Copy time from staging buffer(receive_buffer) to stack = timestamp marker (software) Rx DMA/PIO Rx_NIC_to_Stack App receives pkt. Rx_receive_buffer _to_stack Stack Process Time (TBD)

8 June Fourth Space Internet Workshop Preliminary Performance Results TX_STACK_TO_NIC* TX_BUFFER_TO_NIC (PIO or DMA)* TX_STACK_ PROCESSING** TX_STACK_TO_ STAGING BUFFER* RX_NIC_TO_STACK* Rx_BUFF_to_STACK* Rx_NIC_to_BUFF (PIO or DMA)* Std. Dev (us) Avg Width (us) Max Width (us) Min. Width (us) * Numbers are for Programmed I/O (which can be up to ~4 times slower than DMA), using TCP (ftp), MTU sizes vary between bytes ** Numbers are for Programmed I/O, UDP, MTU sizes between TBD TBD Std. Dev (us) Avg Width (us) Max Width (us) Min. Width (us) Without LLC With LLC TBD TBD 2.346

8 June Fourth Space Internet Workshop Status LLC working in lab (2 millisecond timeout) Additional performance testing includes: –LLC Bi-directional benchmark Worst case footprint of LLC from CPU & network throughput cost –Fixed packet size (minimum and maximum) –Linux OS (should be worse due to virtual memory model)

8 June Fourth Space Internet Workshop Conclusion Preliminary tests indicate that: the onboard network architecture, using LLC for reliability, will meet the 5 millisecond requirement for packet transfer

8 June Fourth Space Internet Workshop The Team Software Greg Menke Art Ferrer Freemon Johnson Alan Cudmore Jonathan Wilmot Jane Marquart Hardware Mike Lin Scott Edfors For more information contact: Jane Marquart at