1. ACEs ACEs 24/9 - 2004 INF5060: Multimedia data communication using network processors

2. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors ACE  Active Computing Element  Defined by Intel’s SDK  Not part of hardware  Also sometimes called Action/Classification Engine  Idea behind ACEs  Element in a graph  Part of a classification engine

3. Classification

4. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Demultiplexing Ethernet Header IP Header TCP Header Payload TCP UDP Applications IP Header TCP Header Payload TCP Header Payload IP MAC  Packet demultiplexing  Used with layered protocols  Packet proceeds through one layer at a time  On input, software in each layer choose module at next higher layer  On output, type field in each header specifies encapsulation

5. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Classification Ethernet Header IP Header TCP Header Payload TCP UDP Applications IP MAC … Payload  Packet classification  Alternative to demultiplexing  Designed for higher speed  Considers all layers at the same time  Linear in number of fields  Two possible implementation options  Software  Hardware

6. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Classification  Idea  Classifiers perform a logical AND between fields  Hardware classifiers can perform AND operation in parallel  Reality  Need detailed distinctions involving many fields  Fields are not always at the same offset  Requirements for classification change over time

7. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Classification  Need detailed distinctions involving many fields  Multiple categories  Web traffic  Telnet traffic  Mail traffic  ICMP traffic  other traffic

8. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Classification ….local etype immed[etype, ETH_IP] alu_shf[ --, etype, -, $$hdr3, >>16] br!=0[NotWeb#].endlocal … br!=byte[$$hdr5, 0, IPT_TCP, NotWeb#] ….local wprt immed[wprt, TCP_WWW] alu[--, dport, -, wprt] br!=0[NotWeb#].endlocal … dest src (2-4) src (0-1) typevershlenservice total lengthidentificationflgsfrag. offsettttltype header checksumsrc. address dest. address (0-1) dest. address (2-3)src. portdest. portseq. number (0-1) seq. number (2-3)acknowledgementversreserv.code windowchecksumurgent pointer data 0816243240485664 possible IP Options !!.local base boff dpoff dport ld_field_w_clr[dpoff, 0001, $$hdr3, >>6].local mask immed[mask, 0x3c] alu [ dpoff, dpoff, AND, mask ].endlocal alu[dpoff, dpoff, +, 16] Buf_GetData[base, dl_buffer_handle] DL_GetBufferOffset[boff] alu[boff, boff, +, dpoff] alu_shf[boff, --, B, boff, >>3] sdram[read, $$hdr0, base, boff, 1], ctx_swap alu[ dpoff, dpoff, AND, 0x7 ] alu[ --, dpoff, -, 4] br>=0[SecondWord#] ld_field_w_clr[dport, 0011, $$hdr0, >>16] br[GotDstPort#] SecondWord#: ld_field_w_clr[dport, 0011, $$hdr1, >>16] GotDstPort#: Logical AND …

9. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Classification  Need detailed distinctions involving many fields  Multiple categories  Web traffic  Telnet traffic  Mail traffic  ICMP traffic  other traffic  Fields are not always at the same offset  Variable-sized headers  Fields not at fixed offsets  Easily handled with software  Finite cases can be specified in rules  Hybrid classification can be considered (hardware first, software next)

10. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Classification  Desirable 1.Fixed headers  Protocols without any optional fields  XTP  Options in dedicated locations  IPv6 – options follow header 2.Multiple classification stages … Hardware classifier Software classifier

11. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Classification  Static Classification  Flow specified in rule sets  Header fields and values know a priori  Dynamic classification  “Flows” created by observing the packet stream  Values taken from headers  Allows fine-grained flows  Requires state information

12. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Example static classification  Allocate one flow per service type  DiffServ  One header field used to identify flow  IP ToS field / IP priority and DS fields Options (0 or more) Destination Address Source address Time to liveProtocolHeader checksum IdentificationDMFragment offset VersionIHLTotal length Data Padding DS 00 Routing table DiffServ classifiers  Class selector codepoints  If of the form xxx000  Differentiated Services Codepoint  xxxxx0 reserved for standardization  xxxx11 reserved for local use  xxxx01 open for local use, may be standardized later DiffServ-enabled Router

13. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Dynamic classification  Allocate flow per TCP connection  Header fields used to identify flow  IP source address  IP destination address  TCP source port number  TCP destination port number  Applications  Clustered web servers  Load leveller dispatching TCP connection to several web servers  Firewalls  Allow incoming packets only when outgoing connection was made  More complicated firewall cases  FTP data connection  Voice over IP  RTP video streaming ……

14. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Dynamic classification  Flow identification  By observing the packet stream  Packets belonging to the flow  FTP, SSH, …  Packets outside the flow  RSVP, IGMP, …  Connection-oriented network  Per-flow SVC can be created on demand  Flow ID equals connection ID  State  Connection  Allocated resources  Connectionless network  Flow ID used internally  Each flow ID mapped to (next hop, interface)  State  Next hop, interface mapping

15. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Dynamic classification  Dynamic classification  Usually performed in software  State kept in memory  State information created/updated at wire speed

16. ACEs

17. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors ACE Features  Active Computing Element  Defined by Intel’s SDK  Not part of hardware  Also sometimes called Action/Classification Engine  One ACE: one step in dynamic classification  ACEs in Intel’s SDK  Fundamental software building block  Runs on StrongARM, microengine, or host  Handles control plane and fast or slow data path processing  Coordinates and synchronizes with other ACEs  ACEs form a graph  Used to construct packet processing systems  Can have multiple inputs or outputs  Can serve as part of a pipeline

18. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors ACE Terminology  Microengine  One of the specialized processors of the IXP card  Microblock  Code that runs  in one thread  on one microengine  MicroACE  Core component runs on StrongARM  Microblock component runs on microengines  Source microblock  Handles ingress from I/O device  Sink microblock  Handles egress to I/O device  Transform microblock  Intermediate position in a pipeline

19. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Four conceptual parts of an ACE  Initialization  Classification  Actions associated with each classification  Message and event management

20. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Four conceptual parts of an ACE  Initialization  Classification  Actions associated with each classification  Message and event management source/init.c … ix_error ix_init(int argc, char **argv, ix_ace ** ap) { struct wwbump *wwb; ix_error e; (void)argc; *ap = 0; wwb = malloc(sizeof(struct wwbump)); if ( wwb == NULL ) … wwb->ue = atoi(argv[2]); strcpy(wwb->bname, "WWBUMP"); wwb->name[sizeof(wwb->name) - 1] = '\0'; strncpy(wwb->name, argv[1], sizeof(wwb->name) - 1); e = ix_ace_init(&wwb->ace, wwb->name); if (e)… e = RmInit(); if (e)… e = RmRegister(&wwb->tag, wwb->bname,&wwb->ace, exception, wwb, wwb->ue); if (e)… e = cc_init(wwb); if ( e )… *ap = &wwb->ace; return 0; }

21. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Four conceptual parts of an ACE  Initialization  Classification  Actions associated with each classification  Message and event management ucbuild/WWBump.uc ….local etype immed[etype, ETH_IP] alu_shf[ --, etype, -, $$hdr3, >>16] br!=0[NotWeb#].endlocal … br!=byte[$$hdr5, 0, IPT_TCP, NotWeb#] ….local wprt immed[wprt, TCP_WWW] alu[--, dport, -, wprt] br!=0[NotWeb#].endlocal …

22. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Four conceptual parts of an ACE  Initialization  Classification  Actions associated with each classification  Message and event management source/action.c … ix_error exception(void *ctx, ix_ring r, ix_buffer b) { struct wwbump *wwb = (struct wwbump *) ctx; ix_error e; unsigned char c; (void) r; e = RmGetExceptionCode(wwb->tag, &c); if ( e ) … Webcnt++; e = RmSendPacket(wwb->tag, b); if ( e )… return 0; }

23. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Four conceptual parts of an ACE  Initialization  Classification  Actions associated with each classification  Message and event management source/wwbcc.c ix_error getcnt(ix_base_t* bp, long* rv) { (void)bp; *rv = Webcnt; return 0; }

24. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Output targets and late binding  ACE has set of outputs  Not in wwbump – only default  Each output given target name  Outputs bound dynamically at run time  Unbound target corresponds to packet discard ixsys.config-wwbump … microace ifaceInput./ingressAce none 0 1 microace ifaceOutput./egressAce none 1 2 microace wwbump./wwbump none 0 0 … bind static ifaceInput/default wwbump bind static wwbump/default ifaceOutput …

25. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Output targets and late binding  ACE has set of outputs  Create other targets in source/init.c  Each output given target name  Outputs bound dynamically at run time  Unbound target corresponds to packet discard SDK Reference ix_error ix_target_init( ix_target* targp, ix_ace* acep, const char* name);  Targp – where to store the new target handle  Acep – ACE to which the new target belongs  Name – symbolic name of the new target

26. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Conceptual ACE interconnection  Ingress ACE acts as source  Process ACE acts as transform  Egress ACE acts as sink  Connections created at run time ixsys.config-wwbump … bind static ifaceInput/default wwbump bind static wwbump/default ifaceOutput …

27. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Components of MicroACE  Single ACE with two components  StrongARM (core component)  Microengines (microblock component)  Communication possible between components arm-be/wwbump SlowIngressWWBump.uof ucbuild/WWBump.uc … immed[dl_next_block, IX_EXCEPTION] br[Finish#] … ucbuild/WWB_dl.uc … alu[ --, dl_next_block, -, IX_EXCEPTION] br=0[Send_SA#] … Send_SA#: DL_SASink[ ].continue … source/action.c … ix_error exception(void *ctx, ix_ring r, ix_buffer b)

28. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Communication: Exceptions  Packets passed from microblock to core component  Mechanism  Microcode set dl_next_block to IX_EXCEPTION  Dispatch loop forwards packet to core  ACE tag used to identify corresponding component  Exception handler is involed in core component

29. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Actions applied to exceptions  Consume the packet and free the buffer  Modify the packet before sending it on  Send the packet back to the microblock for further processing  Forward the packet to another ACE on the StrongARM SDK Reference ix_error ix_buffer_del(ix_buffer* buf) source/action.c ix_error exception( void *ctx, ix_ring r, ix_buffer b) { … e = RmSendPacket( wwb->tag, b); … } SDK Reference ix_error ix_target_take(ix_target* targp, ix_buffer* buf); targp – bound target to pass to buf – buffer to pass to target

30. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Division of ACE into components  Microblocks form fast path

31. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Illustration of microblock groups  Entire microblock group assigned to same microengine  Set of one or more microblocks  Treated as single unit  Loaded onto one microengine for execution  Can be replicated on microengines for higher speed

32. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Illustration of microblock groups  Set of one or more microblocks  Treated as single unit  Loaded onto one microengine for execution  Can be replicated on microengines for higher speed ucbuild/WWB_dl.uc … DL_Init[] EthernetIngress_Init[] WWBumpInit[].while(1) Top_Of_Loop#: DL_SASource[ ] alu[--, dl_buffer_handle, -, IX_BUFFER_NULL] br=0[Test_Ingress#], guess_branch br[Send_MB#] Test_Ingress#: EthernetIngress[ ] alu[--, dl_buffer_handle, -, IX_BUFFER_NULL] br=0[Top_Of_Loop#] … Dispatch loop WWBump microblock SlowIngress microblock

33. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Microblock structure  Asynchronous model  Neither core ACE nor MicroACE may block  Programmer creates  Initialization macro  Dispatch loop

34. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Event loop  Central to asynchronous programming model  Uses polling  Repeatedly checks for presense of event(s) and calls appropriate handler  Can be hidden from programmer  In ACE model  Explicit  Programmer can modify/extend  Note  Event loops are also used when programming Berkeley sockets with select()

35. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Event loop processing  If event loop stops  All processing stops  The arrival of a new event will not trigger invocation of an event handler  Conclusion: the event loop must go on

36. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Crosscall mechanism  Used between  Core component of one ACE and another  ACE core component and non-ACE application  Not intended for packet transfer  Operates like Remote Procedure Call  Mechanism know as crosscall

37. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Crosscall implementation  Both caller and callee programmed to use crosscall  Not dynamic  Not at run-time  Declaration given in Interface Definition Language (IDL)  IDL compiler  Read specification  Generates stubs that handle calling details include/idl/wwbump.idl interface wwbump { twoway long getcnt(); }; include/wwbump_cb_c.h include/wwbump_cc_c.h include/wwbump_import.h include/wwbump_sk_c.h include/wwbump_stub_c.h

38. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Crosscall implementation  Three types of crosscalls  Deferred: caller does not block; return notification asynchronously  Oneway: caller does not block; no value returned  Twoway: caller blocks; callee returns a value  Twoway call corresponds to traditional RPC  Type of call determined at compile time  Note  ACEs are not allowed to block  Therefore an ACE can not call a twoway crosscall (only answer one)

39. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Packet queues  Packet queues  Generated by the SDK when ACEs communication asynchronously  Buffer packets  Placed between ACE components  Permit asynchronous operation

40. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Illustration of packet queues

41. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Packet queues  Operation  ix_target refers to the ring used to transfer packets from the current ACE to another ACE  The target may be bound or not  If bound, buffers to the target (using ix_target_take ) are placed in the ring  If unbound, buffers to the target are discarded  ix_res_bind allocates a new ring for a target  ix_res_unbind disassociates current packet processing queue from the target  all previously taken packets will still processed by the target ACE

42. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Summary  Intel SDK uses ACE programming model  ACE  Basic unit of computation  Can include code for StrongARM (core) and microengines (microblock)  Packet queues used to pass packets between ACEs  Crosscall mechanism used for nonpacket communication

43. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Resource Manager  Task  Used by StrongARM code  For microACEs and microACE applications to interface with microengines  API  Load code into microengines  Enable/disable microengines  Get/set microengine configuration and resource assignment  Send and receive packets to and from microcode blocks  Allocate and access uncached SRAM, SDRAM and Scratch memory

44. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Resource Manager  Task  Used by StrongARM code  For microACEs and microACE applications to interface with microengines  API  Initialize and configure IXP1200  Send and receive packets to and from microcode blocks  Load code into microengines  Get/set microengine configuration and resource assignment  Enable/disable microengines  Allocate and access uncached SRAM, SDRAM and Scratch memory  Create and statically bind microACEs

45. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Resource Manager  Data structures  RmMemoryHandle  Opaque handle identifying memory allocated by the resource manager  typedef int RmMemoryHandle

46. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Resource Manager  Data structures  RmMemoryHandle  Opaque handle identifying memory allocated by the resource manager  typedef int RmMemoryHandle  RmImportedSymbol  Structured used to patch microcode symbols  typedef struct { int meNumber; char name[MAX_SIZE_OF_IMPORTED_SYMBOL]; int value; } RmImportedSymbol;

47. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Resource Manager  RmMalloc  Allocate a particular kind of memory  RM_SRAM  RM_SDRAM  RM_SCRATCH  Some SRAM and SDRAM is already used by the ASL, some SDRAM is used by Linux, the rest can be used freely by microACEs for data structures of its choosing  The memory is not cached  The memory is not protected by an MMU, and the virtual address is the same for all processes  Returned pointers are always aligned (SDRAM to 8 bytes, SRAM and Scratch to 4 bytes)  Requested sizes are rounded to alignment  This allocation is not efficient  microACEs should allocate all memory they need at once and manage it themselves  ix_error RmMalloc( RmMemoryType in_memory_type, unsigned char* out_mem_handle_ptr, int in_size_in_bytes );  RmFree  Released memory allocated by RmMalloc  ix_error RmFree( unsigned char* ptr );

48. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Resource Manager  RmUengPatchSymbols  A function call that patches an array of variables to the StrongARM side into the microcode running on one microengine.  ix_error RmUengPatchSymbols( int uengNumber, RmImportedSymbol symbols[], int numberOfSymbols );

49. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Resource Manager  Translating between virtual and physical addresses  The microengines map memory differently into their address space then the StrongARM  StrongARM addresses make no sense and have to be translated to offsets from the start of each particular kind of memory (and back)  RmGetPhysOffset  ix_error RmGetPhysOffset( RmMemoryType in_memory_type, unsigned char* in_data_ptr, unsigned int* out_offset );  Translate address in_data_ptr in RmAlloc’d memory to its offset from the given memory type  The offset is in words (4 byte units) for SRAM and Scratch, and in quadwords (8 byte units) for SDRAM  RmGetVirtualAddress  ix_error RmGetVirtualAddress( RmMemoryType in_memory_type, unsigned char** out_buffer_ptr, unsigned int in_offset);  Take the physical offset from the base of the given memory type and translate it into a virtual address valid for the StrongARM

50. 2004 Carsten Griwodz & Pål HalvorsenINF5060 – multimedia communication using network processors Resource Manager  RmGetSysConfig  Retrieve an RmSysconfig structure containing the number of ports and the amount of memory installed on the IXP card in bytes  RmGetMemInfo  Retrieve the currently available memory in bytes