1. Washington WASHINGTON UNIVERSITY IN ST LOUIS jdd@arl.wustl.eduhttp://www.arl.wustl.edu/~jdd The New MSR Plugin Download Environment

2. 2 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 First: The Old Way Compile plugin code on NetBSD Copy plugin directory onto File System destined for SPC Boot SPC Log in to SPC –requires user space and serial line or telnet access Use modload(1) on SPC to load plugin –links plugin against kernel in /netbsd on SPC Use modstat(1) on SPC to see list of loaded plugins

3. 3 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 The New Way Compile plugin code on NetBSD Verify plugin has no undefined symbols –There is no link against running kernel anymore –msr_lkm.h provides macros to needed kernel fucntions and structures We can add more as we need them. Use pluginDownload utility from CP (Linux) to download plugin to SPC –Not sure I’ve even tried it on NetBSD… Use sendcmd utility from CP to configure plugin, filters, etc Use pluginStat utility from CP to view list of loaded plugins

4. 4 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 pluginDownload Uses gnu binary utilities cross-linker –Runs on Linux, builds a NetBSD binary –Used to: prelink to get size information final link to relocate Uses new AAL5 control data channels to download plugin directly into kernel memory. –uses sendcmd API to: allocate kernel memory for plugin set up AAL5 data channel call plugin entry function to complete load operation Side Note: sendcmd API –should be expanded to include all of sendcmd, but right now we have about 15 “commands”.

5. 5 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 The rest of the slides are old notes

6. 6 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Plugin Loading Steps Compile Plugin code on NetBSD Do one of these two, currently Linux seems to work better for later steps: –Pre-link Plugin on NetBSD to get size info ready  ld -e example -T 0 -o pl_combined combined.o –Pre-link Plugin on Linux to get size info ready  ld-cross –o pl_combined.o –e _example –Ttext 0 combined.o

7. 7 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Plugin Loading Steps On Linux CP: open file (pl_combined.o), get size information and allocate MSR/SPC memory for plugin  sendcmd –p # -c rp_pcu –s allocate –d -d -d sendcmd code in kernel calls msr_lkmiotcl(LMRESERV,, FWRITE) points to a (struct lmc_reserv) which passes in: –size = a_text + a_data + a_bss (all from a.out header) »but we don’t need to load the bss since it is all 0 –sym_size = a_syms + stb.st_size – N_STROFF(info_buf) »a_syms is symbol table size »string table is at the end so, the string table size is full file size (stb.st_size minus string tab offset) –sym_symsize = a_syms and returns: –address of block of memory allocated in kernel (resrvp->addr) –slot in lkm table (resrvp->slot) –address of symbols (resrvp->sym_addr) msr_lkmioctl() returns 0 if successful, non-zero if there was an error. –sendcmd will return Kernel address (PlAddr) for plugin to be loaded at Slot number Address of location of symbols (what do we use this for?) –This is used for adjusting the string table pointers before they are loaded (see modload.c for details) –useful command:  objdump

8. 8 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Plugin Loading Steps (continued) On Linux CP: re-link plugin to relocate to PlAddr  ld-cross –o plugin –e _example –Ttext combined.o relocates to plAddr+0x20 and sets ‘_example’ as the entry point –the +0x20 is because we are currently downloading the a.out header –This sets the a_entry field in the a.out header of the file file plugin is now ready for downloading On Linux CP: request two data channels to MSR Kernel –One for plugin text and data:  sendcmd –p # -c dchan -s get -d -d –Another for the plugin symbols (sym_addr was returned by allocate command earlier)  sendcmd –p # -c dchan -s get -d -d –sendcmd returns the dchan# in its responses On Linux CP: download file plugin into MSR/SPC memory:  pluginDownload -f plugin –S -d -d -V -L –This needs to adjust the string table before downloading. Thus it needs the sym_addr which was returned by the allocate sendcmd above –text and data go to dchan#1 and symbol and string table to go dchan#2

9. 9 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Plugin Loading Steps (continued) On Linux CP: Finish the loading of the plugin  sendcmd –p # -c rp_pcu –s load -d -d -d –I don’t think we actually need the PlAddr here. –I think in this command (or an earlier one) we do need to send the entry symbol address –We are also sending the amount of data sent for the plugin (size) and symbols (sym_size) the msr_lkm code will use these two values to update –curp->offset –curp->sym_offset –This sendcmd causes a call to msr_lkmioctl(LMREADY,, FWRITE) –We may also need to do some other stuff. See page of ioctl’s used by modload. On Linux CP: release the data channel  sendcmd –p # -c dchan -s release -d When all done with plugin: –On Linux CP: unload and deallocate the plugin  sendcmd –p # -c rp_pcu –s unload –d We should never have to use the LMLOADBUF and LMLOADSYMS cmds

10. 10 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Data Transfer Options for the download phase, CP side: –hack up modload from NetBSD and build it on Linux modload uses ioctl’s to load the module into the local kernel we want to send AAL5 frames –modify Stage3 of AAL5_download Stage3 already processes NetBSD a.out files Need to do one of these: –change Stage3’s interaction with Stage2, i.e. have it not expect any acks –add a Stage2 like thing to the MSR kernel for accepting plugin frames –Merge modload and Stage3 modload does some manipulations to get the string table correct Stage3 has all the code to send AAL5 frames and wait for ACK/NACK Options for the downloading phase, MSR side: –Build a new AAL5 control data channel structure Uses AAL5 frame buffer pool that the IP fwding channels use Re-uses some of the frame processing code in kernel Arriving frames are copied into previously allocated memory associated with the data channel. –It would be nice if APIC descriptors pointed directly at the allocated memory, but that breaks the current model and would make it hard to build in safeguards against corruption.

11. 11 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Data Transfer (continued) Vaddr Start of Virtual address block where data frames should go (32b) Length Length of block in bytes (32b) Next Addr Where next frame should go. Initially set to (32b) Seq # Next expected frame sequence number (first frame should be 1) (16b) Index(dchan#) 0 1... DCHAN_MAX MSR Kernel Dchannel Table

12. 12 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Data Transfer (continued) MSR Control Data Channel RATM AAL5 Frame Format (as it appears in the MSR Kernel) dchan#(8bits)seq# (16bits) framelength (16bits)pad (16bits) AAL5 Trailer (8 Bytes) AAL5 Frame Data (<= 1992 Bytes) Shim Reserved space (8 Bytes) 1992 B 2000 B 2016 B APIC Rx desc points here flags (8bits) flags: 1: Data 2: Ack 4: Nack

13. 13 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 External Symbols in Plugin We will implement a structure to contain pointers to Kernel symbols that Plugins are allowed to use. –This structure will surely grow over time as we discover more kernel functions that we want to expose to Plugins When a plugin is loaded, it will be given the pointer to this structure in the kernel via the call to its entry function The plugin will therefore not need to link against the running kernel to resolve symbols. This strategy will achieve two goals: –Put bounds on what plugins can do in the kernel –Make the linking and downloading of plugins easier

14. 14 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 External Symbols in Plugin (continued) Current kernel functions used in example plugin: –malloc related: free() –via FREE() macro from sys/malloc.h malloc() –via MALLOC() macro from sys/malloc.h –MSR Related: msr_printf() PCU Related: –pcu_deregister_class() –pcu_free_all_instances() –pcu_register_class() –LKM Related: lkm_nofunc() lkm_dispatch() –via DISPATCH() macro from sys/lkm.h lkm_exists()

15. 15 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 The rest of the slides are supporting notes some from other places

16. 16 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Data Transfer Options for the download phase, CP side: –hack up modload from NetBSD and build it on Linux modload uses ioctl’s to load the module into the local kernel we want to send AAL5 frames –modify Stage3 of AAL5_download Stage3 already processes NetBSD a.out files Need to do one of these: –change Stage3’s interaction with Stage2, i.e. have it not expect any acks –add a Stage2 like thing to the MSR kernel for accepting plugin frames Options for the downloading phase, MSR side: –When we get a request to load a plugin of size  sendcmd –p # -c rp_pcu –s allocate –d Allocate an APIC buffer of that size associate the plugin load descriptor with that buffer Tie the descriptor to the plugin load VCI Go –Once we have the plugin in the allocated buffer: in response to a  sendcmd –p # -c rp_pcu –s load –d -S “example use calls to lkmioctl() to do the rest

17. 17 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Implementing modload in MSR Issues: –Where will we download plugin from? presumably just from the CP –Will the plugin on the CP already be pre-linked with the current MSR kernel? if not, we may have Linux vs. NetBSD problems. if so –how do we plan to keep plugins and MSR kernels in sync –how do we know the address where the plugin will go in the kernel –Dynamic scheme: allocate memory on demand in the kernel for a new plugin –Static scheme: predefined slots at predefined addresses in kernel Each plugin is built to fit in a specific slot at a predefined address Still need to resolve symbols with actual kernel –if we allow this at all…

18. 18 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Implementing modload in MSR Option 1: Use sendcmd utility to execute modload on SPC –Download plugin binary to a file on MSR e.g. /var/run/rp/plugin/fileABC.o –and run modload e.g. modload –o /var/run/rp/plugin/fileABC.o –e example fileABC.o –Something like:  sendcmd –p # –c rp_pcu –s load –S “fileABC.o” –S “example” uses proposed new option –S for including strings causes file “fileABC.o” to be downloaded to MSR and loaded as kernel module named “example” –Requires presence on MSR file system of: /sbin/modload /usr/bin/ld up to date symbol version of kernel being executed

19. 19 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Implementing modload in MSR Option 2: Prelink on CP, use sendcmd to download directly into memory –Link plugin against copy of kernel on CP  ld -A netbsd.MSR -e _ -o -T –Reserve Kernel memory:  sendcmd –p # –c rp_pcu –s allocate –d returns kernel load address –Relink at kernel load address (LA)  ld -A netbsd.MSR -e _ -o -T –Open relinked module and load it into kernel  sendcmd –p # –c rp_pcu –s load –S uses proposed new option –S for including a strings –Adjusting symbol table entry pointers and load it. What does this actually do in modload(1) –sync(); What does this actually do in modload(1) –Call the module’s entry function to test it. –Post-install if called for... ioctl(lkm_fd, LMSTAT,...)

20. 20 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Implementing modload in MSR Option 3: Prelink on CP, Plugin must use kernel function pointer array to access predefined, limited number of kernel functions, then use sendcmd to download directly into memory –plugin code uses local pointer for function pointer array which will be reset later when kernel calls entry function –Reserve Kernel memory:  sendcmd –p # –c rp_pcu –s allocate –d returns kernel load address –Relink at kernel load address (LA)  ld -A netbsd.MSR -e _ -o -T –Open relinked module and load it into kernel  sendcmd –p # –c rp_pcu –s load –S uses proposed new option –S for including strings –Call the module’s entry function to test it. kernel will provide the actual location of the function pointer array –Post-install if called for... ioctl(lkm_fd, LMSTAT,...)

21. 21 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Implementing modload in MSR (continued) Current ld commands issued by modload on NetBSD: –modload links it the first time so it can get size information:  ld -A /netbsd -e _example -o /var/run/msr/example -T 0 combined.o –modload links it the second time to resolve symbols and relocate:  ld -A /netbsd -e _example -o /var/run/msr/example -T 100000 combined.o Equivalent command on Linux: –First do this on NetBSD (‘pl_’ implies pre-linked)  ld -e _example -o pl_combined -T 0 combined.o –This is just for relocating, everything else should be done on NetBSD:  ld-cross –o plugin –e _example –Ttext 0x100020 pl_combined.o –Linux and NetBSD seem to be off by 0x20 on their location of the text segment, still investigating that. –After this command, the file plugin should be ready for downloading

22. 22 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 modload ioctl’s We need to know exactly what each of these does: –ioctl(LMRESERV) reserves kernel memory reserves lkm slot for module reserves kernel memory for the symbols –ioctl(LMLOADBUF) loads module into memory, uses and affects lkm struct curp –ioctl(LMLOADSYMS) loads symbol table into memory, uses and affects lkm struct curp –ioctl(LMLOADSYMS) loads string table into memory, uses and affects lkm struct curp –ioctl(LMREADY) if DDB is turned on, adds modules symbols to the DB symbol table clears modules BSS space calls entry procedure of module, uses and affects lkm struct curp –ioctl(LMSTAT) post install retrieval of stats from module –ioctl(LMUNRESRV) release slot for module if something goes wrong after LMRESERV.

23. 23 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 (From Crossbow Tutorial) NetBSD Kernel Topics IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous

24. 24 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Loadable Kernel Modules Mechanism in NetBSD to dynamically load code into running kernel NetBSD System Utilities: –modload(8) –modunload(8) –modstat(8) Module Types supported: –System Call modules When unloaded, returns to original system call Any system call can be replace Take care when replacing ioctl(2) since LKM uses it to load/unload modules!! –Virtual File System modules –Device Driver modules: Block and character device drivers –Execution Interpreters: For binaries not normally usable by OS –Miscellaneous modules No defined interfaces Up to user/developer to provide hooks to get to the code This is what Router Plugins uses

25. 25 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Loadable Kernel Modules: modload Open /dev/lkm –ioctl’s will be performed on the open file descriptor Prelink module, open it and calculate size info –system(“ld -A /netbsd -e _ -o -T ”); Reserve Kernel memory: –ioctl(lkm_fd, LMRESERV, size_info) –returns kernel load address Relink at kernel load address Open relinked module and load it into kernel –ioctl(lkm_fd, LMLOADBUF,...) Adjusting symbol table entry pointers and load it. sync(); Call the module’s entry function to test it. Post-install if called for... –ioctl(lkm_fd, LMSTAT,...)

26. 26 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Overview (From MSR Tutorial) Introduction to hardware environment APIC core processing and buffer management Overview of SPC kernel software architecture and processing steps Plugin environment and filters Command Facility

27. 27 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Packet Classification & Plugins Classification provides and opportunity to bind flows to registered plugin instances. General classifier - Network Management –classification using 5-tuple, Prefix match on address, exact match port and proto 0 is a wildcard for all fields –input and output ports –filters added/removed via the command facility

28. 28 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Flow Bound to a Plugin... DQ/ In Queuing plugin Plugin Plugin Environment... DRR/ Out Queuing Flow Classifier/ (channel map) Route Lookup (Shim, FIPL, Simple, cache)... Frame/Buffer and IP Processing Ingress/ Egress ? NM Filter instance->handle_packet(instance, packet, flags) Call packet handler for bound instance with pointer to IP packet (struct ip *). AAL5 Frame Fragment offset VersionH-lenTOSTotal length Identificationflags TTLprotocolHeader checksum Source Address Destination Address Options ?? IP data (transport header and transport data) AAL5 padding (0 - 40 bytes) CPCS-UU (0) Length (IP packet + LLC/SNAP) CRC Shim pkt (struct ip *) handle_packet(inst, pkt, flags) { /* Plugin may read and/or * modify content but not * delete it unless COPY. * On return the framework * forwards packet */... return;} Rule 1 Rule 2 Rule 3 Rule 4 Rule 5 Rule 6 Rule 7 Rule 8 Rule 9 Rule 10 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 Search Invoke instance handler General Match Classifier: Linear search of {src_addr, dst_addr, src_port, dst_port, proto}. General Classifier options: {First, Last, All} Rule Actions: {Deny, Permit, Active}. Rule flags {All, Copy, Stop} Send packet to exact match classifier Flow Table flow hash Instance 1 {Active} Flow entry to plugin has a one-to-one relationship. Exact Match Classifier: Hash {src_addr, dst_addr, src_port, dst_port}, then linear search for flow spec. Exact Match Classifier options: None. Rule Actions: {Deny, Permit, Active, Reserve}. Rule flags {Pinned, Idle, Remove} Exact Match: active processing same as general match. The AAL5 length is and IP header checksum are calculated so plugin does not have to perform these operations.

29. 29 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Search Rule 1 Rule 2 Rule 3 Rule 4 Rule 5 Rule 6 Rule 7 Rule 8 Rule 9 Rule 10 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 i1i2i3i4i5 Invoke instance handler General Match Classifier: Linear search of {src_addr, dst_addr, src_port, dst_port, proto} General Classifier options: {First, Last, All} Rule Actions: {Deny, Permit, Active}. Rule flags {All, Copy, Stop} General Match Classifier Notes

30. 30 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Flow Table flow hash Instance 1 {Active} Flow entry to plugin has a one-to-one relationship General Match Classifier: Linear search of - {src_addr, dst_addr, src_port, dst_port, proto}. Exact Match Classifier options: None. Rule Actions: {Deny, Permit, Active, Reserve}. Rule flags {Pinned, Idle, Remove} Exact Match Classifier Notes

31. 31 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Instance 1 {Active} Instance 2 {Active, All} Instance 1 {Deny} Rule N General/Exact Match Classifier Class A “plugin x” Class B “plugin y” Class C “plugin z” Rule P Instance 1 {Active} Plugin instance maps to at most one rule/filter. General classifier: rule maps to at most 5 instances. Exact match classifier: rule maps to at most 1 instance. Active Processing Environment

32. 32 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Creating an Instance Class A classid = 100 inst_t *create_instance(class_t *, inst_id) Instance of Class A - (Base Class extended by Developer) class_t*class inst_t*next inst_idid fid_tbound_fid void (*handle_packet) (inst_t *, ip_t *, flag32_t); void (*bind_instance) (inst_t *); void (*unbind_instance) (inst_t *); void (*free_instance) (inst_t *); int (*handle_msg) (inst_t *, buf_t *, flag8_t, seq_t, len_t *)... create class instance Return reference to instance create_instance() Called by PCU framework in response to receiving command. struct my_inst { inst_t base; subclass defs };

33. 33 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Plugin Class Specific Interface All plugins belong to a class. At run time a class (i.e. plugin) must be instantiated before it vcan be referenced. Plugin is passed its instance pointer (like c++) as the first argument. Developer may extend the base class (struct rp_instance) to include additional fields which are local to each instance. Plugin developer must implement the following methods: –void(*handle_packet)(struct rp_instance *, struct ip *, u_int32_t); –void(*bind_instance)(struct rp_instance *); –void(*unbind_instance)(struct rp_instance *); –void(*free_instance)(struct rp_instance *); –int (*handle_msg)(struct rp_instance *, void *, u_int8_t, u_int8_t, u_int8_t);

34. 34 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart - 10/9/2015 Plugin Framework Enhancements Integrated with Command framework –send command cells to PCU: create instance, free instance, bind instance to filter, unbind instance –Send command cells to particular plugin instances –Send command cells to plugin base class Enhanced interface to address limitation noticed in crossbow: –instance access to: plugin class, instance id, filter id –pcu reports describing any loaded classes, instances and filters