Kernel HW KABI Tools Tony Camuso Aug 20, 2015 INTERNAL ONLY EXERCISE - STEP 1 INTRO: I'm here today to talk to you about the value of having a relationship with Red Hat. [ Whiteboard Exercise ] Before we get started, I want to capture a few thoughts you may have about what Red Hat is today. Possibly some of the following words: [ Write examples in a single column as you say them ] Linux Open source Jboss Low-cost Performance Freedom This is all true, and it's part of what of we offer, but but what we want to talk to you about today is something more compelling that Red Hat has to offer from a value and relationship perspective. INTERNAL ONLY

What's the problem? No easy way to determine KABI dependencies kernel wide. Need to view the dependency tree from exported symbols down to all the compound data types they depend on. Need to view the dependency tree from compound data types up to their dependent exported symbols. Need to determine if changes to a subordinate compound data type can affect a whitelisted symbol. [ EXERCISE - STEP 2 ] But rather than have me alone tell this story. I want to take a few moments have you hear from one of our customers, Steve Rubinow from the New York Stock Exchange. [PLAY NYSE VIDEO. - While the video is playing, write key words or themes spoken by Rubinow, but write them in a second, column next to the first list. ] Performance Mission critical Brainstorming Our best minds, their best minds together Technology innovation Good interoperability Advising us the best way to use it We can count on Leader Support Extremely strategic What you heard from Steve is that Red Hat to the New York Stock Exchange is more than just particular products or particular technologies. It's about working closely with a trusted advisor to understand technology and the impact in your organization. About creating a relationship that combines your expertise and Red Hat's knowledge and leadership. Ultimately, this is what we'd like you to understand about the type of relationship we'd like to have with you as a customer. Internal Only

Proposed Solution Create a database of all exported symbols and their dependencies. Create the database as part of the build process or independently as an ancillary process. Create a tool to query the data base for exported symbols and the data structures in their hierarchy. [ EXERCISE - STEP 2 ] But rather than have me alone tell this story. I want to take a few moments have you hear from one of our customers, Steve Rubinow from the New York Stock Exchange. [PLAY NYSE VIDEO. - While the video is playing, write key words or themes spoken by Rubinow, but write them in a second, column next to the first list. ] Performance Mission critical Brainstorming Our best minds, their best minds together Technology innovation Good interoperability Advising us the best way to use it We can count on Leader Support Extremely strategic What you heard from Steve is that Red Hat to the New York Stock Exchange is more than just particular products or particular technologies. It's about working closely with a trusted advisor to understand technology and the impact in your organization. About creating a relationship that combines your expertise and Red Hat's knowledge and leadership. Ultimately, this is what we'd like you to understand about the type of relationship we'd like to have with you as a customer. Internal Only

Proposed Solution (cont) Uses preprocessor output files, which include all headers with all typedef and macro expansions for each corresponding source file. The generated data files comprise a comprehensive graph of all exported symbols and all their dependencies. Internal Only

Why use THIS tool? Able to relate lowest level data structures to any exported symbols that depend on them. Can detail the entire dependency tree of any exported symbol or any data structure. Provides command line options to limit search scope to directories, files, and the kabi whitelists. Provides a verbose option to display entire hierarchies. Imposes a 'dups' policy to limit output when encountering duplications. Internal Only

Internal Only

Internal Only

Show exported symbols affected by data structure -slw : struct, limited to whitelist, whole word search (exact match) -m fs/bio : limit search to path having substring “fs/bio” Internal Only

Another struct search Show exported symbols affected by structs with 'struct device' as part of their name. Limit search to files and directories with 'pci' substring, and limit output to first 20 lines. Search for declaration of 'struct class' and grep for 'struct device'. Internal Only

Search for exported symbol Search for exported 'struct bio_set' and see its top level dependencies. See if 'struct bio_set' is in the kabi whitelist. Internal Only

Find whitelisted function pci_iomap Then show its dependency tree Internal Only

RHEL7 Kernel Compile Times System Without kabi-parser With kabi-parser % change intel workstation core 860 @ 2.8GHz 4 cores (8 HT) 16 G DDR3 15:49 (15.82) 17:07 (17.12) 8.2 amd dinar opteron 6386 @ 1.4GHz 32 cores 32 G DDR3 4:25 (4.42) 4:57 (4.95) 12.1 Internal Only

Search Times Substring search for popular struct (generates lots of output) Exact match search for same takes about 2 seconds less. Internal Only

Search Times (cont) Longest search times are typically less than a minute. At this time, searches are disk intensive, so cpu cores and memory have little impact on performance. SSD should improve search times considerably. Slurping the whole database for each search is impractical, though it might be useful for an interactive version of the tool. Internal Only

Improving Search Performance Under consideration Second level database that maps symbols to the files that contain them. Threaded search Memory mapped database files Internal Only

Implementation kabi-parser Uses sparse semantic parser to tokenize precompiled sources that contain exported symbols. Used by kernel make for source checking (C=1, C=2) Maintained upstream. Uses boost to serialize the sparse output for precompiled source files. Invoked by kernel make for each source file. makei.sh and kabi-data.sh are scripts for creating the precompiled sources and data files outside of the kernel build. [ EXERCISE - STEP 2 ] But rather than have me alone tell this story. I want to take a few moments have you hear from one of our customers, Steve Rubinow from the New York Stock Exchange. [PLAY NYSE VIDEO. - While the video is playing, write key words or themes spoken by Rubinow, but write them in a second, column next to the first list. ] Performance Mission critical Brainstorming Our best minds, their best minds together Technology innovation Good interoperability Advising us the best way to use it We can count on Leader Support Extremely strategic What you heard from Steve is that Red Hat to the New York Stock Exchange is more than just particular products or particular technologies. It's about working closely with a trusted advisor to understand technology and the impact in your organization. About creating a relationship that combines your expertise and Red Hat's knowledge and leadership. Ultimately, this is what we'd like you to understand about the type of relationship we'd like to have with you as a customer. Internal Only

Implementation (cont) kabi-lookup Uses boost to deserialize the data files into a container (std::map). Partial or whole word searches. Searches can be limited to symbols in the kabi whitelists. Searches can be limited to specific directories or files by using a path mask option. [ EXERCISE - STEP 2 ] But rather than have me alone tell this story. I want to take a few moments have you hear from one of our customers, Steve Rubinow from the New York Stock Exchange. [PLAY NYSE VIDEO. - While the video is playing, write key words or themes spoken by Rubinow, but write them in a second, column next to the first list. ] Performance Mission critical Brainstorming Our best minds, their best minds together Technology innovation Good interoperability Advising us the best way to use it We can count on Leader Support Extremely strategic What you heard from Steve is that Red Hat to the New York Stock Exchange is more than just particular products or particular technologies. It's about working closely with a trusted advisor to understand technology and the impact in your organization. About creating a relationship that combines your expertise and Red Hat's knowledge and leadership. Ultimately, this is what we'd like you to understand about the type of relationship we'd like to have with you as a customer. Internal Only

Where is it? Latest rpms: http://people.redhat.com/tcamuso/kabitools/ Git repo: https://github.com/camuso/kabitools Report problems and feature requests to https://github.com/camuso/kabitools/issues Internal Only