1. Perf with the Linux Kernel

2. annotate – annotate source code with profile info
perf commands
annotate – annotate source code with profile info
kmem – kernel memory profiling
kvm – profile guests
list – list kinds of events
lock – analyze lock events
record – save profile data to a file
report – display profile report
stat – gather data while running a command
timechart – visualize system behavior
top – system profiling

3. ~/perf-stuff$ perf list |grep Hardware
cpu-cycles OR cycles [Hardware event]
instructions [Hardware event]
cache-references [Hardware event]
cache-misses [Hardware event]
branch-instructions OR branches [Hardware event]
branch-misses [Hardware event]
bus-cycles [Hardware event]
stalled-cycles-frontend OR idle-cycles-frontend [Hardware event]
ref-cycles [Hardware event]
Much more available if run as root.

4. ~/perf-stuff$ perf stat -e instructions date
User space example:
~/perf-stuff$ perf stat -e instructions date
Wed Feb 18 21:49:03 PST 2015
Performance counter stats for 'date':
669,408 instructions
seconds time elapsed
Subsequent runs gave:
668,858 instructions, seconds time elapsed
667,037 instructions, seconds time elapsed

5. perf calls perf_event_open(2) – kernel/events/core.c
perf list/stat kernel
sudo perf list
perf calls perf_event_open(2) – kernel/events/core.c
sudo perf stat -e net:netif_rx wget (24MB)
Performance counter stats for 'wget lecture7.mp4':
net:netif_rx
seconds time elapsed
For stat -e net:netif_receive_skb
3, net:netif_receive_skb
seconds time elapsed
int netif_receive_skb(struct sk_buff *skb) {
trace_netif_receive_skb_entry(skb);
return netif_receive_skb_internal(skb); }

6. perf_event_paranoid is not mentioned in ./Documentation
top
run perf top with sudo
perf_event_paranoid is not mentioned in ./Documentation
defined in kernel/sysctl.c
depends on CONFIG_PERF_EVENTS

7. timechart
sudo perf timechart record find /usr -name core >&/dev/null
sudo perf timechar
display output.svg

8. sudo perf record wget http://tahoot.com/cs172a-lecture7.mp4
report
sudo perf record wget
sudo perf report

9. sudo perf lock record wget http://tahoot.com/cs172a-lecture7.mp4
tracepoint lock:lock_acquire is not enabled. Are CONFIG_LOCKDEP and CONFIG_LOCK_STAT enabled?
under kernel hacking

10. perf kvm Perf uses CPU performance counters
These need to be virtualized to properly account for performance in a Guest VM
perf kvm [--host] [--guest] [--guestmount=<path> [--guestkallsyms=<path> - -guestmodules=<path> | --guestvmlinux=<path>]] {top|record|report|diff|buildid-list} perf kvm [--host] [--guest] [--guestkallsyms=<path> --guestmodules=<path> | --guestvmlinux=<path>] {top|record|report|diff|buildid-list}

11. Virtualization Overhead
VM layer overhead, Linux guest:
CPU and Memory: 14.36%
Network I/O: 24.46%
Disk I/O: 8.84%
Disk latency for reading: 2.41 times slower
Micro-operations execution time: times slower
Redhat reports 85% efficiency in VM's (RHEL6 Virtualization Getting Started Guide)
Reported by:

12. sudo perf kmem record wget http://tahoot.com/cs172a-lecture7.mp4
you may lose events...
Warning:
Processed events and lost 10 chunks!
Check IO/CPU overload!

13. kmem events
sudo perf record -g -e kmem:mm_page_alloc -c 1 wget

14. sudo perf annotate -l -P

15. annotate in kernel

16. report of looper with dup2()