1. CS4723 Software Validation and Quality Assurance Lecture 7 Non-Functional Testing

2. 2  Performance Testing  Test whether the efficiency (time and space) of a software meets requirements  Security Testing  Test whether the software is vulnerable to attacks (special invalid inputs designed to control the software or reveal info from the software)

3. 3 Performance Testing  Load Testing  Soak Testing  Stress Testing  Spike Testing

4. 4 Input loads  Size of inputs  An extremely long SQL query  An extremely large html file for a browser  Number of inputs provided  Number of students supported in a school management system  Number of web pages opened in a browser  Frequency of inputs provided  Number of SQL queries made per second  Number of Http requests made per second

5. 5 Performance Measures  Input Lag  Response Time  Throughput

6. 6 Performance Measures

7. 7 Load Testing  Provide input under the maximal designed load of software and observe behavior  Purpose:  See whether the software works normally  Find potential bottlenecks of performance: Profiling  Instrument each major component (e.g., method) to see how much time / memory is spent on it  Sampling is sometimes used to reduce overhead

8. 8 Load Testing  Test steps  Determine the content of inputs  Usually can be a large amount of identical or similar inputs  The input can be simple or very complex (to check the performance of software when handling complex input)  Determine the frequency of input feeding  Determine how long the input feeding lasts time Design Load Input Load

9. 9 Demo  Usage of JMeter to perform load testing for web application and databases

10. 10 Input feeding  A multi thread program to feed inputs randomly in a given period of time  Sometimes require multiple machines to feed inputs  Usually only consider valid inputs

11. 11 Stress Testing  Provide input OVER the maximal designed load of software and observe behavior  The software is expected to fail  Purpose:  Observe when (how much load) the software is going to fail  Observe the how the failure looks like: crash? CPU or memory used up? Can be recovered or not?  Observe whether the system can partially work when failure happens

12. 12 Stress Testing  Illustration time Design Load Input Load

13. 13 Soak Testing  Provide heavy input load (slightly under designed maximal load) for a long time  Purpose:  Testing for how long time the software can work normally under heavy input load  Usually memory and disk oriented  Observe the memory / disk usage trend (abnormal increase in the usage)

14. 14 Soak Testing  Illustration time Design Load Input Load

15. 15 Spike Testing  Provide extremely heavy input load (OVER designed maximal load) for a very short time  Purpose:  Test how the software can handle a input load burst  Probable expected behavior:  Temporarily refuse inputs that cannot be handled  Provide some temporary services for the inputs to wait until the burst ends

16. 16 Spike Testing  Illustration time Design Load Input Load

17. 17 Performance Diagnosis  Find out why performance problems happen  Figure out how to optimize software to achieve higher performance

18. 18 Profilers

19. 19 Memory Profilers

20. 20 Performance Testing: Review  Load Testing  Stress Testing  Soak Testing  Spike Testing

21. 21 Tools for Test Measurement  EclEmma  Update site: http://update.eclemma.org/http://update.eclemma.org/  Install in eclipse  Setup coverage configuration: do not check the test folder  Run test using Emma coverage as…  Read test coverage  Enhance test coverage

22. 22 Tools for Test Measurement  VisualVM  http://visualvm.java.net http://visualvm.java.net  Start VisualVM  Start a Java software  Open the tab for the process  Do profiler for memory and computation

23. 23 Security Testing  Major security concerns  Vulnerabilities  Penetration Testing

24. 24 Major Security Concerns  Undermine usability  DOS attacks  Peculiar inputs causing crashes, bloats, …  Information Leaking  SQL Injection, Cross-site Scripting, unencrypted data, side channels, …  Command and Control  OS Injection, Cross-site Scripting, Return Oriented Programming, …

25. 25 Vulnerabilities  Avoid common vulnerabilities  Buffer Overflow  Injection  Cross-Site Scripting

26. 26 Buffer Overflow  Quite many languages (C, C++) are memory unsafe  You define a buffer, and it is your responsibility to keep your data in the buffer  If you read or write to the place out of a buffer  Semantic errors  Crashes  What else? Anything related to security? char buffer[12];

27. Review of OS course: call stacks  Function calls are traced by call stacks main f f f g int main(int argc, char args**){ int result; if(argc >= 1){f(args[0]);} } void f(char* data){ char buffer[12]; strcpy(buffer, data) if(g()){return;} else{…} } bool g(){... }

28. Call stack of the function f  The local variable buffer  The parameter data  The return address to go back to the call-site at main function char[12] buffer

29. Feed in a valid input  Example “username” char[12] buffer

30. Feed in an invalid input  Example “usernameusername”  The parameter data is covered  So it is no longer usable  The return value is covered  So can not return normally  Still just a bug  Minor security problem  Undermines usability char[12] buffer

31. Feed in a malicious input  Idea to do the trick  Feed in an input with 20 chars  Cover the return address  f() will return to the code we Specify  Consider the program is on a server, accessing user requests  How to make it possible?  Where to put the code?  How to specify the return value to our code? char[12] buffer

32. Feed in a malicious input  Use the buffer itself to store the code  Set the return value to the buffer address  Example  Run exec(“/bin/sh”) to open a shell  Translate to machine code char[12] buffer mov $a0 15 mov $a1 data syscall data: /, b, i, n, /, s, h 0x20, 0x42, 0x00,...

33. Feed in a malicious input  Other issues  How to know the address of buffer[]: Programs are executed in virtual memory, so install the software and check memory state  Buffer is too small to hold your code? Jump through return value to the stack frame of parent function char[12] buffer

34. 34 The state of practice  Buffer overflow is very common in C / C++ programs  About 50% of new attacks are related to buffer overflow  Known bugs are being exploited from time to time

35. 35 How to deal with buffer overflow  Boundary check for input-reachable buffers  Not so easy in practice  Check too many places: slow the software down  Check too few places: buffer overflow risk  Automatic supports  Buffer Overflow Detection: libsafe, stackguard, …  Runtime protection: weak memory safe  Runtime protection: split stack

36. 36 A real-world example  If you are interested  Here is a real world example:  https://www.rcesecurity.com/2011/11/buffer -overflow-a-real-world-example/

37. 37 Injection  Directly inject user input into code to be executed  SQL Injection  Inject code to SQL queries  OS Injection  Inject code to OS commands

38. 38 SQL Injection  An example  A student information system  You can query your grade for certain course, year, …  You login to your session, and say you are going to search for the grade of “CS4723”  What does the server do?

39. 39 SQL Injection  The malicious Input  We want to inject code into the SQL query  Say we want it to be “select * from Grade”  It is the same with “select * from Grade where username = ‘you’ and course = ‘CS4723’ or ‘a’ = ‘a’”

40. 40 OS Injection  Quite Similar  Consider a server is going to make a dir for you as a new user, and it will execute exec(“mkdir path/to/” + username)  What username you should make up? An example: HahaGotyou | \bin\sh

41. 41 Injection Protection  Injection works by passing user inputs into back- end engines  Can we simply cut off the path?  Definitely NO  We have to do some filtering  We are going to work on the example: select * from Grade where username = ‘you’ and course = ‘CS4723’/**/or/**/‘a’=‘a’

42. 42 User Input Filtering  What to filter?  or ? => “oorr” can bypass it  Space? => use /**/ can bypass it  Quotes? A little bit difficult, we can search by year, and use year = 2009 or 1=1  Want more?  See select * from Grade where username = ‘you’ and course = ‘CS4723’ or ‘a’ = ‘a’ http://websec.wordpress.com/2010/03/19/exploiting-hard-filtered-sql-injections/

43. 43 User Input Filtering: Other Issues  Functioning of the software  Filter ‘or’ will affect username ‘George’  Cannot filter space when space is allowed for the field  Other string manipulations  In web applications, there are HTML/URL escape characters  &quot for “, &#39 for ‘, &nbsp / %20 for space, …  Replacing escaping characters are common  So &#39 may be used if quotes are disabled…  Always be clear about how user inputs flows in your code

44. 44 Cross Site Scripting  One of the most popular attacks to web applications  Everything is about where the input goes to  This time it goes to a web page  This becomes more popular with so-called web 2.0 (let users do the work, e.g., wiki, youtube, blogs)

45. 45 XSS: Scenario

46. 46 Example  Bob wants to get all the login information of his friends in a social network  So Bob writes a blog, in the blog, he writes: xxxxxxx, xxxx, email(“bob@gmail.com”, getcookie()) bob@gmail.com  Now Mary reads the blog, so the script runs, Bob will get the cookie, and will be able to login with Mary’s cookie…

47. 47 Protection against XSS  Limit the usage of cookies: may result in much inconvenience  Quite similar to SQL Injection  Try to filter dangerous things such as “ ” from user’s input  Also quite similar to SQL Injection  There are a lot of ways to bypass the filtering, so always hard to do a correct filtering  Even harder because HTML is more complex than SQL, and much more web page generations than database query points…

48. 48 Core idea: Devil inputs  Software Security is almost all about the malicious inputs  Buffer Overflow, Injection, and XSS accounts for 70% to 80% of attacks each year…  Also for DOS (Denial of Service) attacks  An example: you may want to filter ‘\’ for security reasons, but if so, handling a input like ‘\\\....\\\\’ with n ‘\’s will take n 2 CPU time…  Consider all possible values of user inputs  Never make assumptions to user inputs

49. 49 Penetration Testing  Random testing (or fuzzing) is often useful for security testing, because it can generate inputs that you cannot imagine  Have security checks during the testing  Buffer Overflow: whether any “out of boundary” happens  Use boundary checker in testing, and disable them in distribution  SQL Injection & XSS: whether user inputs reach syntax tree part of the HTML or SQL code  Use taints during testing to track the user inputs along the execution

50. 50 Review of Non-Functional Testing  Performance Testing  Test whether the efficiency (time and space) of a software meets requirements  Security Testing  Test whether the software is vulnerable to attacks (special invalid inputs designed to control the software or reveal info from the software)