Integer Overflows James Walden Northern Kentucky University

CSC 666: Secure Software Engineering Topics 1.Computer Integers 2.Integers in C and Java 3.Overflow Examples 4.Checking for Overflows

CSC 666: Secure Software Engineering Comair Integer Overflow December 25, 2004 Flight crew scheduling software stopped. Cancelled all 1100 flights that day. What happened? Winter weather led to many crew changes. Number of changes > 32,767.

CSC 666: Secure Software Engineering Integers Computer integers are not the same set of numbers as mathematical integers.  Finite set, not infinite.  What happens when integer calculations result in a number outside that set?

CSC 666: Secure Software Engineering Unsigned Integers

CSC 666: Secure Software Engineering Two’s Complement Two’s complement = One’s complement + 1. Sign is represented by most significant bit. Range: -2 n-1..2 n-1 -1, only one representation of Comp

CSC 666: Secure Software Engineering Two’s Complement

CSC 666: Secure Software Engineering C Integers TypeBitsMin ValueMax Value signed char unsigned char80255 short unsigned short int32-2,147,483,6482,147,483,647 unsigned int3204,294,967,295 long32-2,147,483,6482,147,483,647 unsigned long3204,294,967,295 long x x unsigned long long x 10 19

CSC 666: Secure Software Engineering Java Integers TypeBitsMin ValueMax Value byte short char int32-2,147,483,6482,147,483,647 long x x 10 18

CSC 666: Secure Software Engineering Java Factorial Program public static void main(String args[]) { long product = 1; for(int i = 1; i <= 21; i++) { System.out.print(i); System.out.print("! = "); product *= i; System.out.println(product); }

CSC 666: Secure Software Engineering Output 1! = 1 2! = 2 3! = 6 …. 20! = ! =

CSC 666: Secure Software Engineering Java BigInteger Class import java.math.BigInteger; public class BigFactorials { public static void main(String args[]) { BigInteger product = BigInteger.ONE; BigInteger index = BigInteger.ONE; for(int i = 1; i <= 21; i++) { System.out.print(i); System.out.print("! = "); product = product.multiply(index); System.out.println(product); index = index.add(BigInteger.ONE); }

CSC 666: Secure Software Engineering Output 1! = 1 2! = 2 3! = 6 …. 20! = ! =

CSC 666: Secure Software Engineering Problems of Integer Overflows Difficult to detect after they’ve happened. –Compilers generally ignore them. –Assembly code can check carry flag, but high level languages can’t without calling assembly code. Difficult to avoid. –Subtle bugs can result in integer overflows.

CSC 666: Secure Software Engineering Integer Overflows in Voting Broward County 2004 election Amendment 4 vote was reported as tied. Software from ES&S Systems reported a large negative number of votes. Discovery revealed that Amendment 4 had passed by a margin of over 60,000 votes.

CSC 666: Secure Software Engineering TKADV Integer overflows in Amarok media player.  Reads input size + input from file.  Allocates input size + 1 bytes, which can be very small.  Reads file data into very small buffer, leading to a buffer overflow.

CSC 666: Secure Software Engineering Strip Extension void StripExtension(char * filename) { unsigned short int newSize = strlen(filename) - 4; char * buffer = (char *)malloc(newSize + 1); strncpy(buffer, filename, newSize); buffer[newSize] = ‘\0’; printf(“%s”, buffer); free(buffer); }

CSC 666: Secure Software Engineering Valid Use What would happen if StripExtension were called as follows? StripExtension(“a.txt”);

CSC 666: Secure Software Engineering Invalid Use What would happen if StripExtension were called as follows? // User failed to include the extension. StripExtension(“a”);

CSC 666: Secure Software Engineering Answer  newSize = 0xffffd = (1 minus 4) = -3  newSize is an unsigned short integer  This value is  The function creates a byte buffer.

CSC 666: Secure Software Engineering Unsigned Addition An unsigned addition unsigned int x, y, sum; sum = x + y; Precondition if( x > UINT_MAX – y) /* error */ Postcondition if( (x >= 0 && sum y) ) /* error */

CSC 666: Secure Software Engineering Signed Addition Preconditions xyPrecondition Positive if (x > INT_MAX – y) /* error */ PositiveNegativeNone NegativePositiveNone Negative if (x < INT_MIN – y) /* error */

CSC 666: Secure Software Engineering Integer Multiplication Overflow CESA : libpng info_ptr->row_pointers = (png_bytepp)png_malloc(png_ptr, info_ptr- >height * sizeof(png_bytep)); If height > INT_MAX / sizeof(png_bytep) Size of new buffer will be a small integer. User data in image file can be used to generate a buffer overflow attack.

CSC 666: Secure Software Engineering Widening Conversions A conversion from a type with a smaller range of values to type with a larger range of values. Examples: byte -> short, short -> long Sign extension Propagates signed bit from source type to all unused bits in destination type. Magnitude and sign are preserved.

CSC 666: Secure Software Engineering Widening Conversion Example Source type: byte Value: Destination type: short Value:

CSC 666: Secure Software Engineering Narrowing Conversions Conversions from a wider type to a narrower type. Examples: long -> byte, int -> short Truncation Bits from source type that don’t fit into narrower destination type are discarded. Magnitude and sign may change.

CSC 666: Secure Software Engineering Narrowing Conversion Example Source Type: short Value: Destination Type: byte Value:

CSC 666: Secure Software Engineering Sign Extension Vulnerability CERT CA : bash yy_string_get() reads user data as chars. Each char converted to an int when parsed. A char value of 255 sign extended to int -1. Integer -1 means command separator. Example exploit bash -c 'ls\377who'

CSC 666: Secure Software Engineering Range Checking Check that integer ranges are valid. Be more specific than INT_MIN, INT_MAX. Liquid water temperatures range Use type system to check. Some languages allow integer ranges. Create abstract data types in languages that don’t provide integer range types.

CSC 666: Secure Software Engineering Proposal: Ranged Integers in C All integer types can be ranged. Static: range determined at compile time. Dynamic: range determined at run time. Semantics Saturation: values beyond range = max. Wrap: values wrap to bottom of range. Examples Saturation: int 0|..|100 temperature = 0 Wrap: long min max circular;

CSC 666: Secure Software Engineering Compiler Checks Microsoft VS 2005 CL  Runtime integer error checks: /RTCc  Use highest warning level /W4  Check for #pragma warning(disable, C####) GCC  Runtime integer error checks: -ftrapv  Use integer-relevant warnings: -Wconversion –Wsign-compare  Check for #pragma GCC diagnostic ignored option

CSC 666: Secure Software Engineering Secure Integer Libraries IntegerLib –Designed for C, but usable in C++. –Available from CERT. IntSafe –C library written by Michael Howard. –Uses architecture specific inline assembly. SafeInt –C++ template class from David LeBlanc.

CSC 666: Secure Software Engineering SafeInt C++ Class int main(int argc, char *const *argv) { try { SafeInt s1(strlen(argv[1])); SafeInt s2(strlen(argv[2])); char *buff = (char *) malloc(s1 + s2 + 1); strcpy(buff, argv[1]); strcat(buff, argv[2]); } catch(SafeIntException err) { abort(); }

CSC 666: Secure Software Engineering When to use Secure Int Libraries? Use Secure Integer libraries when Integers come from untrusted sources. Don’t use Secure Integer libraries when Integers not influenced by external sources. Tight loops: check int values before loop.

CSC 666: Secure Software Engineering Integer Overflow: Key Points Integer arithmetic. –Two’s complement format signed ints. –Know your language’s integer conversions. Impact of integer overflows –Can be used to defeat bounds checks. –Influence important data, like vote counts. Mitigating integer overflows. –Precondition or postcondition testing. –Use safe integer libraries where possible.

References 1.Brian Chess and Jacob West, Secure Programming with Static Analysis, Addison-Wesley, Jeff Gennari et. al., Ranged Integers for the C Programming Language. CMU/SEI-2007-TN-027, Michael Howard and David LeBlanc, Writing Secure Code, 2 nd edition, Microsoft Press, Robert C. Seacord, Secure Coding in C and C++, Addison- Wesley, Robert C. Seacord, CERT Secure Coding Standards: Integers, +Integers+(INT), Integers+(INT) 6.John Viega and Gary McGraw, Building Secure Software, Addison-Wesley, David Wheeler, Secure Programming for UNIX and Linux HOWTO, Programs-HOWTO/index.html, Programs-HOWTO/index.html