Edge Case Testing for the Database Professional Vicky Harp

About Me – Vicky Harp  Product Manager at Idera  Community Manager at Idera   

Let’s talk bugs  All software has failure conditions  Corruption  Tampering  Most software has bugs  Errors in logic  Errors in execution

Let’s talk bugs  Grey areas  Scalability  Platform and Hardware Compatibility  Language and Regional Support  If a customer experiences a problem, it’s no longer a grey area – it’s a bug

The Blame Game  Blame the developer  All development done as sysadmin on a local Developer edition server with no test data  Data access layer design treated as an afterthought  Cowboy code  “Cool tricks” that rely on insecure, unstable, or deprecated functionality  Continuing to use deprecated features  Absurdly low fault tolerance  Overuse of ad hoc SQL

The Blame Game  Blame the DBA  No access to reasonable development environments  No availability of QA environments  QA environments used as semi-production  Uneducated use of obscure configuration options, trace flags, and data topology  Database compatibility modes  Failure to maintain server as a stable code platform  Mismatched tempdb collation  Triggers in msdb

The Blame Game  Blame the designer  Preposterous object names  Trailing spaces  Special characters  Over/Under-normalization  User objects in system databases  Underdocumentation

The Blame Game  Blame management  No time allotted for QA  Bugs are expected to be fixed too quickly  Insufficient personnel  Insufficient hardware  Unreasonable expectations

Setting Boundaries  Define expectations for what is and is not supported in your application  Document, communicate, enforce, and maintain these boundaries  Example: SQL 2005 SP2 through SQL 2008 R2 on case insensitive, US-regionalized English language instances

Enter QA  All software, internal or external, application or database, needs QA  Professionals test their code  Not testing your code is unprofessional

Creating Test Cases  Most testing done in a small environment is ad hoc  Did this change work?  More robust testing relies on test cases and use cases

Use Cases  Expected behavior for the application  Example:  When User clicks the Cancel button:  The window will close  No data will be changed  When User clicks the OK button:  The window will close  Data will be saved  If there is an exception, retry twice before returning an error message

Test Cases  Test definition with expected output  Example:  Precursors: Product is running on a supported platform but the SQL Server is offline  User clicks Cancel  Expected: Window closes  User clicks OK  Expected: Exception returned to user

Using Test Cases  Almost no limit to how highly documented test cases can be  If you’re starting from 0, a good start is to start making checklists  Things to check before accepting a build  Things to check after failing over a cluster

Test Plans  A collection of test cases is a test plan  The best case scenario is to write the test plan before the feature  Realistically this often needs to be done with an application in-situ

Types of Test Cases  Main Success Cases  Edge Cases  Corner Cases  Others? It depends

Main Success Cases  What most people think of when they think of testing  Test case that supports a use case  Example:  Use Case: User runs sales forecasting report  Main Success Case: Report returns correct data with no error messages within 30 seconds

Edge Cases  Test cases which explore the boundaries of an application  The boundary may be based on the use case or based on technology  Based on use case: User clicks all the buttons on a view  Based on technology: What is the behavior of the application on SQL 2000?

Edge Cases  Ideally edge cases should be identified at design time  This is part of good programming practice, regardless of language  DBAs should be aware of where their configuration is introducing new edge cases that may not have been accounted for

Edge Cases  Most common edge for databases is datatypes  Name field backed by an nvarchar(100) column  Edge case: NULL  Edge case: 0-length string  Edge case: 100 character Unicode string  Primary key on a smallint column  Edge case: key seeded at 0 and 32,768 rows inserted  Edge case: key seeded at -32,768 and rows inserted

Corner Cases  The intersection of two cases  Not necessarily the intersection of two edges  Example:  Mail merge – inserting a customer’s name into a message, then saving the text to a column  Corner cases  Special character in either name or message, plus  Extremely long string for either name or message

Corner Cases  SQL Server is especially prone to corner cases  Shared resources are a common source of corner cases  Shared disks  Shared tempdb  Shared memory  Be aware that configuration decisions can introduce corner cases that a developer cannot possibly anticipate

Common Edge Cases  Data Types  Language and Regionalization  Date/Time Considerations  Performance  Usability/Maintainability  Security  Integration  Recoverability  Compatibility

Data Types  Column and variable types  int, bigint, nvarchar, char, xml, etc.  Most basic of all test cases, so no excuse not to test this

Data Types  Max and min values  Collation and sort order  Special characters  NULL values

Data Type Overflows  Overflow during aggregation  select sum(intcolumn)  select sum(cast(intcolumn as bigint))  Overflow or truncation with isnull  Use same data types or coalesce()

DEMO

Data Types  Let the engine help you out – use table constraints!  Name your constraints such that any error message returned will help you understand the problem

Language/Regionalization  Behavior when using different languages and regional settings  Particularly affects numeric and date fields, may also cause other issues

Afrikaans Albanian Amharic Armenian Arabic Assamese Azeri (Latin) Bangla (Bangladesh) Basque Bengali (India) Bosnian (Cyrillic) Bosnian (Latin) Bulgarian Catalan Chinese (Simplified) Chinese (Traditional) Croatian Czech Danish Dari English Estonian Finnish Filipino French Galician Georgian German Greek Gujarati Hausa (Latin) Hebrew Hindi Hungarian Icelandic Igbo Indonesian Inuktitut (Latin) Irish isiXhosa isiZulu Italian Japanese Kannada Kazakh Khmer KiSwahili Konkani Korean Kyrgyz Lao Latvian Lithuanian Luxembourgish Malay (Brunei Darussalam) Malay (Malaysia) Malayalam Maltese Maori Marathi Mongolian (Cyrillic) Nepali Norwegian (Bokmål) Norwegian (Nynorsk) Oriya Persian Polish Portuguese (Brazil) Portuguese (Portugal) Punjabi Quechua Romanian Serbian (Cyrillic) Serbian (Latin) Sesotho sa Leboa Setswana (South Africa) Sinhala Slovak Slovenian Spanish Swedish Tamil Tatar Telugu Thai Turkish Turkmen Ukrainian Urdu Uzbek (Latin) Vietnamese Ting Vit Welsh Yoruba Language/Regionalization

DEMO

Language/Regionalization  Make your code language and region agnostic  :00:00 AM  1000 instead of 1,000  Use “set language”  set language us_english  As a DBA, seriously consider the region settings for your SQL Server  Mismatch between SQL and Windows can cause heartache

Collation Conflicts  Using different collations on the same database or on the same server can cause conflicts  Use a COLLATE statement when comparing two strings  Rule of thumb: if you are going to normalize case with LOWER or UPPER, use COLLATE  Never assume rows will come back in a specified order

Date/Time Considerations  Inaccuracies in scheduling and in date arithmetic  Can become a real nightmare on a WAN

Date/Time Considerations  Daylight savings time  When DST ends, the clock goes from 1:59 AM - 1:00 AM  SQL Agent will pause for an hour  February and Leap Day  Day-of-year calculations after the 60 th day

Date/Time Considerations  Timezones  Not all timezones are on even hour boundaries so you need to be able to account for 30 and 15 minute differences  Date Arithmetic  Region settings can affect functions like datepart()  Date math functions can and do overflow. It takes less than 25 days in milliseconds.

DEMO

Performance  This is a very large topic  DBAs sometimes take this more seriously than developers  Making your development environment look and behave more like production goes a long way toward identifying these cases  Unless it is a single-user application, test with multiple users

Usability/Maintainability  How hard is it to track down bugs?  Are error messages helpful?  Classic problems:  Huge stored procedures  TSQL embedded in other application code  Obfuscated table names  Encrypted stored procedures

Security  Doing all dev work with an administrator login is not realistic  Default databases for logins  Compliance issues  xp_cmdshell

Integration  Does this application play well with others?  Do two projects that live together in prod have separate test environments?  Does maintenance overlap?

Recoverability  Are you backing up your data? Can it be restored?  Is your application mirror or cluster aware?

Compatibility  Compatibility with SQL Server and Windows versions  Non-default configurations  Database compatibility modes  The master database will be set at 80 compatibility on a 2000 instance that has been upgraded to 2005  DMF functions, COLLATE statements, and JOIN statements are all subject to failure in certain compatibility modes  Know what you support and be sure those collations exist in your test environment

How To Test?  Dev and Test Environments  Ad Hoc testing  Automated testing  Bug tracking

Dev and Test Environments  You should have a dev and/or test environment  This environment should be more challenging than production, not less  Having a difficult dev environment means that code is more likely to perform in prod

Dev and Test Environments  Example  Korean language and regionalization  case sensitive collation  large number of databases  many agent jobs  long database and object names with special characters

Ad Hoc Testing  Set aside time to “poke around” in code  Enter long strings into fields  Input negative numbers  Change the clock and see what happens on Leap Day  Set high seed values for primary keys and change default values for tables

Automated Testing  Can be as simple as SQL Agent jobs  Developers can add test cases to their product  Many tools  If you do not have source code, just profile  Build up a list of test cases that need to be run before going to production

Bug Tracking  Post it notes won’t cut it  Start with a spreadsheet  Keep track of how long bugs take to fix  Try to prevent bug fixes from taking over your schedule

Wrap-Up  Define your application boundaries  Define use and test cases  Test your software  Refine, repeat, improve

Discussion

More Information  Contact me:  Forums: community.idera.com    Slides will be available at