1. Testing ColdFusion Applications
John Paul Ashenfelter
CTO/Transitionpoint
Test-driven development is very popular in the Java and Ruby worlds and becoming moreso in the ColdFusion world. This session covers the tools that are available to test ColdFusion applications and discusses how to implement them into your existing workflow. We'll specifically cover:
* CFCUnit/CFUnit for testing ColdFusion code
* Selenium for testing web pages
* DBUnit for managing the database during testing
* Grinder (and similar) load-testing tools
Together, these tools can provide unit, functional, regression, and load tests for your applications. Finally, we'll touch on automating these tests so you can ensure that you are delivering higher-quality, well-tested code.
CFUNITED – The premier ColdFusion conference

2. What is Testing? Verifying the software works correctly
Validating that the reliability and stability of the software
Ensuring that the software solves the problem it is supposed to solve
Making sure the user can use the software
Unit testing the most 'micro' scale of testing; to test particular functions or code modules. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. Not always easily done unless the application has a well-designed architecture with tight code; may require developing test driver modules or test harnesses.
incremental integration testing Continuous testing of an application as new functionality is added; requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed, or that test drivers be developed as needed; done by programmers or by testers.
Integration testing Testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems.
functional testing Black-box type testing geared to functional requirements of an application; this type of testing should be done by testers. This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing.)
sanity testing or smoke testing Typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing systems every 5 minutes, bogging down systems to a crawl, or corrupting databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state
System testing System testing is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of Black box testing, and as such, should require no knowledge of the inner design of the code or logic. It takes, as its input, all of the "integrated" software components that have successfully passed Integration testing and also the software system itself integrated with any applicable hardware system(s). The purpose of Integration testing is to detect any inconsistencies between the software units that are integrated together called assemblages or between any of the assemblages and hardware. System testing is more of a limiting type of testing, where it seeks to detect both defects within the "inter-assemblages" and also the system as a whole.
Regression testing Re-testing after fixes or modifications of the software or its environment. It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Automated testing tools can be especially useful for this type of testing.
Load testing Is the practice of modeling the expected usage of a software program by simulating multiple users accessing the program's services concurrently.
Performance testing Measures how fast some aspect of a system performs under a particular workload. It can serve different purposes. It can demonstrate that the system meets performance criteria. It can compare two systems to find which performs better. Or it can measure what parts of the system or workload cause the system to perform badly
Stress testing Is a subset of load testing. Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results.
recovery testing Testing how well a system recovers from crashes, hardware failures, or other catastrophic problems.
failover testing Typically used interchangeably with 'recovery testing'
Security testing Testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques.
Model-based testing
install/uninstall testing Will frequently occur on the computer system the software product will eventually be installed on.
compatability testingtesting how well software performs in a particular hardware/software/operating system/network/etc. environment.
exploratory testing Often taken to mean a creative, informal software test that is not based on formal test plans or test cases; testers may be learning the software as they test it.
ad-hoc testing Similar to exploratory testing, but often taken to mean that the testers have significant understanding of the software before testing it.
context-driven testingtesting driven by an understanding of the environment, culture, and intended use of software. For example, the testing approach for life-critical medical equipment software would be completely different than that for a low-cost computer game.
mutation testing A method for determining if a set of test data or test cases is useful, by deliberately introducing various code changes ('bugs') and retesting with the original test data/cases to determine if the 'bugs' are detected. Proper implementation requires large computational resources.
PLUS
Usability testing Testing for 'user-friendliness'. Clearly this is subjective, and will depend on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers
Stability testing
Authorization testing
User acceptance testing is a process to obtain confirmation by an SME, preferably the owner of the object under test, through trial or review, that the modified/new object under test meets mutually agreed-upon requirements.
Conformance testing is testing to determine whether a system meets some specified standard.
Portability testing is testing the ease with which an application or component can be moved from one environment (e.g., hardware, operating system, and browser) to another.
Playtest
June 28th – July 1st 2006

3. Why Test?
There is an old saying with software that three years from now, no one will remember if you shipped an awesome software release a few months late. What customers will still remember three years from now is if you shipped a software release that wasn’t ready a few months too soon. It takes multiple product releases to change people’s quality perception about one bad release.
-Scott Guthrie
Scott Guthrie runs the development teams that build IIS, ASP.NET, and Visual Web Developer 2005
June 28th – July 1st 2006

4. Who Should Do the Testing?
Good Testers
Developers
QA Staff
Help Desk
Internal Staff
Bad Testers
Users
Customers
The public
June 28th – July 1st 2006

5. When Should I Test? When you: write code change code fix a bug
integrate code
deploy code
think of a new edge case
take a coffee break…
Art of Software Testing was written by Glenford J. Myers in 1979, a seminal work on testing
We try to solve the problem by rushing through the design process so that enough time is left at the end of the project to uncover the errors that were made because we rushed through the design process. -Glenford J. Myers, "Code Complete: A Practical Handbook of Software Construction" by Steve C McConnell
June 28th – July 1st 2006

6. How Should I Test? Code: Unit testing Application: Functional testing
System: Load/performance testing
And if possible, user acceptance testing
This should be the standard for *all* web applications. We’ll discuss the first three in detail
June 28th – July 1st 2006

7. Testing Your Code
Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. -Brian Kernighan
Coinvented C, involved in many Unix apps (is the K in awk)
June 28th – July 1st 2006

8. Unit Testing In A Nutshell
A developer writes a unit test to verify an extremely specific area (unit) of functionality (code) behaves as they expect
One unit test is not sufficient!
Good data
Bad data
Edge cases
Note that this is a developer activity. This is a form of “white box” testing which requires access to the source code.
June 28th – July 1st 2006

9. Classic Unit Test Example
Problem: return the largest value in an array of numbers
Scenarios to test:
normal sets of values (the “good” case)
duplicates
negative numbers
empty array
single value
This is from Pragmatic Unit Testing (Hunt/Thomas)
June 28th – July 1st 2006

10. Unit Test Code Example We’ll use one of Sean’s GGCC CFCs
ggcc3 to be specific
We’ll use CFCUnit for the testing
have I finished the Ant task yet?
June 28th – July 1st 2006

11. What’s Next for Unit Tests
Test cases become test suites
Test suites can be automated
Ant
Commit hooks in SCC (eg SVN)
Continuous integration (eg CruiseControl)
Quality of code should improve
June 28th – July 1st 2006

12. Testing the Application
The trouble with programmers is that you can never tell what a programmer is doing until it's too late. -Seymour Cray
Yes, that Cray.
June 28th – July 1st 2006

13. Functional Testing in a Nutshell
Functional tests verify the application works as expected from a user’s point of view
Written by developers and/or QA staff from test cases
This is “black-box” testing – no internal knowledge of code
June 28th – July 1st 2006

14. Test Cases Single step (or small series of steps) with single outcome
precondition (known input)
postcondition (expected output)
Need at least one test per requirement/use case
June 28th – July 1st 2006

15. Functional Test Code Example
We’ll use Selenium
We’ll reset the tests with DBUnit
Example is basic login process
June 28th – July 1st 2006

16. Scenario testing Based on a story Any failure would motivate the fix
Is credible in real-world usage
Involves a non-trivial task
The results are easy to evaluate
June 28th – July 1st 2006

17. Testing the System
Premature optimization is the root of all evil in programming. -C. A. R. Hoare
Turing Award Winner (ACM) for Hoare logic, among other things. Also invented the Quicksort algorithm
June 28th – July 1st 2006

18. Load Testing in a Nutshell
Measures the performance of the application with multiple concurrent “users”
Flavors of load testing:
Load testing expected usage patterns
Performance testing known workloads
Stress testing to the breaking point
Load testing Is the practice of modeling the expected usage of a software program by simulating multiple users accessing the program's services concurrently.
Performance testing Measures how fast some aspect of a system performs under a particular workload. It can serve different purposes. It can demonstrate that the system meets performance criteria. It can compare two systems to find which performs better. Or it can measure what parts of the system or workload cause the system to perform badly
Stress testing Is a subset of load testing. Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results.
June 28th – July 1st 2006

19. Testing Environment
Load testing is WYTIWYM (what you test is what you measure)
identical hardware
identical software
identical configuration
Results rarely (i.e. never) scale as expected to different hardware or software configurations
Load testing is complex. For example, one former employer had problems with web server integration (particularly slow response from one of three partners). Another employer found that code performing admirably on a JRun server slowed significantly when clustered (apparently due to overhead from session replication)
Always need to verify that you measuring what you *think* you’re measuring – so must monitor lots of things, including network throughput at the server and switch/firewall/router, etc.
June 28th – July 1st 2006

20. Load Testing Code Example
We’re using Grinder
Java based
Scripted in Jython (Java-based Python)
Can simulate multiple servers with VMWare, but beware results!
June 28th – July 1st 2006

21. User Testing
Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning. -Rich Cook
Fantasy author
June 28th – July 1st 2006

22. Other Kinds of Testing Regression Security
Validity/conformance/compliance
June 28th – July 1st 2006

23. Final Thoughts
There has never been an unexpectedly short debugging period in the history of computers. -Steven Levy
Author, Senior tech editor for Newsweek. Found Einstein’s brain…
June 28th – July 1st 2006

24. More Final Thoughts Testing IS your job
Testing needs to be integrated into the entire development process
Tests build on one another into comprehensive suite(s)
Automated testing is something computers are really good at, so take advantage of that!
June 28th – July 1st 2006

25. Resources
The average software developer, for example, doesn't own a single book on the subject of his or her work, and hasn't ever read one. That fact is horrifying for anyone concerned about the quality of work in the field, for folks like us who write books, it is positively tragic.
-Tom DeMarco and Timothy Lister , "Peopleware : Productive Projects and Teams, 2nd Ed."
June 28th – July 1st 2006

26. Booklist The Pragmatic Programmer, Hunt/Thomas
Code Complete, Steve McConnell
Agile Software Development, Robert Martin
Writing Effective Test Cases, Alistair Cockburn
June 28th – July 1st 2006

27. Software CFUnit CFCUnit Selenium Grinder Subversion Trac Ant Eclipse
June 28th – July 1st 2006

28. Questions/Thanks!!! John Paul Ashenfelter
June 28th – July 1st 2006