Trnsport Test Suite Project Tony Compton, Texas DOT Charles Engelke, Info Tech

Topics Charlie will provide overview –Genesis of project –Strategic plans for testing Tony will give the details –Just completed Phase 1 –Phase 2 underway –Next steps

What is the Trnsport Test Suite? The elements needed to perform thorough and affordable testing of all of Trnsport Plansa uniform, integrated test plan for all Trnsport modules (today there are many separate test plans) Dataa single comprehensive test database covering the entire Trnsport project lifecycle Toolsthat enable automated testing both during development and at agency sites

Project Genesis – the 2001 Strategic Directions Presentation Theme: Focus on Processes –Process changes are key to support Trnsport’s growing scope without breaking the bank First Target: Testing –Traditionally, each Trnsport component had developed its own test database and procedures –We now need a single unified test database and methodology –The benefits of this are very large, and surprising broad

Original Goals Develop and package a standard test environment –Database(s), templates and interface files –Tools to reset all to initial states Create automated regression tests –GUI is hardest, so do it last Add testing to the installation procedures Use testing environment for all regular builds during development Provide tools for agency acceptance testing

Record/Replay Test Method Prior testing was mostly task driven –Task analyzed, script written, user follows script –Expensive to design and perform Recording actual use is easier –Looks at sessions, not tasks –User starts the application, performs many typical tasks, recorded by new tools Replaying is easier still –Tool replays every action in the session, compares results to recorded responses

Some Potential Applications Database platform validation –Use proxy to record database traffic –Replay proxy recording to new database Continual regression testing –Maintain a large library of recordings –After every new build, replay entire library Agency acceptance tests –Replay recordings made by Info Tech at your own site with a delivered test database –Make your own recordings under old release, replay against new installation

Start with the Servers Windows programs are hard to record and replay –The user can “talk” to the program in lots of ways But servers are generally driven by a single connection point to the client –ODBC for databases, HTTP for all others This is a well defined target for recording –And proxies, a common tool for other purposes, can record as they proxy

2001 Recommendation Develop phased work plan –Create basic test database, develop test scripts, create ODBC proxy and recorder, create HTTP proxy and recorder, develop automated build and test environment, package end-user test tools, expand test database and scripts Proceed as aggressively as possible Benefits will start to accrue very early

Motivation – Improved Quality Much more extensive regression testing can be performed much more often than would otherwise be practical Automation ensures repeatability of procedures, documentation of results Results are immediately visible to all interested parties –Potentially including licensees via web –Likely to improve unit testing before code check- in

Be Optimistic, but Also Realistic Test database requires deep analysis GUI automation requires a proxy for a human being – hard to do! Test data and scripts require careful maintenance Source management procedures must be uniform to automate Response to replays can have many minor variations, so test logs require human interpretation

Some Things The Test Suite Won’t Do Run standard tests against agency data –Recordings can only be replayed against the exact same database to be meaningful Migrate recordings to new releases –Some recordings will remain valid in new releases, some won’t. New recordings would have to be made in those cases Always give a “Pass” or “Fail” on a test –Unless replay is identical in every way to the original, a person will have to examine deviations for significance

Progress on the Test Suite Components Phases Processes Lessons Learned

Trnsport Test Suite Components Test Plan –Test processes –Test cases and Test Case Repository Test Database –Spanning all platforms, all modules Test Automation Framework –Automated execution of regression tests –Learning and replaying tests

Multiple Phases Needed Phase 1 – “Low Hanging Fruit” –General framework –Automated batch testing –HTTP proxy record and playback Phase 2 – Expand Scope –More of everything in Phase 1 –Investigate ODBC and GUI issues Phase 3 – Complete Toolset –Covers all planned areas –Not “finished”, always maintain and expand

Phase 1 October 2002 to June 2003 Focused scope –Covers only PES and LAS –One contract, five functions per module –Oracle 8i on Windows Results –Test plan and database –Testing framework Automated batch testing HTTP record and playback tools –Accepted by TTF

Phase 2 April 2003 to December 2004 Broaden scope –Finish PES/LAS test plans –Cover all other client-server modules –20 contracts, 10 functions per module –Some high volume test cases Cover other databases Results –ODBC proxy prototype –Agency testing tools (verify behavior matches release tests at Info Tech)

Phase 3 All of Trnsport –Add Estimator, SitePad, etc. –More test data and volume testing Results –Test database representative of three years of production use –ODBC and GUI testing –Binary data covered –More Agency testing tools (record/playback)

TRT and ITI Process Several teleconferences per phase Regular with attachments Multiple iterations of draft deliverable documents for review and approval ITI’s workplan, processes and methodologies also reviewed by TRT Thorough and detailed documentation delivered Learning experience

Trnsport Test Suite Project Tony Compton, Texas DOT Charles Engelke, Info Tech