1. Durham Blackboard User’s Conference
9th January 2014
Automated Testing Saving Time & Money Link to the paper:
David Barrett Blayn Parkinson
University of York, UK

2. Functional testing Automated testing is is necessary is time consuming
Why?
Functional testing
is necessary
is time consuming
can be ambiguous
Automated testing is
consistent
easily repeatable
reusable
David: Why consider automated testing?
- We had looked at automated testing a number of years ago, but were not that convinced it would add much value. Tests didn’t seem to run very well on Blackboard and the test playback was often hindered by the frameset.
- We decided to have another look at this, and the reasons behind that were:
- Functional testing - is necessary (we do not deploy software changes without testing them first).
- is time consuming, to test in different combinations of browser and OS takes a long time
- can be ambiguous: human testers, when asked to perform a test can go about the same test in different ways. This could result in a problem being undetected.
- What automated testing offers - it is consistent, it always performs the same test
- it is easily repeatable, you can keep on performing the same tests over and over again
- tests can be reused across different browsers and platforms

3. How? Speed Command Target (Locator) Tests Value David:
There are many testing frameworks around, many of which integrate with an IDE, however we wanted a simple IDE to create tests in and this is what is offered by the Firefox Selenium IDE plugin.
ThoughtWorks Selenium is the testing framework most web browser automated testing products are based upon and the IDE has good export facilities should we want to export a test script out to run in Junit, or Nunit, or Ruby.
The plugin ide is simple. Tests cases are lists here. Each test is run line by line and is made up of a command (open, click, type), a target (html element), and if appropriate an input value. You can also set the speed at which the test should run. You don’t want it to run too quickly, otherwise the page may not have loaded completely and the test could fail.
Target (Locator)
Tests
Value

4. Organising tests
David: Once we had selected Selenium as our automated testing framework of choice we then had to consider how tests would be stored, understandable and how results would be recorded.
Pre automated testing tests were stored in Google spreadsheets, and a different set of spreadsheets for each test run. Selenium tests are stored in separate files and it’s not always obvious what a test is doing by looking at the code.
So we had a challenge in neatly matching up the file Selenium creates and our more formal way of organising tests.

5. Test Manager
David
The solution was to create a database to store test scripts and results in that we named ‘Test Manager’. The system has a no frills GUI via which we can commission tests, stored test scripts and access results.
Even if you weren’t going to automate any test scripts this isn’t a bad way of storing tests, results, etc.
Explain what test scripts, test cases are – which form test runs….
There is a web service that is available to the tests which interprets results and updates the system accordingly.
This part of the process doesn’t use Selenium itself, it’s just our way of organising the test scripts created using Selenium and making sense of the results.

6. Constructing good tests
A good test
Mirror user behaviour
Independent
Report accurately
HTML present?
Text present?
Screenshot
Blayn:
Writing a good automated test.
A good test should be realistic and mirror user behavior – mirror what a user would do and expect to interact with. For example, if you were testing a specific tool that would normally be reached by navigating the left hand menu, then the test should navigate the left hand menu too, not just go to the page containing the tool you actually want to test.
Independent of other tests. If a test fails for whatever reason you don’t want this to impact all the other tests in the suite and prevent them from executing. Where possible tests should be self contained and not dependent on the success of other tests to run.
Report accurately. So careful planning about how the test will determine success is important. Using multiple verification commands is best with the test script analysing the page for expected html or text, or taking a screen shot for inspection later

7. Source Control
TEST SCRIPT (A1.3)
ADD BLOG POST (ROLE STUDENT)
BLOCK 1
Login as test student
BLOCK 2
Navigate to the test course site
BLOCK 3
Navigate the left hand menu to access the blog tool
NEW SCRIPT
Enter the blog and click the Add New Entry button
BLOCK 4
Add content to the content editor
BLOCK 5
Click the submit button
BLOCK 6
Take a screenshot for human inspection/verification
Verify text on the page is present
BLOCK 7
Logout of the VLE
Blayn:
When developing scripts we noticed that there were some actions that cropped up in nearly every test script. For example: logging in, logging out, navigating the left hand menu, or using the Content Editor. To save having to recode these every time a test script was developed we created BLOCKS of reusable code. This speeded up the development of tests scripts.
The example on screen is a test scripts for testing the blog tool, and adds a new post to the blog with the student role. As you can see when this script was developed it only required 2 new set of actions, ‘entering the blog and clicking the add new entry button’, and ‘verifying the success of the text by searching for text on the page’. The rest of the test script is made up of blocks of code that have already been written.

8. Test Results Web Service
Test Manager Results
Blayn:
This diagram illustrates how tests are executed and results returned. Selenium will execute a suite of test scripts. Each test will run one at a time, line by line. During the test if any of the verification tests fail the test will fail. If however the tool or feature under test is working then the test will get to the last line of the test which contacts the Test Results Web Service and reports back. When contacting the web service the test will pass a result and it’s ID. The Test Manager then imports incoming results against a results set.
Execute
Test Suite
Test Script ID
Pass
Test Results Web Service
Pass pending check
Fail

9. Pros and Cons Saved time Maintenance burden Reusable
Human intervention
David:
To start wrapping up, these are the pros and cons and we see them. Automated testing has saved time and allowed staff who would be performing functional testing, to spend that time looking at new features. The test are reusable, across different browsers.
The cons, there is a maintenance burden keeping the scripts up and running that should be factored in, it’s not a write once operation. Human intervention is still required in some cases to review screen shots taken by tests, and reviewing the results.

10. Blackboard client assisted testing?
Taking this forward
Automate more
Maintenance
Continuous testing
Load testing
Blackboard client assisted testing?
David:
We have automated about half our tests so we still have some more to automate.
We need to evaluate what kind of a maintenance burden keeping these tests up and running will bring.
Can we use a continuous testing tool, such as Jenkins, to run tests continuously. This would effectively be providing a service monitoring and services availability capability
Can the tests be exported to Junit and then incorporated into Jmeter to perform realistic load testing of our VLE.
Finally could Blackboard not provide automated testing facilitates that perform functional tests on our installations? They already use automated tests in a development environment, but with the power of local data, then surely they could identify far more bugs.

11. Questions? Link to paper: http://goo.gl/Cp17CL
David:
That concludes our presentation, we did promise to talk about exporting data from Campus Pack, but then realised that was probably too ambitious with a 15 minute slot. We do have a slide on that and I’m happy to talk about how we did that after the presentation.
You may also be interested in reading our paper of this presentation, URL on screen.
Whilst we take questions I thought I’d play a screen capture recording of some automated tests taking place, just to help you visualise what it looks like when Selenium is running a test suite.
Time for questions.
David Barrett Blayn Parkinson
University of York, UK

12. Exporting data from Campus Pack 4
Export from CP4 hosted solution
Ruby Watir and AutoIT
Script generator
Actions
Visiting every CP4 user/course page
Scraping html
Triggering an export
Downloading zip files
For more info: