1. ePREM Training Overview

2. https://experiencesurveys.nationalresearch.ca/login
Logging onto ePREM
New users will receive an from when their account is first created prompting them to set their password.
If you do not remember your password, click on the “Forgot your password?” link.
If there are additional account issues, contact:
Additional info: ePREM – First Time login.pdf and ePREM - Reset a Lost Password.pdf
Note: If you are using IE you have to add the website to the Trusted Sites zone.

3. User Roles and Access Report Viewer Survey Manager
Can view and test surveys
Can view all shared reports and crosstabs for a project
Can create new reports and crosstabs
Can view and test surveys
Can view all shared reports and crosstabs for a project
Can create new reports and crosstabs
Can create new surveys and manage user access to these surveys
Note: The “Your Voice Matters” survey can only be edited by CCO

4. ePREM Tool Terminology
Within the ePREM tool, the following terms are used to describe and display total numbers of responses:
Complete Surveys a.k.a Qualified Completes:
The patient reaches the final page of the survey and clicks “Continue” button
Incomplete Surveys a.k.a Partial Completes : patients that do not reach the final page
Active quit a.k.a Terminates: the patient clicks the “Quit” button to exit the survey
Passive quit: the patient stops answering and the survey page timeout period is reached. (ex. Patient walks away from kiosk)
The timeout period for “Your Voice Matters” is 2 minutes.
Partial Completes = active quit (Terminates) + passive quit
Partial Completes are also referred to as Drop-Outs
Note: Bold+Italic terms are ePREM terms (visible on dashboard e.g. Field Report)

5. Review of Helpful Resources
First Time Login
ePREM Quick Reference Guide: Knowledge Base
ePREM: Quick Terminology Guide for Regional Cancer Centres

6. Portal Screen – Listing of Projects/Surveys
After logging into the ePREM tool, a list of all projects/surveys is presented: 1
Use these buttons to search, access your profile and the knowledge base for help 2
Use these buttons to view, sort, order the projects/survey as well as create a new project/survey 3
The headers in the default list view of all projects/surveys
This button brings up the “Project/Survey Controls” menu, with a number of actions related to the project (will be covered in detail later in the training) 4 5
Click on the project name to open the “Project/Survey Details” page

7. Project/Survey Details
The “Project/Survey Details” screen provides an overview of survey responses and serves as a hub from which to access other parts of a survey.
There are two main components to this screen:
A summary of survey responses on the left of the screen
Tools for survey managers and report viewers on the right of the screen

8. Project/Survey Details
A summary of all survey responses
Total – “Qualified/completes” responses + “Terminates” responses
Qualified – Complete survey responses
Today – The number of responses today
Questions – The total number of questions in the survey
Completion Time – The average time to complete the survey
Qualification – The percentage of “Total” responses that are “Qualified”
Drop-Outs – The percentage of incomplete surveys from the number of surveys started
Languages – Filter displayed data based on language
Sample Sources – Filter displayed data based on how the user reached the survey
Note: Incidence is not used within the “Your Voice Matters” survey

9. Project/Survey Details
Shared Documents
Files can be uploaded to the project directory and shared amongst users -
Description
A place to make notes regarding the survey
Tags
Can be used to search for surveys from the Portal screen

10. Project/Survey Details
Users
Shows a list of all users assigned to a project, and their project permissions.
Add new users
Modify a user’s permissions for this project/survey
Remove users 1 2 3 4
Note: User permissions for the “Your Voice Matters” survey can only be modified by Cancer Care Ontario

11. Project/Survey Details
Reports
Shows a list of saved reports and crosstabs for the project

12. Questions?

13. Project/Survey Controls
The “Project Controls” menu displays available actions for a particular survey. This menu can be displayed from the “Portal” screen or the “Project/survey Details Screen” by clicking on the menu icon
Portal
Project/Survey details screen

14. Reports
There are three main types of reports available from the “Project Controls” menu:
Field Reports
Reports (2010)
Crosstabs

15. Reports Crosstabs robust reporting tool
many options for data filtering and presentation
recommended option for report creation
Field Reports
allows the user to monitor the progress of the survey
does not include survey responses
Reports (2010)
basic reports with survey response data
basic filtering and presentation tools
older report format (legacy), replaced by crosstabs

16. Crosstabs
Crosstabs are the recommended option for viewing response data and creating custom reports, and they are the most robust option.
There are three options available for crosstabs:
Run Total
Run Quick Split Crosstab
Build Crosstab

17. Types of Crosstabs Run Total
displays the aggregate totals of each question – the total frequency report
presents the same information as “Report (2010) > Standard”
filters or conditions can be applied to this report after it is run (e.g. “Only responses from the past week”)
Run Quick Split Crosstab
allows for response data to be split according to the answer(s) given to one or more questions e.g. split by gender if a survey question asked for the gender of the respondent
Build Crosstab
allows for the creation of a crosstab from scratch
most robust option
greater control on how data is split and what responses are included in the report

18. Crosstabs
From the main “Crosstab” page, clicking “Run Total” will create the following:

19. Crosstabs – Run Total
Change the filter settings for crosstabs by clicking here 1
Custom date ranges can be changed using the drop-down menu 2
Change table settings or add a chart to a question 3

20. Crosstabs – Edit Settings
Change the name and description of a crosstab 1
Custom date ranges can be changed using the drop-down menu 2
Adjust which response types to include in the report (ex. Qualified Only, Including Partial Responses) 3
A means of comparing two samples and determining whether the differences are valid or due to chance

21. Crosstabs
From the main “Crosstab” page, clicking “Run Quick Split Crosstab” will display the following screen:

22. Crosstabs – Quick Split
Select conditions to split response data by in the top section.
In this example, the crosstab will split data based on the respondents answer to question one.
A preview of the crosstab that will be created is shown below.
Click this button to create the crosstab report. 1 2 3

23. Crosstabs – Quick Split
A sample segment from a quick split crosstab

24. Crosstabs
From the main “Crosstab” page, clicking “Build Crosstab” will display the following screen:

25. Crosstabs – Build Crosstab
Custom groupings of respondents can be created
Multiple logic rules can be combined to filter data

26. Crosstabs – Build Crosstab
A sample segment from a custom built crosstab:

27. Crosstabs – Statistical testing
The Stat Test Groups: indicate what segments are tested together e.g. B-J will test the month segments against each other
The Stat Test Levels: set the thresholds for indicating statistical significance e.g. 95%/90%
The red notations indicate the statistical significance: capital letters indicate statistical significance at the 95% level and lower case letters at the 90% level

28. Crosstabs – Statistical testing – cont
Best5 April 2016: in statistical terms we would interpret “We are 95% certain that the difference in Ratings of 5 between April and July and April and September respectively, is a true difference between the months, but only 90% certain that the difference between April and August is a true difference
There is no red C/c (or D/d) in the April cell: while there is a difference between April and May (4%) and April and June (6%), the difference is not statistically significant, meaning it is likely due to chance/randomness.

29. Questions?

30. Field Report
The field report provides an overview of response volume over the duration of the survey.
You can , print the page or download the field report to Excel.

31. Field Report - Completions tab
Click-Through – Respondents who opened the first page of the survey.
Partial Completes – Respondents who abandoned the survey (hit the Quit button (active quit) or walked away (passive quit) )
Terminates – Respondents who actively quit the survey (i.e. Quit button); subset of Partial Completes
Overquota – Respondents who exceeded survey quotas
Note: Not used in “Your Voice Matters”
Qualified Completes – Respondents who completed the survey 1 2
An ISAAC patient that accepted the YVM in ISAAC, when clicks “Accept” the patient is redirected to the first page of the YVM survey and that is the moment the YVM survey starts
Due to the nature of the tool, the way the Quit button was implemented, the terminates are counted twice. The real number of the survey participants is not equal click-through but equal click-through minus terminates: =1027=
Partial completes: hit the Quit button or walk away letting the page to timeout 420
Terminates: hit the Quit button, a subset of partial completes 142
Overquota: ignore, CCO didn’t establish quota logic (how many respondents or how long the survey is live)
Qualified completes: aka completes, respondents that reached the last page of the YVM and hit Finish button 3 4 5

32. Field Report - Completions tab cont.
Participation Rate – Equal to: (Terminates + Overquota + Qualified) / Click-Through
Note: Not used in “Your Voice Matters”
Incidence – Equal to: (Qualified + Overquota) / (Click-Through – Partial)
Completion Time (average) – Average completion time for qualified responses
Completion Time (median) – Median completion time for qualified responses 6 7
Participation rate: ( )/1169 = 64%; don’t use it, will be provided in the CCO reports; real rate would be qualified/real total = 607/( )=607/1027=59%
Incidence: don’t use it, will be provided in the CCO reports
Completion time average: 5 surveys with completion times (1,2,4,5,7) => ( )/5=3.8
Completion time median: for above example = 4
The completion times are calculated by removing first the top and bottom 1% (i.e., any time outside of 3 standard deviations), then recalculating the average/median with the remaining 99%. 8 9

33. Field Report - Completions tab: Click - Through
The number of respondents that entered the survey/opened the first page of the survey
Available as hyperlink in Field Report, “Completions” tab:
Click to open the partial data in tab-delimited form in a new browser window; does not contain the patient answers
To save the partial data you can either copy and paste the contents into a text editor, or select "save page as" for the browser window and save the contents to your computer in a .txt file. You can then import the file into Excel using the import wizard; simply select the tab-delimited .txt file and import the partial data into Excel.
Partial data example and data dictionary:

34. Field Report - Completions tab cont.
Clicking on an individual data-point will “zoom in” to display an hourly breakdown of survey responses for that day.
NOTE: All times are displayed in Pacific Standard Time (PST).

35. Field Report – Terminates and Drop-outs tabs
Clicked Quit Link - respondents that hit the Quit button
Overquota and Dupe - ignore, not used in YVM
Drop-outs
Helpful in seeing where (page/question) the patient quits
CCO investigating ways to make improvements in partnership with vendor; first 3 rows represent the first/intro page of YVM
e.g. from the 1437 that got on the first/intro YVM page only 1116 continued to the 2nd page (labeled Page 3 but in reality 2nd page of YVM)
The number of terminates is available on the Completions tab
The Drop-outs though inaccurate can provide a high level understanding of the quit trend; one can notice that the biggest percentage of drop-outs is on the first page (represented by the first 3 rows, page 0, 1 and 2 represent same first YVM intro page)

36. Questions?

37. Report (2010)
The “Report (2010)” menu allows for creation of basic reports.
A brief summary of responses is displayed at the top of the page.
A default “Standard” report is always available.
Custom reports can be created with the “New Report” button. Reports can be set to display only specific questions, or to include results from a subsection of all respondents.
To view a report, click the “Run Report” button. 1 2 3 4

38. Report (2010)
To add a graph to the report, click the icon here

39. Report (2010)
Report data can be exported in a number of formats, using the options on the toolbar in the upper-left
Note: The PPT option was discontinued, use PPTX.

40. Report (2010)
Report(2010) can be used to create a custom report with defined date ranges that can be further used in the field report as custom banner (split by date)
Select “+New Report” and create segments that fit he desired date ranges using the “start of” and “end of” dropdowns and click “Add Date Restrictions”. Once all segments created click Save. Each segment of the saved report will be a column in field report split.

41. Questions?

42. Useful links Knowledge Base: https://support.focusvision.com/Decipher
Portal overview:
User management:
Project controls:
Crosstab:
Field Report:
Report (2010):
Build new project/survey:

43. Useful articles Access Project/Survey Reports
Details downloaded from KB complementary to the info on the slides

44. Demo of Survey Builder
Double click on screen for video to start

45. Please send your questions to: ePREM@cancercare.on.ca