ePREM Training Overview

https://experiencesurveys.nationalresearch.ca/login Logging onto ePREM https://experiencesurveys.nationalresearch.ca/login New users will receive an e-mail from support@nationalresearch.ca when their account is first created prompting them to set their password. If you do not remember your password, click on the “Forgot your password?” link. If there are additional account issues, contact: ePREM@cancercare.on.ca Additional info: ePREM – First Time login.pdf and ePREM - Reset a Lost Password.pdf Note: If you are using IE you have to add the website https://experiencesurveys.nationalresearch.ca to the Trusted Sites zone.

User Roles and Access Report Viewer Survey Manager Can view and test surveys Can view all shared reports and crosstabs for a project Can create new reports and crosstabs Can view and test surveys Can view all shared reports and crosstabs for a project Can create new reports and crosstabs Can create new surveys and manage user access to these surveys Note: The “Your Voice Matters” survey can only be edited by CCO

ePREM Tool Terminology Within the ePREM tool, the following terms are used to describe and display total numbers of responses: Complete Surveys a.k.a Qualified Completes: The patient reaches the final page of the survey and clicks “Continue” button Incomplete Surveys a.k.a Partial Completes : patients that do not reach the final page Active quit a.k.a Terminates: the patient clicks the “Quit” button to exit the survey Passive quit: the patient stops answering and the survey page timeout period is reached. (ex. Patient walks away from kiosk) The timeout period for “Your Voice Matters” is 2 minutes. Partial Completes = active quit (Terminates) + passive quit Partial Completes are also referred to as Drop-Outs Note: Bold+Italic terms are ePREM terms (visible on dashboard e.g. Field Report)

Review of Helpful Resources First Time Login ePREM Quick Reference Guide: Knowledge Base ePREM: Quick Terminology Guide for Regional Cancer Centres

Portal Screen – Listing of Projects/Surveys After logging into the ePREM tool, a list of all projects/surveys is presented: 1 Use these buttons to search, access your profile and the knowledge base for help 2 Use these buttons to view, sort, order the projects/survey as well as create a new project/survey 3 The headers in the default list view of all projects/surveys This button brings up the “Project/Survey Controls” menu, with a number of actions related to the project (will be covered in detail later in the training) 4 5 Click on the project name to open the “Project/Survey Details” page

Project/Survey Details The “Project/Survey Details” screen provides an overview of survey responses and serves as a hub from which to access other parts of a survey. There are two main components to this screen: A summary of survey responses on the left of the screen Tools for survey managers and report viewers on the right of the screen

Project/Survey Details A summary of all survey responses Total – “Qualified/completes” responses + “Terminates” responses Qualified – Complete survey responses Today – The number of responses today Questions – The total number of questions in the survey Completion Time – The average time to complete the survey Qualification – The percentage of “Total” responses that are “Qualified” Drop-Outs – The percentage of incomplete surveys from the number of surveys started Languages – Filter displayed data based on language Sample Sources – Filter displayed data based on how the user reached the survey Note: Incidence is not used within the “Your Voice Matters” survey

Project/Survey Details Shared Documents Files can be uploaded to the project directory and shared amongst users - Description A place to make notes regarding the survey Tags Can be used to search for surveys from the Portal screen

Project/Survey Details Users Shows a list of all users assigned to a project, and their project permissions. Add new users Modify a user’s permissions for this project/survey Remove users 1 2 3 4 Note: User permissions for the “Your Voice Matters” survey can only be modified by Cancer Care Ontario

Project/Survey Details Reports Shows a list of saved reports and crosstabs for the project

Questions?

Project/Survey Controls The “Project Controls” menu displays available actions for a particular survey. This menu can be displayed from the “Portal” screen or the “Project/survey Details Screen” by clicking on the menu icon Portal Project/Survey details screen

Reports There are three main types of reports available from the “Project Controls” menu: Field Reports Reports (2010) Crosstabs

Reports Crosstabs robust reporting tool many options for data filtering and presentation recommended option for report creation Field Reports allows the user to monitor the progress of the survey does not include survey responses Reports (2010) basic reports with survey response data basic filtering and presentation tools older report format (legacy), replaced by crosstabs

Crosstabs Crosstabs are the recommended option for viewing response data and creating custom reports, and they are the most robust option. There are three options available for crosstabs: Run Total Run Quick Split Crosstab Build Crosstab

Types of Crosstabs Run Total displays the aggregate totals of each question – the total frequency report presents the same information as “Report (2010) > Standard” filters or conditions can be applied to this report after it is run (e.g. “Only responses from the past week”) Run Quick Split Crosstab allows for response data to be split according to the answer(s) given to one or more questions e.g. split by gender if a survey question asked for the gender of the respondent Build Crosstab allows for the creation of a crosstab from scratch most robust option greater control on how data is split and what responses are included in the report

Crosstabs From the main “Crosstab” page, clicking “Run Total” will create the following:

Crosstabs – Run Total Change the filter settings for crosstabs by clicking here 1 Custom date ranges can be changed using the drop-down menu 2 Change table settings or add a chart to a question 3

Crosstabs – Edit Settings Change the name and description of a crosstab 1 Custom date ranges can be changed using the drop-down menu 2 Adjust which response types to include in the report (ex. Qualified Only, Including Partial Responses) 3 A means of comparing two samples and determining whether the differences are valid or due to chance

Crosstabs From the main “Crosstab” page, clicking “Run Quick Split Crosstab” will display the following screen:

Crosstabs – Quick Split Select conditions to split response data by in the top section. In this example, the crosstab will split data based on the respondents answer to question one. A preview of the crosstab that will be created is shown below. Click this button to create the crosstab report. 1 2 3

Crosstabs – Quick Split A sample segment from a quick split crosstab

Crosstabs From the main “Crosstab” page, clicking “Build Crosstab” will display the following screen:

Crosstabs – Build Crosstab Custom groupings of respondents can be created Multiple logic rules can be combined to filter data

Crosstabs – Build Crosstab A sample segment from a custom built crosstab:

Crosstabs – Statistical testing The Stat Test Groups: indicate what segments are tested together e.g. B-J will test the month segments against each other The Stat Test Levels: set the thresholds for indicating statistical significance e.g. 95%/90% The red notations indicate the statistical significance: capital letters indicate statistical significance at the 95% level and lower case letters at the 90% level

Crosstabs – Statistical testing – cont Best5 April 2016: in statistical terms we would interpret “We are 95% certain that the difference in Ratings of 5 between April and July and April and September respectively, is a true difference between the months, but only 90% certain that the difference between April and August is a true difference There is no red C/c (or D/d) in the April cell: while there is a difference between April and May (4%) and April and June (6%), the difference is not statistically significant, meaning it is likely due to chance/randomness.

Questions?

Field Report The field report provides an overview of response volume over the duration of the survey. You can email, print the page or download the field report to Excel.

Field Report - Completions tab Click-Through – Respondents who opened the first page of the survey. Partial Completes – Respondents who abandoned the survey (hit the Quit button (active quit) or walked away (passive quit) ) Terminates – Respondents who actively quit the survey (i.e. Quit button); subset of Partial Completes Overquota – Respondents who exceeded survey quotas Note: Not used in “Your Voice Matters” Qualified Completes – Respondents who completed the survey 1 2 An ISAAC patient that accepted the YVM in ISAAC, when clicks “Accept” the patient is redirected to the first page of the YVM survey and that is the moment the YVM survey starts Due to the nature of the tool, the way the Quit button was implemented, the terminates are counted twice. The real number of the survey participants is not equal click-through but equal click-through minus terminates: 1169-142=1027=607+420 Partial completes: hit the Quit button or walk away letting the page to timeout 420 Terminates: hit the Quit button, a subset of partial completes 142 Overquota: ignore, CCO didn’t establish quota logic (how many respondents or how long the survey is live) Qualified completes: aka completes, respondents that reached the last page of the YVM and hit Finish button 3 4 5

Field Report - Completions tab cont. Participation Rate – Equal to: (Terminates + Overquota + Qualified) / Click-Through Note: Not used in “Your Voice Matters” Incidence – Equal to: (Qualified + Overquota) / (Click-Through – Partial) Completion Time (average) – Average completion time for qualified responses Completion Time (median) – Median completion time for qualified responses 6 7 Participation rate: (142+0+607)/1169 = 64%; don’t use it, will be provided in the CCO reports; real rate would be qualified/real total = 607/(1169-142)=607/1027=59% Incidence: don’t use it, will be provided in the CCO reports Completion time average: 5 surveys with completion times (1,2,4,5,7) => (1+2+4+5+7)/5=3.8 Completion time median: for above example = 4 The completion times are calculated by removing first the top and bottom 1% (i.e., any time outside of 3 standard deviations), then recalculating the average/median with the remaining 99%. 8 9

Field Report - Completions tab: Click - Through The number of respondents that entered the survey/opened the first page of the survey Available as hyperlink in Field Report, “Completions” tab: Click to open the partial data in tab-delimited form in a new browser window; does not contain the patient answers To save the partial data you can either copy and paste the contents into a text editor, or select "save page as" for the browser window and save the contents to your computer in a .txt file. You can then import the file into Excel using the import wizard; simply select the tab-delimited .txt file and import the partial data into Excel. Partial data example and data dictionary:

Field Report - Completions tab cont. Clicking on an individual data-point will “zoom in” to display an hourly breakdown of survey responses for that day. NOTE: All times are displayed in Pacific Standard Time (PST).

Field Report – Terminates and Drop-outs tabs Clicked Quit Link - respondents that hit the Quit button Overquota and Dupe - ignore, not used in YVM Drop-outs Helpful in seeing where (page/question) the patient quits CCO investigating ways to make improvements in partnership with vendor; first 3 rows represent the first/intro page of YVM e.g. from the 1437 that got on the first/intro YVM page only 1116 continued to the 2nd page (labeled Page 3 but in reality 2nd page of YVM) The number of terminates is available on the Completions tab The Drop-outs though inaccurate can provide a high level understanding of the quit trend; one can notice that the biggest percentage of drop-outs is on the first page (represented by the first 3 rows, page 0, 1 and 2 represent same first YVM intro page)

Questions?

Report (2010) The “Report (2010)” menu allows for creation of basic reports. A brief summary of responses is displayed at the top of the page. A default “Standard” report is always available. Custom reports can be created with the “New Report” button. Reports can be set to display only specific questions, or to include results from a subsection of all respondents. To view a report, click the “Run Report” button. 1 2 3 4

Report (2010) To add a graph to the report, click the icon here

Report (2010) Report data can be exported in a number of formats, using the options on the toolbar in the upper-left Note: The PPT option was discontinued, use PPTX.

Report (2010) Report(2010) can be used to create a custom report with defined date ranges that can be further used in the field report as custom banner (split by date) Select “+New Report” and create segments that fit he desired date ranges using the “start of” and “end of” dropdowns and click “Add Date Restrictions”. Once all segments created click Save. Each segment of the saved report will be a column in field report split.

Questions?

Useful links Knowledge Base: https://support.focusvision.com/Decipher Portal overview: https://support.focusvision.com/Decipher/0002_Portal/0001_Overview/020_Introduction_to_the_Research_Hub User management: https://support.focusvision.com/Decipher/0002_Portal/0003_User_Management Project controls: https://support.focusvision.com/Decipher/0002_Portal/0002_Project_Management/0003_Project_Controls Crosstab: https://support.focusvision.com/Decipher/0006_Report/004_Crosstabs https://support.focusvision.com/Decipher/0006_Report/004_Crosstabs/080_Configuring_Crosstab_Settings Field Report: https://support.focusvision.com/Decipher/0006_Report/001_Field_Report https://support.focusvision.com/Decipher/0006_Report/001_Field_Report/011_Accessing_Click-Through_Partial_Data https://support.focusvision.com/Decipher/0006_Report/001_Field_Report/030_Splitting_the_Field_Report Report (2010): https://support.focusvision.com/Decipher/0006_Report/003_Report_(2010) Build new project/survey: https://support.focusvision.com/Decipher/0003_Build/0002_Getting_Started%3A_Build_a_Survey/040_Starting_a_New_Project#

Useful articles Access Project/Survey Reports Details downloaded from KB complementary to the info on the slides

Demo of Survey Builder Double click on screen for video to start

Please send your questions to: ePREM@cancercare.on.ca