SRDR Quarterly Training Brown Evidence-based Practice Center Brown University December 5 th, :00pm-2:00pm Best Practices: Improving the efficiency and effectiveness, and consistency of data entry SRDR™ was initially developed by the Tufts EPC and currently maintained by the Brown EPC under contract with AHRQ (Contract No. HHSA I & HHSA I ).

12/5/2014http://srdr.ahrq.gov2 Presentation outline  Summarize common difficulties / challenges encountered by users entering data into SRDR  Demonstrate best practices to mitigate these challenges

12/5/2014http://srdr.ahrq.gov3  Inefficient design of extraction forms  Extraction forms with lots of empty space  Unintuitive question titles on extraction forms  Unrestrained extraction forms giving too much latitude to extractors  Entering data prospectively into SRDR  Pitfalls and limitations for copying and pasting data into extraction forms  Losing track of data extractor progress, and questions they might have  Entering data retrospectively into SRDR  Many different file formats (MS Word, MS Excel, PDFs)  Preparing results data for importing (coming feature) Challenges entering data into SRDR

12/5/2014http://srdr.ahrq.gov4  Exporting data from SRDR in a format usable to others (statisticians, analysts) or in a format ready to be used as evidence tables in appendix of an EPC report  To be addressed in a future Webinar in January  new features added to SRDR since the last training Webinar held in September Challenges exporting data from SRDR

12/5/2014http://srdr.ahrq.gov5  Compilations of strategies used by SRDR Users, EPCs, and the SRDR Team  Recommendations that will enable you to overcome the challenges associated with entering data into SRDR  Benefits of using best practices include:  Reduce time and effort required to enter data into SRDR  Improve accuracy and consistency of data collected  Increase accessibility of data for other SRDR contributors and public users  Take advantage of SRDR Tools and features Best Practices

12/5/2014http://srdr.ahrq.gov6 Best Practices for Designing Efficient Extraction Forms

12/5/2014http://srdr.ahrq.gov7  Background  Key Questions (KQ) entered into an SRDR project should parallel the KQs addressed in your systematic review or EPC Report.  Each KQ can only be associated with a single (primary) extraction form  Several KQs may use the same extraction form  Challenge  Since a KQ can only be associated with one (primary) extraction form, collection of certain data across all eligible studies for the KQ (i.e. study quality information for multiple study designs) can result in extraction forms with long lists of mutually exclusive questions (i.e questions relevant to only one type of study design) that could be unwieldy to navigate Create accessory forms to complement primary extraction form

12/5/2014http://srdr.ahrq.gov8  Best Practice  Create “accessory” extraction forms to complement the data collected on a primary extraction form (each accessory extraction form will require adding a new KQ to your project)  Note: You will need to describe in the “Project notes” field of the Project Information page of the SRDR project, which KQs were added to the project to create accessory extraction forms and the KQs from the report they address.  Benefit  Creating “accessory” extraction forms will shorten the number of data fields that an extractor need enter data into for any one extraction form. Create accessory forms to complement primary extraction form (Cont.)

12/5/2014http://srdr.ahrq.gov9 Example: Denoting addition of accessory extraction forms in “Project Notes” field.

12/5/2014http://srdr.ahrq.gov10  Background  A question title (data header) is required for each question added to the extraction form  Displayed as column (spreadsheet) headers in simple and advanced export of data from SRDR  Challenge  It is difficult to understand arbitrarily named or unintuitive question titles (e.g., BMI_X instead of Average BMI). This difficulty adds time required by the data extractor to interpret the meaning of the question and presents confusion to other users. Use self-explanatory question titles on extraction forms

12/5/2014http://srdr.ahrq.gov11  Best Practice  Question titles should be self-explanatory and include: units and/or specific data ranges / formats (i.e. only whole number values; or DD/MM/YYYY date format)  Benefits  Minimize data extraction errors  Improve accessibility of data by users Use self-explanatory question titles (Cont.)

12/5/2014http://srdr.ahrq.gov12 Example: Intuitive and unintuitive question titles Self-explanatory: Not self-explanatory:

12/5/2014http://srdr.ahrq.gov13  Background  Users can add a list of “suggested” arm and outcome titles to the Arm and Outcome tabs of SRDR extraction forms.  During individual data abstraction, data extractors are given the option to use these pre-specified arm and outcome titles or create their own.  Challenge  When extractors have the latitude to name the arms and outcomes on their own, they may come up with different names (e.g., Cardiovascular disease could be named as: CVD, CV disease, etc.. ). Variations in the naming of arms and outcomes is unwanted and require time to standardize. It will also confuse users of the data unfamiliar with the project. Pre-specify arm and outcome titles for data extractors to use on the extraction form

12/5/2014http://srdr.ahrq.gov14  Best Practice  Step 1: The project lead should pre-specify a list of the arm and outcome titles and then add the titles from this list for other extractors to use as “suggested” arms and outcomes on the Arms and Outcomes tabs of the SRDR extraction form.  Step 2: Train data extractors in the team to use suggested arms and outcomes, unless it is absolutely necessary to create a new name (and then the new name should be added to the suggested list).  Benefits  Reduce data entry time  Reduce spelling errors, case errors and unwanted redundancy  Reduce data cleaning needs Pre-specify arm and outcome titles for data extractors to use on the extraction form

12/5/2014http://srdr.ahrq.gov15 Example of the unwanted variation that can result in Outcome titles

12/5/2014http://srdr.ahrq.gov16  Background  Users may choose from the following selection of question types to add to an SRDR extraction form:  Text  Dropdown menu  Multiple choice (select one) aka. Radio button question  Multiple choice (select any) aka. Checkbox question  Matrix of choices (only one answer allowed) aka. Matrix of Radio buttons  Matrix of choices (select more than one answer) aka. Matrix of Checkboxes  Matrix of dropdowns (specify option in each cell)  Challenge  SRDR extraction forms relying primarily on text, dropdown menu, and multiple choice questions could result in very long forms. Long forms are a challenge to use because of excessive vertical scrolling which increases the time needed by data extractors to enter data. Shortening extraction forms using dropdown and matrix-type questions

12/5/2014http://srdr.ahrq.gov17  Best Practice  When possible, consolidate multiple questions into a single matrix question. (e.g. when the same set of descriptive statistics and other information is asked for multiple risk factors on the Baseline tab of SRDR extraction forms, the individual questions can be consolidated into separate cells within a single matrix question).  Benefits  Any number of columns or rows of cells may be added to matrix questions on SRDR extraction forms enabling users to minimize the length of the form.  Values entered into each cell of a matrix question are exported top separate columns of simple export spreadsheets allowing for easy analysis/isolation of data. Using Dropdown and Matrix-type questions (Cont.)

12/5/2014http://srdr.ahrq.gov18 Example contrasting vertical space used by matrix and non-matrix questions Multiple Text Qs Single Matrix Q Multiple Matrix Qs

12/5/2014http://srdr.ahrq.gov19  Background  Arm and outcome descriptions can be added to arms and outcomes entered into the Arm and Outcomes tabs of SRDR extraction forms.  Challenge  When exporting data from SRDR, Arm Title and Arm Description (same for Outcome) are concatenated into a single character string (e.g. the title,” Vitamin D”, with description, “consumed”, will appear on export as Arm: “Vitamin D [consumed]”) to distinguish arms that may have the same title but different descriptions. This concatenated string may require additional processing steps before analysis. Save time cleaning data by creating arm and outcome description questions on your forms

12/5/2014http://srdr.ahrq.gov20  Best Practice  Instead of entering information into arm description and outcome description fields on Arm and Outcome tabs, create a question on Arm Details, Outcome Details tabs to capture this information.  Note: To use this best practice, Arm Details and Outcome Details tabs on extraction forms should be set to display questions for each arm or outcome, respectively.  Benefit  Less time needed to clean arm and outcome titles exported from SRDR Avoid using arm, outcome description fields on Arm, Outcome tab (Cont.)

12/5/2014http://srdr.ahrq.gov21 Example: Adding arm description value to arm title on Arm tab Arms tab of extraction form Simple export of Arms tab SRDR Simple Export tool used to export data entered into extraction form

12/5/2014http://srdr.ahrq.gov22 Example: Adding arm description value to arm title on Arm tab (Cont.) Export of extraction form using Report Builder tool

12/5/2014http://srdr.ahrq.gov23 Example: Adding arm description value to question on Arm Details tab Arm Details tab of extraction form Simple export of Arm Details tab SRDR Simple Export tool used to export data entered into extraction form

12/5/2014http://srdr.ahrq.gov24 Example: Adding arm description value to question on Arm Details tab (Cont.) Export of extraction form using Report Builder tool

12/5/2014http://srdr.ahrq.gov25  Background  Questions asked on an extraction form (excluding questions concerning study design or study quality), are typically asked for a specific arm or outcome defined in a study.  This is typically accomplished by creating questions for specific arms or outcomes on the form, or instead creating blank data fields into which miscellaneous arms or outcomes may be entered (the same applies when creating extraction forms in SRDR).  Challenge  An extraction form (e.g., paper, software) seeking to predefine all possible arms/outcomes could be lengthy and unwieldy.  To accommodate unexpected arms/outcomes during data extraction, the extraction form will need to be revised (this problem also applies to SRDR when features are not used). Use SRDR’s feature to dynamically create arm/outcomes questions in tabs

12/5/2014http://srdr.ahrq.gov26  Best Practice  Rather than enumerating each question explicitly on an SRDR extraction form, use the “dynamic” arm/outcome question feature to ask a question on extraction form tabs ( Arm Details, Baseline, Outcome Details, Results, Adverse Events) for just the arms and outcome that are defined in a particular study.  Pro-tip: After constructing an extraction form, always check to see how data fields are presented in simple and advanced data export. Also, pilot the extraction form.  Benefits  Minimizes the number of times questions need to be asked on a tab, resulting in a smaller extraction form and quicker navigation  Extraction form will readily accommodate unforeseen arm/outcomes keeping the number of revisions to the extraction form at a minimum Use SRDR’s feature to dynamically create arm/outcomes questions in tabs (Cont.)

12/5/2014http://srdr.ahrq.gov27 Example: Extraction form that collects data for a predefined number of arms

12/5/2014http://srdr.ahrq.gov28 Example: Replicating the extraction form in SRDR and exporting data Simple export of Arm Details tab of SRDR extraction form

12/5/2014http://srdr.ahrq.gov29 Simple export of Arm Details tab of SRDR extraction form Example: Incorporating feature to dynamically create arm questions in form and exporting data

12/5/2014http://srdr.ahrq.gov30 Best Practices for Entering Data Prospectively into SRDR

12/5/2014http://srdr.ahrq.gov31  Definition of prospective data entry  Copying data from the full text of studies included in a systematic review, directly into data fields on SRDR extraction forms.  Method  Manual entry: Requires typing or copying and pasting data values, individually, into data fields on the extraction form. Background

12/5/2014http://srdr.ahrq.gov32  Background  Symbols, such as those listed below are commonly found in the full text of studies and can be typed or copied and pasted into extraction forms in SRDR  Mathematical operations (÷, ×, ±, °, >, <, ≥, ≤, ≠, =, ≅,≈, …)  Greek Letters ( α, β, γ, ε, ζ, η, θ, λ, μ, ξ, ω, …)  Other Symbols ( &, ®, ™, $, €, –, —, ^, ©, …)  Challenge  Among the symbols that can be entered into extraction forms, only a slim minority can be exported from SRDR. Among the symbols listed above as examples, only >, <, &, $, and ^ can be exported to a simple export spreadsheet. The remainder are substituted with a “blank” space on the simple export spreadsheet.  Note: All symbols imported into SRDR using the import tool may be viewed upon export. Use work-arounds when entering symbols unsupported by SRDR into extraction form data fields

12/5/2014http://srdr.ahrq.gov33  Best Practice  Data extractors should be trained to use the following commonly accepted work-arounds for symbols that can otherwise not be exported from SRDR  ≤, ≥, and ≠ should be written as “ =“, “/=“  “cm 3” can instead be written as “cm^3”  Otherwise, extractors should refrain from entering symbols into SRDR or when necessary, spell symbols out. (i.e. “ α ”entered as “alpha”, or “And” instead of “&”)  Benefits  Use of work-arounds or spelling out symbols means that data from a project can be exported by other users and the absence of specific symbols will not throw off the meaning of data values. Use work-arounds when entering symbols unsupported by SRDR into extraction form data fields (Cont.)

12/5/2014http://srdr.ahrq.gov34 Example: How string of symbols is not exported completely from SRDR Response as it appears on Simple Export spreadsheet Response as it appears on SRDR extraction form SRDR Simple Export tool used to export data entered into extraction form

12/5/2014http://srdr.ahrq.gov35  Background  Superscript and subscript text are often used in full text studies for footnotes (x i ), mathematical operations (cm 3 ) and citations (Smith, ).  Challenge  When superscript or subscript text is copied and pasted into data fields on an extraction form in SRDR, the super/subscripted text reverts to a regular sized font which can change the entire meaning of a particular response thereby reducing the accuracy of data collected (e.g. “Unadjusted RR = ” appears as “Unadjusted RR = 1.34” on extraction form) or otherwise make the data entirely incomprehensible to other users (e.g. “x i ” appears as “xi” on extraction form) Copying and pasting super/subscripted text into extraction form data fields

12/5/2014http://srdr.ahrq.gov36  Best Practice  Data extractors should either avoid copying and pasting super- subscripted text into data fields on SRDR extraction forms or find an alternative way to enter these values (i.e. enter cm 3 as cm^3) such that their meaning is not lost to other users.  Benefits  More accurate data collection  Less confusion for other users Copying and pasting super/subscripted text into extraction form data fields

12/5/2014http://srdr.ahrq.gov37  Background  When a question on an SRDR extraction form is left with a blank response it is recognized by SRDR as a null-value.  Challenge  When reviewing the export of data from a project in SRDR, it is therefore impossible to know whether the response to an extraction form question was 1) left blank because the full-text of the study simply did not contain the data requested by the question OR 2) left blank because the data extractor forgot to answer the question. Use of “No Data” (ND) response to answer questions on form when no data is available

12/5/2014http://srdr.ahrq.gov38  Best Practice  Data extractors should enter “ND” into all extraction form data fields for which no data is given in the article to respond.  Benefit  Project leads are able to more readily identify data fields that were missed or forgotten by extractors and have them corrected, thus improving accuracy of data collected. Use of “ND” for questions on form for which no data is available (Cont.)

12/5/2014http://srdr.ahrq.gov39  Background  At the bottom of each tab on an SRDR extraction form is the button “Mark Section Complete” which data extractors may use to keep track of the sections they have completed on the extraction form.  Clicking the button twice will change the status of the tab from “Complete” to “Incomplete”  The completion status of all tabs in the extraction form may be viewed as well as “toggled” by the extractor on the Finalize tab of the extraction form.  Challenge  Keeping tabs on the progress of data extractors during data abstraction can be a difficult and time-consuming process Use of “Mark Section Complete” button during individual data extraction

12/5/2014http://srdr.ahrq.gov40  Best Practice  Each time a data extractor completes entering data into data fields on an extraction form in SRDR, they should mark the tab “Complete” using the Mark Section Complete button and remember to review the completion status of each tab from the Finalize tab of the extraction form before exiting.  Benefits  Enables data extractors to remember whether there is a section they have yet to complete, thereby improving accuracy of data collected.  Pro-tip: Allows the Project Lead to use the Progress Report tool located under project Tools to view a report of each extractor’s overall progress. Use of “Mark Section Complete” button during individual data extraction (Cont.)

12/5/2014http://srdr.ahrq.gov41 Example Report generated by the SRDR Progress Report Tool

12/5/2014http://srdr.ahrq.gov42  Background  Notes can be left by data extractors on the Finalize tab of the extraction form for a particular study, in the “Note” data field.  Notes may be viewed by project leads from the Complete Study List of the project.  Challenge  It is important that project leads are able to address questions extractors may have while extracting data from studies.  Best Practice  Data extractors should be trained to leave notes on the extraction form for project lead, when needed.  Benefit  Faster data extraction Use of “Note” data field on extraction forms during individual data extraction

12/5/2014http://srdr.ahrq.gov43 Example of Notes left by extractor for a study on study list

12/5/2014http://srdr.ahrq.gov44 Best Practices for Entering Data Retrospectively into SRDR

12/5/2014http://srdr.ahrq.gov45  Definition of retrospective data entry  Transferring data that has already been collected from the full text of multiple studies and saved in an external file (MS Word, MS Excel, PDF file…) into data fields on SRDR extraction forms OR otherwise uploading the external files to a project for easy downloading by other SRDR contributors and public users  Method(s)  Manual entry: Requires typing or copying and pasting data values, individually, into data fields on the extraction form  Importing: Requires, first, formatting and saving the data as a Microsoft Excel spreadsheet. Data from the spreadsheet is then read and transferred, in batch, by the SRDR Import tool into data fields on the extraction form  Uploading: Requires using the SRDR Manage Reports tool to “attach” files (PDF, text, MS Word, MS Excel …) such as EPC report appendices to SRDR projects. These files will then be available to download by other users Background

12/5/2014http://srdr.ahrq.gov46 1. Log into SRDR with your SRDR username and password. 2. Check that you are in the MySRDR tab. 3. Click the edit link under the project to which you would like to upload the PDF files. 4. In the Navigation menu, on the left side of your project, click the Data Export Tool button located under Project Tools. 5. Next, on the Export Data Tools page, click the Manage Reports link, located in the box labelled Project Reports to access the Manage Reports Tool. 6. On the following page, click the Add Report button. 7. Under Upload Report, in the Description field, enter a brief description of the PDF file you want to upload to your project. 8. Next to the Description field, click the Choose File button to select the PDF file from your computer. 9. Select the File. 10. Below the Description field, click the Upload button, to upload the file to your project. 11. Repeat steps 1 through 10, to add additional PDF files to your project, 12. At the top of the page, below the MySRDR tab, click the Edit Project tab to return to your project information page. Steps for Uploading a file to SRDR

12/5/2014http://srdr.ahrq.gov47  Manual Entry  Pro: Data can be entered into every tab of SRDR extraction form  Con: Data must be entered into tabs on extraction form individually (time intensive process)  Importing  Pro: Batch Process, can enter data into tab for any number of studies at once  Con: Data on Excel spreadsheets has to be formatted to meet requirements for importing (see prev webinar for requirements); Data cannot yet be imported into tabs supporting “dynamic” arm/outcome question feature, including the Results tab (feature coming soon)  Uploading  Pro: Files are uploaded quickly; Files are viewable by other SRDR users  Con: Data values are not incorporated into the SRDR database Methods for Entering data saved on a Microsoft Excel file into SRDR

12/5/2014http://srdr.ahrq.gov48  Manual Entry  Same Pros/Cons listed for entering data saved on MS Excel files  Importing  Pro: Batch Process, can enter data into tab for any number of studies at once.  Con: Additional step required to transfer data to MS Excel files; After which, same cons listed for importing data from MS Excel files apply.  Pro-tip: Watch for the inclusion of returns when copying and pasting data from MS Word into excel, or else each new line in the copied text will be pasted into an individual cell.  Uploading  Same Pros/Cons listed for entering data saved on MS Excel files Entering data saved on a Microsoft Word file into SRDR

12/5/2014http://srdr.ahrq.gov49  Manual Entry  Same Pros/Cons listed for entering data saved on MS Excel files  Importing  Pro: Batch Process, can enter data into tab for any number of studies at once.  Con: Additional step required to transfer data from to MS Excel files (best accomplished using PDF to Excel conversion software, i.e. Able2Extract Pro distributed by Investintech Inc.); After which, same cons listed for importing data from MS Excel files apply.  Pro-tips:  Can print MS Word files to PDFs and then use PDF conversion software to convert PDFs to MS Excel files.  Using Conversion software some symbols on text of PDF file may not convert correctly to the MS Excel file (i.e. ≥, ≤ symbols in particular)  Uploading  Same Pros/Cons listed for entering data saved on MS Excel files Entering data saved on a PDF file into SRDR

12/5/2014http://srdr.ahrq.gov50  Manual Entry  Same Pros/Cons listed for entering data saved on MS Excel files  Uploading  Same Pros/Cons listed for entering data saved on MS Excel files  Importing – Not Yet Available  Requirements:  Data must be saved in a Microsoft Excel spreadsheet  Data arranged in a vertical fashion with:  Multiple rows per study record  Column headers for Arm, Outcome, Time point, Time point units, Subgroup, and the desired outcome measures Methods for entering results data into SRDR

12/5/2014http://srdr.ahrq.gov51 Example: Importing into the Results tab of an SRDR Extraction form PMIDTitleAuthorYearArmOutcomeTime Point Time Point Unit MeanSD Comparative efficacy of LEAP, TEACCH and non-model-specific special education programs for preschoolers with autism spectrum disorders Boyd2013 LEAP IQBaselineN/A Comparative efficacy of LEAP, TEACCH and non-model-specific special education programs for preschoolers with autism spectrum disorders Boyd2013 LEAP IQ1years Comparative efficacy of LEAP, TEACCH and non-model-specific special education programs for preschoolers with autism spectrum disorders Boyd2013 LEAP AttentionBaselineN/A Comparative efficacy of LEAP, TEACCH and non-model-specific special education programs for preschoolers with autism spectrum disorders Boyd2013 LEAP Attention1years93 Template of Microsoft Excel spreadsheet to be imported:

12/5/2014http://srdr.ahrq.gov52 Example: Importing into the Results tab of an SRDR Extraction form (Cont.) Result of Import as seen in the Results tab of the SRDR extraction from:

12/5/2014http://srdr.ahrq.gov53 Reminder These slides may be downloaded from the Quarterly Training tab of the SRDR Help & Training page (

12/5/2014http://srdr.ahrq.gov54 Questions?