1. ICD-10 Testing January 27-30, 2015 Puerto Rico ICD-10 Implementation Assistance Site Visit ICD-10 implementation segments to assist the Puerto Rico with the transition

2. Agenda Definition of testing types, components and artifacts The testing life-cycle Test plans, testing strategy and testing governance Scenario based testing Testing templates Managing recursive testing and remediation External testing 1

3. 2

4. 3 Health Data Consulting © 2010 Components

5. Testing Components Definitions Test Plan – The test plan is the overall document that describes how testing will be done for the enterprise relative to a change in business or system implementation. Test Strategy – The test strategy, also referred to as the test approach, speaks to the strategic goals of testing and includes a reference to what, how and why various testing components will be accomplished based on the stated goals and rationale for the approach. 4 Source: Health Data Consulting 2013HResourcesc

6. Testing Components Definitions Test Cases – Test cases are scenarios that represent aspects of the business or intended system functions where failure may occur. These are generally prioritized based on financial impact, likelihood, critical path business functions and any other parameters that are important to the business enterprise. Test Data – Test data is specific data that is created in a variety of formats to be uploaded or entered into the system based on defined test cases. 5 Source: Health Data Consulting 2013HResourcesc

7. Testing Components Definitions Test Environment – The test environment traditionally refers to the system platforms or configurations technically that have been set up outside of the production work flow to allow applications of test cases and evaluate results in the new system design. Testing Governance – A structure defines the charter that sets parameters for decisions on prioritization of testing and remediation and resolution of issues. 6

8. Testing Components Definitions Testing Documentation – All aspects of the testing process will need to be documented including test plans, test cases, governance charters, results reporting, project plans and all other aspects of testing that are important to the business. Error Management – Tracking the identification, prioritization and resolution of errors and issues discovered during the testing process Testing Remediation – Testing remediation refers to the process of solving system problems or “fixing bugs”. 7 Source: Health Data Consulting 2013HResourcesc

9. Testing Components Definitions Regression Testing – Regression testing refers to the ongoing process of system fixes followed by retesting. Internal Testing – Testing of internal systems and processes. External Testing – Testing transactions with trading partners outside of the internal enterprise. 8 Source: Health Data Consulting 2013HResourcesc

10. Testing Components Definitions Testing Phases – For the State Medicaid Agency testing check list, NGS (National Government Services) has defined 3 levels of testing as part of the published testing check list. Level 1: Internal testing readiness Level 2: testing with external trading partners is ready Level 3: Testing with external trading partners has been demonstrated to be “production ready” 9 Source: Health Data Consulting 2013HResourcesc

11. Testing Components Definitions Known Issues – Issues that have been documented and remain after final remediation and release of the system version. Unit Testing – Testing of isolated units, modules or sub-systems independent of overall system function. Integration Testing – Testing of the entire system after all remediated modules have been tested 10 Source: Health Data Consulting 2013HResourcesc

12. Testing Components Definitions User Acceptance Testing – Testing by end users to make sure that the system meets functional needs. End to End Testing – Testing from a defined point of entry into the system to the defined ending point on the system. Most commonly this means testing “round trip” internally and externally. Business Testing – Testing to assure that the business will operate properly after change even though the system is functioning as specified. 11 Source: Health Data Consulting 2013HResourcesc

13. Testing Components System Testing Types Functional Testing – User Interface Testing – System Interface Testing – Report Testing – User Access Testing – Data Validation Testing – Import/Export Testing 12 Source: Health Data Consulting 2013HResourcesc

14. Testing Components System Testing Types Non-functional Testing – Load/Stress Testing – Performance Testing – Accessibility Testing – Compatibility Testing – Recovery Testing – Scalability Testing – Security Testing – Monitor testing 13 Source: Health Data Consulting 2013HResourcesc

15. 14 Health Data Consulting © 2010 Testing Lifecycle PlanningImplementationResolution

16. 15 Source: Health Data Consulting 2013HResourcesc

17. The Testing Lifecycle Overview Focused on internal testing Prerequisites – Defining the testing strategy and goals – Establishing the testing environment – Executive levels understanding and support – Testing is a project; you need a project plan and project management Understanding the difference between business testing and system testing 16 Source: Health Data Consulting 2013HResourcesc

18. 17 Source: Health Data Consulting 2013HResourcesc

19. The Testing Lifecycle 1. Define change 18 Source: Health Data Consulting 2012HResourcesc Defining the change – Change in business requirements – Change in the system to support the same business requirements Defining the risk of change – Likelihood of failure – Impact of failure Money Time Relationships – What it takes to fix failure

20. 19 Source: Health Data Consulting 2013HResourcesc

21. The Testing Lifecycle 2. Establish Governance 20 Source: Health Data Consulting 2012HResourcesc Strategy and Goals – What is the purpose of the change in the business or system – What are the expected outcomes at a strategic level – What are the key strategic priorities Ownership and accountability – Who sets the agenda (including scope) – Who monitors the agenda – Who has the authority to enforces the agenda Acceptable levels of risk or failure

22. 21 Source: Health Data Consulting 2013HResourcesc

23. The Testing Lifecycle 3. Create test plan 22 Source: Health Data Consulting 2012HResourcesc Defining the test plan – Test environment – Tactical definition of the scope – Tactical definition of the priorities – Defining and managing the project plan – Allocating resources – Assigning authority and accountability – Setting expectations (acceptable risk and limitations) – Contingencies Test Plan Template

24. 23 Source: Health Data Consulting 2013HResourcesc

25. Predicting the future is difficult if we have no historical reference for what’s coming.

26. Reference Implementation Model Virtual worlds are safer that real ones

27. The Testing Lifecycle 4. Create Scenarios 26 Source: Health Data Consulting 2012HResourcesc Scenarios simulate the business environment They are based on what we know today – High dollar – High volume – High complexity – Important to other business functions Quality metrics Population risk Effectiveness Other initiatives Scenario Development Example

28. 27 Source: Health Data Consulting 2013HResourcesc

29. The Testing Lifecycle 5. Define test cases 28 Source: Health Data Consulting 2012HResourcesc Based on scenarios and system response to those scenarios Different scenario based test cases may be needed for different functional and non-functional requirements – Example user Interface test case Example user Interface test case – Example report test case Example report test case – And other functional non functional requirements Test cases should be developed from business requirements not from system development

30. 29 Source: Health Data Consulting 2013HResourcesc

31. The Testing Lifecycle 6. Create Test Data 30 Source: Health Data Consulting 2012HResourcesc Test data is directly linked to the test cases for each business requirement Test data should be based on existing data with known system response and then altering parameters of that data to demonstrate business and system changes as defined in the test cast Example of test case development

32. 31 Source: Health Data Consulting 2013HResourcesc

33. The Testing Lifecycle 7. Execute Test Cases 32 Source: Health Data Consulting 2012HResourcesc Once test cases are defined they need to be executed and the results assessed against the expected result. Key questions to consider – Is the environment ready to execute the test cases – Are there blocking issues – What is the turnaround time for remediation in order to re-test the scenario – Does executing the test cases reveal the need for new requirements of new scenarios and test cases

34. 33 Source: Health Data Consulting 2013HResourcesc

35. The Testing Lifecycle 8. Document Findings 34 Source: Health Data Consulting 2012HResourcesc Documenting test findings is important: – Creates an ongoing work list – Identifies changes in scope and resource requirement for the testing project – Provides the basis for adjusting scope, resources or time to completion – Establishes decision points for governance – Helps to set expectations for end users and executive management – Provides a basis for requirements that may be pushed to the next scheduled system release.

36. 35 Source: Health Data Consulting 2013HResourcesc

37. The Testing Lifecycle 9. Manage Errors 36 Source: Health Data Consulting 2012HResourcesc Testing will result in a list of errors – Tools will be needed to record errors, – Assign source – Attach to known requirements – Assign accountable resource for remediations – Assign priorities for remediations – Track progress – Share with development that may be impacted in other modules

38. 37 Source: Health Data Consulting 2013HResourcesc The Testing Lifecycle Bug Tracking Software Example

39. 38 Source: Health Data Consulting 2013HResourcesc The Testing Lifecycle Bug Tracking Software Example

40. The Testing Lifecycle 10. Remediate Errors 39 Source: Health Data Consulting 2012HResourcesc

41. The Testing Lifecycle 10. Remediate Errors 40 Source: Health Data Consulting 2012HResourcesc Once identified errors will need to be remediated – Identifying causes – Scheduling fixes – Finding resources – Who’s responsible? – Relationships – where and who may be impacted – Reporting – progress and challenges – Accepting – fix is good enough?

42. The Testing Lifecycle 10. Remediate Errors 41 Source: Health Data Consulting 2012HResourcesc

43. The Testing Lifecycle 11. Re-test 42 Source: Health Data Consulting 2012HResourcesc A number of iterations will be needed to make sure the fix worked and continued to work – The fix may not have worked – The fix may have worked from a system perspective but still not meet the business requirement – The fixed may have impacted other aspects of the module and created other errors – The module may function properly but fail on system integration testing – The level of regression test acceptable will need to be established through the governance structure.

44. The Testing Lifecycle 10. Remediate Errors 43 Source: Health Data Consulting 2012HResourcesc

45. The Testing Lifecycle 12. Document Final State 44 Source: Health Data Consulting 2012HResourcesc Summary of testing – Summary of initial strategy and Goals – Summary of key plan components – Summary of successful requirements test – Summary of requirements discovered during the testing process – Itemization of known issues – Plan for remediation of known issues Ignore Workaround Scheduled for future release

46. The Testing Lifecycle 10. Remediate Errors 45 Source: Health Data Consulting 2012HResourcesc

47. The Testing Lifecycle 13. Define Next Release Cycle 46 Source: Health Data Consulting 2012HResourcesc Identify all of the planned features to be added to the next system release Identify if the feature is an known issue remaining after the testing process Identify if the planned feature is based on a discovery of the need for the new feature noted during the testing cycle but not defined in pre-testing requirements

48. Defining Trading Partners Who are the data trading partners that will be impacted by the transition? Which trading partners are at risk and how can risk be determined? What’s the level of awareness? Are they friendly? What’s the scope of interaction? What’s the risk to your organization? Where are the priorities? 47

49. Establishing the Testing Rationale SMAs want to know how providers are going to code. Providers want to know how the SMAs are going to pay. Only cross external testing will address both questions Identify those trading partners that represent a significant part of the business on both sides or a significant risk for both 48

50. The External Testing Environment Define relevant scenarios for testing Provide a portal for testing Provide ongoing support during the testing process Make sure that internal system remediation is ready to support external testing Provider feed back on testing progress Provide a forum to discuss options for addressing problems in a way that benefits both sides 49

51. External Testing What is external testing? Who benefits? Universal or selective testing? Platforms and portals for testing? Managing the testing process Certification for production? Sharing results with all stakeholders 50

52. Post Implementation Testing will not be done October 1, 2015 Many errors will be discovered that demand a solution Solutions will need to be tested to make sure that the problem is resolved and the solution did not create another problem Assuming a problem free implementation, there will be many changes after the ICD-10 transition date that will require new testing of new system functions 51

53. Critical success factors CMS define 5 critical success factors 52 1. Accept electronic claims 2. Adjudicate claims 3. Payment 4. Coordination of benefits 5. MSIS/T-MSIS data submission

54. Critical success factors 1. Accept electronic claims 53 Process question: – Will the SMA be able to accept electronic claims with ICD-9 and ICD-10 coding based on the dates of service and dates of discharge using Version 5010 transactions on October 1, 2015? Confirm: – Testing EDI gateways with of a variety or common test claim scenarios from key clearinghouse and direct provider submitters to assure that ICD-9 and ICD-10 claims can be accepted into the MMIS system. – Testing to assure that claim rejections associated with invalid codes provide the correct response transactions to the submitter. – Assure that processes are in place to provide feedback and training support for submitters who are having challenges with test transactions.

55. Critical success factors 2. Adjudicate claims 54 Process question: – Will the SMA be able to adjudicate diagnosis dependent claims? Confirm: – Create and implement test cases with claim scenarios that will be impacted by ICD-10 based policies, coverage logic, and other ICD-10 related edits; and rules function according to the intent of the policy, rule, or edit. – Test claims in ICD-9 and ICD-10 given the same claims scenario to assure that processing results in the intended outcomes. – Assure there is a process for ongoing remediation of the SMA, MCO, and provider side to address issues with coding as well as rule remediation as early as possible.

56. Critical success factors 3.a Payment (Provider claim payment) 55 Process question: – Will the SMA be able to make provider payments to include professional and institutional providers for ICD-9/ICD-10 codes on October 1, 2015? Confirm: – Create and implement test claim scenarios with emphasis on high dollar and high volume cases to assure that payment will occur as anticipated. – Create test claims in ICD-9 and ICD-10 given the same claims scenario to assure that claim payment will be comparable before and after transition. – Assure there is a process for ongoing remediation of the SMA, MCO, and provider side to address issues with payment as early as possible to assure revenue neutrality.

57. Critical success factors 3.b Payment (MCOs) 56 Process question: – Will SMA be able to pay MCOs for ICD-9/ICD-10 codes on October 1, 2015? Confirm: – Test to evaluate MCO payments including risk adjustments, ‘kicker’ payments, or other condition related carve out models based on the use of ICD data operate as anticipated in an ICD-10 environment. – Assure revenue neutrality for MCOs payments modeled on ICD- 10 assumptions. – Assure that all claim/encounter data is submitted from MCOs and that data content and quality is acceptable to support.

58. Critical success factors 4. Coordination of benefits 57 Process question: – Will the SMA be able to complete coordination of benefit processes and exchange claims with partners including Medicare and others on October 1, 2015? Confirm: – Create test case scenarios where coordination of benefits may occur and test with key trading partners to assure transaction acceptance inbound and outbound. – Use test cases to assure that claims scenarios that should trigger coordination of benefits do so.

59. Critical success factors 5. T-MSIS and MSIS reports 58 Process question: – Will the SMA be able to create and send MSIS and/or TMSIS reports (data exchanged from MMIS to CMS MSIS) for ICD-10 claims on October 1, 2015? Confirm: – Test to assure that all required data elements associated with ICD-10 code and code type submission are populated according to specifications. – Work with the CMS to assure that submitted MSIS and T-MSIS files meet required validation for content and format (especially related to ICD-10 codes and code type data elements).

60. Questions 59