AMTSO Standards Operational Pilot SUMMARY REPORT AMTSO Standards Working Group October 24, 2017 – Version 1.1 AMTSO Standards Operational Pilot SUMMARY REPORT

Agenda Artifacts and Key Findings Tester Feedback Post-Operational Pilot

Artifacts and Key Findings AMTSO Operational Pilot Process Execution Artifacts and Key Findings

Test Plan Templates Multiple Test Plan Creation Tools Tested Word Edition (Produces DOC/PDF Files) On-Line Edition (Produces CSV Files) Detailed Usage Guidelines 6 Test Plans Built During Operational Pilot AV-Comparatives (3 Test Plans, 3 Tests) Mac, Mobile, Business SE Labs (1 Test Plan, 3 Tests) Endpoint (Consumer, SMB, Enterprise) SKD Labs (1 Test Plan, 1 Test) Endpoint (Consumer, China Market) AppEsteem/AV-C (1 Test Plan, Commissioned) Deceptor Test

Test Plan Structure Introduction Scope Methodology Participation Environment Schedule Control Procedures Dependencies Scoring Process Dispute Process Tester Attestations On-Line Change Submission Process http://amtso.org/amtso-standards-change-request-system/

Test Plan Feedback Initial Response (June 2017 – Start of Pilot) “Too heavyweight” “We already have all of these things” Subsequent Feedback (October 2017) Adds structure to discussions between Testers and Participants Collects needed information in a single document Adds transparency Enables better decision-making regarding Test Quality Continue work on streamlining and adding common cross-Tester content to Template Sign-off process between Commissioner and Tester requires additional work Test Plan commentary period requires configuration management

Vendor Attestations Sensitivity to Question Wording Solicited vs. Vendor-Sought Participation Disclosure of Third Party Components Need Fixed Window for Responses Proposed that 5 Business Days Be Allowed Default Response will be Test Support E-Mail Automation / Notifications

Voluntary / Involuntary Feedback Different Presentations Required < 2 Minutes for Voluntary Participants Meets the Posting Feedback Requirements

One-Page Test Review Summary Test Plan At-a-Glance Feedback Click-Through Interface to Drill-Down for Detailed Feedback Tester Report Links Compliance Report

Compliance Confirmation ~50 Checklist Items Compares all Test Artifacts against Standard Relaxed requirements for publishing links Operational Pilot Confirmed Compliant Tests AV-Comparatives 2017 Mac Security Review SE Labs Q3 2017 Endpoint Security Test AV-Comparatives 2017 Mobile Security Review AV-C : http://amtso.org/compliance-confirmation-av-comparatives/ SE Labs : http://amtso.org/amtso-compliance-confirmation-se-labs/

Assorted Additional Findings Auto-Generate Participant E-Mail Invitations Individual Surveys Per Participant Discoverability Search Engines over AMTSO Web Sites Password Protection Members-Only Web Site (AMTSO.NET) Introduce More Automation Manual Transformation of Attestation Data

Engaging with AMTSO Operational Pilot Process Tester Feedback

AV-Comparatives Feedback from Peter Stelzhammer SWG Meeting, October 10th Positive on AMTSO help in building initial test plans and navigating (defining) the whole process However AV-C not in acceptance of standards citing that still too Vendor friendly Cited that investment cost of two man-days per test is considered too high AMTSO Standards must accommodate “Test Series” Annual Notification, Monthly Execution AMTSO Compliance Certification Potentially Negatively received by his Consumer Base PR Push needed to raise profile of AMTSO compliance

SE Labs Feedback from Simon Edwards General Support of Mission E-Mail from Monday, October 23rd General Support of Mission Provide tangible benefits for Testers for following demands made by Vendors Some Assumptions Busted Different methods used among Testers Timelines not consistent between Testers Future Automation Process driven nature of the program should make it easier to semi-automate going forward Testing is so process driven that adding a few extra steps to link up with the AMTSO Standards Program should be realtively easy Eliminate as much manual intervention as possible Hold on … there’s more …

SE Labs Tester Engagement Vendor Bias Glad to see that multiple organizations were willing to be part of the Operational Pilot Announce Certification Program Widely Compliant Testers should advertise success Vendor Bias Be sure not to allow Vendors too much power in shaping the program Be sure to recognize that the Testers have power as you do not want them to ignore the program. “Great job, guys. It has been a genuine pleasure.” – Simon Edwards

AppEsteem Deceptor Feedback Feedback from Dennis Batchelder Notes from SWG Meeting, October 24th Commissioner vs Tester Role Clarification Rules for Notification of Revised Test Plan Strengthen Tester Recourse Against Aggressive Vendor Threats Publication of Final Participants Listing Feedback from Peter Stelzhammer E-Mail from Monday October 23rd Numerous Hours were Required

SKD Feedback Feedback from Jesse Song

Post-Operational Pilot AMTSO Full Operational Mode Post-Operational Pilot

Moving into 2018 Endorsement Working Group Handover Process Between Compliance and Endorsement Web Site Infrastructure for Automation Expanded Testing Organization Participation Public Disclosure of Confirmed Compliance and Endorsement Activities