1. AMTSO Standards Operational Pilot SUMMARY REPORT
AMTSO Standards Working Group
October 24, 2017 – Version 1.1
AMTSO Standards Operational Pilot SUMMARY REPORT

2. Agenda Artifacts and Key Findings Tester Feedback
Post-Operational Pilot

3. Artifacts and Key Findings
AMTSO Operational Pilot Process Execution
Artifacts and Key Findings

4. Test Plan Templates Multiple Test Plan Creation Tools Tested
Word Edition (Produces DOC/PDF Files)
On-Line Edition (Produces CSV Files)
Detailed Usage Guidelines
6 Test Plans Built During Operational Pilot
AV-Comparatives (3 Test Plans, 3 Tests)
Mac, Mobile, Business
SE Labs (1 Test Plan, 3 Tests)
Endpoint (Consumer, SMB, Enterprise)
SKD Labs (1 Test Plan, 1 Test)
Endpoint (Consumer, China Market)
AppEsteem/AV-C (1 Test Plan, Commissioned)
Deceptor Test

5. Test Plan Structure Introduction Scope Methodology Participation
Environment
Schedule
Control Procedures
Dependencies
Scoring Process
Dispute Process
Tester Attestations
On-Line Change Submission Process

6. Test Plan Feedback Initial Response (June 2017 – Start of Pilot)
“Too heavyweight”
“We already have all of these things”
Subsequent Feedback (October 2017)
Adds structure to discussions between Testers and Participants
Collects needed information in a single document
Adds transparency
Enables better decision-making regarding Test Quality
Continue work on streamlining and adding common cross-Tester content to Template
Sign-off process between Commissioner and Tester requires additional work
Test Plan commentary period requires configuration management

7. Vendor Attestations Sensitivity to Question Wording
Solicited vs. Vendor-Sought Participation
Disclosure of Third Party Components
Need Fixed Window for Responses
Proposed that 5 Business Days Be Allowed
Default Response will be Test Support
Automation / Notifications

8. Voluntary / Involuntary Feedback
Different Presentations Required
< 2 Minutes for Voluntary Participants
Meets the Posting Feedback Requirements

9. One-Page Test Review Summary
Test Plan
At-a-Glance Feedback
Click-Through Interface to Drill-Down for Detailed Feedback
Tester Report Links
Compliance Report

10. Compliance Confirmation
~50 Checklist Items
Compares all Test Artifacts against Standard
Relaxed requirements for publishing links
Operational Pilot Confirmed Compliant Tests
AV-Comparatives 2017 Mac Security Review
SE Labs Q Endpoint Security Test
AV-Comparatives 2017 Mobile Security Review
AV-C :
SE Labs :

11. Assorted Additional Findings
Auto-Generate Participant Invitations
Individual Surveys Per Participant
Discoverability
Search Engines over AMTSO Web Sites
Password Protection
Members-Only Web Site (AMTSO.NET)
Introduce More Automation
Manual Transformation of Attestation Data

12. Engaging with AMTSO Operational Pilot Process
Tester Feedback

13. AV-Comparatives Feedback from Peter Stelzhammer
SWG Meeting, October 10th
Positive on AMTSO help in building initial test plans and navigating (defining) the whole process
However AV-C not in acceptance of standards citing that still too Vendor friendly
Cited that investment cost of two man-days per test is considered too high
AMTSO Standards must accommodate “Test Series”
Annual Notification, Monthly Execution
AMTSO Compliance Certification
Potentially Negatively received by his Consumer Base
PR Push needed to raise profile of AMTSO compliance

14. SE Labs Feedback from Simon Edwards General Support of Mission
from Monday, October 23rd
General Support of Mission
Provide tangible benefits for Testers for following demands made by Vendors
Some Assumptions Busted
Different methods used among Testers
Timelines not consistent between Testers
Future Automation
Process driven nature of the program should make it easier to semi-automate going forward
Testing is so process driven that adding a few extra steps to link up with the AMTSO Standards Program should be realtively easy
Eliminate as much manual intervention as possible
Hold on … there’s more …

15. SE Labs Tester Engagement Vendor Bias
Glad to see that multiple organizations were willing to be part of the Operational Pilot
Announce Certification Program Widely
Compliant Testers should advertise success
Vendor Bias
Be sure not to allow Vendors too much power in shaping the program
Be sure to recognize that the Testers have power as you do not want them to ignore the program.
“Great job, guys. It has been a genuine pleasure.” – Simon Edwards

16. AppEsteem Deceptor Feedback
Feedback from Dennis Batchelder
Notes from SWG Meeting, October 24th
Commissioner vs Tester Role Clarification
Rules for Notification of Revised Test Plan
Strengthen Tester Recourse Against Aggressive Vendor Threats
Publication of Final Participants Listing
Feedback from Peter Stelzhammer
from Monday October 23rd
Numerous Hours were Required

17. SKD Feedback
Feedback from Jesse Song

18. Post-Operational Pilot
AMTSO Full Operational Mode
Post-Operational Pilot

19. Moving into 2018 Endorsement Working Group
Handover Process Between Compliance and Endorsement
Web Site Infrastructure for Automation
Expanded Testing Organization Participation
Public Disclosure of Confirmed Compliance and Endorsement Activities