ATML Test Description Orlando, FL January 2007

ATML Test Description2 September 2006 Agenda Status Feedback from Candidate evaluation Review standard draft Discuss examples to be included in Annex A Schedule for review and ballot

ATML Test Description3 September 2006 Status Created Draft 1 and Draft 2 (internal review) Completed review At ATML Meeting  Discuss document contents & structure  Discuss contents of Annex A Remaining work  Incorporate changes in Draft 3, post for public review  Create Annex A, include in Draft 4, post for public review  Continue Candidate review, collect feedback, update schemas & standard

ATML Test Description4 September 2006 Feedback from Candidate evaluation Web Services using elements from Test Description & Test Results (prototyping)  No problem Using Test Description schema for functional testing (analysis)  A Test may verify multiple UUT characteristics this may result in multiple Outcomes; all the Outcomes may be used for a diagnostic decision  Possible solutions: Support for multiple outcomes for tests, steps, etc. Use “combination” outcomes Use schema extensions (would occur in multiple places) Current design is consistent with Test Results and Test Results schema supports recording of individual measurements & limits; “partial” outcomes can be inferred if needed. Test Description schema supports the description of Test Results and limits. This appears to be sufficient. No change.

ATML Test Description5 September 2006 Standard Draft - Structure Overview Normative references Definitions, acronyms and abbreviations Test Description schema Conformance Extensibility Annex A (informative)– Users information and examples Annex B (informative) – Glossary Annex C (informative) - Bibliography

ATML Test Description6 September 2006 Standard Draft – Structure… Test Description schema  Elements Root element Child elements (expanded down to XML schema types, Common types and schema- defined types)  Attribute Groups  Simple Types  Complex Types (organized by functionality); Sort alphabetically? Or describe functional classification? Add class diagram? Type Child elements (expanded down to XML schema types, Common types and schema- defined types) Sort types alphabetically Include classification in introductory subsection; for each type, provide reference to the section describing it (should be converted into PDF hyperlink!)

ATML Test Description7 September 2006 Standard Draft – Structure… For each element and complex type (as applicable):  Description  Attributes Name Type Description  Child elements Name Type Description Use – make sure cardinality is recorded where applicable (ex. max 2 elements) Postfix names of abstract types with “(abstract)”  Description of “choice” situations Use typographic convention for elements that belong to a choice; is there a quasi-standard convention?  References to Common types Description of constraints not enforced by schema (in descriptive text of attributes or elements) Describe uniqueness and referential integrity constraints  In specialized table (for the element where they are defined), or  In descriptive text of attributes or elements  Both

ATML Test Description8 September 2006 Standard Draft - Contents Power requirements – can be described in two formats  1. Under GeneralData TRD-inspired format (DC Power Requirements and AC Power Requirements)  Ripple under DCPowerRequirement – is this amplitude or RMS?  What is PhaseReference under ACPowerRequirement? Adequate for documentation  2. Under SignalRequirements (along with stimuli & measurements) IEEE 1641 signal format Adequate for programmatic use Not all characteristics of Format 1 are supported (ex. current capability of voltage source) If both descriptions are used, they must be consistent; this is not enforced by the schema Keep both formats? Remove Format 1 if format 2 can accommodate all data (Matt will investigate)

ATML Test Description9 September 2006 Standard Draft – Contents… PerformanceCharacteristics  Descriptive text includes “The description shall contain sufficient detail that any "black box" with the characteristics described could be inserted in the system without degrading the system performance.” Keep this? YES  Descriptive text includes “Also used to assure consistency at all levels of maintenance as well as compatibility between maintenance and QA test results.” Not sure what this means.  Inputs/Input/Characteristics – these are acceptable values; different word?  Outputs/Output/Characteristics – these are expected values; or guaranteed?

ATML Test Description10 September 2006 Standard Draft – Contents… Initialization  Descriptive text includes “A “Failed” outcome returned by the referenced test indicates that initialization has failed. When this situation occurs, the execution of the test group should not continue.” OK? Yes Termination  Descriptive text includes “A “Failed” outcome returned by the referenced test indicates that termination has failed. When this situation occurs, the behavior of the test program may be application-specific. For example, the test program may reset the instrumentation, to bring it in a known state.” OK? Yes

ATML Test Description11 September 2006 Standard Draft – Annex A Available sample instance documents – may be used to build examples:  Basic example: sequence, tests, outcomes, parameters, test results  SequenceCall: calling a sequence from another sequence  FaultExample: fault detection and isolation  FailureExample: failure detection and isolation  Params: passing data between tests via parameters  PrePostConditions: usage of Pre-conditions and Post-conditions  StdSignals: use of IEEE 1641 signals as test parameters  PortExample: use of signal ports as test parameters  GlobalSignals: signals that exist for the duration of multiple tests  Actions: use of actions to describe test behavior  ActionsRepeatAndConditional: use of ActionRepeat and ActionConditional

ATML Test Description12 September 2006 Standard Draft – Annex A… Proposed examples:  One or more use cases for extensions (José)  Examples showing all data types (José)  Sequence call example with balanced tree (José)  Test group with different values from parameters; using test group parameters in tests and actions (José)  Extend Basic Example to include more UUT Description, General Data, Signal Requirements & Performance Characteristics data (Ion) Would be nice to use a real(istic) UUT. Do we have to include the complete XML in the specification? Or just make it avilable on the IEEE web site? This determines the complexity of the examples. Jose will look into this. Can have sample instance documents on IEEE web site; not required to have them in spec (they are pretty useless as the XML cannot be transferred in a file, for use w/ tools). Will probably have short snippets where appropriate. Look at 1671 Annex A for a disclaimer applicable to examples!

ATML Test Description13 September 2006 Basic example  Test, Outcomes  Parameters Using data types (may have a few types in XML example; should have all exemplified through XML snippets in document)  Test Results & Limits Using data types  TestGroup – sequence  Faults  Calling sequence from another sequence Example 1: Steps reference Test Groups; 3 levels (Root Test Group, called Test Groups; called Tests Example 2 (maybe in Actions instance document): “TestGroupCall” Behavior; parameterized; 2 calls w/ different actual parameters  Conditions  Passing data between Tests via parameters  Passing parametric (numeric) data between Test Group and Test  Extensibility – using Possibly: a custom format for describing Test Behavior

ATML Test Description14 September 2006 IEEE 1641 signals example  Signals as Test Parameters  BSC & TSF (from 1641 example TSF Library)  Ports as Test Parameters  Measurements as Test Results  Global Signals  Extensibility – custom TSF library Actions example  Signal Actions  Repeat & Conditional  MessageIn & MessageOut  Passing parametric data (signal definitions) between Test Group, Test and Actions  Extensibility – type derivation

ATML Test Description15 September 2006 Schedule Remaining work  Incorporate changes in Draft 3, post for public review  Create Annex A, include in Draft 4, post for public review (for Madrid)  Continue Candidate review, collect feedback, update schemas & standard Schedule  Finalize candidate evaluation Use of capabilities in conjunction with Test Equipment capabilities  Finalize standard review  Ballot Tentative schedule for balloting other ATML components .3 and.4 by April '07 .2 by May '07 .5 and.6 by July '07