1. NMFS Use Case 1 review/ evaluation and next steps
April 19, 2012
Woods Hole, MA
Peter Fox (RPI* and WHOI**) and Andrew Maffei (WHOI)
*Tetherless World Constellation, ** AOP&E

2. Modern informatics enables a new scale-free framework approach
Use cases
Stakeholders
Distributed authority
Access control
Ontologies
Maintaining Identity
Look at scale free network definitions in wikipedia, web, semantic networks. (

3. What’s happened since .. Implementation Now Evaluation
Technology assessment
Leverage infrastructure
Rapid prototyping
Now
Evaluation
Open world, iteration

4. Review and evaluation

5. References Twidale, Randall and Bentley (1994) and references therein
Twidale, M., Randall, D. and Bentley, R. 1994, Situated evaluation for cooperative systems, Proceedings, Comp. Supp. Coop. Work 1994, Chapel Hill, NC, pp

6. Metrics Things you can measure (numerical) Things that are categorical
Could not do before
Faster, more complete, less mistakes, etc.
Wider range of users
Measure or estimate the baseline before you start

7. Result/ outcome We will refer to the use case document
Outcome (and value of it) is a combination of data gathering processes, that may include surveys, interviews, focus groups, document analysis and observations that will yield both qualitative and quantitative results. I.e. a discussion (today)
Did we meet the goal?

8. Example: what we wanted to know about VSTO
Evaluation questions are used to determine the degree to which the VSTO enhanced search, access, and use of data for scientific and educational needs and effectively utilized and implemented a template for user-centric utilization of the semantic web methodology.
VO – appears to local and integrated and in the end-users language (this is one of the metrics)

9. Evaluation (Twidale et al.)
An assessment of the overall effectiveness of a piece of software, ideally yielding a numeric measure by which informed cost-benefit analysis of purchasing decisions can be made.
An assessment of the degree to which the software fulfils its specification in terms of functionality, speed, size or whatever measures were pre-specified.
Role is singular

10. Evaluation
An assessment of whether the software fulfils the purpose for which it was intended.
An assessment of whether the ideas embodied in the software have been proved to be superior to an alternative, where that alternative is frequently the traditional solution to the problem addressed.
An assessment of whether the money allocated to a research project has been productively used, yielding useful generalizeable results.
Role is singular

11. Evaluation
An assessment of whether the software proves acceptable to the intended end-users.
An assessment of whether end-users continue to use it in their normal work.
An assessment of where the software fails to perform as desired or as is now seen to be desirable.
An assessment of the relative importance of the inadequacies of the software.
Role is singular

12. (Orthogonal) Dimensions of evaluations
Structured
Less structured
Quantitative
Qualitative
Summative
Formative
Controlled experiments
Ethnographic observations
Formal and rigorous
Informal and opportunistic

13. Formative and Summative
evaluation carried out for two reasons:
grading translations = summative evaluation (when the guests taste the soup)
giving feedback = formative evaluation (when the cook tastes the soup)

14. Iterating Evolve, iterate, re-design, re-deploy Small fixes
Full team aware of the evaluation results and implications
Decide what to do about the new use cases, or if the goal is not met
Determine what knowledge engineering is required and who will do it (participants in the evaluation may become domain experts)
Determine what new knowledge representation
Assess need for an architectural re-design

15. Summary Project evaluation has many attributes
Structured and less-structured
Really need to be open to all forms
A good way to start is to get members of your team to do peer evaluation
This is a professional exercise, treat it that way at all times
Other possible techniques for moving forward on evolving the design, what to focus upon, priorities, etc.: SWOT, Porter’s 5 forces

16. Summary
By now, the reality of going into complete detail for the prototype implementation should be apparent
Keeping it simple is also very important as you begin to implement
Being prepared to iterate is essential
Now is the time to validate the models with domain experts and the team

17. Questions?

18. Back shed

19. Use case!

20. Use Case
… is a collection of possible sequences of interactions between the system under discussion and its actors, relating to a particular goal.
The collection of Use Cases should define all system behavior relevant to the actors to assure them that their goals will be carried out properly.
Any system behavior that is irrelevant to the actors should not be included in the use cases.
is a prose description of a system's behavior when interacting with the outside world.
is a technique for capturing functional requirements of business systems and, potentially, of an ICT system to support the business system.
can also capture non-functional requirements

21. Developed for NASA TIWG
Use case format
Use case name
Goal
Summary
Triggers
Basic flow
Alternate flow
Post conditions
Activity diagram
Preconditions in tabular form
Notes
Developed for NASA TIWG