1. Measuring Team Shared Understanding: Using Analysis-Constructed Shared Mental Model Methodology
Tristan E. Johnson
Learning Systems Institute, Florida State University, Tallahassee, FL, USA
International Workshop and Mini-conference on
Extending Cognitive Load Theory and Instructional Design
to the Development of Expert Performance
August 29-30, 2005
Open University of the Netherlands

2. Background Team Performance
Team Cognition
Link between SMM and Team Performance
Shared Understanding and Shared Mental Models
Over the last decade, interest in the concept of shared mental model in teams has grown dramatically (Cannon-Bowers & Salas, 2001; Klimoski & Mohammed, 1994; Rentsch & Heffner, & Duffy (1994). This interest undoubtedly stems from the tremendous potential value the concept has with regard to team performance. Shared mental model in teams not only servers as an explanatory mechanism (explaining differences between effective and ineffective teams), but it also has the potential to be a valuable predictor of team performance (Cannon-Bowers & Salas, 2001).
Development of SMM and its relation to team performance

3. Team Cognition Elaborated view of team cognition including
team interactions and SMM development

4. Shared Knowledge Types
Task Knowledge—domain specific
Team Knowledge—5 factors

5. Team Knowledge Factors
Knowledge about team members and tasks that they need to perform
Teammates knowledge, Task knowledge
Team Skills
Abilities associated with successful job performance
Communication skills, Interpersonal skills, Leadership skills, Skills to deal with conflict and team cohesion
Team Attitudes
Internal state that influences team members’ choices or decision to act in a certain way under particular circumstances
Shared belief, Shared value
Team Dynamics
Combination of dynamic processes of team coordination and team cohesion
Team coordination, Team cohesion
Team Environment
External conditions affecting the foundation of the team mental model
Technology, Organization, Synchrony & Geographic dispersion

6. Measuring Task Knowledge
Measuring Shared Understanding—measuring concept relatedness
Card sorting, cognitive interviewing, MDS, Pathfinder, surveys, casual maps (Langan-Fox, Code, Langfield-Smith, 2000; Trochim, 1989)
Concept Mapping
Statistical analysis
Descriptive analysis
Analysis Constructed - Shared Mental Model (AC-SMM)

7. SMM Elicitation Techniques
TmC-SMM—Whole team elicitation (1 map)
AC-SMM—Individual elicitation with aggregation (n maps)
Alternative methods of measuring team shared mental models. ICMM—Individually Constructed Mental Models;
SMM i— desired shared mental model state
TmC-SMM — involves team negotiation and interaction
SMM∂— altered team shared mental model state
AC-SMM —retains the initial ICMM state

8. AC-SMM Methodology Rationale
Knowledge Elicitation
Process allows simultaneous consideration of concepts
Reflection and changes during elicitation
Analysis
Allows for explication of implicit relationships—considering 1) logic, 2) structure, and 3) spatial orientation
Relatedness
Specific to three levels
Concepts
Links
Clusters
Appropriate for studying shared understanding in applied settings

9. AC-SMM Methodology Overview
Instrument Design
Structured/Semi-Structured/Unstructured
Task Analysis (Generate Concepts)
Data Collection
Guided Practice
Individually Constructed Mental Model (ICMM) Elicitation
Data Analysis
Phase I: ICMM Analysis/Coding
Relatedness at concepts, links, clusters levels
Allows for explication of implicit relationships
Implicit coding has [logic and spatial] or [logic and structural] support
Phase II: Shared Analysis
Determine sharedness level—number or percentage of team members
Phase III: AC-SMM Construction
Generates SMM

10. Phase I: ICMM Analysis Factor 1: Concepts Factor 2: Links
Explicit individual nodes
Factor 2: Links
Two concepts joined explicitly [connector] or implicitly
Factor 3: Clusters
Two or more connectors explicitly bridging three or more concepts
May have implicit connections with evidence
Combination of clusters—Sub- and Super- clusters
Factor 4: Emphasis and Sequence
Explicit notation of node emphasis or node order

11. ICMM Coding—Links

12. ICMM Coding—Clusters

13. ICMM Coding—Emphasis and Sequence

14. ICMM Coding Example

15. Phase II: Shared Analysis
Determine Sharedness Level Criterion—Number or Percentage
Shared Data Used for AC-SMM Construction

16. Phase III: AC-SMM Construction

17. Research General Research Focus What task knowledge is shared?
How does shared understanding change over time?
What are the patterns of change?
What is the affect of task performance on the shared understanding of the team?

18. Data Collection Timeline

19. Concepts

20. Team Profiles Findings

25. Shared Data Findings, Team 1 Only

28. Shared Data Summary Per Team

37. Cross Case Findings

40. Pre, Mid, Post Analysis
As teams engage in cognitive activities, we would expect to see improvements in individual mental models. In order to show this change, pre and post AC-SMMs would be analyzed to show a comparison between the mental models. This analysis has an emphasis on how things are different rather than the shared focus emphasis that we have described so far. By comparing these differences with other measures such as performance, efficacy, or communication, we can start to understand if indeed the change is related to specific types of cognitive activity.
Based on initial findings?, as teams work together, the similarity among ICMMs tends to increase as does the number of clustered concepts, even though the tendency is for the number of concepts used to decrease. These factors provide evidence that ICMMs were becoming more structured and more representative of the team task in addition to becoming more similar to the ICMMs of other team members. These ideas are not yet proven. We have designed a set of studies to try and validate our hypothesis. Currently, we are looking at concept maps collected from three content domains: performance improvement (performance standards development), instructional systems (formative evaluation), and science education (mentoring). This work is intended to not only learn about teams that work in the various settings, but to validate the AC-SMM analysis model as delineated in this article.
The use of qualitative analysis we hope provides a richer description of the detail included in the AC-SMM than would have been found with other qualitative and quantitative methods. However, this methodology lacked the weighted measures and precise distances between concepts in the resulting AC-SMM maps as is found in shared maps generated using quantitative methods such as Pathfinder or MDS. Also, as is often the case in concept mapping, there was a lack of prepositional descriptors to define the exact relationship between concepts in ICMMs, requiring the rater to engage in a more exhaustive analysis procedures that are based on other supportive data. In our current studies, we have supportive data from the non-participant researcher observations to support many of these assumptions and decisions. Most importantly, we are in the process of validating the idea that the AC-SMM is a more accurate representation of the SMMi. In addition, we are able to create this shared mental model with minimal disturbance to the team’s cognitive activities.
Future steps we are considering include the combination of the AC-SMM methodology with quantitative analysis. This could provide the weighted measures needed for greater precision in the resulting team concept maps in addition to the qualitative descriptions representing fluctuations in team cognition. Once we have a more precise and descriptive analysis of shared mental models, we can utilize the new knowledge to better describe, explain, and understand team cognition. We can also use this deeper understanding about the development of team mental models for determining how to train team members in developing shared mental models. This in turn will facilitate team training with this intent to improve team performance outcomes.

41. ACSMM Scores

42. General Findings
Similarity among ICMMs tends to increase as does the number of clustered concepts, the tendency is for the number of concepts used to decrease.
ICMMs were becoming more structured and more representative of the team task
These ideas are not yet proven.
We have designed a set of studies to try and validate our hypothesis
This work is intended to not only learn about teams that work in the various settings, but to validate the AC-SMM analysis model

43. Summary Richer description of shared understanding in teams
AC-SMMs compared over time to determine change in shared understanding
Lacks weighted measures and precise distances between concepts, but future work will include descriptive statistics of the key factors
Lack of prepositional descriptors
As we become more precise and descriptive we can utilize this new knowledge to better explain and understand team cognition
Facilitate team training with intent to improve team performance outcomes

44. Thanks for your attention. Questions?

56. Findings and Extra Slides

57. Findings Across Pre, Mid, Post

58. Participants & Context
Personnel Qualification Standards (PQS) Team
Team Task
Team Members
Team Member Roles
Context
U.S. Navy Training Center, Pensacola, FL
Face to face workshop
Equipment

59. Findings From AC-SMM Analysis
What is shared?
Does shared understanding change over time?

60. Secondary Analysis Sequence Links Clusters
Where concepts were placed within each ICMM
Focus on key concepts
Team member roles
Sections of PQS book
Referencing
Questions
Links
Relationships between concepts without directionality
Clusters
Commonalities between related clusters of concepts

61. with Secondary Clusters [B,D,E,I] and [D,E,I]
Secondary Analysis
Links - Relationships between concepts without directionality
Clusters - Relationships between concepts without directionality and commonalities between related clusters of concepts
Example of Cluster
[[B,D], [B,E], [B,I]]
with Secondary Clusters [B,D,E,I] and [D,E,I]
without Related Links
[D,E], [D,I], and [E,I]

62. All Data, Shared by ≥ 2

63. All Data, Shared by ≥ 50%

64. Follow-up Analysis Sequence
Started with original data submitted by each team member
Where concepts were placed within each ICMM
Focus on key concepts
Team member roles (concepts [D], [E], [I])
Sections of PQS book (concepts [J], [K], [L])
Referencing (concepts [H], [O], [S])
Questions (concepts [N], [Q], [T])

65. All Data, Shared by ≥ 2 Secondary Analysis

66. All Data, Shared by ≥ 50% Secondary Analysis

67. Complete Datasets, Shared by ≥ 50%

68. Complete Datasets, Shared by ≥ 2 Secondary Analysis

69. Complete Datasets, Shared by ≥ 50% Secondary Analysis