Measuring Team Shared Understanding: Using Analysis-Constructed Shared Mental Model Methodology Tristan E. Johnson Learning Systems Institute, Florida State University, Tallahassee, FL, USA tjohnson@lsi.fsu.edu International Workshop and Mini-conference on Extending Cognitive Load Theory and Instructional Design to the Development of Expert Performance August 29-30, 2005 Open University of the Netherlands

Background Team Performance Team Cognition Link between SMM and Team Performance Shared Understanding and Shared Mental Models Over the last decade, interest in the concept of shared mental model in teams has grown dramatically (Cannon-Bowers & Salas, 2001; Klimoski & Mohammed, 1994; Rentsch & Heffner, & Duffy (1994). This interest undoubtedly stems from the tremendous potential value the concept has with regard to team performance. Shared mental model in teams not only servers as an explanatory mechanism (explaining differences between effective and ineffective teams), but it also has the potential to be a valuable predictor of team performance (Cannon-Bowers & Salas, 2001). Development of SMM and its relation to team performance

Team Cognition Elaborated view of team cognition including team interactions and SMM development

Shared Knowledge Types Task Knowledge—domain specific Team Knowledge—5 factors

Team Knowledge Factors Knowledge about team members and tasks that they need to perform Teammates knowledge, Task knowledge Team Skills Abilities associated with successful job performance Communication skills, Interpersonal skills, Leadership skills, Skills to deal with conflict and team cohesion Team Attitudes Internal state that influences team members’ choices or decision to act in a certain way under particular circumstances Shared belief, Shared value Team Dynamics Combination of dynamic processes of team coordination and team cohesion Team coordination, Team cohesion Team Environment External conditions affecting the foundation of the team mental model Technology, Organization, Synchrony & Geographic dispersion

Measuring Task Knowledge Measuring Shared Understanding—measuring concept relatedness Card sorting, cognitive interviewing, MDS, Pathfinder, surveys, casual maps (Langan-Fox, Code, Langfield-Smith, 2000; Trochim, 1989) Concept Mapping Statistical analysis Descriptive analysis Analysis Constructed - Shared Mental Model (AC-SMM)

SMM Elicitation Techniques TmC-SMM—Whole team elicitation (1 map) AC-SMM—Individual elicitation with aggregation (n maps) Alternative methods of measuring team shared mental models. ICMM—Individually Constructed Mental Models; SMM i— desired shared mental model state TmC-SMM — involves team negotiation and interaction SMM∂— altered team shared mental model state AC-SMM —retains the initial ICMM state

AC-SMM Methodology Rationale Knowledge Elicitation Process allows simultaneous consideration of concepts Reflection and changes during elicitation Analysis Allows for explication of implicit relationships—considering 1) logic, 2) structure, and 3) spatial orientation Relatedness Specific to three levels Concepts Links Clusters Appropriate for studying shared understanding in applied settings

AC-SMM Methodology Overview Instrument Design Structured/Semi-Structured/Unstructured Task Analysis (Generate Concepts) Data Collection Guided Practice Individually Constructed Mental Model (ICMM) Elicitation Data Analysis Phase I: ICMM Analysis/Coding Relatedness at concepts, links, clusters levels Allows for explication of implicit relationships Implicit coding has [logic and spatial] or [logic and structural] support Phase II: Shared Analysis Determine sharedness level—number or percentage of team members Phase III: AC-SMM Construction Generates SMM

Phase I: ICMM Analysis Factor 1: Concepts Factor 2: Links Explicit individual nodes Factor 2: Links Two concepts joined explicitly [connector] or implicitly Factor 3: Clusters Two or more connectors explicitly bridging three or more concepts May have implicit connections with evidence Combination of clusters—Sub- and Super- clusters Factor 4: Emphasis and Sequence Explicit notation of node emphasis or node order

ICMM Coding—Links

ICMM Coding—Clusters

ICMM Coding—Emphasis and Sequence

ICMM Coding Example

Phase II: Shared Analysis Determine Sharedness Level Criterion—Number or Percentage Shared Data Used for AC-SMM Construction

Phase III: AC-SMM Construction

Research General Research Focus What task knowledge is shared? How does shared understanding change over time? What are the patterns of change? What is the affect of task performance on the shared understanding of the team?

Data Collection Timeline

Concepts

Team Profiles Findings

Shared Data Findings, Team 1 Only

Shared Data Summary Per Team

Cross Case Findings

Pre, Mid, Post Analysis As teams engage in cognitive activities, we would expect to see improvements in individual mental models. In order to show this change, pre and post AC-SMMs would be analyzed to show a comparison between the mental models. This analysis has an emphasis on how things are different rather than the shared focus emphasis that we have described so far. By comparing these differences with other measures such as performance, efficacy, or communication, we can start to understand if indeed the change is related to specific types of cognitive activity. Based on initial findings?, as teams work together, the similarity among ICMMs tends to increase as does the number of clustered concepts, even though the tendency is for the number of concepts used to decrease. These factors provide evidence that ICMMs were becoming more structured and more representative of the team task in addition to becoming more similar to the ICMMs of other team members. These ideas are not yet proven. We have designed a set of studies to try and validate our hypothesis. Currently, we are looking at concept maps collected from three content domains: performance improvement (performance standards development), instructional systems (formative evaluation), and science education (mentoring). This work is intended to not only learn about teams that work in the various settings, but to validate the AC-SMM analysis model as delineated in this article. The use of qualitative analysis we hope provides a richer description of the detail included in the AC-SMM than would have been found with other qualitative and quantitative methods. However, this methodology lacked the weighted measures and precise distances between concepts in the resulting AC-SMM maps as is found in shared maps generated using quantitative methods such as Pathfinder or MDS. Also, as is often the case in concept mapping, there was a lack of prepositional descriptors to define the exact relationship between concepts in ICMMs, requiring the rater to engage in a more exhaustive analysis procedures that are based on other supportive data. In our current studies, we have supportive data from the non-participant researcher observations to support many of these assumptions and decisions. Most importantly, we are in the process of validating the idea that the AC-SMM is a more accurate representation of the SMMi. In addition, we are able to create this shared mental model with minimal disturbance to the team’s cognitive activities. Future steps we are considering include the combination of the AC-SMM methodology with quantitative analysis. This could provide the weighted measures needed for greater precision in the resulting team concept maps in addition to the qualitative descriptions representing fluctuations in team cognition. Once we have a more precise and descriptive analysis of shared mental models, we can utilize the new knowledge to better describe, explain, and understand team cognition. We can also use this deeper understanding about the development of team mental models for determining how to train team members in developing shared mental models. This in turn will facilitate team training with this intent to improve team performance outcomes.

ACSMM Scores

General Findings Similarity among ICMMs tends to increase as does the number of clustered concepts, the tendency is for the number of concepts used to decrease. ICMMs were becoming more structured and more representative of the team task These ideas are not yet proven. We have designed a set of studies to try and validate our hypothesis This work is intended to not only learn about teams that work in the various settings, but to validate the AC-SMM analysis model

Summary Richer description of shared understanding in teams AC-SMMs compared over time to determine change in shared understanding Lacks weighted measures and precise distances between concepts, but future work will include descriptive statistics of the key factors Lack of prepositional descriptors As we become more precise and descriptive we can utilize this new knowledge to better explain and understand team cognition Facilitate team training with intent to improve team performance outcomes

Thanks for your attention. Questions?

Findings and Extra Slides

Findings Across Pre, Mid, Post

Participants & Context Personnel Qualification Standards (PQS) Team Team Task Team Members Team Member Roles Context U.S. Navy Training Center, Pensacola, FL Face to face workshop Equipment

Findings From AC-SMM Analysis What is shared? Does shared understanding change over time?

Secondary Analysis Sequence Links Clusters Where concepts were placed within each ICMM Focus on key concepts Team member roles Sections of PQS book Referencing Questions Links Relationships between concepts without directionality Clusters Commonalities between related clusters of concepts

with Secondary Clusters [B,D,E,I] and [D,E,I] Secondary Analysis Links - Relationships between concepts without directionality Clusters - Relationships between concepts without directionality and commonalities between related clusters of concepts Example of Cluster [[B,D], [B,E], [B,I]] with Secondary Clusters [B,D,E,I] and [D,E,I] without Related Links [D,E], [D,I], and [E,I]

All Data, Shared by ≥ 2

All Data, Shared by ≥ 50%

Follow-up Analysis Sequence Started with original data submitted by each team member Where concepts were placed within each ICMM Focus on key concepts Team member roles (concepts [D], [E], [I]) Sections of PQS book (concepts [J], [K], [L]) Referencing (concepts [H], [O], [S]) Questions (concepts [N], [Q], [T])

All Data, Shared by ≥ 2 Secondary Analysis

All Data, Shared by ≥ 50% Secondary Analysis

Complete Datasets, Shared by ≥ 50%

Complete Datasets, Shared by ≥ 2 Secondary Analysis

Complete Datasets, Shared by ≥ 50% Secondary Analysis