Automated scoring of student trees Two models of algorithmic judgement.

Automated scoring of student trees Two models of algorithmic judgement. (how to tell if a tree is any good, without using your brain)

Last winter, some two hundred freshman biology students were asked to perform a simple task of sorting organisms to indicate how they are related to each other.

Last winter, some two hundred freshman biology students were asked to perform a simple task of sorting organisms to indicate how they are related to each other. The goal was to find out, first, how well they understood evolutionary relationships prior to taking Biology 101.

Last winter, some two hundred freshman biology students were asked to perform a simple task of sorting organisms to indicate how they are related to each other. The goal was to find out, first, how well they understood evolutionary relationships prior to taking Biology 101. The more interesting goal was to understand what sorts of mistakes they made, and to try to understand what they believe about phylogenetic relationships when they enter college.

Last winter, some two hundred freshman biology students were asked to perform a simple task of sorting organisms to indicate how they are related to each other. The goal was to find out, first, how well they understood evolutionary relationships prior to taking Biology 101. The more interesting goal was to understand what sorts of mistakes they made, and to try to understand what they believe about phylogenetic relationships when they enter college. Here are some of the trees they produced.

Some of the trees are pretty good...

Some of the trees are pretty good...

Some of the trees are pretty good...

... and some are not so good.

... and some are not so good.

... and some are not so good.

We’d like to analyze these trees to determine which ones show some understanding of the relevant classifications, and which do not.

We’d like to analyze these trees to determine which ones show some understanding of the relevant classifications, and which do not. Ideally, we’d like to do this without using expensive human brains to do the sorting.

We’d like to analyze these trees to determine which ones show some understanding of the relevant classifications, and which do not. Ideally, we’d like to do this without using expensive human brains to do the sorting. The following slides will show two ways to determine whether trees have organisms grouped into vertebrates and invertebrates, one based on graph relationships and the other based on the organisms’ locations on the screen.

Automated analysis of student trees based on graph relations: the shortest-path method.

How can we tell if organisms are well-grouped?

How can we tell if organisms are well-grouped? In a tree structure, a group is a set of nodes descended from a common ancestor.

How can we tell if organisms are well-grouped? In a tree structure, a group is a set of nodes descended from a common ancestor. This method makes use of this fact to determine group relationships based on nodes’ distances from one another on the graph.

How can we tell if organisms are well-grouped? In a tree structure, a group is a set of nodes descended from a common ancestor. This method makes use of this fact to determine group relationships based on nodes’ distances from one another on the graph. We begin by examining the distances between members of the same group – vertebrates – in a well-formed tree.

Distance from rat to human: 2 Distance from rat to bird: 4 Average distance between vertebrates: 6

Distance from rat to human: 2 Distance from rat to bird: 4 Average distance between vertebrates: 6 Correct groupings.

Distance from rat to human: 2 Distance from rat to bird: 4 Average distance between vertebrates: 6 Distance from rat to human: 2 Distance from rat to bird: 4 Average distance between vertebrates: 3 Correct groupings.

Next, we look at the distances between members of the vertebrates and the invertebrates on the tree.

Distance from rat to snail: 6 Distance from rat to bird: 4 Average distance between vertebrates: 6 Correct groupings.

Distance from rat to snail: 6 Distance from rat to beetle: 6 Average distance between vertebrates: 6 Correct groupings.

Distance from rat to snail: 6 Distance from rat to beetle: 6 Average distance from vertebrates to invertebrates: 6 Correct groupings.

Average distance from vertebrates to invertebrates: 6 Average distance between invertebrates: 3 Correct groupings.

Next, we look at the same distances for an incorrectly-structured tree.

Distance between rat and human: 6 One of these trees has the right idea.... And the other hasn’t Incorrect groupings.

Distance between rat and human: 6 Distance between rat and bird: 4 One of these trees has the right idea.... And the other hasn’t Incorrect groupings. Incorrect groupings.

Distance between rat and human: 6 Distance between rat and bird: 4 Average distance between vertebrates: 5.3 One of these trees has the right idea.... And the other hasn’t Incorrect groupings.

Distance from rat to beetle: 2 One of these trees has the right idea.... And the other hasn’t Incorrect groupings. Incorrect groupings.

Distance from rat to beetle: 2 Distance from rat to snail: 6 One of these trees has the right idea.... And the other hasn’t Incorrect groupings. Incorrect groupings.

Distance from rat to beetle: 2 Distance from rat to snail: 6 Average distance from vertebrate to invertebrate: 4.5 One of these trees has the right idea.... And the other hasn’t Incorrect groupings.

In a well-formed tree, the members of a group will be clustered under one branch of the tree.

In a well-formed tree, the members of a group will be clustered under one branch of the tree. This means that the average distance between members of a group will be smaller than the average distance between any member of the group and the non-group members.

In a well-formed tree, the members of a group will be clustered under one branch of the tree. This means that the average distance between members of a group will be smaller than the average distance between any member of the group and the non-group members. If a tree is not well-formed, the in-group distance and the out-of-group distances will be similar.

In a well-formed tree, the members of a group will be clustered under one branch of the tree. This means that the average distance between members of a group will be smaller than the average distance between any member of the group and the non-group members. If a tree is not well-formed, the in-group distance and the out-of-group distances will be similar. We can derive a grouping score by dividing the out-of-group distances by the in-group distances.

In a well-formed tree, the members of a group will be clustered under one branch of the tree. This means that the average distance between members of a group will be smaller than the average distance between any member of the group and the non-group members. If a tree is not well-formed, the in-group distance and the out-of-group distances will be similar. We can derive a grouping score by dividing the out-of-group distances by the in-group distances. A score of one indicates random placement, higher scores indicate greater clustering of vertebrates or invertebrates.

In a well-formed tree, the members of a group will be clustered under one branch of the tree. This means that the average distance between members of a group will be smaller than the average distance between any member of the group and the non-group members. If a tree is not well-formed, the in-group distance and the out-of-group distances will be similar. We can derive a grouping score by dividing the out-of-group distances by the in-group distances. A score of one indicates random placement, higher scores indicate greater clustering of vertebrates or invertebrates.

Incorrect groupings: out-of-group distance: 5.3 in-group distance: 4.5 Grouping score = 1.1 Correct groupings: out-of-group distance: 6 in-group distance: 3 Grouping score = 2

This works well, if the student is kind enough to connect all of the nodes together.

This works well, if the student is kind enough to connect all of the nodes together. But what do we do if the student's tree is not connected?

Using convex hulls to determine groupings purely from spatial relationships.

This is the “correct grouping” tree, with all of its connections removed.

Clearly, we can no longer count connections to establish groups.

However, the grouping information can be recovered by examining the convex hulls enclosing the organisms.

A convex hull is the smallest convex curve enclosing a set of points.

A convex hull is the smallest convex curve enclosing a set of points. If we draw the convex hulls surrounding two sets of points, for example vertebrates and invertebrates.

A convex hull is the smallest convex curve enclosing a set of points. If we draw the convex hulls surrounding two sets of points, for example vertebrates and invertebrates, we know that the points are separate groups if their hulls do not collide.

Invertebrates Invertebrates Vertebrates

A convex hull is the smallest convex curve enclosing a set of points. If we draw the convex hulls surrounding two sets of points, for example vertebrates and invertebrates, we know that the points are separate groups if their hulls do not collide. This gives us a simple test for whether groups are ideally separated or not.

A convex hull is the smallest convex curve enclosing a set of points. If we draw the convex hulls surrounding two sets of points, for example vertebrates and invertebrates, we know that the points are separate groups if their hulls do not collide. This gives us a simple test for whether groups are ideally separated or not. But what if the groups are only a little bit mixed?

A convex hull is the smallest convex curve enclosing a set of points. If we draw the convex hulls surrounding two sets of points, for example vertebrates and invertebrates, we know that the points are separate groups if their hulls do not collide. This gives us a simple test for whether groups are ideally separated or not. But what if the groups are only a little bit mixed? If we can figure out the minimum number of nodes we have to remove to eliminate the collision

A convex hull is the smallest convex curve enclosing a set of points. If we draw the convex hulls surrounding two sets of points, for example vertebrates and invertebrates, we know that the points are separate groups if their hulls do not collide. This gives us a simple test for whether groups are ideally separated or not. But what if the groups are only a little bit mixed? If we can figure out the minimum number of nodes we have to remove to eliminate the collision We have an indication of how badly the two groups are mixed.

A convex hull is the smallest convex curve enclosing a set of points. If we draw the convex hulls surrounding two sets of points, for example vertebrates and invertebrates, we know that the points are separate groups if their hulls do not collide. This gives us a simple test for whether groups are ideally separated or not. But what if the groups are only a little bit mixed? If we can figure out the minimum number of nodes we have to remove to eliminate the collision We have an indication of how badly the two groups are mixed. Here’s how our collision elimination algorithm does that:

First, the convex hulls are identified First, the convex hulls are identified. The area of overlap is shaded in red.

Next, the organisms are identified for elimination.

The algorithm seeks the node closest to the centroid of the collision area, and removes it.

First, the convex hulls are identified First, the convex hulls are identified. The area of overlap is shaded in red.

The process is repeated until there is no area of collision.

The misplaced nodes are highlighted, identifying the problem areas for the student. Further analysis can identify patterns of mistakes, showing areas needing attention from teachers of biology.