Western EcoSystems Technology (WEST), Inc. Comparison of Full-Spectrum and Zero-Crossing Automated Bat Call Classifiers Donald Solick, Matthew Clement, Kevin Murray, Christopher Nations, and Jeffery Gruver Western EcoSystems Technology (WEST), Inc. Good day!

Identification of bat echolocation calls to species is becoming increasingly important, particularly for assessing the risk posed to bats by wind-energy development and for monitoring the spread of White-Nose Syndrome. However, acoustic monitoring typically yields large amounts of data, and manually classifying calls can be subjective and time-consuming. To make these analyses more objective and manageable, several software programs have been developed in recent years that perform automatic classification of calls to species.

Full-Spectrum (FS) Zero-Crossing (ZC) Time and Frequency Amplitude Multiple frequency content -harmonics, multiple bats, calls against background noise Zero-Crossing (ZC) Time and Frequency Dominant frequency content -loudest sound gets recorded Acoustic data for bat studies come in two main flavors. Full-spectrum data—which you might record on a Pettersson detector--are a digitized representation of the full acoustic soundscape, and provide information on time, frequency, and amplitude of a bat call, as well as multiple frequency content. That is, all competing sounds such as harmonics, multiple bats, or background noise are fully rendered. In contrast, zero-crossing data that you might record on an Anabat provide just time and frequency information, and only render the dominant frequency –or loudest sound—so that harmonics, other bats, or the calls themselves may be incomplete or obscured.

More information = better species discrimination Assumption More information = better species discrimination ? Objective To determine which type of classifier is better at species discrimination given the same set of known calls The fact that full-spectrum data contain more information has led to a prevailing notion that these data are better suited for discriminating species than are zero-cross data. However, this has never been tested. Therefore, our objective was to determine—given the same set of known calls—whether full-spectrum automated classifiers are better at species discrimination than existing zero-cross methods.

FS Reference Calls Perimyotis subflavus PESU 97 Lasiurus borealis LABO Species Abbreviation Number of files Perimyotis subflavus PESU 97 Lasiurus borealis LABO 40 Eptesicus fuscus EPFU 115 Lasionycteris noctivagans LANO 23 Lasiurus cinereus LACI 18 Myotis leibii MYLE Myotis septentrionalis MYSE 46 Myotis lucifugus MYLU 263 Myotis sodalis MYSO 26 We addressed this objective by obtaining 652 full-spectrum reference calls for nine eastern US bat species. We used only high quality files by hand-released bats that contained a minimum of 5 calls and were free of interlopers or other extraneous noise.

SonoBat 3.04 Northeast Mean Classification: average of all calls in file By Vote: majority of calls in file Consensus: when Class and Vote agree We first analyzed calls using SonoBat Northeast, which is currently the only software available for automated analysis of full-spectrum data in the US. Sonobat reports three types of classification output:

SonoBat 3.04 Northeast Mean Classification: average of all calls in file By Vote: majority of calls in file Consensus: when Class and Vote agree x = Lano Mean Classification makes decisions based on the average parameter values for all calls within a file; Lano 0.9991

SonoBat 3.04 Northeast Mean Classification: average of all calls in file By Vote: majority of calls in file Consensus: when Class and Vote agree By Vote looks at each call and determines which species had the majority of calls within the file; Lano 4 of 5 Lano Lano Epfu Lano Lano

SonoBat 3.04 Northeast Mean Classification: average of all calls in file By Vote: majority of calls in file Consensus: when Class and Vote agree Lano 4 of 5 and Consensus classifies a file as a species when both the Mean Classification and By Vote decisions agree. Lano Lano 0.9991

ZC Classifiers Bat Classification and Identification (BCID) East v2.4mAC www.batcallid.com EchoClass 64 v1 www.fws.gov/midwest/Endangered/mammals/inba/inbasummersurveyguidance.html Discriminant Function Analysis for New York Developed by Eric Britzke for use by NY Dept.of Environmental Conservation The zero-cross classifiers we tested were BCID, EchoClass, and a Discriminant Function Analysis developed for use by the New York Department of Environmental Conservation.

Converting FS to ZC Anabat Converter 0.8 (http://bertrik.sikken.nl/anabat/) AnalookW 3.8e Applied filter and extracted parameters Except EchoClass We converted the full-spectrum files to zero-cross data for use in the zero-cross classifiers using Anabat Converter 0.8. Using AnalookW, we then applied a general filter to files and extracted call parameters for use in BCID and the NY DFA. This step was skipped for EchoClass, as it operates directly on the zero-cross files themselves.

Overall Classification Rates (%) SonoBat BCID Echo Class NY DFA Class Vote Consensus % Correct 49 56 45 53 43 % Incorrect 6 7 4 38 36 % Unknown 14 2 15 And here are some results. This table shows the overall classification rates—expressed as percentages--for each of the methods. For example, for the SonoBat Mean Classification output, 49% of the files were correctly classified to species, 6% were misclassified, and 43% were not classified at all. I want to point out a few things with this table. First, none of the classifiers did a great job overall, correctly classifying between 43 and 60% of all calls.

Overall Classification Rates (%) SonoBat BCID Echo Class NY DFA Class Vote Consensus % Correct 49 56 45 53 43 % Incorrect 6 7 4 38 36 % Unknown 14 2 15 Second, the incorrect classification rates—or misclassified calls—are much higher for the zero-cross methods than for Sonobat. These methods are much less likely to classify files as unknown, and do not instill a lot of confidence in the data, as essentially any given file classification has a 40-50% chance of being wrong. In contrast, Sonobat is more conservative, and the By Vote output looks to provide the best trade-off in terms of correct, incorrect, and unknown classifications.

Correct Classification Rates (% Correct) Species SonoBat BCID Echo Class NY DFA Class Vote Consensus PESU 96 98 77 95 LABO 43 70 30 60 65 63 EPFU 88 93 86 51 57 78 LANO 74 83 52 26 LACI 53 68 58 MYLE 17 9 4 MYSE 33 46 MYLU 31 24 56 25 MYSO 38 35 This table shows the correct classification rates for each species, again in percentages. Overall, the classifiers performed the best on tricolored bats, correctly classifying between 77 and 98 percent of calls, and performed the worst on eastern small-footed bats, only classifying 0 to 26 percent of calls correctly.

Correct Classification Rates (% Correct) Species SonoBat BCID Echo Class NY DFA Class Vote Consensus PESU 96 98 77 95 LABO 43 70 30 60 65 63 EPFU 88 93 86 51 57 78 LANO 74 83 52 26 LACI 53 68 58 MYLE 17 9 4 MYSE 33 46 MYLU 31 24 56 25 MYSO 38 35 Here, the highlighted cells indicate which classifier performed best for each species. For example, both the Sonobat By Vote and BCID performed best for tricolored bats, correctly classifying 98 percent of calls. By Vote performed the best for the eastern red bat, and so on. In general, Sonobat performed best for non-Myotis species. Sonobat was particularly successful at discriminating big brown and silver-haired bats, which are notoriously difficult to separate, correctly classifying 93 and 83 percent of calls, respectively. With the exception of eastern small-footed bats, the zero-cross classifiers performed best for the Myotis species. That said, classification rates still only ranged between 31 and 56 percent.

Correct Classification Rates (% Correct) Species SonoBat BCID Echo Class NY DFA Class Vote Consensus PESU 96 98 77 95 LABO 43 70 30 60 65 63 EPFU 88 93 86 51 57 78 LANO 74 83 52 26 LACI 53 68 58 MYLE 17 9 4 MYSE 33 46 MYLU 31 24 56 25 MYSO 38 35 Of particular interest is that the zero-cross methods only classified 31-38% of Indiana bat calls correctly, and Sonobat did not identify any. This suggests that automated methods may not be very reliable for monitoring endangered Indiana bats, and that Sonobat in particular may be too conservative for this task.

Summary None of the classifiers performed well overall SonoBat better for non-Myotis, ZC better for Myotis SonoBat more conservative Caution when using automated classification for Indiana bat surveys To summarize, none of the classifiers performed well overall. This is a bit surprising, as we only used high-quality reference calls in this study. We would expect performance to be worse for calls that were passively collected in the field. Performance varied by species, with Sonobat performing better for non-Myotis species in general and zero-cross classifiers performing better for Myotis species. This difference may be because discriminating non-Myotis calls relies more on harmonics and amplitude, while Myotis classification relies more on slope and duration, but this remains unclear. Sonobat is also more conservative than zero-cross methods, which is perhaps its greatest strength. However, Sonobat may be too conservative when it comes to Indiana bats. Zero-cross methods also did not perform very well for classifying Indiana bats. This is of concern, as the U.S. Fish and Wildlife Service has proposed a protocol to determine the presence of Indiana bats solely from acoustic data. Our results suggest that acoustic data and automated classifiers are not yet sufficient for this task.

Caveats Low sample size for some species Lost in translation? Different Analook filters could improve or worsen ZC classifier performance Focus on trends, not absolute comparisons That said, there are some caveats to mention. First, we had a low sample size for several species which is not ideal, but highlights the lack of full-spectrum reference libraries available in the US and Canada. It is also possible that some information was lost in translation when converting to zero-cross and may have affected species ID. However, the fact that zero-cross methods outperformed Sonobat for some species suggests this may not have been much of an issue. The filter we used for extracting parameters in Analook, however, likely did affect performance. Matthew Clement will be presenting research later today that demonstrates how different filters can lead to very different classifications. A filter was not used on the data for EchoClass, which might explain why it did not perform well relative to the other classifiers. That said, I also want to emphasize that the point of this study was not to see which classifier was best, but to see how one type of classifiers compared to another. We are certain that the performance of each of these classifiers will vary with different filters, different species, and different datasets, but we are also confident that the trends revealed by our study represent real differences between full-spectrum and zero-cross automated classifiers.

Conclusion Illustrates limitations of automated classification of bat acoustic data Species presence/probable absence should be based on multiple lines of evidence In conclusion, our results illustrate some of the limitations of automated classification for bats, and suggest that species presence and probable absence should be determined from multiple lines of evidence rather than any single data source.

Thank You! Ryan Allen, Bat Call Identification, Inc. Eric Britzke, US Army Engineer Research & Development Center John Chenger, Bat Conservation and Management Carl Herzog, New York Dept. of Environmental Conservation Amie Shovlain, Montana Natural Heritage Program Craig Stihler, West Virginia Dept. of Natural Resources Joe Szewczak, Humboldt State University Thank you!