Current methods in forensic speaker ID: results of the fake case Tina Cambier-Langeveld Dutch Ministry of Justice formerly employed by the Netherlands Forensic Institute currently employed by the Immigration and Naturalisation Service IAFPA 24 July 2006

Outline Main goal Set-up: preparations, material, time path The participants Results summarized per file: Q1-10 Summary of the results Other findings Conclusion

Main goal To document the different methodologies and strategies that are used in different laboratories / by different experts to come to a final conclusion in a speaker verification task, based on an examination of the same set of materials Have as many experts participate as possible (representative of the field, i.e. including all methods that are currently being used) Not a proficiency test but a collaborative exercise

Preparations 20 experts were found willing to participate Erik (a former colleague) asked three Canadian friends if we could record some telephone conversations with them for this purpose The friend who could also be reached at work was picked as ‘suspect’ or Reference speaker Jos Bouten acted as potential customer and speaking partner for the Reference recordings

Material (audio) The Questioned recordings (Q1-10) were telephone conversations between friends, very informal, varying in duration between 22 seconds and 11 minutes (file duration, i.e. including ringing) The Reference recordings (R1-2) were telephone conversations with a potential customer asking about their services in detail, more formal, quite long: about 5 and 11 minutes (file duration), each containing > 2 minutes of net speech

Material (questionnaire) A questionnaire was added, asking about the experience of the experts (in general and in English casework), the software that was used, the time spent on the fake case, any restrictions that the institute or the law imposes, comments, etc.

Time path The materials were sent out to 20 potential participants in August-September 2004 Reports were due back before 1 February 2005 The final (12 th ) report was received on 29 April 2005, i.e. three months later Definite results are presented to you now (as planned) Paper for the International Journal of Speech, Language and the Law is currently under review

The participants (1) 12 participants from 10 different countries, within and outside of Europe, employed by forensic laboratories, universities, police labs, the military etc. or working as private experts Methods: 2 fully automatic 5 auditory-phonetic, all members of IAFPA 5 semi-automatic (variable) pretty representative of the field :-)

The participants (2) Experience of the participants ranges between 28 years and one case; average >10 years 5 of the 12 participants had never worked on a case in English before 5 of the 12 participants had done between 1 and 3 cases in English before 2 of the 12 participants had worked on cases in English a lot

Pooling of the results The strength of the conclusion is not reported, only: whether a conclusion was drawn the direction of the conclusion (the same/not the same speaker as in the Reference recordings) …since the strength of the conclusion largely depends on the familiarity with the language, and so it would not be a fair comparison.

Exempted from the results Two of the reports are excluded from this first section because the experts involved explicitly state that they have assumed that all of the ten questioned recordings came from the same source (1 semi-automatic, 1 auditory-phon.) One of the reports is excluded from this section because the expert involved, after first ‘ensuring’ that all the questioned material is from one speaker (?), also examined the ten questioned recordings as a whole (1 auditory-phon.)

Results summarized per file…...for 9 reports (2 x fully automatic, 4 x semi- automatic, 3 x auditory-phonetic)

Results summarized per file: Q1 -Net duration > 2 minutes -not the same speaker 8 x correct conclusion 1 x no decision (automatic analysis)

Results summarized per file: Q2 -Net duration: approx. 10 seconds -the same speaker 3 x correct conclusion (2 x aud-phon, 1 x semi) 6 x too short / not enough information / no decision

Results summarized per file: Q3, 4, 6 -Net duration: more than 50 sec. each -the same speaker 9 x correct conclusion

Results summarized per file: Q5 -Net duration: approx. 14 seconds -the same speaker 7 x correct conclusion 2 x too short / not enough information (2 x semi)

Results summarized per file: Q7 -Net duration: approx. 5 seconds -the same speaker 2 x correct conclusion (1 x aud-phon, 1 x semi) 7 x too short / not enough information / no decision

Results summarized per file: Q8 -Net duration: approx. 15 seconds -not the same speaker 6 x correct conclusion 3 x too short / not enough information / no decision (2 x semi, 1 x aut.)

Results summarized per file: Q9 -Net duration: approx. 14 seconds -the same speaker 7 x correct conclusion 2 x too short / not enough information (2 x semi)

Results summarized per file: Q10 -Net duration: approx. 18 seconds -not the same speaker 2 x correct conclusion (1 x aut, 1 x semi) 5 x too short / not enough information / no decision 2 x wrong conclusion (1 x aud-phon, 1 x semi)

Summary of the results 3 participants (1 semi-automatic, 2 auditory- phonetic), assuming that all of the Q-files were from one speaker, failed to pick out the three conversations that were from different speakers 2 other participants (1 semi-automatic, 1 auditory-phonetic) identify Q1 and Q8 as being from a different speaker, but incorrectly conclude that Q10 is from the Reference speaker.

Summary of the results I have not been able to detect any obvious relation between the results and the experience of the participants, in general or in English casework their proficiency in English the number of experts involved in the examination

Some other findings: all 12 reports F0 measurements are done in 9 reports Formant measurements are done in 7 reports Speech rate / articulation rate / speaking rate measurements are done in 4 reports …but measurements are often not comparable, because they are implemented differently, done on different selections of the material or reported differently.

Some other findings In 4 of the 5 auditory-phonetic reports, a difference in communicative context is noted In 2 reports, the accent is correctly identified as Canadian, “possibly Ontario” / “maybe in the area of Toronto” Documented feature ‘Canadian raising’ is referred to in 1 report (not a native of English) Other accent remarks: RP, general American

Some other findings Gradations in the conclusion: 4 x Bayesian format (Likelihood Ratio, or an amount of support for hypothesis H0 vs. H1) 6 x in terms of likelihood, possibility etc. (all of these 6 scales are different! The number of possible conclusions ranges from 6 to 11…) 1 x categorical (due to the legal system) 1 x no gradations, just free text (private expert)

Questionnaire results Time spent on the case ranges from 28 hours to 3 x 40 hours; average 65 hours Time spent on an average case ranges from “8 hours for a 1:1 comparison” to “one month”. Average for the other 10 is 48 hours Number of researchers involved: one (4x), two (3x), three (3x), four (2x), including technicians

Questionnaire results Software/speech analysis tools: Praat (5x), Cool Edit Pro/Adobe Audition (5x), SoundForge (3x), Kay CSL (3x), Wavesurfer (2x), SIVE (2x), Matlab (1x), x-waves Entropic (1x), DCLive Forensics (1x), own systems (5x) Some are typically suitable for editing and listening (Adobe Audition, SoundForge), others are better analysis tools (Praat, SIVE)

Questionnaire results “Is your analysis or report in any way restricted or enforced by e.g. the law, guidelines of your institute, accreditation boards, or previous court decisions?” - Four experts mention their laboratory / institute in their answer to this question - Only one participant mentioned the IAFPA Code of Conduct…….

Questionnaire results Comments include: examination limited due to the language (2x) examination limited for other reasons (3x) took up too much time (2x) “This was a very typical case for me” “well chosen material, not too difficult, not too easy!” words of appreciation (5x) ;-)

Questionnaire results Suggestions for the next exercise: mismatched recording conditions (2x) another language / various languages / multilingual speech (4x) straight 1:1 comparison different speaking styles

Conclusion General comments: Gives great insight into the different methods that are used Reports vary widely on almost every aspect you can think of, and overlap is very limited, also between experts using the same method Some of the reports may be made available (selectively), together with the audio material