Presentation is loading. Please wait.

Presentation is loading. Please wait.

SpeechWeb & Adobe Captivate towards a revolution in education Richard Frost School of Computer Science University of Windsor ITS 2012 Windsor.

Published byLee Barrett Modified over 9 years ago

Similar presentations

Presentation on theme: "SpeechWeb & Adobe Captivate towards a revolution in education Richard Frost School of Computer Science University of Windsor ITS 2012 Windsor."— Presentation transcript:

1 SpeechWeb & Adobe Captivate towards a revolution in education Richard Frost School of Computer Science University of Windsor ITS 2012 Windsor

2 Questions 1.Given that speech is a fundamental method of communication: Why are there so few web-based “speech applications”. Why are there so few natural-language “English” interfaces to web applications and data? Why are there hardly any “speech games” on the web? 2.Given that YouTube is so easy to use: Why do we not have more “college and university lessons” available on YouTube?

3 Possible Answers 1.Speech and natural-language applications Speech technology is immature. NL theories cannot be computerized. There is no market for such applications Few people are interested in creating speech & NL apps. Speech and NL technologies are extremely difficult. 2.YouTube lessons Instructors are not interested in creating on-line lessons. Video capture technology is difficult to use.

4 A different perspective Speech technology is very mature (e,g, Google speech apps, iPhone 4S) Compositional theories of natural language are available. The market for NL speech applications is huge, as is on-line learning. Many people are interested in these technologies BUT think that they are very difficult.

5 My Thesis Technology, interest and NOTATION is now available for non-experts to create natural-language speech applications and deploy them on the web. Video capture technology is available that allows non- experts to build computer based lessons and deploy them on YouTube and elsewhere. In the next few years we will see a massive increase in NL speech interfaces to knowledge and access to on- line lessons which will revolutionize education. We begin with an analogy

6 An old tune goes global Pachelbel composed the “Canon” (late 1600s) http://www.youtube.com/watch?v=8Af372EQLck Jerry C (Chang) re-arranged for electric guitar around “Canon Rock” (2005) http://www.youtube.com/watch?v=by8oyJztzwo A youtube user, Impeto, spliced together 39 excerpts of musicians playing and called it the “Ultimate Canon Rock” (2007) http://www.youtube.com/watch?v=dMWl_5NujBw

7 What helped Jerry C teach a wide range of people to play the Canon and participate in the “Ultimate Canon Rock” Electric guitar (1930’s) The Web (Tim Berners-Lee 1990’s) YouTube Guitar TAB reborn in 40’s, widely used now ------------------------|------------------------|------------------------ --19~--------15~--------|--16~--------13~--------|--14~--------11~-------- ------------------------|------------------------|------------------------

8 www.youtube.comwww.youtube.com (and type in “speechweb”) or go directly: http://www.youtube.com/watch?v=Axa-n4etdZE http://www.youtube.com/watch?v=Axa-n4etdZE And now for something completely different A video demonstration of SpeechWeb created using Adobe Captivate Software.

9 A Brief Overview of SpeechWeb Technology The SpeechWeb architecture The speech browser interface How to create a SpeechWeb application and deploy on the web. The mathematical basis of natural language processing. A summary of the notation which has made it possible.

10 Local Recognition Remote Processing (LRRP) Architecture

11 Applications in the cloud synthesized speech outhuman speech in speech recognizer X + V browser application recognition grammar Application program text query OR hyperlink request text answer OR address of new application application hyperlinked applications residing on conventional web servers X+V browser

12 To Create a SpeechWeb Application Copy three files into a web directory 1.The X+V browser 2.A sample grammar 3.A sample program Modify four lines in the X+V browser Change the grammar for your application’s input language. Modify the sample program or replace with a program, written in any language to process the input. ALL SIMPLE NOTATION?

13 The X+V Browser <html xmlns="http://www.w3.org/1999/xhtml" xmlns:vxml……………………………… var appName = “Monty"; var appFullName = “speechweb.cs.uwindsor.ca/applications/Monty"; var greeting = “Hello. My name is Monty. I know a joke."; 0"> etc ………………………….

14 Recognition Grammars Guide Search = what is your name | where do you live | what do you know | tell me a joke | can I talk to | etc ; = judy | solarman | pete …

15 The Programs can be as simple as you want interpret "what is your name" = "My name is Monty.“ interpret "where do you live" = "I hang out in one of Frosties computers. “ interpret "what do you know" = "I got a joke or two. Not much else.“ interpret “tell me a joke" = “Did you hear about the two professors……………."

16 The Basis of the Natural language Technology Variation of Montague’s NL semantics (1970’s) developed in the λ-calculus (Church 1930’s), and implemented in set-theory. [[Mars]] = λs e mars s [[spin]] = {e earth, e mars, e luna, …} [[moon]] = {e luna, e phobos, …..} [[Mars]] [[spins]] => (λs e mars s) {e earth,e mars,…} => e mars {e earth, e mars, …} => True [[every]] = λp λq p subset q

17 The result is a fully compositional semantics The composition rule is always simple function application, e.g. (hall or kuiper) (discovered (every moon)) The semantics covers a large sub-set of classical first-order English. does every moon and every planet spin how many moons that orbit a red planet were discovered by the person who discovered Nereid which planet is orbited by no moon The meaning of words can be defined in terms of other words. [[ discoverer ]] = [[ person who discovered a thing ]]

18 The notation which simplifies creation and deployment of NL speech applications VXML (X+V) to configure/interface to the speech recognizer BNF notation for recognizer grammars Declarative/equational programming languages λ calculus and set theory for NL

19 Adobe Captivate Captures all screen activity and voice over (and sounds from a computer session). Clever capture minimizes resulting video. Publish as.pdf,.mp4 etc and directly to YouTube. Can edit video and sound. Learning curve similar to PowerPoint. Can be used with tablets to create “Khan-style” online lessons: http://www.khanacademy.org/http://www.khanacademy.org/

20 Use of speech and captivate technology in Education Non experts can add speech interfaces to their web applications. Non experts can create lessons about anything and deploy them on the web. In the future we will be able to create interactive on-line lessons with spoken natural-language interfaces. Multi-Modal Online Education

21 Using speech games to create cognitive profiles Video games are being used to develop cognitive profiles of users. Can help identify learning strengths and weaknesses in children. Speech games can add another “dimension” to the cognitive profiles. We are currently designing speech-only games for children aged 6 and above.

22 Acknowledgements Graduate Students: Sanjay Chitte, William Ma, Fadi Hanna, Jack Su, Shahriar Chandon, Nabil Abdullah, (Sunny) Yue Shi, and Rahmatullah Hafiz. Undergraduate students: Ali Karaki, David Dufour, Josh Greig, S. Daichendt, Justin Barolak, Randy Fortier, Bryan St Amour, Jon Donais, Paul Meyer and Matthew Clifford. The research is funded by NSERC discovery grants, NSERC USRAs, and U. of Windsor Outstanding Scholar awards.

Download ppt "SpeechWeb & Adobe Captivate towards a revolution in education Richard Frost School of Computer Science University of Windsor ITS 2012 Windsor."

Similar presentations

Discipline-Specific Ideas Language Arts- Speech classes could easily benefit from the use of PowerPoint, while any grammar class could benefit from the.

Discipline-Specific Ideas Language Arts- Speech classes could easily benefit from the use of PowerPoint, while any grammar class could benefit from the.
An Event-based Approach for Querying Graph- Structured Data Using Natural Language Richard Frost, Wale Agboola, Eric Matthews and Jon Donais School of.

An Event-based Approach for Querying Graph- Structured Data Using Natural Language Richard Frost, Wale Agboola, Eric Matthews and Jon Donais School of.
Video Game Design Lesson 1. Game Designer Person involved in the development of a video game Person involved in the development of a video game Usually.

Video Game Design Lesson 1. Game Designer Person involved in the development of a video game Person involved in the development of a video game Usually.
 To publish information for global distribution, one needs a universally understood language, a kind of publishing mother tongue that all computers may.

 To publish information for global distribution, one needs a universally understood language, a kind of publishing mother tongue that all computers may.
A Framework for Creating Speech Games for Cognitive Profiling Richard Frost School of Computer Science University of Windsor MHCI’13.

A Framework for Creating Speech Games for Cognitive Profiling Richard Frost School of Computer Science University of Windsor MHCI’13.
Richard Yu.  Present view of the world that is: Enhanced by computers Mix real and virtual sensory input  Most common AR is visual Mixed reality virtual.

Richard Yu.  Present view of the world that is: Enhanced by computers Mix real and virtual sensory input  Most common AR is visual Mixed reality virtual.
PHONEXIA Can I have it in writing?. Discuss and share your answers to the following questions: 1.When you have English lessons listening to spoken English,

PHONEXIA Can I have it in writing?. Discuss and share your answers to the following questions: 1.When you have English lessons listening to spoken English,
Master’s course Bioinformatics Data Analysis and Tools Lecture 6: Internet Basics Centre for Integrative Bioinformatics.

Master’s course Bioinformatics Data Analysis and Tools Lecture 6: Internet Basics Centre for Integrative Bioinformatics.
CAMEL at Greenwich “Sailing the Seven Seas” Off shore navigation with OSCARS and friends Information & Library Services.

CAMEL at Greenwich “Sailing the Seven Seas” Off shore navigation with OSCARS and friends Information & Library Services.
Improving Technology Expertise in Mathematics Instructors Maria H. Andersen, Muskegon Community College We live in a society exquisitely dependent on science.

Improving Technology Expertise in Mathematics Instructors Maria H. Andersen, Muskegon Community College We live in a society exquisitely dependent on science.
The Internet & The World Wide Web Notes

The Internet & The World Wide Web Notes
Designing Web Pages Getting to know HTML... What is HTML? Hyper Text Markup Language HTML is the major language of the Internet’s World Wide Web. Web.

Designing Web Pages Getting to know HTML... What is HTML? Hyper Text Markup Language HTML is the major language of the Internet’s World Wide Web. Web.
COMPUTING IN THE NATIONAL CURRICULUM. WHY?  The 2014 national curriculum introduces a new subject, computing, which replaces ICT. This represents continuity.

COMPUTING IN THE NATIONAL CURRICULUM. WHY?  The 2014 national curriculum introduces a new subject, computing, which replaces ICT. This represents continuity.
Glenys Crane-Emerson FSLP Coordinator Kathy Vesey Director

Glenys Crane-Emerson FSLP Coordinator Kathy Vesey Director
Web Design Basic Concepts.

Web Design Basic Concepts.
Ryann Kramer EDU 521.02 Prof. R. Moroney Summer 2010.

Ryann Kramer EDU Prof. R. Moroney Summer 2010.
Paul Trani Adobe Certified Instructor/Expert Resources:

Paul Trani Adobe Certified Instructor/Expert Resources:
Lecture 1, 7/21/2005Natural Language Processing1 CS60057 Speech &Natural Language Processing Autumn 2005 Lecture 1 21 July 2005.

Lecture 1, 7/21/2005Natural Language Processing1 CS60057 Speech &Natural Language Processing Autumn 2005 Lecture 1 21 July 2005.
IT Applications Theory Slideshows By Mark Kelly Vceit.com Types and contents of On-screen user documentation.

IT Applications Theory Slideshows By Mark Kelly Vceit.com Types and contents of On-screen user documentation.
1. Human – the end-user of a program – the others in the organization Computer – the machine the program runs on – often split between clients & servers.

1. Human – the end-user of a program – the others in the organization Computer – the machine the program runs on – often split between clients & servers.

Similar presentations

Ads by Google