CIS 470 Mobile App Development Lecture 22 Wenbing Zhao Department of Electrical Engineering and Computer Science Cleveland State University wenbing@ieee.org 11/17/2018 CIS 470: Mobile App Development

Android Speech Recognition Android speech recognition: the speech input will be streamed to a server, on the server voice will be converted to text and finally text will be sent back to our app Internet connection is required to perform speech recognition 11/17/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Speech Recognition Create a new app and name it MySpeechRecognizer Add permissions in manifest <uses-permission android:name="android.permission.INTERNET"/> <uses-permission android:name="android.permission.RECORD_AUDIO"/> <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" /> <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" /> 11/17/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development <?xml version="1.0" encoding="utf-8"?> <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android" android:layout_width="match_parent" android:layout_height="match_parent" android:orientation="vertical" > <ProgressBar android:id="@+id/progressBar1" style="?android:attr/progressBarStyleHorizontal" android:layout_width="match_parent" android:layout_height="wrap_content" android:layout_alignParentLeft="true" android:layout_below="@+id/toggleButton1" android:layout_marginTop="28dp" android:paddingLeft="10dp" android:paddingRight="10dp" /> <TextView android:id="@+id/textView1" android:layout_width="wrap_content" android:layout_height="wrap_content" android:layout_below="@+id/progressBar1" android:layout_centerHorizontal="true" android:layout_marginTop="47dp" /> <ToggleButton android:id="@+id/toggleButton1" android:layout_width="wrap_content" android:layout_height="wrap_content" android:layout_alignParentTop="true" android:layout_centerHorizontal="true" android:layout_marginTop="26dp" android:text="ToggleButton" /> </RelativeLayout> Speech Recognition Modify activity_main.xml layout: 11/17/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Speech Recognition Modify MainActivity.java (download the entire java file from the course webpage public class MainActivity extends AppCompatActivity implements RecognitionListener { private static final int REQUEST_RECORD_PERMISSION = 100; private TextView returnedText; private ToggleButton toggleButton; private ProgressBar progressBar; private SpeechRecognizer speech = null; private Intent recognizerIntent; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); returnedText = (TextView) findViewById(R.id.textView1); progressBar = (ProgressBar) findViewById(R.id.progressBar1); toggleButton = (ToggleButton) findViewById(R.id.toggleButton1); .... 11/17/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development @Override protected void onCreate(Bundle savedInstanceState) { .... progressBar.setVisibility(View.VISIBLE); speech = SpeechRecognizer.createSpeechRecognizer(this); speech.setRecognitionListener(this); recognizerIntent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH); recognizerIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_PREFERENCE, "en"); recognizerIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM); recognizerIntent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 3); toggleButton.setChecked(true); speech.startListening(recognizerIntent); toggleButton.setOnCheckedChangeListener(new CompoundButton.OnCheckedChangeListener() { @Override public void onCheckedChanged(CompoundButton buttonView, boolean isChecked) { if (isChecked) { progressBar.setVisibility(View.VISIBLE); progressBar.setIndeterminate(true); speech.startListening(recognizerIntent); } else { progressBar.setIndeterminate(false); progressBar.setVisibility(View.INVISIBLE); speech.stopListening(); } } }); } 11/17/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development @Override public void onBeginningOfSpeech() { Log.i(LOG_TAG, "onBeginningOfSpeech"); progressBar.setIndeterminate(false); progressBar.setMax(10); } @Override public void onBufferReceived(byte[] buffer) { Log.i(LOG_TAG, "onBufferReceived: " + buffer); } @Override public void onEndOfSpeech() { Log.i(LOG_TAG, "onEndOfSpeech"); } @Override public void onError(int errorCode) { String errorMessage = getErrorText(errorCode); Log.d(LOG_TAG, "FAILED " + errorMessage); speech.cancel(); speech.startListening(recognizerIntent); } @Override public void onEvent(int arg0, Bundle arg1) { Log.i(LOG_TAG, "onEvent"); } @Override public void onPartialResults(Bundle arg0) { Log.i(LOG_TAG, "onPartialResults"); } @Override public void onReadyForSpeech(Bundle arg0) { Log.i(LOG_TAG, "onReadyForSpeech"); progressBar.setVisibility(View.VISIBLE); } @Override public void onResults(Bundle results) { Log.i(LOG_TAG, "onResults"); ArrayList<String> matches = results.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION); float[] confs = results.getFloatArray(SpeechRecognizer.CONFIDENCE_SCORES); String text = ""; int i = 0; for (String result : matches) { if(confs[i] > 0.2) // to include text only if the confidence > 20% text += confs[i]+": "+ result + "\n"; i++; } returnedText.setText(text); speech.cancel(); progressBar.setVisibility(View.INVISIBLE); speech.startListening(recognizerIntent); } @Override public void onRmsChanged(float rmsdB) { progressBar.setProgress((int) rmsdB); } 11/17/2018 CIS 470: Mobile App Development

Challenge Task (May use this as the project) Change the basic app for speech recognition to an app that performs continuous and voice-activated speech recognition. Log both the raw audio and the recognized text to two separate files. 11/17/2018 CIS 470: Mobile App Development