1. CIS 470 Mobile App Development
Lecture 22
Wenbing Zhao
Department of Electrical Engineering and Computer Science
Cleveland State University
11/17/2018
CIS 470: Mobile App Development

2. Android Speech Recognition
Android speech recognition: the speech input will be streamed to a server, on the server voice will be converted to text and finally text will be sent back to our app
Internet connection is required to perform speech recognition
11/17/2018
CIS 470: Mobile App Development

3. CIS 470: Mobile App Development
Speech Recognition
Create a new app and name it MySpeechRecognizer
Add permissions in manifest
<uses-permission android:name="android.permission.INTERNET"/> <uses-permission android:name="android.permission.RECORD_AUDIO"/> <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" /> <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
11/17/2018
CIS 470: Mobile App Development

4. CIS 470: Mobile App Development
<?xml version="1.0" encoding="utf-8"?> <RelativeLayout xmlns:android=" android:layout_width="match_parent" android:layout_height="match_parent" android:orientation="vertical" > <ProgressBar style="?android:attr/progressBarStyleHorizontal" android:layout_width="match_parent" android:layout_height="wrap_content" android:layout_alignParentLeft="true" android:layout_marginTop="28dp" android:paddingLeft="10dp" android:paddingRight="10dp" /> <TextView android:layout_width="wrap_content" android:layout_height="wrap_content" android:layout_centerHorizontal="true" android:layout_marginTop="47dp" /> <ToggleButton android:layout_width="wrap_content" android:layout_height="wrap_content" android:layout_alignParentTop="true" android:layout_centerHorizontal="true" android:layout_marginTop="26dp" android:text="ToggleButton" /> </RelativeLayout>
Speech Recognition
Modify activity_main.xml layout:
11/17/2018
CIS 470: Mobile App Development

5. CIS 470: Mobile App Development
Speech Recognition
Modify MainActivity.java (download the entire java file from the course webpage
public class MainActivity extends AppCompatActivity implements RecognitionListener { private static final int REQUEST_RECORD_PERMISSION = 100; private TextView returnedText; private ToggleButton toggleButton; private ProgressBar progressBar; private SpeechRecognizer speech = null; private Intent recognizerIntent; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); returnedText = (TextView) findViewById(R.id.textView1); progressBar = (ProgressBar) findViewById(R.id.progressBar1); toggleButton = (ToggleButton) findViewById(R.id.toggleButton1);
11/17/2018
CIS 470: Mobile App Development

6. CIS 470: Mobile App Development
@Override protected void onCreate(Bundle savedInstanceState) { progressBar.setVisibility(View.VISIBLE); speech = SpeechRecognizer.createSpeechRecognizer(this); speech.setRecognitionListener(this); recognizerIntent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH); recognizerIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_PREFERENCE, "en"); recognizerIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM); recognizerIntent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 3); toggleButton.setChecked(true); speech.startListening(recognizerIntent); toggleButton.setOnCheckedChangeListener(new CompoundButton.OnCheckedChangeListener() { @Override public void onCheckedChanged(CompoundButton buttonView, boolean isChecked) { if (isChecked) { progressBar.setVisibility(View.VISIBLE); progressBar.setIndeterminate(true); speech.startListening(recognizerIntent); } else { progressBar.setIndeterminate(false); progressBar.setVisibility(View.INVISIBLE); speech.stopListening(); } } }); }
11/17/2018
CIS 470: Mobile App Development

7. CIS 470: Mobile App Development
@Override public void onBeginningOfSpeech() { Log.i(LOG_TAG, "onBeginningOfSpeech"); progressBar.setIndeterminate(false); progressBar.setMax(10); public void onBufferReceived(byte[] buffer) { Log.i(LOG_TAG, "onBufferReceived: " + buffer); public void onEndOfSpeech() { Log.i(LOG_TAG, "onEndOfSpeech"); public void onError(int errorCode) { String errorMessage = getErrorText(errorCode); Log.d(LOG_TAG, "FAILED " + errorMessage); speech.cancel(); speech.startListening(recognizerIntent); public void onEvent(int arg0, Bundle arg1) { Log.i(LOG_TAG, "onEvent"); }
@Override public void onPartialResults(Bundle arg0) { Log.i(LOG_TAG, "onPartialResults"); }
@Override public void onReadyForSpeech(Bundle arg0) { Log.i(LOG_TAG, "onReadyForSpeech"); progressBar.setVisibility(View.VISIBLE); } @Override public void onResults(Bundle results) { Log.i(LOG_TAG, "onResults"); ArrayList<String> matches = results.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION); float[] confs = results.getFloatArray(SpeechRecognizer.CONFIDENCE_SCORES); String text = ""; int i = 0; for (String result : matches) { if(confs[i] > 0.2) // to include text only if the confidence > 20% text += confs[i]+": "+ result + "\n"; i++; } returnedText.setText(text); speech.cancel(); progressBar.setVisibility(View.INVISIBLE); speech.startListening(recognizerIntent); }
@Override public void onRmsChanged(float rmsdB) { progressBar.setProgress((int) rmsdB); }
11/17/2018
CIS 470: Mobile App Development

8. Challenge Task (May use this as the project)
Change the basic app for speech recognition to an app that performs continuous and voice-activated speech recognition. Log both the raw audio and the recognized text to two separate files.
11/17/2018
CIS 470: Mobile App Development