CIS 470 Mobile App Development Lecture 19 Wenbing Zhao Department of Electrical Engineering and Computer Science Cleveland State University wenbing@ieee.org 11/14/2018 CIS 470: Mobile App Development

Google Mobile Vision API Face tracking 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Face Tracking <?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.wenbing.facetracker"> <uses-feature android:name="android.hardware.camera" /> <uses-permission android:name="android.permission.CAMERA" /> <application android:allowBackup="true" android:hardwareAccelerated="true" android:theme="@style/Theme.AppCompat" android:label="FaceTracker"> <meta-data android:name="com.google.android.gms.version" android:value="@integer/google_play_services_version"/> <meta-data android:name="com.google.android.gms.vision.DEPENDENCIES" android:value="face" /> <activity android:name=".MainActivity" android:label="Face Tracker" android:theme="@style/Theme.AppCompat.NoActionBar" android:screenOrientation="fullSensor"> <intent-filter> <action android:name="android.intent.action.MAIN" /> <category android:name="android.intent.category.LAUNCHER" /> </intent-filter> </activity> </application> </manifest> Create a new app and name it FaceTracker Modify manifest 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Face Tracking Modify build.gradle (Module: app) dependencies { compile fileTree(dir: 'libs', include: ['*.jar']) androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', { exclude group: 'com.android.support', module: 'support-annotations' }) compile 'com.android.support:appcompat-v7:26.+' compile 'com.android.support:design:26.+' compile 'com.google.android.gms:play-services-vision:9.4.0+' testCompile 'junit:junit:4.12' } 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Face Tracking <?xml version="1.0" encoding="utf-8"?> <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android" android:id="@+id/topLayout" android:orientation="vertical" android:layout_width="match_parent" android:layout_height="match_parent" android:keepScreenOn="true"> <com.wenbing.facetracker.CameraSourcePreview android:id="@+id/preview" android:layout_width="match_parent" android:layout_height="match_parent"> <com.wenbing.facetracker.GraphicOverlay android:id="@+id/faceOverlay" android:layout_width="match_parent" android:layout_height="match_parent" /> </com.wenbing.facetracker.CameraSourcePreview> </LinearLayout> Add three Java classes: CameraSourcePreview, FaceGraphic, GraphicOverlay. Populate all four classes with the java files posted on the web Modify activity_main.xml layout: 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Face Tracking <?xml version="1.0" encoding="utf-8"?> <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android" android:id="@+id/topLayout" android:orientation=”horizontal" android:layout_width="match_parent" android:layout_height="match_parent" android:keepScreenOn="true"> <com.wenbing.facetracker.CameraSourcePreview android:id="@+id/preview" android:layout_width="match_parent" android:layout_height="match_parent"> <com.wenbing.facetracker.GraphicOverlay android:id="@+id/faceOverlay" android:layout_width="match_parent" android:layout_height="match_parent" /> </com.wenbing.facetracker.CameraSourcePreview> </LinearLayout> Add one more layout with the same name but change the default directory name to: layout-land 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Face Tracking Add one more layout with the same name but change the default directory name to: layout-land 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Face Tracking Change values/strings.xml <resources> <string name="app_name">FaceTracker</string> <string name="ok">OK</string> <string name="permission_camera_rationale">Access to the camera is needed for detection</string> <string name="no_camera_permission">This application cannot run because it does not have the camera permission. The application will now exit.</string> <string name="low_storage_error">Face detector dependencies cannot be downloaded due to low device storage</string> </resources> 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Face Tracking 11/14/2018 CIS 470: Mobile App Development

Google Mobile Vision Face Tracking https://developers.google.com/vision/android/getting-started Face detection is the process of automatically locating human faces in visual media (digital images or video) A face that is detected is reported at a position with an associated size and orientation Once a face is detected, it can be searched for landmarks (points of interest within a face) such as the eyes and nose Classification is determining whether a certain facial characteristic is present. For example, a face can be classified with regards to whether its eyes are open or closed. Another example is whether the face is smiling or not 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Face Tracking Pose angle examples where y==Euler Y, r==Euler Z 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Face Tracking Landmarks: The left eye, right eye, and nose base Landmark detection is not done by default, since it takes additional time to run. You can optionally specify that landmark detection should be done: Euler Y angle detectable landmarks < -36 degrees left eye, left mouth, left ear, nose base, left cheek -36 degrees to -12 degrees left mouth, nose base, bottom mouth, right eye, left eye, left cheek, left ear tip -12 degrees to 12 degrees right eye, left eye, nose base, left cheek, right cheek, left mouth, right mouth, bottom mouth 12 degrees to 36 degrees right mouth, nose base, bottom mouth, left eye, right eye, right cheek, right ear tip > 36 degrees right eye, right mouth, right ear, nose base, right cheek 11/14/2018 CIS 470: Mobile App Development

Face Tracking: Classification The Android Face API currently supports two classifications: eyes open and smiling Classification is expressed as a certainty value, indicating the confidence that the facial characteristic is present. For example, a value of 0.7 or more for the smiling classification indicates that it is likely that a person is smiling “eyes open” and “smiling” classification only works for frontal faces, that is, faces with a small Euler Y angle (at most about +/- 18 degrees) 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Face Tracking APIs Creating the face detector: Detecting faces with facial landmarks Classification: FaceDetector detector = new FaceDetector.Builder(context) .setClassificationType(FaceDetector.ALL_CLASSIFICATIONS).build(); for (Landmark landmark : face.getLandmarks()) {         int cx = (int) (landmark.getPosition().x * scale);         int cy = (int) (landmark.getPosition().y * scale);         canvas.drawCircle(cx, cy, 10, paint);     } face.getIsSmilingProbability() face.getIsRightEyeOpenProbability() face.getIsLeftEyeOpenProbability() 11/14/2018 CIS 470: Mobile App Development

CIS 470: Mobile App Development Exercise Draw all landmarks on the detected face(s). 11/14/2018 CIS 470: Mobile App Development