Android media Part 2: Pictures and Video Cosc 5/4730 Android media Part 2: Pictures and Video

Emulators and Samsung The emulators all have problems, much of the following code has been tested on the actual phones Video playback code works, but the video may not always display in the emulators android Video/picture capture does not work as documented. Samsung hardware! Some Samsung devices have many problems. Something about their hardware… search for the video and samsung for some possible answers.

Android android.media Play media

Supported formats In general Android’s support is consistent with other mobile phones. It supports the 3GP (.3gp) and MPEG-4 (.mp4) file formats. 3GP is a video standard derived from MPEG-4 specifically for use by mobile devices. As far as codecs go, Android supports H.263, a codec designed for low-latency and low-bitrate videoconferencing applications. H.263 video is supported in either MPEG-4 (.mp4) or 3GP (.3gp) files. Android also supports MPEG-4 Simple Profile in 3GP files (.3gp) as well as H.264.

Android First method The second method uses a VideoView to display. For greater flexibility you will need to use the mediaPlayer and a surfaceView (or TextureView API 14+) MediaPlayer like the audio and use a surfaceView to display the video. There are examples in the API demo, plus several of the books. The second method uses a VideoView to display. The VideoView uses a MediaController to start and stop and provide functions to control the video play back. With a MediaPlayer like the audio. prepare(), then start() This is one I’ll cover in this lecture.

VideoView VideoView is a View that has video playback capabilities and can be used directly in a layout. We can then add controls (play, pause, forward, back, etc) with the MediaController.

ViewView example Get the ViewView out of the layout vv = (VideoView) this.findViewById( R.id.VideoView); Setup where the file to play is Uri videoUri = Uri.parse(Environment.getExternalStorageDirectory().getPath() + "/example.mp4"); vv.setVideoURI(videoUri); play the video vv.start();

Adding media controllers. Very simple vv = (VideoView) this.findViewById(R.id.VideoView); vv.setMediaController(new MediaController(this)); Now media controls will show up on the screen.

Using native media player Call for an Intent and send it. Uri data = Uri.parse(VideoFile); intent.setDataAndType(data, "video/mp4"); startActivity(intent); Remember, your app is now in the background.

Example code The VideoPlay example code This will play a video from the internet. If you can uncomment the code to have it play the video from the sdcard, but you will need to copy the file to the sdcard.

Displaying a picture

Displaying Pictures See code already covered to how display pictures. ImageView for example…

Android Taking a picture

Camera vs Camera2 API Starting in API 21 (Lollipop) there is a new set of APIs. There is no backward compatibly either. So first we look at Camera v1 APIs And then Camera2 APIs These are more flexible and should allow for “filters” to be added on the fly to image. Finally, you can just use an intent have the native camera app take the picture.

Taking A Picture Camera V1 APIs

What to use Android packages Permissions and features import android.hardware.CameraDevice; import android.hardware.CameraDevice.CaptureParams; Permissions and features <uses-permission android:name="android.permission.CAMERA" /> Note, This one will requirement permission checking in API 23+ This too, if you change the how the camera is functioning. <uses-feature android:name="android.hardware.camera" /> <uses-feature android:name="android.hardware.camera.autofocus" />

Taking a picture In brief CameraDevice cameraDevice = CameraDevice.open() To preview you need a surfaceHolder then use setPreviewDisplay(surfaceHolder) and cameraDevice.startPreview() Finally use the takePicture(…) to get the picture release() and close() the CameraDevice to release it.

“View Finder” The view finder is implemented via a SurfaceHolder and SurfaceView In the layout, the a surfaceView is used. Example: public class PicCapture extends Activity implements OnClickListener, SurfaceHolder.Callback, Camera.PictureCallback { … cameraView = (SurfaceView) this.findViewById(R.id.CameraView); surfaceHolder = cameraView.getHolder(); surfaceHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS); surfaceHolder.addCallback(this); To finally take the picture we need all this too. cameraView.setFocusable(true); cameraView.setFocusableInTouchMode(true); cameraView.setClickable(true); cameraView.setOnClickListener(this);

“View Finder” (2) The code to implement the surfaceHolder can be very simple. When created open the camera and set the display surfaceCreated() { camera = Camera.open(); try { camera.setPreviewDisplay(holder); catch (IOException exception) { camera.release(); } } Once it’s ready, start the preview surfaceChanged() { camera.startPreview();

“View Finder” (3) When we are done surfaceDestroyed() { camera.stopPreview(); camera.release(); }

Get the Picture Using the Camera.PictureCallBack we implement, we can get the data for the picture and decide what to do it with it. In its simplest form, which doesn’t nothing with the picture public void onPictureTaken(byte[] data, Camera camera) { byte[] data is the picture that was taken this just restarts the preview. camera.startPreview(); }

Get the Picture (2) To take the picture we use the camera.takePicture method in the onClick method for the SurfaceView public void onClick(View v) { camera.takePicture(null, null, null, this); } takePicture (Camera.ShutterCallback shutter, Camera.PictureCallback raw, Camera.PictureCallback postview , Camera.PictureCallback jpeg) We only need the last to get the picture and it show on the previous slide. shutter the callback for image capture moment, or null raw the callback for raw (uncompressed) image data, or null postview the callback with postview image data, may be null jpeg the callback for JPEG image data, or null

Taking A Picture Camera V2 APIs

Camera2 APIs Like v1, this breaks up into 2 major pieces Viewfinder Connect it to a SurfaceView or TextureView. Taking the picture

Camera2 ViewFinder First find the “id” for the camera to use CameraManager manager = (CameraManager) getActivity().getSystemService(Context.CAMERA_SERVICE); String cameraId = manager.getCameraIdList()[0]; In this case just use the first one (tends to be the back camera) Then is gets complex, because we open the camera via this manager.openCamera (String cameraId, CameraDevice.StateCallback callback, Handler handler) StateCallback is a listener “first” and the handler is this case is likely null. The handler is the thread to use, null for main. Another thread is likely where you would apply a filter to the image, before it is displayed. Note, someone want to figure this out?

Camera2 ViewFinder (2) The call back is pretty simple, but the complex code is here. CameraDevice.StateCallback mStateCallback = new CameraDevice.StateCallback() { @Override public void onOpened(CameraDevice camera) { mCameraDevice = camera; //We now have the camera, so setup the capture of the current surface. startPreview(); //in a method since it will be called several times. } public void onDisconnected(CameraDevice camera) { Log.e(TAG, "onDisconnected"); public void onError(CameraDevice camera, int error) { Log.e(TAG, "onError"); };

startPreview First we need a surface (from the SurfaceView or Textview). We are using a surfaceView Surface surface = mHolder.getSurface(); mHolder is our surfaceview mPreviewBuilder = mCameraDevice.createCaptureRequest( CameraDevice.TEMPLATE_PREVIEW); mPreviewBuilder.addTarget(surface); Note, if wanted to display it on more then 1 surfaceview, we then add the rest here. (I think). mCameraDevice.createCaptureSession(Arrays.asList(surface), CameraCaptureStateCallback); The CameraCapture is listed on the next slide.

startPreview(2) CameraCaptureSession.StateCallback CameraCaptureStateCallback = new CameraCaptureSession.StateCallback() { @Override public void onConfigured(CameraCaptureSession session) { mPreviewSession = session; //now setup the update preview. mPreviewBuilder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO); HandlerThread thread = new HandlerThread("CameraPreview"); //this time we need a thread. thread.start(); Handler backgroundHandler = new Handler(thread.getLooper()); mPreviewSession.setRepeatingRequest(mPreviewBuilder.build(), null, backgroundHandler); // Request endlessly repeating capture of images by this session and place it in the surfaceview } public void onConfigureFailed(CameraCaptureSession session) { Toast.makeText(context, "onConfigureFailed", Toast.LENGTH_LONG).show(); }, null); //null again is for a handler we don’t need.

Now we have preview To take a picture, it has a very similar method but we start with the createCaptureSession method. We need to setup an ImageReader, so we can save it as JPEG (or other format), another handler, Surfaces, plus another captureBuilder

Taking the picture First setup some variables we need reader = ImageReader.newInstance(width, height, ImageFormat.JPEG, 1); Width and height handled in the code, but not shown here. outputSurfaces = new ArrayList<Surface>(2); outputSurfaces.add(reader.getSurface()); //here the surface is a reader, which will save to a file later on. Again add more surfaces here as needed. Setup our handler threads here. HandlerThread thread = new HandlerThread("CameraPicture"); thread.start(); backgroudHandler = new Handler(thread.getLooper()); Readlistner is shown later. reader.setOnImageAvailableListener(readerListener, backgroudHandler); //this is where the save is. configure the catureBuilder, which is built in listener later on. captureBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE); captureBuilder.addTarget(reader.getSurface()); captureBuilder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO); Fix Orientation, getJpegOreientation is shown in the code, but not here. The build for capture is in a lister. int deviceorientation = context.getResources().getConfiguration().orientation; captureBuilder.set(CaptureRequest.JPEG_ORIENTATION, getJpegOrientation(characteristics, deviceorientation));

readerListener A new image is available, ie the capture, so save file. ImageReader.OnImageAvailableListener readerListener = new ImageReader.OnImageAvailableListener() { @Override public void onImageAvailable(ImageReader reader) { Image image = null; image = reader.acquireLatestImage(); ByteBuffer buffer = image.getPlanes()[0].getBuffer(); byte[] bytes = new byte[buffer.capacity()]; buffer.get(bytes); save(bytes); } This is just a helper method to save the file which at this point is just an array of bytes. private void save(byte[] bytes) throws IOException { OutputStream output = null; try { output = new FileOutputStream(file); output.write(bytes); } finally { if (null != output) { output.close(); } };

Taking the picture (2) This is the line you call to actually take the picture mCameraDevice.createCaptureSession( outputSurfaces, mCaptureStateCallback, backgroudHandler); Like in the preview, we need the stateCallback The outputSurfaces, and handler were declared 2 slides back.

Taking the picture (3) So this CameraCaptureSession.StateCallback doesn’t deal with the preview, instead is readies to save a file. CameraCaptureSession.StateCallback mCaptureStateCallback = new CameraCaptureSession.StateCallback() { @Override public void onConfigured(CameraCaptureSession session) { session.capture(captureBuilder.build(), captureListener, backgroudHandler); } public void onConfigureFailed(CameraCaptureSession session) { }; And the last call back CameraCaptureSession.CaptureCallback captureListener = new CameraCaptureSession.CaptureCallback() { public void onCaptureCompleted(CameraCaptureSession session, CaptureRequest request, TotalCaptureResult result) { super.onCaptureCompleted(session, request, result); startPreview(); //start the preview again.

Taking A Picture With an INTENT

Via an Intent. You can also have the default “camera app” take the picture and return it to your app. Also doesn’t require any permissions. Intent intent = new Intent( android.provider.MediaStore.ACTION_IMAGE_CAPTURE); startActivityForResult(intent, myID); Picture is returned in onActivityResult

Via an Intent (2) protected void onActivityResult(int requestCode, int resultCode, Intent data) { super.onActivityResult(requestCode, resultCode, data); //get the picture out of the intent and display in an image view. Bitmap bp = (Bitmap) data.getExtras().get("data"); iv.setImageBitmap(bp); }

Android Example Code A note for the example code PicCapture Remember it uses touch to take the picture. Both a V1 and v2 APIs are used in separate fragments. PicCapture2 uses a button and separates the code a little better. PicCapture3 uses an intent to take the picture. CameraPreview is only camera2 APIs and it separated up into multiple classes to make the code easier to read. Similar to how it shown in the slides.

Android android.media Recording VIDEO

Recording VIDEO Camera V1 APIs

First… Most examples and code on the web and from the android books, DO NOT work. lots of subtle errors debugging is made more difficult, because the camera throws errors CameraInput Recording is not ready … frame dropped. the AudioFlinger shows constant buffer overflow warnings. And this is when the app is working correctly.

Uses a surfaceView Like a taking a picture, we need a view finder which uses a surfaceView. Call for the MediaRecorder and setup the encoding. both audio and video.

Example SurfaceView in onCreate recorder = new MediaRecorder(); //setup recorder settings Next Slide SurfaceView cameraView = (SurfaceView) findViewById(R.id.CameraView); holder = cameraView.getHolder(); holder.addCallback(this); holder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS); implemented methods. public void surfaceCreated(SurfaceHolder holder) { recorder.setPreviewDisplay(holder.getSurface()); recorder.prepare(); }

Example SurfaceView (2) public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) { } public void surfaceDestroyed(SurfaceHolder holder) { if (recording) { recorder.stop(); recording = false; recorder.release(); finish();

Recorder Settings We need to set the sources for audio and video recorder.setAudioSource( MediaRecorder.AudioSource.MIC); recorder.setVideoSource( MediaRecorder.VideoSource.DEFAULT); MediaRecorder.VideoSource.CAMERA should work as well. Now we need to setup encoders. In android 2.2 we can use profiles instead of setting everything manually. CamcorderProfile.QUALITY_HIGH or QUALITY_LOW CamcorderProfile cpHigh = CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH); recorder.setProfile(cpHigh); And set the output location recorder.setOutputFile("/sdcard/videocapture_example.mp4");

Recorder Settings (2) Manual settings could look like this: recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4); recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB); recorder.setVideoEncoder(MediaRecorder.VideoEncoder.MPEG_4_SP); QUALITY_HIGH settings are MP4 file QUALITY_LOW settings are 3gp file The manual list is very long, see the android doc’s or Apress - Pro Android Media Developing Graphics, Music, Video, and Rich Media Apps for Smartphones and Tablets, Chapter 11 for a full list of settings.

Recording To record recorder.start(); To stop recorder.stop(); The file should be there at this point. Remember when you are done recorder.release();

Example code The example code need an Sdcard to test code and the file will be located /sdcard/videocapture_example.mp4 The code uses an extended surfaceView call captureSurface, instead of just a surfaceView The code is all there, but rearranged from the slides. Honesty, the code just didn’t work without an extended SurfaceView The code starts up with the viewfinder, touch the screen to start recording, again to stop recording. It will then launch the native media player to replay the video.

Recording VIDEO Camera V2 APIs

Camera2 APIs. Not ready yet.

References http://developer.android.com/intl/zh-CN/guide/topics/media/index.html http://www.brighthub.com/mobile/google-android/articles/43414.aspx (a difficult example to follow and it’s for 1.6) Apress - Pro Android Media Developing Graphics, Music, Video, and Rich Media Apps for Smartphones and Tablets Chapter 2 for taking pictures, chapter 9 for video playback. API examples, com.example.android.apis.media Camera2 https://github.com/googlesamples/android-Camera2Video http://developer.android.com/reference/android/hardware/camera2/package-summary.html

Q A &