1. Android media Part 2: Pictures and Video
Cosc 5/4730
Android media Part 2:
Pictures and Video

2. Emulators and Samsung
The emulators all have problems, much of the following code has been tested on the actual phones
Video playback code works, but the video may not always display in the emulators
android
Video/picture capture does not work as documented.
Samsung hardware!
Some Samsung devices have many problems. Something about their hardware… search for the video and samsung for some possible answers.

3. Android android.media
Play media

4. Supported formats
In general Android’s support is consistent with other mobile phones.
It supports the 3GP (.3gp) and MPEG-4 (.mp4) file formats.
3GP is a video standard derived from MPEG-4 specifically for use by mobile devices.
As far as codecs go, Android supports H.263, a codec designed for low-latency and low-bitrate videoconferencing applications. H.263 video is supported in either MPEG-4 (.mp4) or 3GP (.3gp) files. Android also supports MPEG-4 Simple Profile in 3GP files (.3gp) as well as H.264.

5. Android First method The second method uses a VideoView to display.
For greater flexibility you will need to use the mediaPlayer and a surfaceView (or TextureView API 14+)
MediaPlayer like the audio and use a surfaceView to display the video.
There are examples in the API demo, plus several of the books.
The second method uses a VideoView to display.
The VideoView uses a MediaController to start and stop and provide functions to control the video play back.
With a MediaPlayer like the audio. prepare(), then start()
This is one I’ll cover in this lecture.

6. VideoView
VideoView is a View that has video playback capabilities and can be used directly in a layout.
We can then add controls (play, pause, forward, back, etc) with the MediaController.

7. ViewView example Get the ViewView out of the layout
vv = (VideoView) this.findViewById( R.id.VideoView);
Setup where the file to play is
Uri videoUri = Uri.parse(Environment.getExternalStorageDirectory().getPath() + "/example.mp4");
vv.setVideoURI(videoUri);
play the video
vv.start();

8. Adding media controllers.
Very simple
vv = (VideoView) this.findViewById(R.id.VideoView);
vv.setMediaController(new MediaController(this));
Now media controls will show up on the screen.

9. Using native media player
Call for an Intent and send it.
Uri data = Uri.parse(VideoFile);
intent.setDataAndType(data, "video/mp4");
startActivity(intent);
Remember, your app is now in the background.

10. Example code The VideoPlay example code
This will play a video from the internet.
If you can uncomment the code to have it play the video from the sdcard, but you will need to copy the file to the sdcard.

11. Displaying a picture

12. Displaying Pictures See code already covered to how display pictures.
ImageView for example…

13. Android
Taking a picture

14. Camera vs Camera2 API
Starting in API 21 (Lollipop) there is a new set of APIs.
There is no backward compatibly either.
So first we look at Camera v1 APIs
And then Camera2 APIs
These are more flexible and should allow for “filters” to be added on the fly to image.
Finally, you can just use an intent have the native camera app take the picture.

15. Taking A Picture
Camera V1 APIs

16. What to use Android packages Permissions and features
import android.hardware.CameraDevice;
import android.hardware.CameraDevice.CaptureParams;
Permissions and features
<uses-permission android:name="android.permission.CAMERA" />
Note, This one will requirement permission checking in API 23+
This too, if you change the how the camera is functioning.
<uses-feature android:name="android.hardware.camera" />
<uses-feature android:name="android.hardware.camera.autofocus" />

17. Taking a picture In brief
CameraDevice cameraDevice = CameraDevice.open()
To preview you need a surfaceHolder then use setPreviewDisplay(surfaceHolder) and cameraDevice.startPreview()
Finally use the takePicture(…) to get the picture
release() and close() the CameraDevice to release it.

18. “View Finder”
The view finder is implemented via a SurfaceHolder and SurfaceView
In the layout, the a surfaceView is used.
Example:
public class PicCapture extends Activity implements OnClickListener, SurfaceHolder.Callback, Camera.PictureCallback { …
cameraView = (SurfaceView) this.findViewById(R.id.CameraView);
surfaceHolder = cameraView.getHolder();
surfaceHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
surfaceHolder.addCallback(this);
To finally take the picture we need all this too.
cameraView.setFocusable(true);
cameraView.setFocusableInTouchMode(true);
cameraView.setClickable(true);
cameraView.setOnClickListener(this);

19. “View Finder” (2)
The code to implement the surfaceHolder can be very simple.
When created open the camera and set the display
surfaceCreated() {
camera = Camera.open();
try { camera.setPreviewDisplay(holder);
catch (IOException exception) {
camera.release(); } }
Once it’s ready, start the preview
surfaceChanged() {
camera.startPreview();

20. “View Finder” (3) When we are done surfaceDestroyed() {
camera.stopPreview();
camera.release(); }

21. Get the Picture
Using the Camera.PictureCallBack we implement, we can get the data for the picture and decide what to do it with it.
In its simplest form, which doesn’t nothing with the picture
public void onPictureTaken(byte[] data, Camera camera) {
byte[] data is the picture that was taken
this just restarts the preview.
camera.startPreview(); }

22. Get the Picture (2) To take the picture we use the
camera.takePicture method in the onClick method for the SurfaceView
public void onClick(View v) {
camera.takePicture(null, null, null, this); }
takePicture (Camera.ShutterCallback shutter, Camera.PictureCallback raw, Camera.PictureCallback postview , Camera.PictureCallback jpeg)
We only need the last to get the picture and it show on the previous slide.
shutter
the callback for image capture moment, or null
raw
the callback for raw (uncompressed) image data, or null
postview
the callback with postview image data, may be null
jpeg
the callback for JPEG image data, or null

23. Taking A Picture
Camera V2 APIs

24. Camera2 APIs Like v1, this breaks up into 2 major pieces Viewfinder
Connect it to a SurfaceView or TextureView.
Taking the picture

25. Camera2 ViewFinder First find the “id” for the camera to use
CameraManager manager = (CameraManager) getActivity().getSystemService(Context.CAMERA_SERVICE);
String cameraId = manager.getCameraIdList()[0];
In this case just use the first one (tends to be the back camera)
Then is gets complex, because we open the camera via this
manager.openCamera (String cameraId, CameraDevice.StateCallback callback, Handler handler)
StateCallback is a listener “first” and the handler is this case is likely null.
The handler is the thread to use, null for main. Another thread is likely where you would apply a filter to the image, before it is displayed.
Note, someone want to figure this out?

26. Camera2 ViewFinder (2)
The call back is pretty simple, but the complex code is here.
CameraDevice.StateCallback mStateCallback = new CameraDevice.StateCallback() {
@Override
public void onOpened(CameraDevice camera) {
mCameraDevice = camera;
//We now have the camera, so setup the capture of the current surface.
startPreview(); //in a method since it will be called several times. }
public void onDisconnected(CameraDevice camera) {
Log.e(TAG, "onDisconnected");
public void onError(CameraDevice camera, int error) {
Log.e(TAG, "onError"); };

27. startPreview
First we need a surface (from the SurfaceView or Textview). We are using a surfaceView
Surface surface = mHolder.getSurface();
mHolder is our surfaceview
mPreviewBuilder = mCameraDevice.createCaptureRequest( CameraDevice.TEMPLATE_PREVIEW);
mPreviewBuilder.addTarget(surface);
Note, if wanted to display it on more then 1 surfaceview, we then add the rest here. (I think).
mCameraDevice.createCaptureSession(Arrays.asList(surface), CameraCaptureStateCallback);
The CameraCapture is listed on the next slide.

28. startPreview(2)
CameraCaptureSession.StateCallback CameraCaptureStateCallback = new CameraCaptureSession.StateCallback() {
@Override
public void onConfigured(CameraCaptureSession session) {
mPreviewSession = session;
//now setup the update preview.
mPreviewBuilder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO);
HandlerThread thread = new HandlerThread("CameraPreview"); //this time we need a thread.
thread.start();
Handler backgroundHandler = new Handler(thread.getLooper());
mPreviewSession.setRepeatingRequest(mPreviewBuilder.build(), null, backgroundHandler);
// Request endlessly repeating capture of images by this session and place it in the surfaceview }
public void onConfigureFailed(CameraCaptureSession session) {
Toast.makeText(context, "onConfigureFailed", Toast.LENGTH_LONG).show();
}, null); //null again is for a handler we don’t need.

29. Now we have preview
To take a picture, it has a very similar method but we start with the createCaptureSession method.
We need to setup an ImageReader, so we can save it as JPEG (or other format), another handler, Surfaces, plus another captureBuilder

30. Taking the picture First setup some variables we need
reader = ImageReader.newInstance(width, height, ImageFormat.JPEG, 1);
Width and height handled in the code, but not shown here.
outputSurfaces = new ArrayList<Surface>(2);
outputSurfaces.add(reader.getSurface()); //here the surface is a reader, which will save to a file later on.
Again add more surfaces here as needed.
Setup our handler threads here.
HandlerThread thread = new HandlerThread("CameraPicture"); thread.start();
backgroudHandler = new Handler(thread.getLooper());
Readlistner is shown later.
reader.setOnImageAvailableListener(readerListener, backgroudHandler); //this is where the save is.
configure the catureBuilder, which is built in listener later on.
captureBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE);
captureBuilder.addTarget(reader.getSurface());
captureBuilder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO);
Fix Orientation, getJpegOreientation is shown in the code, but not here. The build for capture is in a lister.
int deviceorientation = context.getResources().getConfiguration().orientation;
captureBuilder.set(CaptureRequest.JPEG_ORIENTATION, getJpegOrientation(characteristics, deviceorientation));

31. readerListener A new image is available, ie the capture, so save file.
ImageReader.OnImageAvailableListener readerListener = new ImageReader.OnImageAvailableListener() {
@Override
public void onImageAvailable(ImageReader reader) {
Image image = null;
image = reader.acquireLatestImage();
ByteBuffer buffer = image.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.capacity()];
buffer.get(bytes);
save(bytes); }
This is just a helper method to save the file which at this point is just an array of bytes.
private void save(byte[] bytes) throws IOException {
OutputStream output = null;
try {
output = new FileOutputStream(file);
output.write(bytes);
} finally {
if (null != output) {
output.close(); } };

32. Taking the picture (2)
This is the line you call to actually take the picture
mCameraDevice.createCaptureSession( outputSurfaces, mCaptureStateCallback, backgroudHandler);
Like in the preview, we need the stateCallback
The outputSurfaces, and handler were declared 2 slides back.

33. Taking the picture (3)
So this CameraCaptureSession.StateCallback doesn’t deal with the preview, instead is readies to save a file.
CameraCaptureSession.StateCallback mCaptureStateCallback = new CameraCaptureSession.StateCallback() {
@Override
public void onConfigured(CameraCaptureSession session) {
session.capture(captureBuilder.build(), captureListener, backgroudHandler); }
public void onConfigureFailed(CameraCaptureSession session) { };
And the last call back
CameraCaptureSession.CaptureCallback captureListener = new CameraCaptureSession.CaptureCallback() {
public void onCaptureCompleted(CameraCaptureSession session,
CaptureRequest request, TotalCaptureResult result) {
super.onCaptureCompleted(session, request, result);
startPreview(); //start the preview again.

34. Taking A Picture
With an INTENT

35. Via an Intent.
You can also have the default “camera app” take the picture and return it to your app.
Also doesn’t require any permissions.
Intent intent = new Intent( android.provider.MediaStore.ACTION_IMAGE_CAPTURE);
startActivityForResult(intent, myID);
Picture is returned in onActivityResult

36. Via an Intent (2)
protected void onActivityResult(int requestCode, int resultCode, Intent data) { super.onActivityResult(requestCode, resultCode, data); //get the picture out of the intent and display in an image view. Bitmap bp = (Bitmap) data.getExtras().get("data"); iv.setImageBitmap(bp); }

37. Android Example Code A note for the example code PicCapture
Remember it uses touch to take the picture.
Both a V1 and v2 APIs are used in separate fragments.
PicCapture2 uses a button and separates the code a little better.
PicCapture3 uses an intent to take the picture.
CameraPreview is only camera2 APIs and it separated up into multiple classes to make the code easier to read.
Similar to how it shown in the slides.

38. Android android.media
Recording VIDEO

39. Recording VIDEO
Camera V1 APIs

40. First…
Most examples and code on the web and from the android books, DO NOT work.
lots of subtle errors
debugging is made more difficult, because
the camera throws errors
CameraInput Recording is not ready … frame dropped.
the AudioFlinger shows constant buffer overflow warnings.
And this is when the app is working correctly.

41. Uses a surfaceView
Like a taking a picture, we need a view finder which uses a surfaceView.
Call for the MediaRecorder
and setup the encoding.
both audio and video.

42. Example SurfaceView in onCreate recorder = new MediaRecorder();
//setup recorder settings Next Slide
SurfaceView cameraView = (SurfaceView) findViewById(R.id.CameraView);
holder = cameraView.getHolder();
holder.addCallback(this); holder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
implemented methods.
public void surfaceCreated(SurfaceHolder holder) {
recorder.setPreviewDisplay(holder.getSurface());
recorder.prepare(); }

43. Example SurfaceView (2)
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) { }
public void surfaceDestroyed(SurfaceHolder holder) {
if (recording) {
recorder.stop();
recording = false;
recorder.release();
finish();

44. Recorder Settings We need to set the sources for audio and video
recorder.setAudioSource( MediaRecorder.AudioSource.MIC); recorder.setVideoSource(
MediaRecorder.VideoSource.DEFAULT);
MediaRecorder.VideoSource.CAMERA should work as well.
Now we need to setup encoders.
In android 2.2 we can use profiles instead of setting everything manually.
CamcorderProfile.QUALITY_HIGH or QUALITY_LOW
CamcorderProfile cpHigh = CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH);
recorder.setProfile(cpHigh);
And set the output location
recorder.setOutputFile("/sdcard/videocapture_example.mp4");

45. Recorder Settings (2) Manual settings could look like this:
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB);
recorder.setVideoEncoder(MediaRecorder.VideoEncoder.MPEG_4_SP);
QUALITY_HIGH settings are MP4 file
QUALITY_LOW settings are 3gp file
The manual list is very long, see the android doc’s or Apress - Pro Android Media Developing Graphics, Music, Video, and Rich Media Apps for Smartphones and Tablets, Chapter 11 for a full list of settings.

46. Recording To record recorder.start(); To stop recorder.stop();
The file should be there at this point.
Remember when you are done
recorder.release();

47. Example code The example code
need an Sdcard to test code and the file will be located /sdcard/videocapture_example.mp4
The code uses an extended surfaceView call captureSurface, instead of just a surfaceView
The code is all there, but rearranged from the slides.
Honesty, the code just didn’t work without an extended SurfaceView
The code starts up with the viewfinder, touch the screen to start recording, again to stop recording. It will then launch the native media player to replay the video.

48. Recording VIDEO
Camera V2 APIs

49. Camera2 APIs.
Not ready yet.

50. References
(a difficult example to follow and it’s for 1.6)
Apress - Pro Android Media Developing Graphics, Music, Video, and Rich Media Apps for Smartphones and Tablets
Chapter 2 for taking pictures, chapter 9 for video playback.
API examples, com.example.android.apis.media
Camera2

51. Q A
&