As our smartphones become more powerful, we can do more advanced things that previously required a high-end PC. One way to make use of the robust processing power of your smartphone is Computer Vision – the ability for a device to acquire, process, analyse and understand images the same way images are perceived by human eyes. Basically, we can use the powerful CPU in modern smartphones to interpret the images captured through the camera. Examples of use cases are face detection and recognition or simple post-processing of photographs. The best approach to using Computer Vision on Android is through library called OpenCV. Read on as Erik Hellman, research engineer at Sony, explains more.

Erik Hellman, research engineer at Sony.

The OpenCV library of programming functions has become the de-facto industry standard for software based computer vision. It is completely open-sourced (BSD licensed) and has support for most major platforms, including Android. The current version is 2.4.3, and you can download the Open CV SDK for free and start writing your OpenCV-based Android apps from the OpenCV website.

In this tutorial I will cover three things that will help you to get started with OpenCV on Android:

  1. How to acquire images from the Android camera through OpenCV.
  2. How to process an image to retrieve the parts within a certain colour range only.
  3. How to display the result of the processed image.

Before you can do any of the above, you need to load the OpenCV library. OpenCV for Android is designed as a standalone library APK that you install from Google Play. You also need to include the OpenCV SDK as an Android library project to your own Android project. Instructions on how to do this can be found in the OpenCV online documentation.

How to acquire camera images using OpenCV
In this first example, we will rely on the preview frames from the camera since we can retrieve these images relatively fast. We need to pick an appropriate size for the preview frames, as too small frames will result in a bad result when we do the processing, and too large frames will slow down everything to an unacceptable level. Since smartphone cameras often have a different set of supported sizes for preview frames, we need to pick a minimum acceptable size and go through all of them to find the correct one. OpenCV has its own Java API, and doesn’t rely on the Java-based Android Camera API. The following code opens the camera and sets the preview size to an acceptable level.

  1. private void setupCamera() {
  2.  
  3.     if (mCamera != null) {
  4.        VideoCapture camera = mCamera;
  5.        mCamera = null; // Make it null before releasing...
  6.       camera.release();
  7.     }
  8.  
  9.     mCamera = new VideoCapture(mCameraId);
  10.  
  11.     List<Size> previewSizes = mCamera.getSupportedPreviewSizes();
  12.     double smallestPreviewSize = 1280 * 720; // We should be smaller than this...
  13.     double smallestWidth = 480; // Let's not get smaller than this...
  14.     for (Size previewSize : previewSizes) {
  15.         if (previewSize.area() < smallestPreviewSize && previewSize.width >= smallestWidth) {
  16.             mPreviewSize = previewSize;
  17.         }
  18.     }
  19.  
  20. mCamera.set(Highgui.CV_CAP_PROP_FRAME_WIDTH, mPreviewSize.width);
  21. mCamera.set(Highgui.CV_CAP_PROP_FRAME_HEIGHT, mPreviewSize.height);
  22. }

An important thing to remember when working with OpenCV is that we need a lot of RAM for the images we are processing. OpenCV uses matrices to represent all images so to be as efficient as possible, we will allocate the needed matrices once and re-use them for every frame.

Once we have the camera setup and the matrices allocated, we can start to grab preview frames for processing. The following code shows how we do this with OpenCV.

  1. while (mDoProcess && mCamera != null) {
  2.     boolean grabbed = mCamera.grab();
  3.     if (grabbed) {
  4.         mCamera.retrieve(mCurrentFrame,
  5. Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGB);
  6.         // Process mCurrentFrame…
  7.     }
  8. }

Screen shot showing the camera frame. In this example app, you can tap a colour to display that colour only.

How to process a camera frame
We now have a preview frame in mCurrentFrame that we can use for processing. In this case, we will use the function inRange to create a mask that covers only a specific colour interval. For the best result, we will first convert the grabbed frame to HSV-format. Once we have the mask, we draw the original frame to a new matrix using the mask we got from the inRange function. The following code shows how to convert the format from RGB to HSV, generate the colour mask and drawing the result.

  1. // Convert mCurrentFrame to HSV format and store
  2. the result in mCurrentFrameHsv
  3. Imgproc.cvtColor(mCurrentFrame, mCurrentFrameHsv,
  4. Imgproc.COLOR_RGB2HSV);
  5. // Generate the color mask based on the lower and upper
  6. color limits
  7. Core.inRange(mCurrentFrameHsv, mLowerColorLimit,
  8. mUpperColorLimit, mInRangeResult);
  9. // Clear the final image before we copy the new filtered image
  10. mFilteredFrame.setTo(new Scalar(0, 0, 0));
  11. // Copy the original frame using the color mask
  12. mCurrentFrame.copyTo(mFilteredFrame, mInRangeResult);

Screen shot showing the processed camera frame when a colour has been selected. As you can see, only the red colour range is displayed.

How to display the result of the processed image
The resulting bitmap can now be used to draw to a SurfaceView or similar to show the result. The easiest way to do this is to convert the OpenCV matrix to an Android Bitmap object. There is a Util class that provides a function for this.

  1. // Convert the matrix to an Android bitmap
  2. Utils.matToBitmap(result, resultBitmap, true);

From this point, you can simply draw the Bitmap as usual on a SurfaceView, or assign it to an ImageView.

Deeper dive and more OpenCV code examples are available
This was a very quick walkthrough of OpenCV. To get a deeper look into this, you can download more extensive versions of these OpenCV code examples. Feel free to use this anyway you want to experiment with different OpenCV functions. There are lots of more tutorials available on the OpenCV website, and you can find additional code examples at other resources online.

So are you interested in OpenCV? Do you plan to try it out? Let us know in the comments below!

More information

  • Download the OpenCV code examples.
  • Learn more about OpenCV.
  • Read about Computer Vision.