手持设备上的摄像头

1. 硬件和基本物理原理

a) CMOS & CCD

笼统的看,CMOS和CCD都属于传感器的集合，用于将光信号转换为电信号。通常所说的物理像素(非插值运算像素)指的就是单位面积上集成的感光元件的数量。比如Canon eos 5D的参数：

• 传感器类型：CMOS
• 传感器尺寸：35.8 × 23.9 mm
• 最高分辨率：4368×2912
• 图像分辨率：4368×2912/3168×2112/2496×1664

这就意味着在35.8x23.9mm的面积上集成了4368x2912=12719616个光电传感器。实际的有效像素也就是大约1280万。

CCD的成像原理

CCD（Charge Coupled Device），中文名字叫电荷耦合器，是一种特殊的半导体材料。它由大量独立的光敏元件组成，这些光敏元件通常是按矩阵排列的。光线透过镜头照射到CCD上，并被转换成电荷，每个元件上的电荷量取决于它所受到的光照强度。当你按动快门，CCD将各个元件的信息传送到模/数转换器上，模拟电信号经过模/数转换器处理后变成数字信号，数字信号以一定格式压缩后存入缓存内，此时一张数码照片诞生了。然后图像数据根据不同的需要以数字信号和视频信号的方式输出。

CMOS的成像原理

CMOS即互补金属氧化物半导体，它在微处理器、闪存和ASIC（特定用途集成电路）的半导体技术上占有绝对重要的地位。CMOS和CCD一样都是可用来感受光线变化的半导体。CMOS主要是利用硅和锗这两种元素所做成的半导体，通过CMOS上带负电和带正电的晶体管来实现基本的功能。这两个互补效应所产生的电流即可被处理芯片记录和解读成影像。

通常情况下，CMOS成本较低。因为低端的CMOS存在晶体管电流快速变化是产生过热，容易出现噪点的问题。CCD相对来说比较高端，很少见智能手机上的摄像头采用CCD作为感光元件。

b)背照式

在传统CMOS感光元件中，感光二极管位于电路晶体管后方，进光量会因遮挡受到影响。所谓背照式CMOS就是将它掉转方向，让光线首先进入感光二极管，从而增大感光量，显著提高低光照条件下的拍摄效果。

c)YUV &  RGB & RAW数据格式

从CMOS/CCD采集到的数据被称为raw data。RAW Data直接决定于各种图像采集的硬件。Camera Sensor输出的图像既可以是YUV数据，也可以是JPEG数据，也可以是RAW数据。具体要看硬件设计。

尽管jpeg是非常优秀的图片存储格式，但是jpeg为了满足在较小容量的文件中存储比较多的颜色信息，压缩的过程中也丢掉了很多图像信息。这也是高端的数码相机选择输出RAW Data,并提供专门的处理软件的原因。

针对RAW格式数据的处理，windows和mac上的很多专用软件都是闭源的。看到老外搞了一个开源的raw数据编码项目 DCRAW后续再专门读一下相关的代码。

YUV颜色空间是从亮度(灰度级)来记录颜色信息，并且Y(亮度)跟U(色度)和V(对比度)是分离的。以下是从MSDN上摘录的YUV采样的描述：

One of the advantages of YUV is that the chroma channels can have a lower sampling rate than the Y channel without a dramatic degradation of the perceptual quality. A notation called the A:B:C notation is used to describe how often U and V are sampled relative to Y:

• 4:4:4 means no downsampling of the chroma channels.
• 4:2:2 means 2:1 horizontal downsampling, with no vertical downsampling. Every scan line contains four Y samples for every two U or V samples.
• 4:2:0 means 2:1 horizontal downsampling, with 2:1 vertical downsampling.
• 4:1:1 means 4:1 horizontal downsampling, with no vertical downsampling. Every scan line contains four Y samples for every U or V sample. 4:1:1 sampling is less common than other formats, and is not discussed in detail in this article.

从上述描述可以看出：YUV颜色空间确保RAW Data记录的亮度信息不丢失，通过downsample U/V分量来降低编码的负载。

Figure 1 shows the sampling grid used in 4:4:4 pictures. Luma samples are represented by a cross, and chroma samples are represented by a circle.

Figure 1. YUV 4:4:4 sample positions

The dominant form of 4:2:2 sampling is defined in ITU-R Recommendation BT.601. Figure 2 shows the sampling grid defined by this standard.

Figure 2. YUV 4:2:2 sample positions

There are two common variants of 4:2:0 sampling. One of these is used in MPEG-2 video, and the other is used in MPEG-1 and in ITU-T recommendations H.261 and H.263. Figure 3 shows the sampling grid used in the MPEG-1 scheme, and Figure 4 shows the sampling grid used in the MPEG-2 scheme.

Figure 3. YUV 4:2:0 sample positions (MPEG-1 scheme)

Figure 4. YUV 4:2:0 sample positions (MPEG-2 scheme)

Compared with the MPEG-1 scheme, it is simpler to convert between the MPEG-2 scheme and the sampling grids defined for 4:2:2 and 4:4:4 formats. For this reason, the MPEG-2 scheme is preferred in Windows, and should be considered the default interpretation of 4:2:0 formats.

下面是从MSDN上摘录下来的YUV的数据排列情况的描述：

This section describes the 8-bit YUV formats that are recommended for video rendering. These fall into several categories:

• 4:4:4 Formats, 32 Bits per Pixel
• 4:2:2 Formats, 16 Bits per Pixel
• 4:2:0 Formats, 16 Bits per Pixel
• 4:2:0 Formats, 12 Bits per Pixel

First, you should be aware of the following concepts in order to understand what follows:

• Surface origin. For the YUV formats described in this article, the origin (0,0) is always the upper-left corner of the surface.
• Stride. Thestride of a surface, sometimes called the pitch, is the width of the surface in bytes. Given a surface origin at the upper-left corner, the stride is always positive. 
• Alignment. The alignment of a surface is at the discretion of the graphics display driver. The surface must always be DWORD aligned, that is, individual lines within the surface are guaranteed to originate on a 32-bit (DWORD) boundary. The alignment can be larger than 32 bits, however, depending on the needs of the hardware.
• Packed format versus planar format. YUV formats are divided intopacked formats andplanarformats. In a packed format, the Y, U, and V components are stored in a single array. Pixels are organized into groups of macropixels, whose layout depends on the format. In a planar format, the Y, U, and V components are stored as three separate planes.

A single 4:4:4 format is recommended, with the FOURCC code AYUV. This is a packed format, where each pixel is encoded as four consecutive bytes, arranged in the following sequence.

Figure 5. AYUV memory layout

The bytes marked A contain values for alpha.

Two 4:2:2 formats are supported, with the following FOURCC codes:

• YUY2
• UYVY

Both are packed formats, where each macropixel is two pixels encoded as four consecutive bytes. This results in horizontal downsampling of the chroma by a factor of two.

In YUY2 format, the data can be treated as an array of unsignedchar values, where the first byte contains the first Y sample, the second byte contains the first U (Cb) sample, the third byte contains the second Y sample, and the fourth byte contains the first V (Cr) sample, as shown in Figure 6.

Figure 6. YUY2 memory layout

If the image is addressed as an array of two little-endianWORD values, the firstWORD contains Y0 in the least significant bits (LSBs) and U in the most significant bits (MSBs). The secondWORD contains Y1 in the LSBs and V in the MSBs.

YUY2 is the preferred 4:2:2 pixel format for Microsoft DirectX® Video Acceleration (DirectX VA). It is expected to be an intermediate-term requirement for DirectX VA accelerators supporting 4:2:2 video.

This format is the same as YUY2, except the byte order is reversed — that is, the chroma and luma bytes are flipped (Figure 7). If the image is addressed as an array of two little-endianWORD values, the first WORD contains U in the LSBs and Y0 in the MSBs, and the secondWORD contains V in the LSBs and Y1 in the MSBs.

Figure 7. UYVY memory layout

Two 4:2:0 16-bits per pixel formats are recommended, with the following FOURCC codes:

• IMC1
• IMC3

Both FOURCC codes are planar formats. The chroma channels are subsampled by a factor of two in both the horizontal and vertical dimensions. 

All of the Y samples appear first in memory as an array of unsignedchar values. This is followed by all of the V (Cr) samples, and then all of the U (Cb) samples. The V and U planes have the same stride as the Y plane, resulting in unused areas of memory, as shown in Figure 8.

Figure 8. IMC1 memory layout

This format is identical to IMC1, except the U and V planes are swapped:

Figure 9. IMC3 memory layout

Four 4:2:0 12-bpp formats are recommended, with the following FOURCC codes:

• IMC2
• IMC4
• YV12
• NV12

In all of these formats, the chroma channels are subsampled by a factor of two in both the horizontal and vertical dimensions. 

This format is the same as IMC1 except that the V (Cr) and U (Cb) lines are interleaved at half-stride boundaries. In other words, each full-stride line in the chroma area starts with a line of V samples, followed by a line of U samples that begins at the next half-stride boundary (Figure 10). This layout makes more efficient use of address space than IMC1. It cuts the chroma address space in half, and thus the total address space by 25 percent. Among 4:2:0 formats, IMC2 is the second-most preferred format, after NV12.

Figure 10. IMC2 memory layout

This format is identical to IMC2, except the U (Cb) and V (Cr) lines are swapped:

Figure 11. IMC4 memory layout

All of the Y samples appear first in memory as an array of unsignedchar values. This array is followed immediately by all of the V (Cr) samples. The stride of the V plane is half the stride of the Y plane, and the V plane contains half as many lines as the Y plane. The V plane is followed immediately by all of the U (Cb) samples, with the same stride and number of lines as the V plane (Figure 12).

Figure 12. YV12 memory layout

All of the Y samples are found first in memory as an array of unsignedchar values with an even number of lines. The Y plane is followed immediately by an array of unsignedchar values that contains packed U (Cb) and V (Cr) samples, as shown in Figure 13. When the combined U-V array is addressed as an array of little-endianWORD values, the LSBs contain the U values, and the MSBs contain the V values. NV12 is the preferred 4:2:0 pixel format for DirectX VA. It is expected to be an intermediate-term requirement for DirectX VA accelerators supporting 4:2:0 video.

Figure 13. NV12 memory layout

d) JEPG压缩算法

JPEG是图像存储的常用格式。一幅RGB/YUV颜色空间的图像经过JPEG压缩，尽管会丢掉一小部分的颜色信息，但是文件的大小大大的缩小了，并且图像的画质也不会有太明显的变化。

JPEG编码算法粗略的分为如下几个步骤：

a) 色彩空间转换(转换到YUV颜色空间),此时一幅图像是一矩阵的形式记录着，并按YUV格式排列。

b)缩减采样，实际上就是根据YUV颜色空间的特性downsample U/V分量，从而减小要压缩数据的大小。

c) DCT(离散余弦变化)，把一个大矩阵划分成多个8x8的小矩阵，进行从时域到频域的数学变换，进一步缩小矩阵中数据的大小。

d) 量化，进一步压缩高频分量，把矩阵变换成对角阵。
至此一幅图像就压缩成JPEG格式的文件，体积大大的缩小。而解码则执行相反的操作。通常的额SOC都有专门的JPEG IP来完成这一步的操作，实现JPEG硬编和硬解，速度也非常快。当然也有专门的开源库来完成这些工作。譬如libjpeg。

JPEG压缩还涉及很多数学知识，可参阅JPEG的WIKI。

2.通讯和数据接口
a) i2c & usb
b) MIPI & YUV
c) hsync & vsync & mclk & pclk
d) frontend 和 backend 硬件结构


3.功能调试开发
a)preview的基本原理
b)capture的基本原理
自动对焦，记录当前预览参数，切换到拍照分辨率，配置拍照曝光和感光参数，等待两帧数据，记录第三帧数据，返回到预览
c)record的基本原理
d)autoFocus
--主动式 主动探测光强，反馈结果
--被动式
获取光线信息，驱动马达，调整焦距
e)flash
--camera模组本身有接口




4.高级功能与调优
a)场景模式
b)图像质量<亮度/饱和度/对比度/白平衡/ISO/曝光补偿>
c)场景模式<夜间模式...>
d)帧率优化
e)camera APK启动时间优化
f)camera 预览现场和拍照现场切换优化
g)拍照速度优化




5.软件结构
a) android.hardware.Camera
b) Camera Service
c) Camera HAL 
d)V4L

第一部分：控制流程

CameraHardwareInterface.h定义了用于实现设定，获取参数，预览，拍照的硬件抽象层接口。

CameraHardwareInterface的基类是RefBase。CameraService调用openCameraHardware()方法就能获得一个指向硬件抽象层类的实例的强类型指针，并且openCameraHardware()方法能够多次被调用来获得多个实例。

 

正常的流程如下：

CameraService调用openCameraHardware()之后，getParameters()和setParameters()将会被调用来初始化camera hardware abstract layer(HAL)实例。CameraService接着调用getPreviewHeap()方法来建立对preview堆内存的访问，接着用SurfaceFlinger来注册这段堆内存，这样在预览的时候就能够更高效率的刷新屏幕。接下来调用startpreview()方法，预览线程被启动，camera HAL在收到底层的一桢完整数据的时候就会发送一个CAMERA_MSG_PREVIEW_FRAME消息，预览线程回接收并处理相应的消息。

拍照的过程也很类似，有拍照的指令下来后，CameraService先调用autofocus()方法，自动对焦线程被启动，再完成自动对焦线程后，camera HAL送出一个CAMERA_MSG_FOCUS消息，自动对焦线程接收这个消息，并做相应的处理。接下来CameraService调用takePicture()方法来实现拍照的动作，takePicture回销毁预览线程同时打开拍照线程，拍照线程调用camera HAL完成拍照动作，拍照线程同样接收并处理Shutter,raw,compressed等消息。

 

Tcc8900 bsp camera硬件接口控制实例

 

1.硬件接口实例化

Tcc8900 android 2.3 bsp中TelechipsCameraHardware采用一种单例模式来设计。构造函数被设成私有，透过静态方法createInstance来获取类的实例。

TelechipsCameraHardware的构造函数如下：TelechipsCameraHardware::TelechipsCameraHardware(int cameraId) : mParameters(), mPreviewHeap(NULL), mRawHeap(NULL), #if defined(SENSOR_TVP5150) || defined(SENSOR_RDA5888) mRecHeap(NULL),#endif mV4l2Camera(0), mPreviewRunning(0), mPreviewFrameSize(0), mPreviewRealFrameSize(0), mCaptureRealFrameSize(0), mNotifyCb(0), mDataCb(0), mDataCbTimestamp(0), mCallbackCookie(0), mMsgEnabled(0), mCurrentPreviewFrame(0), mRecordRunning(false), mOverlay(NULL), mUseOverlay(false), mOverlay_Setuped(false),mRecordFrmSend_Total(0),mRecordFrmSend_Last(0),mRecordFrmRelease(false),mCamStatus(0){struct fb_var_screeninfo info;char value[PROPERTY_VALUE_MAX];unsigned intuiOutputMode = 0; mCameraIndex = cameraId;if(mV4l2Camera != NULL)delete mV4l2Camera;mV4l2Camera = new TCC_V4l2_Camera(mCameraIndex);pmap_get_info("temp", &mTempPmap);pmap_get_info("prior_temp", &mPriorTempPmap);mUseOverlay = useOverlay();memset(&mScaler_arg, 0x00, sizeof(mScaler_arg));#if defined(SENSOR_TVP5150) || defined(SENSOR_RDA5888)memset(&mScaler1_arg, 0x00, sizeof(mScaler_arg));#endifmemset(&mGrp_arg, 0x00, sizeof(mGrp_arg));CameraCmdQ = CameraAckQ = NULL;mPreviewThread = NULL;createPreviewThread();mFirst_Frame = mParallel = mOrder_Parallel = false;mNeed_scaler = mNeed_transform = mNeed_rotate = mNeed_crop = false;mRt_fd = mM2m_fd = mFb_fd = mOverlay_fd = mComposite_fd = mComponent_fd = -1;#if defined(SENSOR_TVP5150) || defined(SENSOR_RDA5888)mM2m_fd1 = -1;#endifmHDMIOutput = false;mCompositeOutput = false;mComponentOutput = false;mHdmi_width = mHdmi_height = 0;mFb_fd = open(FB0_DEVICE, O_RDWR);if (mFb_fd <= 0) {LOGE("can't open[%s] '%s'", strerror(errno), FB0_DEVICE);}ioctl( mFb_fd, FBIOGET_VSCREENINFO, &info);mLcd_width = info.xres;mLcd_height = info.yres;for (int i = 0; i < kBufferCount; i++) {mBuffers[i] = NULL;}#if defined(SENSOR_TVP5150) || defined(SENSOR_RDA5888)for (int i = 0; i < kBufferCount; i++) {mRecBuffers[i] = NULL;}#endif#ifdef USE_OUTPUT_DISPLAYproperty_get("persist.sys.output_mode", value, "");uiOutputMode = atoi(value);#if defined(TCC_HDMI_UI_SIZE_1280_720)if( uiOutputMode == OUTPUT_HDMI || uiOutputMode == OUTPUT_NONE)#elseif( uiOutputMode == OUTPUT_HDMI)#endif {#if !defined(TCC_HDMI_UI_SIZE_1280_720)if(mLcd_height > mLcd_width){TCC_HDMI_M hdmi_pause;hdmi_pause = TCC_HDMI_SUSEPNED;ioctl( mFb_fd, TCC_LCDC_HDMI_MODE_SET, &hdmi_pause);}else#endif{LOGV("HDMI output enabled");mHDMIOutput = true;}} if (uiOutputMode == OUTPUT_COMPOSITE){// COMPOSITE Device Open!!mComposite_fd = open(COMPOSITE_DEVICE, O_RDWR);if (mComposite_fd <= 0) {LOGE("can't open[%s] '%s'", strerror(errno), COMPOSITE_DEVICE);}mCompositeOutput = true;}else if (uiOutputMode == OUTPUT_COMPONENT){// COMPOSITE Device Open!!mComponent_fd = open(COMPONENT_DEVICE, O_RDWR);if (mComponent_fd <= 0) {LOGE("can't open[%s] '%s'", strerror(errno), COMPONENT_DEVICE);}mComponentOutput = true;}if(mHDMIOutput || mCompositeOutput || mComponentOutput){TCC_HDMI_M hdmi_video;hdmi_video = TCC_HDMI_VIDEO_START;if(ioctl( mFb_fd, TCC_LCDC_HDMI_MODE_SET, &hdmi_video) != 0){LOGE("Fb0 HDMI ctrl Error!");}}#else{unsigned int uiHdmi = TCC_HDMI_SUSEPNED;if( ioctl( mFb_fd, TCC_LCDC_HDMI_MODE_SET, &uiHdmi) != 0){LOGE("Fb0 HDMI ctrl Error!");}}#endif initDefaultParameters();}