“智能高清”试验

文章作自2016年8月11日

“智能高清”只是一个想法，效果有多少，我心里还没有底，既然这样，那就做一个试验吧。

"Smart HD" is just an idea, but I don't know what's the result of it, let's have a try.


我的计划是找一个1080p的MV（时间短，处理起来快，视频质量一般很高），将其缩小到640x360，这个就是低清视频。然用相同的方法压缩一次，保持其大小，将这个作为高清视频。观看这2个视频，比较其差异。
My plan is to find a 1080p HD video. Resize it to 640X360, this is low resolution video. And then, compress it the same way, keep it in full size, and use this as a HD video. Watch the 2 videos and compare the differences.


然后将高清视频切割成4x4=16个小视频，背景播放低清视频，中间2x2=4个块播放高清视频，观看者尽量注视画面中间，观看这个视频和原高清视频，比较其差异。
Then split it into 4x4=16 small HD video blocks, play low resolution video in background, play 2x2=4 HD video blocks in centre, audience look at the middle of the screen, watch this video and origin HD video, compare the difference.
 



这里以windows系统为例子。由于我们用到了ffmpeg，首先要下载ffmpeg（http://ffmpeg.org/download.html），解压到本地，然后把里面bin路径加到环境变量。
Here, take the windows system as an example. Since we use ffmpeg, we first download ffmpeg (http://ffmpeg.org/download.html), extract it locally, and then add the bin path inside to the environment variable.
  
首先下载一个高清mv(比如我们下载了origin.mp4)，从这里（https://trac.ffmpeg.org/wiki/Scaling%20(resizing)%20with%20ffmpeg）找到放大缩小的方法。去到mv所在的文件夹，新建一个scale.bat文件，内容如下：
First, download an HD MV (for example, we downloaded origin.mp4) and find ways to zoom in from here (https://trac.ffmpeg.org/wiki/Scaling%20 (resizing)%20with%20ffmpeg). Go to the folder where MV is located and create a new scale.bat file, which reads as follows:
ffmpeg -i origin.mp4 -vf scale=1920:1080 input.mp4
ffmpeg -i input.mp4 -vf scale=640:360 input_small.mp4
保存bat文件，打开windows命令行工具，用cd命令去到mv的文件夹，执行scale.bat，就会得到分辨率为1920x1080的高清视频"input.mp4"（我这里一个3分20秒的视频文件大小为77M），和一个分辨率为640x360的低清视频"input_small.mp4"（同样时长的视频文件大小为14M）。然后观看一下这2个文件，差距挺大吧？
Save the bat file, open the windows command-line tool, use the CD command to execute scale.bat to the MV folder, and get a resolution of 1920x1080 HD video "input.mp4" (here is a 3 minute 20 seconds of video file size is 77M), and a resolution of 640X360 low resolution video "input_small.mp4" (the same time the video file size is 14M). Then look at these 2 documents, the gap is quite large, right?


然后从这篇文章（http://video.stackexchange.com/questions/4563/how-can-i-crop-a-video-with-ffmpeg）里找到了切割视频的方法。在windows里，新建一个batch_crop.bat文件，内容如下:
Then we found the way to cut video from this article (http://video.stackexchange.com/questions/4563/how-can-i-crop-a-video-with-ffmpeg). In windows, create a new batch_crop.bat file that reads as follows:
set inFile=%1
set width=%2/%4
set height=%3/%5


for /L %%j in (1,1,%5) do (
for /L %%i in (1,1,%4) do (
ffmpeg -i %inFile% -filter:v "crop=%width%:%height%:(%%i-1)*%width%:(%%j-1)*%height%" -an output_%%j_%%i.mp4
)
)
保存bat文件，打开windows命令行工具，用cd命令去到mv的文件夹，执行下面的命令：
Save the bat file, open the windows command line tool, use the CD command to go to the MV folder, and execute the following command:
batch_crop input.mp4 1920 1080 4 4
完成后你将会得到16个小视频文件。
When you're done, you'll get 16 small video files.


现在我们可以合成视频了，参考这篇文章（http://www.daniel-mitchell.com/blog/video-in-a-video-ffmpeg/）。在windows里，新建一个scale_overlay.bat文件，内容如下（只有一行）:
Now we can synthesize video, and refer to this article (http://www.daniel-mitchell.com/blog/video-in-a-video-ffmpeg/). In windows, create a new scale_overlay.bat file with the following contents (only one line):
ffmpeg -i input_small.mp4 -vf "[in] scale=1920:1080 [sin]; movie=output_2_2.mp4[inner22];movie=output_2_3.mp4[inner23];movie=output_3_2.mp4[inner32];movie=output_3_3.mp4[inner33]; [sin][inner22] overlay=480:270 [step1]; [step1][inner23] overlay=960:270 [step2]; [step2][inner32] overlay=480:540 [step3]; [step3][inner33] overlay=960:540 [out]" scale_overlay.mp4
这里解释一下这行命令，
-i input_small.mp4 ：输入视频文件
-vf ：后面的是滤镜
[in] scale=1920:1080 [sin] ：将输入的小视频文件"in"放大为"sin"
movie=output_2_2.mp4[inner22] ：加载小视频22为"inner22"
movie=output_2_3.mp4[inner23] ：加载小视频23为"inner23"
movie=output_3_2.mp4[inner32] ：加载小视频32为"inner32"
movie=output_3_3.mp4[inner33] ：加载小视频33为"inner33"
[sin][inner22] overlay=480:270 [step1] ：合成"sin"和"inner22"为"step1"
[step1][inner23] overlay=960:270 [step2] ：合成"step1"和"inner23"为"step2"
[step2][inner32] overlay=480:540 [step3] ：合成"step2"和"inner32"为"step3"
[step3][inner33] overlay=960:540 [out] ：合成"step3"和"inner33"为"out"
保存bat文件，打开windows命令行工具，用cd命令去到mv的文件夹，执行下面的命令：
scale_overlay
完成后你将会得到scale_overlay.mp4。你可以观看视频"scale_overlay.mp4"并和"input.mp4"进行比较了。


Here, explain this line of command,


-i input_small.mp4: input the video file
-vf: the filters
[in] scale=1920:1080 [sin]: enlarge the small video file "in" to "sin""
Movie=output_2_2.mp4[inner22]: load small video 22 for "inner22""
Movie=output_2_3.mp4[inner23]: load small video 23 for "inner23""
Movie=output_3_2.mp4[inner32]: load small video 32 for "inner32""
Movie=output_3_3.mp4[inner33]: load small video 33 for "inner33""
[sin][inner22] overlay=480:270 [step1]: synthesize "sin" and "inner22" for "step1""
[step1][inner23] overlay=960:270 [step2]: synthesize "step1" and "inner23" for "Step2""
[step2][inner32] overlay=480:540 [step3]: synthesize "Step2" and "inner32" for "Step3""
[step3][inner33] overlay=960:540 [out]: synthesize "Step3" and "inner33" for "out""
Save the bat file, open the windows command line tool, use the CD command to go to the MV folder, and execute the following command:
Scale_overlay
When you have finished, you will get scale_overlay.mp4. You can watch the video "scale_overlay.mp4" and compare it with "input.mp4".




首先比较一下文件大小。这里要注意一下，你会发现"scale_overlay.mp4"并不比"input.mp4"小很多，因为我没找到一个播放器可以设置固定的播放尺寸，并在特定的位置叠加播放多个小视频。所以就只能模拟，将低清视频"input_small.mp4"放大，一经放大，其视频大小就接近"input.mp4"了，但其实清晰度是很低的。然后在上面叠加4个小视频，合成一个视频"scale_overlay.mp4"。而实际应用的时候，我们是仅传输"input_small.mp4"到客户端，在客户端放大播放，并叠加播放小视频的，这样总文件大小就小很多了。这里我进行了2次尝试，(低清视频+4个小视频)/高清视频的比例分别为57%和51%，也就是说节省了进一半的大小。这仅仅是一个试验，实际上视频可以切割得更小，并且动态调整和加载，不像这个例子是固定的25%高清。
First, compare the file size. It should be noted here, you will find that "scale_overlay.mp4" is not much smaller than the "input.mp4", because I did not find a player can set the playback fixed size, and play multiple small video in particular superposition. So we can only simulate the low resolution video "input_small.mp4" and zoom it in, after that, the video size close to "input.mp4", but in fact the clarity is very low. Then stack 4 small HD videos blocks above to synthesize a video "scale_overlay.mp4"". In real case, we only transport the "input_small.mp4" to the client, zoom in on the client, and play a small HD video blocks, so the total file size is much smaller. Here I have 2 attempts, (low resolution video + 4 small HD video blocks) / HD video ratio of 57% and 51%, that is, save half of the size. This is just a test, in fact, video can be cut smaller and dynamically adjusted and loaded, unlike this example is fixed 25% hd.


然后比较一下效果，注意这个例子是固定中心高清的，所以你要尽量将视线集中在中心，以便比较效果。就笔者而言，"scale_overlay.mp4"的效果是要比"input_small.mp4"好很多的，感觉基本接近"input.mp4"，但有些地方还是感觉到差异，估计是有时候控制不了，看了中心以外的地方。我们可以截个图来比较一下。
Then compare the effects. Note that this example HD blocks are is fixed in center, try to focus your eyes on the center so that you can compare the results. "scale_overlay.mp4," effect is much better than "input_small.mp4", feel close to the "input.mp4", but in some places still feel the difference, maybe we can't control our gaze, and see the outside of the center. We can compare it with a graph.


打开windows命令行工具，用cd命令去到mv的文件夹，执行下面命令，得到"input.mp4"位于1分08秒的24张截图：
ffmpeg -i input.mp4 -qscale:v 2 -f image2 -ss 00:01:08 -vframes 24  one_input_frame-%02d.jpg
再执行下面命令，得到" scale_overlay.mp4"位于1分08秒的24张截图：
ffmpeg -i scale_overlay.mp4 -qscale:v 2 -f image2 -ss 00:01:08 -vframes 24  one_scale_overlay_frame-%02d.jpg
从中抽出2张进行比较：
Open the windows command line tool, use the CD command to go to the MV folder, and execute the following command to get 24 screenshots of "input.mp4" in 1 minutes and 08 seconds:
Ffmpeg, -i, input.mp4, -qscale:v, 2, -f, image2, -ss, 00:01:08, -vframes,, one_input_frame-%02d.jpg
Then execute the following command and get 24 screenshots of "scale_overlay.mp4" in 1 minutes and 08 seconds:
Ffmpeg, -i, scale_overlay.mp4, -qscale:v, 2, -f, image2, -ss, 00:01:08, -vframes,, one_scale_overlay_frame-%02d.jpg
Draw 2 out of them for comparison:
 


 


 


截图的效果非常接近，如果看了一张图，间隔一会再看另外一种，没感觉到差异，以至我一开始怀疑自己是不是搞错了，2次截图是同一个视频文件。但同时打开2个图片进行比较，才发现真的是不同。这可能是人的心理作用吧，看到中心清晰的地方，边缘就不那么关注了，感觉有点像摄影的虚化效果。
The screenshot is pretty close, and if you look at a picture, and then look at the other one, you don't feel the difference, so I started wondering if I made a mistake, guess the 2 shot were from the same video file. But if we open 2 pictures for comparison, we can find the difference. This may be the psychological of human, focus on centre, the edge is less concerned about.


好了，试验结束，是不是挺好玩的？不过如果要实际应用，还有很多事情要处理，比如人眼的观看点检测，用什么算法，用什么设备（建议用手机，平板电脑，或者生产带摄像头的智能电视，不需要消费者单独购买额外的设备），而且摄像头装在设备的什么位置，对算法也有影响，是不是需要定一个行业标准？还有就是视频传输，是不是要改进现在的传输协议，定一个新的标准？然后就是视频的播放，怎么发送请求到服务器，获得多个小视频后，怎么同步播放，怎么根据视线移动，选择新的小视频位置，怎么管理和调整这些小视频的缓存，以便更快地将用户关注的高清视频加载下来？现实情况是高清视频的分辨率和设备的分辨率可能不一样，还需要进行缩放，然后小视频的大小和位置也要相应的缩放。还有可以根据大数据分析，知道大部分观众关注的是那些地方，可以预测地加载一些一下高清小视频块。
Well, the experiment is over, isn't it fun? But if the actual application, there are a lot of things to deal with, such as eye viewing point detection, with what algorithm, what equipment to use (mobile phone, tablet, or TV box), Then is the video player, how to send a request to the server, after getting a number of small video blocks, how to synchronous play them, how to change blocks according to gaze movement, how to manage and adjust these small video blocks cache for faster HD video loading? The resolution of HD video and the resolution of the device may not be the same, we also need to resize it, and then the size and location of small video blocks should be scaled accordingly. Also, according to big data analysis, you can predict to load some HD video blocks.


这个方案并不绑定任何编码技术，默认用ffmpeg的h.264编码，但是你也可以用h.265编码。以上面的2个例子来试验，
MV1: h.264高清视频为77M，h.264低清视频+4个小视频为44M，h.265低清视频+4个小视频为19M，效果明显。
MV2: h.264高清视频为135M，h.264低清视频+4个小视频为69M，h.265低清视频+4个小视频为21M，效果明显。
This solution does not rely any coding techniques, and the default is to use encoding of ffmpeg H.264, but you can also encode with h.265. Experiment with 2 examples above,
MV1: H.264 HD video for 77M, H.264 low clear video, +4 small video for 44M, h.265 low clear video, +4 small video for 19M, the effect is obvious.
MV2: H.264 HD video for 135M, H.264 low clear video, +4 small video for 69M, h.265 low clear video, +4 small video for 21M, the effect is obvious.


最后，关于文件切分，因为视频有帧内压缩，如果切分的太小，压缩效果就会变差。经过实验，对于1080p的视频，切分到6x6=36或者8x4=32个小视频，总文件大小基本不变。当切分到8x8甚至更大，总文件大小明显增大。
Finally, about file segmentation, because the video has inter frame compression, if the segmentation is too small, compression effect will become worse. After the experiment, for 1080p video, cut into 6x6=36 or 8x4=32 small video, the total file size is basically unchanged. When splitting to 8x8, or even greater, the total file size is significantly increased.