Tango 开发指南 — 区域学习原理

原文链接：https://developers.google.com/tango/overview/area-learning

工作原理

单用运动追踪，设备可以追踪自己在 3D 空间中的运动和方向，也能上报自己的位置和前进方向，但是无法保存看过的事物。区域学习技术能让设备找到并记住物理空间的关键特征——边缘，角落和其他特征点——在稍后故地重游时识别出该空间。原理是，系统将物理空间唯一的视觉特征点以数学描述的形式存储到设备上，并建立索引。所以设备能够快速匹配正在观察的特征点和已存储的特征点，而该过程无需借助任何远程服务。

当设备学习了某个区域，设备会做两个关键操作来优化运动追踪提供的数据：

1. 使用“漂移修正”来提高轨迹的精确程度；
2. 使用“定位”来确定自己在某个已经学习的区域内的位置和方向；

优化轨迹

前面章节说过，运动追踪的时间越长，测量结果越不精确。设备会让自己面朝竖直方向以此来修正误差，但是其他方面的 pose 数据误差，单靠运动追踪是无法修正的。

开启区域学习功能时，设备能够记住它浏览过的区域的视觉特征点，并使用这些特征点来修正位置、方向、运动方面的误差。这些特征点能让系统完成漂移修正（也称为循环闭包（loop closures））。当设备来到一个曾经来过的地方并意识到这一点时，它会明白自己已经绕了一圈，然后通过自我调节以使轨迹与之前的记录更加吻合。在应用中我们可以使用这些修正措施来调整设备的轨迹和位置。

用下面的例子说明漂移修正。当你开始通过某个区域时，实际有两条不同的轨迹同时存在——你走的那条轨迹（真实轨迹）和设备测量出来的轨迹（测量轨迹）。绿线是设备的真实轨迹。红线是测量轨迹，红线向我们展示了随着时间的推移，测量轨迹偏离真实轨迹的过程。当设备回到原点，并认出原点时，它会修正漂移误差，同时调整测量轨迹使之与真实轨迹更加吻合。

没有漂移修正，在将现实空间映射到游戏或应用中时将会出现误差。比如，游戏中的门洞应该对应现实世界中门洞，但是漂移误差将会造成游戏中的门洞对应现实世界中其他非门洞的位置。

区域描述和定位

开启区域学习功能的前提下，当你通过一个区域后，你可以把设备看到的东西保存到一个区域描述文件中（ADF）。学习某个区域然后以 ADF 文件的形式加载之，具有很多优点。比如，你可以在应用中将设备坐标帧和之前保存的坐标帧进行对齐，以保证也游戏或应用中的地点总是出现在相同的物理位置。

有两种创建 ADF 文件的方法。第一种是使用任何一款能够保存区域描述的应用，包括区域学习的 demo 工程。第二种是使用 Tango API 在你的应用中完成区域学习、保存和加载操作。

如果想在同一个空间内保持一致的用户体验，比如让虚拟虚体出现在它上次用户看见它的地方，你必须使用自定位。自定位包括两个阶段：

1. 加载之前保存的 ADF 文件；
2. 移动设备至被保存的区域内。当设备“看到”当前正处于 ADF 文件描述的区域时，它会立即知道自己相对于 ADF 文件中起点（被保存区域中区域学习开始的起点）的位置——这就是自定位。没有自定位，每次区域学习过程结束时设备就会丢失起点。

使用建议

• Tango 设备依赖区域的视觉多样性去实现自定位。如果某个区域内有很多相同的房间或者空荡荡的房间内只有空荡荡的墙壁，设备很难做到自定位；
• 横看成岭侧成峰，同一个区域从不同的角度和位置看，差别是很大的，而且随着时间的不同而不同（比如家具被移动位置，一天中的不同时间光线的差异等）。如果你实施自定位时的条件跟保存在 ADF 文件中条件类似时，设备更有可能成功完成自定位；
• 因为环境时刻在变化，你可以为同一个物理区域创建多个不同条件下的 ADF 文件。这样能让用户选出最接近其当时条件的文件。此外，你还可以将多个区域学习会话添加到同一个 ADF 文件中，来获得从各个方位和各种光线和环境变化下该区域的视觉描述。

我们的用户体验最佳实践页面中有其他的关于创建 ADF 文件和使用区域学习的建议。

常见案例

多用户体验：如果处于同一个区域的多个用户借助云服务使用同一个 ADF 文件，那么他们将定位到同一个坐标帧。这样多个用户将知道彼此的位置并进行互动。原生的 Tango API 并不支持通过云服务共享数据，但是开发者可以通过谷歌云存储或谷歌游戏 API 做到这一点。

基于位置的购物或活动：零售商店的老板创建商店的 ADF 文件并共享输出，然后消费者下载该文件，完成自定位，然后可以使用设备直接导航到感兴趣的货架。

注意：尽管 ADF 文件不能直接录制图像或视频，但是这些数据可以重建成可视图像。因此，为了保护用户隐私，就跟处理图片和视频时候一样，当你将用户被学习的区域描述上传到云或跟其他用户共享之前，必须要获得用户授权。

区域学习和区域描述都是很强大的功能，我们很高兴开发者使用这些功能去开发牛逼应用、做出牛逼的用户体验。

使用区域学习和 ADF

某些 Tango API 的行为跟学习模式的系统设置和是否加载 ADF 文件有关。

下面的表格中，左侧两列表示是否开启学习模式和是否加载已有 ADF。这两列的状态将决定你是否能够保存 ADF。比如，如果你没有开启学习模式，你将无法保存 ADF。如果你开启了学习模式而且加载了 ADF，只有当你借助已加载 ADF 完成自定位后才能保存。

同样，如果你没有开启学习模式而且没有加载 ADF 文件，你将无法获取 TANGO_COORDINATE_FRAME_AREA_DESCRIPTION 参考帧对应的 pose 数据。如果你已经加载了 ADF，在借助已加载 ADF 完成自定义阶段后，你就可以获取 TANGO_COORDINATE_FRAME_AREA_DESCRIPTION 对应的 pose 数据了。

*If tracking is lost, these frame of reference pairs will no longer be available. After service reset, the session functions as if learning mode is True and an ADF was loaded, where the area descriptions are those that were learned up to the loss of tracking. To continue using the area description frame of reference, you must localize against what you learned before the loss of tracking. You must also localize to include what you learned before tracking was lost when saving an ADF.

以下是原文

How it works

With Motion Tracking alone, the device tracks its movement and orientation through 3D space and tells you where it is and which way it’s facing, but it retains no memory of what it sees. Area Learning gives the device the ability to see and remember the key visual features of a physical space—the edges, corners, other unique features—so it can recognize that area again later. To do this, it stores a mathematical description of the visual features it has identified inside a searchable index on the device. This allows the device to quickly match what it currently sees against what it has seen before without any cloud services.

When a Tango device has learned an area, there are two key things it can do to improve upon the information provided by Motion Tracking alone:

1. Improve the accuracy of the trajectory by performing “drift corrections.”
2. Orient and position itself within a previously learned area by performing “localization.”

Improving the trajectory

As mentioned on the Motion Tracking Overview page, motion estimates become less accurate over time. The device corrects for some errors by orienting itself to gravity, but errors in other aspects of its pose cannot be detected through Motion Tracking alone.

With Area Learning turned on, the Tango device remembers the visual features of the area it has visited and uses them to correct errors in its understanding of its position, orientation, and movement. This memory allows the system to perform drift corrections (also called loop closures). When the device sees a place it knows it has seen earlier in your session, it realizes it has traveled in a loop and adjusts its path to be more consistent with its previous observations. These corrections can be used to adjust the device’s position and trajectory within your application.

The illustration below shows an example of drift correction. As you begin walking through an area, there are actually two different trajectories occurring simultaneously—the path you are walking (the “real trajectory”) and the path the device estimates that you are walking (the “estimated trajectory”). The green line is the real trajectory that the device is traveling; the red line shows how, over time, the estimated trajectory has drifted away from the real trajectory. When the device returns to the origin and realizes it has seen the origin before, it corrects the drift errors and adjusts the estimated trajectory to better match the real trajectory.

Without drift correction, a game or application using a virtual 3D space aligned with the real world may encounter inaccuracies in Motion Tracking after extended use. For example, if a door in a game world corresponds with a door frame in the real world, drift errors can cause the game door to appear in the middle of the real-world wall instead of in the door frame.

Area descriptions and localization

After you have walked through an area with Area Learning turned on, you can save what the device has seen in an Area Description File (ADF). Learning an area and loading it as an ADF has a number of advantages; for example, you can use it to intentionally align the device’s coordinate frame with a pre-existing coordinate frame so that content in a game or app always appears in the same physical location.

There are two ways to create an ADF. You can use any application that can save area descriptions, including Tango Area Learning sample projects (see more information about sample projects for C, Java, or Unity). Your second choice is to use the Tango APIs to handle the learning, saving, and loading all within your application.

If you want to create a consistent experience within the same mapped space, such as having virtual objects appear in the same location as the last time the user visited an area, you must perform localization. This is a two-step process:

1. Load a previously saved ADF.
2. Move the device into the area that was saved in the ADF.
   When the device “sees” that it is in the area covered by the ADF, it instantly knows where it is relative to the origin in the file (that is, the point where original learning started in the saved area)—this is localization. Without localizing to an area description, a device’s starting point is lost every time you end the session.

Usability tips

• Tango devices depend on the visual diversity of the area to localize. If you are in an area with many identical rooms or in a completely empty room with blank walls, it is difficult to localize.
• An environment can look quite different from different angles and positions, and can change over time (furniture can be moved around, lighting will be different depending on the time of day). Localization is more likely to succeed if the conditions at the time you localize are similar to the conditions that existed when the ADF was created.
• Because environments can and do change, you might create multiple ADFs for a single physical location under different conditions. This gives your users the option to select a file that most closely matches their current conditions. You could also append multiple sessions onto the same ADF to capture visual descriptions of the environment from every position and angle and under every variation of lighting or environmental change.
  Our UX Best Practices page has additional tips on creating ADFs and using Area Learning.

Common use cases

Multi-player experiences: Two or more users in the same physical location share an ADF through a cloud service and then localize to the same coordinate frame. This allows multiple people to interact in the same physical space where all of their relative positions are known. The Tango APIs do not natively support data sharing in the cloud, but you can implement this through Google Cloud Storage and the Google Play Games API.

Location-aware shopping or other activities: A retail store manager makes an ADF of their store and then makes the ADF publicly available. Customers load the ADF, localize, and then use the device to navigate directly to products they are interested in.

Important: While saved area descriptions do not directly record images or video, the data can be reconstructed into viewable images. Therefore, to protect the user’s privacy, you must ask for permission before saving any of their learned areas to the cloud or sharing areas between users just as you would with any other images or video.
Area Learning and using area descriptions are powerful features, and we’re excited to see how developers use them to offer new user experiences.

Using learning mode and loaded ADFs

The behavior of some aspects of the Tango APIs will vary depending on your settings for learning mode or whether you loaded an ADF.

In the table below, the two left columns specify whether you have learning mode on and whether you have loaded a previously stored ADF. You may or may not be able to save an ADF depending on the status of those two things. For example, if you don’t have learning mode on, you cannot save an ADF. If you have learning mode on and have loaded an ADF, you can only save again after you have localized against the loaded ADF.

Also, if you aren’t in learning mode and don’t have an ADF loaded, you cannot get pose data using the TANGO_COORDINATE_FRAME_AREA_DESCRIPTION frame of reference. If you have an ADF loaded, you can get pose data from that frame of reference after the device localizes to the loaded ADF.

*If tracking is lost, these frame of reference pairs will no longer be available. After service reset, the session functions as if learning mode is True and an ADF was loaded, where the area descriptions are those that were learned up to the loss of tracking. To continue using the area description frame of reference, you must localize against what you learned before the loss of tracking. You must also localize to include what you learned before tracking was lost when saving an ADF.