Future iPhones Could Use Thermal, Infrared Imaging to Aid Regular Cameras

A new patent application shows that future iPhones could use thermal or infrared imaging to aid regular cameras.

The patent application (via Apple Insider), titled “Method and System for Determining at least one Property Related to at least Part of a Real Environment,” goes into great detail about the various ways the technology might work, but focuses on the same issue.

“Computer vision methods that involve analysis of images are often used, for example, in navigation, object recognition, 3D reconstruction, camera pose estimation, and Augmented Reality applications, to name a few,” explains the patent application.

“[However, when] a camera pose estimation, object recognition, object tracking, Simultaneous Localization and Tracking (SLAM) or Structure-from-Motion (SfM) algorithm is used in dynamic environments where at least one real object is moving, the accuracy of the algorithm is often reduced significantly,” it says, “with frequent tracking failures, despite robust optimization techniques employed in the actual algorithms.”

Basically, the issue is that AR can incorrectly position a virtual object within the real environment. Apple says that the problem is more than just skin-deep, and proposed a complex solution.

“This is because various such computer vision algorithms assume a static environment and that the only moving object in the scene is the camera itself, which pose may be tracked,” the application continues. “This assumption is often broken, given that in many scenarios various moving objects could be present in the camera viewing frustum.”

So while an AR system might have positioning details about its own hardware, tracking other things proves to be difficult for a number of reasons, including data processing.

“There exist in the current state of the art many algorithms for detection and segmentation of dynamic (i.e. moving) objects in the scene,” reads the patent application. “However, such approaches are usually computationally expensive and rely on motion segmentation and/or optical flow techniques.”

Apple’s solution to this issue is to augment the AR visual cameras with cameras that rely on thermal imaging or infrared. “[Thermal] properties of real objects in the scene could be used in order to improve robustness and accuracy of computer vision algorithms (e.g. vision based object tracking and recognition).”

“If the system can know the ‘thermal properties’ of unrecognized or what Apple calls ‘unreliable objects,’ then ‘a thermal imaging device could be utilized to detect… objects,'” explains Apple Insider.

Read the entire patent application here.