What the New Photos App in iOS 10 Can Recognize

When Apple SVP Craig Federighi detailed the new features macOS Sierra packs, it already suggested that Photos has undergone a major change. Reddit user vista980622 took a closer look at the Photos app and discovered a few lines of code possibly detailing the features Federighi mentioned onstage: searchable objects and facial recognition (via MacRumors).

Highlighting the new features, Federighi mentioned “advanced computer vision” as the most important enhancement of Photos. The updated app will be able to understand the people, places, and things within the images you take using on-device facial, object, and scene recognition alongside location information.

In a longer post on Medium, Kay Yin detailed what the new Photos app will be able to do: Actually, the app recognizes and distinguishes seven facial expressions and is able to detect 4,432 different scenes and objects.

You may recall that Photos will also “refresh your memory” with Moments, which is a curated collection of various occasions. vista980622 sheds more light on that feature, as well:

Memories from areas of interest, Best of past memories, Memories that break out of routine, Celebration in history, Contextual memories, Crowd, Day in history, Holiday in history, Location of interest, Nearby, New contextual memories, New memories, Person’s Birthdays, Person’s memories, Recent events (calendar, crowd, holiday, people, person, social, trip, weekend), Region of interest, Social group memories, Sometime memories, Special memories, Favourited, Trips, Week in history, Weekend, Year summary, Last week, Last Weekend

The first beta users of macOS Sierra have already been impressed by the search parameters of the Photos app, and vista980622 threw in some other tidbits, as well.

Additionally, you can search for various landmarks. For example, Photos can respond for search query of “Maho” (beach in Saint Martin), despite Photos is not programmed or trained to understand specific landmarks. Behind the scenes, Photos app first generates a generic categorization for the scene, “beach”, then searches through a built-in dictionary for all landmarks that has the name “beach” in its definition.

You can go through the full list of the 432 searchable objects and scenes in the Medium post.