What the New Photos App in iOS 10 Can Recognize

Screen Shot 2016 06 20 at 19 07 57

When Apple SVP Craig Federighi detailed the new features macOS Sierra packs, it already suggested that Photos has undergone a major change. Reddit user vista980622 took a closer look at the Photos app and discovered a few lines of code possibly detailing the features Federighi mentioned onstage: searchable objects and facial recognition (via MacRumors).

Highlighting the new features, Federighi mentioned “advanced computer vision” as the most important enhancement of Photos. The updated app will be able to understand the people, places, and things within the images you take using on-device facial, object, and scene recognition alongside location information.

In a longer post on Medium, Kay Yin detailed what the new Photos app will be able to do: Actually, the app recognizes and distinguishes seven facial expressions and is able to detect 4,432 different scenes and objects.

You may recall that Photos will also “refresh your memory” with Moments, which is a curated collection of various occasions. vista980622 sheds more light on that feature, as well:

Memories from areas of interest, Best of past memories, Memories that break out of routine, Celebration in history, Contextual memories, Crowd, Day in history, Holiday in history, Location of interest, Nearby, New contextual memories, New memories, Person’s Birthdays, Person’s memories, Recent events (calendar, crowd, holiday, people, person, social, trip, weekend), Region of interest, Social group memories, Sometime memories, Special memories, Favourited, Trips, Week in history, Weekend, Year summary, Last week, Last Weekend

The first beta users of macOS Sierra have already been impressed by the search parameters of the Photos app, and vista980622 threw in some other tidbits, as well.

Additionally, you can search for various landmarks. For example, Photos can respond for search query of “Maho” (beach in Saint Martin), despite Photos is not programmed or trained to understand specific landmarks. Behind the scenes, Photos app first generates a generic categorization for the scene, “beach”, then searches through a built-in dictionary for all landmarks that has the name “beach” in its definition.

You can go through the full list of the 432 searchable objects and scenes in the Medium post.

Technology enthusiast, rocker, biker and writer of iPhoneinCanada.ca. Follow me on Twitter or contact me via email: istvan@iphoneincanada.ca

  • jabohn

    I’d rather they add in all that’s missing from iPhoto before putting in new features.

  • Quattro

    In other words, it’s just an attempt to catch up to Google Photos, while tossing in an extra enhancement or two. They had no choice, as this was a “must have” set of features to remain relevant. But it’s also part of a continued move to get you to buy more of Apple’s over-priced cloud storage.

  • Mr Dog

    The big difference and truly amazing part of all of this is, it is all done ON Device. It processes the backlog of your photos when on power but after, every photo you take is instantly analyzed. And all of this without a huge impact on battery.

    Where as good photos, takes all your photos and does the processing on the cloud.

  • Mr Dog

    FYI! I can finally find all pictures of dogs 😀

  • Thierry Marques

    Also, this respects the user privacy (everything is done on the user device) unlike what Google does, taking all the data they get out of the treatment of the pics on their servers to better know you.

  • Quattro

    I don’t see that as a problem.

  • Thierry Marques

    Free storage on one end, payed for by your information.
    Paid storage on the other end, $ given so I keep my privacy.

    It is just a different approach, none is better than the other… unlike what the last sentence of your first message tended to imply.
    Everyone gets to choose how to pay for said storage.