After working on Vision Pro, this AR veteran is returning to mobile

These are Lowpass and Janko RoettgersA newsletter about the ever-changing technology and entertainment industry, created for the Seaside subscribers once a week.

When Mark Drummond was working on Vision Pro at Apple, he had an epiphany that didn’t fit Cupertino’s preferences.

Drummond was in charge of the Character Intelligence Team, which among other things builds Meet the Dinosaurs show. Previously installed on Vision Pro, the show puts viewers face-to-face with the creatures of history.

“We did this with Jon Favreau,” Drummond says, referring to Apple long term relationship and The Mandalorian the director. “Before there were headsets to come down to Burbank, we were using iPhones and iPads,” he recalls. The reliance on mobile devices for demos was felt. VisionOS is basically a fork of iPadOS. “It really worked. We had the best iPhone and iPad experiences,” he says.

Through this experience, Drummond realized that these mobile devices were not just headstands. “What we’ve learned in our search for surprises and delights with people who have different connections is that headphones aren’t the best (device) for this kind of thing,” he says.

“I still think it’s a very beautiful instrument,” says Drummond. However, headphones can also isolate, separating the viewer from the world and the people around them. Drummond says. Having an AR app on your phone, on the other hand, makes it easier to show it to others. “People can lean on your shoulders,” says Drummond.

That’s why, soon after leaving Apple in 2023, Drummond embraced mobile devices for real-time storytelling. For the past two and a half years, he has been working on a new iPhone called AR Pixie That’s all for mobile-first storytelling.

In short, Pixi is working on what can best be described as an AR version of the email greeting card. When it launches in the coming weeks, Pixi will allow anyone to pick a favorite person for an event, add a personalized message, and send it to their contacts via iMessage or WhatsApp. When the recipient opens such a Pixi message, the character appears overlaid with their real-time location camera, and interacts with them.

At a recent show, Pixi’s AR experiences included a cartoon cat and a robot. They can tell jokes, play tiki with you, or challenge you to a game of whack-a-mole, right at your desk.

At first glance, nothing seemed to be too much of a disappointment if you’ve ever played with an AR Snapchat filter or any other AR mobile app. “If you’re not a little bit shy about the first thing, you’re a (late) reboot,” Drummond says, implying that the company will include more characters and more advanced stories in the future.

However, Pixi includes some interesting features: The cat comedian, for example, offers jokes about one dad until the app detects that the viewer is smiling through the phone’s front camera. Afterwards, the artist thanks the audience for their interest as a comedian ends their show and gives a personal greeting.

“For a person to feel like they’re there, they have to be responsive,” says Drummond. This includes not only listening to the audience, but also their environment. “This kind of care is possible through an AI device,” he says. Pixi uses AI and machine learning to recognize facial expressions, and the app also downloads ML models on the fly to recognize objects and integrate them into stories.

When AR phones became popular with Snapchat filters and ARKit-based apps, developers had little to work with when it came to interacting with real-world environments. Basically, software can detect the location and place any object on top of the device. Over time, AR apps have gotten smarter about occlusion, allowing characters to hide behind real objects. In the end, some came to recognize ancient things.

With recent advances in AI, there is the potential to increase object recognition, to the point where software can identify groups of objects, understand how they work, and integrate them into experiences. Here’s how Drummond envisions Pixi’s AR greeting cards will work in the future:

“Let’s say I have a friend (who is) a lawyer. He was promoted, and I want to say thank you. I’m sending him (Pixi with a) golden retriever, which is found on the floor of the office. It looks at him, then it starts to smell. It goes to the filing cabinet, and it’s focused on the drawer in the filing cabinet. Dogs, those are just old taxes. A dog will not give up. It looks, smells, looks great. He opens the drawer. Inside, we have hidden a large dog biscuit (with a message): ‘Thank you, see you on Friday.’

Can all of this be technically possible with Vision Pro software? Maybe. But there are many other reasons that hold the headphones back – including the $3,500 price tag. “It’s expensive,” Drummond admits, adding, “I still think it’s a steal because of the incredible technology it has, just the displays.”

However, the high cost of the device has also hindered adoption, with IDC estimating that the company has shipped it. 45,000 units only last year’s holiday season. “It doesn’t have much market penetration,” Drummond says.

Even the people who own it may not want to install it every time someone sends them a two-minute video or event. “You don’t always have it, and (you have) the setup, the expense,” Drummond says. “We always have calls.”

It’s no secret that the Vision Pro was Apple’s first feature, which is he says he is working his smart glasses are AR. However, Drummond believes that such glasses will not be able to replace smart phones for years to come due to physical problems. Phones can now use complex types of machine learning locally, while glasses will have limited power and computational capabilities. Because of these constraints, it can act as another display of information and movements, similar to the Apple Watch. “I think the Watch and Apple’s smartwatches will do the same,” he says.

This means that AR developers looking to create exciting experiences may find that the phone is still the best platform for the future – a platform that’s getting better every day, thanks to AI.