YouTube
If you have problems with this application, please send an email to stereopub@stereo.jpn.org
This is an iOS application that uses AI to create depth from 2D photos
and 2D videos in 360-degree equirectangular format and convert and display
them in real time as 3D images in SBS.
The viewpoint can be freely moved by using the iPhone's motion sensor or touch scrolling on the screen. Pinch zooming is also supported.
The app also supports output to smart glasses connected via USB-C. However, since the motion sensor information on the smart glasses side cannot be acquired, please use the motion sensor, touch scroll, and pinch zoom on the iPhone side to move the viewpoint and perform zooming operations even when smart glasses are connected.
For instructions on connecting to external monitors such as smart glasses,
please see this Youtube video.
Please note that due to limitations of the AI model used in the application,
the application can only be used on iPhone 12Pro (A14) or later models
with iOS 17.0 or later.
The application performs the following processes within the iPhone
2D photo or 2D video in 360-degree equirectangular format -> For video, frame image extraction by encoding -> Flat projection (viewpoint shift) -> Depth generation by AI -> Creation of left/right 3D image using flat projected image and its depth image -> Side-by-side image creation -> Screen display
As described above, the iPhone's GPU is fully utilized to perform tremendous calculations.
Therefore, even though I call it real-time, it is difficult to run it as smoothly as a normal 360 degrees viewer, but I hope you will find value in being able to view a 2D 360 degrees image in 3D.
Also, if used continuously, the unit will get hot, so please give it a rest.
As mentioned above, this application displays a 3D conversion of a flat projected image from a 360-degree equirectangular format image on the iPhone. Please note that this application does not create a 3D 360-degree equirectangular format image.
How to use (The following operation buttons are toggled show on and off
each time the screen is tapped. If the display is obstructive, please use
it in non-display mode.)
Press and hold the screen to save the displayed image in the display mode at that time.
Acknowledgements
The AI for this depth generation used the Depth-Anything-V2 Small model.
I would like to express my deep appreciation to those who have published such a useful AI model.
Depth-Anything-V2
https://github.com/DepthAnything/Depth-Anything-V2
Apple Core ML Models
https://developer.apple.com/machine-learning/models/