Friday, June 10, 2016
GETTING STARTED WITH THE MICROSOFT HOLOLENS
GETTING STARTED WITH
THE MICROSOFT HOLOLENS
CONTENTS OF THIS ARTICLE:
INTRODUCTION
WHAT IS A HOLOGRAM?
WHAT CAN THE HOLOLENS DO?
SUGGESTIONS
COMPETITION
POTENTIAL APPLICATIONS FOR THE HOLOLENS
LINKS
HOLOLENS SPECS
INTRODUCTION
I have begun using and developing for the HoloLens. In this article I review my impressions of the HoloLens and provide links to extensive resources, both for suggesting the kind of Apps for which the HoloLens is uniquely appropriate, and links to training, documentation, and other aids to developing HoloLens applications.
It is convenient to separate the functions of the HoloLens into five parts:
· Display,
· Sensors, data capture, and content creation,
· A wearable computer,
· External interactions and
· Mixing of real world and computer generated images.
Although not usually considered when talking about the HoloLens, these functions are somewhat independent and could be realized, to an extent, separate from each other.
The display allows you to see and hear the real world you are in, while adding computer generated audio, text and images (called “holograms”) that seem to be in the real world with you.
The HoloLens contains distance sensors and video and audio capture devices that can sense, capture and record the real space around you and save videos that include the real world together with the holograms you perceive looking through the HoloLens; it can use that information to create content.
The computer that controls the HoloLens is in the HoloLens. So you wear it! It controls the display, monitors the sensors, and stores data in real time.
The HoloLens can communicate externally with Bluetooth and Wi-Fi, or via a USB cable to other hardware, and also allows communication between devices, and allows different HoloLenses to share their view of holograms.
It can mix holograms with the real world and allows interaction between the real and computer generated worlds.
It is the fact that the holograms act as if they are in the real world that the system is called Mixed Reality:
- Virtual Reality (VR) creates a completely computer generated world;
- Augmented Reality (AR) allows you to get information while viewing the real world.
- Mixed Reality (MR) allows you to mix the computer generated world and the real world as if they are one.
XR = VR + AR + MR + ….
The most important features that are most unique to the HoloLens are:
- That it is a wearable computer;
- That the holograms seem to be real objects in the real world: It mixes the real world and holograms together seamlessly, and they are shareable between HoloLens devices; and
- That it can capture photos and video of this mixed world with holograms in the real world.
The HoloLens
Photo by Eric Roffman
QPORIT
WHAT IS A HOLOGRAM?
A hologram, as the term is used by Microsoft, has a meaning somewhat different from the “classic” meaning of the term.
The “Classic” definition of a hologram:
A hologram is a photographic recording of a light field, rather than of an image formed by a lens, and it is used to display a fully three-dimensional image of the holographed subject, which is seen without the aid of special glasses or other intermediate optics.
A (“Microsoftic”) hologram:
A 2D or 3D image seen with special glasses which can be controlled and animated by a computer, so that it seems as if it were in the same actual room as the viewer.
A hologram shot directly by my HoloLens.
Video by Eric Roffman
QPORIT
WHAT CAN THE HOLOLENS DO?
THE HOLOLENS...
- Is excellent for Augmented Reality; it can be used to acquire information about anything it sees and present that to the user; it should be able to do anything the first version of Google Glass suggested, but better;
- Is able to create a sense of its local, current environment by using its distance sensors to map the environment and create an internal mesh; software can then allow an interaction between digital, created objects and the real environment;
- Is good for Mixed Reality, where virtual characters and objects (not just text and information) are placed within the field of view; (currently it is the best – actually the only – device optimized for Mixed Reality!)
- Can be immersive, though differently from the Oculus Rift or Gear VR which block out the current environment; instead, the HoloLens can create an immersive aura within the user’s physical, actual environment; (note: live, immersive theater – eg SLEEP NO MORE – takes place in a real place too, not just in your head mounted display; and, of course, a naked person appearing to be lying on your bed in your own bedroom would also be pretty immersive);
- Is a complete and quite powerful computer – which explains why it is expensive (currently $3K); it is untethered to any standalone computer fixed in position, so you can move around with it;
- Is able to connect wirelessly to a local or remote computer, so it can be used for remote communication with other people (for example to collaborate on a repair on a remote object, or for videoconferencing), and it can offload heavy-duty processing (for example, to enable deep searching);
- Includes 3D 360 degree audio headphones, distance sensors, microphone, and head tracking hardware;
- Includes a camera (2D) which can take photos and videos that include both the real world and the holograms that the user sees;
- Can be controlled by voice input, a clicker, Bluetooth connected devices like a mouse or keyboard, and by hand gestures in the air;
- Is very good looking.
SUGGESTIONS
The current version of the HoloLens hardware, and the current software (OS, development software, Apps…) is being provided to developers only. It is not yet ready for consumers. Before making it available to consumers, the hardware and software needs to be improved. Here are some possible features that would make it easier, more useful, and a better value for consumers.
It could be improved:
- By making it more comfortable, including
o Making it softer around the head and the nose
o Making it easier to adjust
o Making it easier for people to use eyeglasses, including switching easily between near glasses, computer glasses, distance glasses, and no glasses
o Changing the location of the USB cable, which tends to get in the way and also fall out;
- By making the field of view (FOV) larger for the portion of the screen where holograms appear;
- By making the setup process, and the development process easier;
- By consolidating the documentation;
- By improving the navigation and the text selection process (including improving the use of gestures) and adding a new controller (eg a 6 degrees of freedom (DOF) device);
- By adding more consumer Apps
o Including, especially, holographic and non-holographic apps that extend the use cases for the on-board computer;
- By continuing to extend the use of Cortana;
- By improving the on-board recording camera, including
o Making it 3D,
o Making it wide angle (even panoramic),
o Making it able to record holograms ready to use in other applications
o Making it easy to stream live (3D, panoramic) video,
o Increasing the target video hosting sites to which it can be easily shared,
o Making it easy to create “selfies” (for example, by providing software to flip a mirror image)
- By implementing the display’s ability to create VR (ie totally immersive, totally computer generated) experiences, as well as MR experiences;
- By implementing modular hardware features (to allow the possibility of either less cost or more use cases) that would allow more or less
o Computing power
o Visual resolution
o Audio quality
o Sensory devices;
- By reducing the cost.
At the present time, the HoloLens is a very specialized device. The general value of Microsoft’s contributions and development of XR would be enhanced with a complete vision for developing software and hardware to support the XR industry.
In addition to partnering with other companies to use MS holographic computing (which has already begun), for example, it would be useful to improve the resolution, the size of the memory, and access to the distance measuring features of the HoloLens so it could be used for very accurate and very large scale measuring and recording of physical spaces. (Note: The ability to scan, measure, and record spaces requires high security to prevent inadvertent disclosure of personal or proprietary information embedded in the spatial information.)
Most important, MS should extend the hardware and software offerings from MR to all of XR, including, for example, a tight linkage to VR, IoT, AI and especially machine intelligence, which can power many of the most important applications, including navigation, and any application in which real world vision is linked to computer assisted processing for searching, identifying and providing relevant information.
If, for example, the Mixed Reality Capture was able to capture high-res panoramic 3D videos and audio, and holographic videos (as opposed to 2D, limited FOV, normal res videos), it would put the HoloLens directly in the center of XR capture technology; and if the display could obscure (as well as view) the real world, it would allow VR as well as MR and put the HoloLens directly in the center of XR display technology as well;
I am suggesting that MS put the HoloLens at the very center of an eXtended Reality ecosystem, rather than leaving it as an outlier.
COMPETITION
At the present time, there is really no competition (for Mixed Reality applications) to the HoloLens. Competition is most likely to come in the future from:
- MAGIC LEAP, whose plans are still mostly a mystery;
- Google’s project TANGO, which is likely to be incorporated into a smart phone, and then may be integrated into Google Daydream;
- HTC VIVE, which uses currently external sensors to locate the user in a physical space and then places the user in a virtual environment within that physical space, could possibly extend their model by integrating the real contents of the physical space with the virtual contents;
- Other smartphones or VR devices (like the Oculus RIFT) could add local cameras and distance sensing, and integrate the views they see and the distances they measure with their VR computer generated views;
- Tethered devices (like the RIFT or Meta) could become untethered by using powerful laptops worn by the user (perhaps in a backpack), or by using fast Wi-Fi connections to a powerful computer; or eventually with fast wearable computers.
POTENTIAL APPLICATIONS FOR THE HOLOLENS
First, some comments:
Note that applications may be designed for...
- The current installed base of HoloLens users – this is a small, but growing number of developers;
- The future installed base of consumers – this is not likely to begin growing for a while; the size and rate of growth will depend on the utility, convenience and uniqueness of the HoloLens, as well as, of course, the quality of the marketing, and the price;
- A targeted group of users who will buy the HoloLens specifically because they need it to use a critical application or suite of applications (for example, NASA);
- A targeted group of users who will, essentially, rent the HoloLens which will be included in the price of the application (for example, players in a location based game, or a game-park).
Notice that, because the computer is specialized (and head-mounted), finding generalized uses for the computer (thus amortizing the cost) may be difficult, and many industrial applications of the HoloLens will need to justify the whole cost of the HoloLens for a particular application.
Types of applications…
- Of course, Augmented and Mixed Reality applications;
- One set of possible generalized applications could be any set of use cases where the ability to carry this powerful computer around on your head can be convenient; for example, students may find it a convenience to have a whole study room available on their head;
- People who need hands-free complex information while operating in a real space could use it;
- The ability to communicate remotely while sharing a view of the same space, object, or hologram may make it very valuable for industrial training;
- Workers in hazardous environments could use coupled sensors analyzed by the powerful onboard computer to monitor environmental conditions while maintaining visual and audio freedom in the space;
- Sensors for handicapped users: visual sensing for the visually challenged; speech-to-text for the hearing impaired;
- Videoconferencing;
- Serious gamers may love it;
- If the device can be passed from user to user in a sanitary way (eg no head lice?), then units could be pooled at a VR theater or gamesite;
- With the same caveat, units could be pooled at museums, tourist attractions, and other similar places. It could be used to give information at art sales, concerts and other places where you need added information while still present in the space;
- Porn;
- Transmedia, interactive, live, theatrical presentations where live actors are coupled with added pre-created content;
- Location specific games;
- Mapping real spaces, then exploring them remotely;
- Exploring mixed real and imaginary places.
- Shopping (for example, a virtual mirror), or placing potential purchases in a real setting -- curtains? couch? How does it look in my living room?
- Advertising;
- While some applications can be symmetric – both the creation of the app content, and the user employ a HoloLens -- other applications may be a-symmetric:
- created WITH a HoloLens, but FOR a different target device;
- created with NO HoloLens, but targeting a user wearing one;
- or (if the HoloLens were modular) created by the developer with some COMPONENTS of the HoloLens for a user with different components; for example, there may be applications that need only the distance sensors, or only the visual input, or only audio output, enabling a simpler, less expensive version of the complete device;
- While it might not be safe for a driver to wear one, mounting the visual input system on a vehicle could supplement the GPS for driving info; or a passenger/navigator could use the device for in-travel directions and other information;
- There’s no reason why the HoloLens should not be able to incorporate (optionally?) other devices, like a phone, a camera, a radio, and be linked to other wearable devices, etc
(And of course I have plans for my own HoloLens applications, particularly in gaming, entertainment, education, advertising and reporting.)
The most needed applications for all of eXtended Reality (not just the HoloLens), might be applications that facilitate content creation, including editing for panoramic and 3D video, and editing and creating XR within an XR application; and applications that facilitate navigation and creating and reading textual content within XR. (To read text, one prerequisite may be higher display resolution.)
CONCLUSION
The kinds of applications for which the HoloLens is not only uniquely qualified, but is actually currently unique in being able to do them, include those for which:
- The user needs a powerful onboard computer;
- The user needs to see the real world, while getting additional information, especially information that is generated by what is in the real world (like identifying information about a building you pass on a trip);
- The user needs to create an interaction between real people and computer generated objects or characters;
- The user needs to wear a camera; especially a camera which is capable of filming the real world and the computer generated world together;
- The user needs to share information about some remote object, while seeing additional information about the object or seeing the real world.
All in all, the bottom line is that there is still work to be done on both HoloLens hardware and holographic software and, because it will be expensive, the applications must be valuable enough to justify the cost of purchase.
But, most important, the HoloLens is potentially a tremendously valuable device, with many important, even vital, use cases and possible applications, and it is REAL: development on the actual HoloLens as well as the HoloLens emulator is now possible.
LINKS
MAIN HOLOLENS SITES
HOLOLENS OFFICIAL SITE
HOLOLENS OFFICIAL DEVELOPER SITE
HOLOLENS DOCUMENTATION
HOLOLENS FORUMS
FAQ
MICROSOFT HOLOLENS CHANNEL ON YOUTUBE
HOLOLENS ON TWITTER:
@HOLOLENS
HOLOLENS ON FACEBOOK
HOLOLENS EVENTS
TO GET THE HOLOLENS AT THE MICROSOFT STORE
HOLOLENS TUTORIALS, TRAINING & EXAMPLES
GETTING STARTED
THE HOLOGRAPHIC ACADEMY
HOLOGRAMS 100 – GETTING STARTED WITH UNITY
HOLOGRAMS 101 – INTRODUCTION TO THE HOLOLENS
HOLOGRAMS 101E – INTRODUCTION TO THE EMULATOR
HOLOGRAMS 210 – GAZE
HOLOGRAMS 211 – GESTURE
HOLOGRAMS 212 – VOICE
HOLOGRAMS 220 – SPATIAL SOUND
HOLOGRAMS 230 – SPATIAL MAPPING
HOLOGRAMS 240 – SHARING HOLOGRAMS
HOLOLENS DEVELOPER DOCUMENTATION
GETTING STARTED
INSTALL
HOLOLENS SHELL OVERVIEW
MAPPING THE ROOM YOU ARE IN
EMULATOR
DEVELOPMENT OVERVIEW
UNITY DEVELOPER OVERVIEW
DIRECT X DEVELOPMENT OVERVIEW
INFORMATION ABOUT THE HOLOLENS …
HOLOLENS ON WIKIPEDIA
… & HOLOGRAMS (classic definition!)
HOLOGRAMS ON WIKIPEDIA
APPS & IDEAS FOR DEVELOPING HOLOLENS APPLICATIONS
HOLOLENS APPS
DESCRIPTION OF THE FIRST APPS
THE GALAXY EXPLORER APP (with code on GitHub)
HOLOPORTATION
HEALTH & SAFETY, LICENSE TERMS
HEALTH AND SAFETY
MICROSOFT STORE TERMS OF USE
WINDOWS INSIDER PROGRAM
MICROSOFT STORE LICENSE TERMS
RELATED STORIES & VIDEO IN QPORIT
EXTENDED REALITY XR = VR + AR + MR + …
QPORIT’S VAMP:ire XRG – OUR NEW XR PRODUCTION GROUP
QPORIT’S DAILY XR NEWSLETTER
VIDEO: MY FIRST HOLOGRAM [*]
[*] Note: this video was shot by the HoloLens through the HoloLens. It shows what I was seeing. The cube is stationary in space, but as I moved – side to side and up and down – I am looking at the hologram from different angles. The video also shows the effect of gestures and voice commands.
The hologram was created in Visual Studio using …
Templates / Visual C# / Windows / Universal / Holographic
THIS STORY...
GETTING STARTED WITH THE MICROSOFT HOLOLENS
http://qporit.blogspot.com/2016/06/getting-started-with-microsoft-hololens.html
http://qporit.blogspot.com/2016/06/getting-started-with-microsoft-hololens.html
HOLOLENS HARDWARE SPECIFICATIONS
THE HOLOLENS HARDWARE (at the link below, click on HARDWARE DETAILS)
OPTICS
See-through holographic lenses (waveguides)
2 HD 16:9 light engines
Automatic pupillary distance calibration
Holographic Resolution: 2.3M total light points
Holographic Density: >2.5k radiants (light points per radian) See-through holographic lenses (waveguides)
2 HD 16:9 light engines
Automatic pupillary distance calibration
Holographic Resolution: 2.3M total light points
Holographic Density: >2.5k radiants (light points per radian)
SENSORS
1 IMU
4 environment understanding cameras
1 depth camera
1 2MP photo / HD video camera
Mixed reality capture
4 microphones
1 ambient light sensor
INTERACTIONS AND HUMAN UNDERSTANDING
Spatial sound
Gaze tracking
Gesture input
Voice support
INPUT/OUTPUT CONNECTIVITY
Built-in speakers
Audio 3.5mm jack
Volume up/down
Brightness up/down
Power button
Battery status LEDs
Wi-Fi 802.11ac
Micro-USB 2.0
Cable
MEMORY
64GB Flash
2GB RAM (2GB CPU and 1GB HPU)
POWER
Battery Life
• 2-3 hours of active use
• Up to 2 weeks of standby time
• Fully functional when charging
Passively cooled (no fans)
WEIGHT
579g
WHAT YOU NEED TO DEVELOP
Windows 10 PC able to run Visual Studio 2015 and Unity 5.4
Labels: AR, Augmented Reality, eXtended Reality, Hololens, Microsoft, Mixed Reality, MR, Virtual Reality, VR, XR