Mixed reality (MR)
Mixed reality is a subset of extended reality that blends the physical and artificial/virtual worlds, superpositioning digital elements into the real-world environment in a manner in which either can interact with the other.
In the MR experience, the user can see and interact with both the digital elements and the physical ones.
In this way, MR gets input from the environment and changes according to that information.
Research given by Paul Milgram and Fumio Kishino in a paper published in 1994 defines mixed reality as “..a particular subclass of VR-related technologies that involve the merging of real and virtual worlds.”
This is the latest immersive technology and is as well known as hybrid reality – a hybrid of both real and virtual worlds.
In contrast to augmented reality, users can interact with virtual objects in an MR experience while the experience is observed in the real world, unlike what is obtainable in virtual reality, the experience being in a virtual world.
Humans, computers, and the environment
A mixed-reality experience is enabled by wearable MR devices.
The experience unlocks a natural interaction among 3D humans, computers, and the environment.
MR is based on advancements in computer vision, graphical processing, display technologies, input systems, and cloud computing.
The applications of mixed reality have gone beyond displays to include environmental understanding (spatial mapping and anchors), human understanding (hand-tracking, eye-tracking, and speech input), spatial sound, locations, and positioning in both physical and virtual spaces as well as collaboration on 3D assets in mixed reality spaces.
To achieve a true mixed reality experience, a combination of three essential elements needs to be at play.
- Cloud-powered computer processing
- Advanced input methods
- Environmental perceptions
Blending the physical and virtual world has to do with mapping real-life human movements in the virtual world as a 3D representation, mapping real-life environmental inputs and perceptions as well, and combining them with computer-generated virtual elements to create an interactive experience between the real and virtual world.
Advancements in input methods have bridged the gap in human and computer interactions, as computers can now receive human inputs using keyboards, computer mice, touch, voice, skeletal tracking, etc, and make use of sensors, to receive input about a person’s body position (head tracking); objects, surfaces, and boundaries (spatial mapping and scene understanding), ambient lighting and sound, object recognition, physical locations, etc.
The mixed reality spectrum
Mixed reality blends both physical and digital worlds, the two realities mark the polar ends of the virtuality continuum mentioned earlier.
Augmented reality overlays graphics, video streams, or holograms in the physical world, virtual reality occludes the user’s view to present a fully immersive digital experience while mixed reality transitions between both augmented and virtual realities.
Types of mixed reality displays
Milgram and Kishino, in a later paper in 1995, classified existing mixed reality displays into seven, considering how real and virtual objects are overlayed.
The classification was based, firstly using a table in which the nature of the underlying scene, how it is viewed, and the observer’s reference to it are compared, and then using a three-dimensional taxonomic framework comprising: the extent of the world knowledge, reproduction fidelity, and extent of presence metaphor.
- Monitor-based (non-immersive) video displays: showing a video of the real world onto which digital images are superimposed
- A head-mounted display (HMD) showing a video: is the same as type 1, but the content is in a head-mounted display.
- Optical see-through HMD: a see-through display that allows virtual images to appear superimposed over the real world
- Video see-through HMD: The same as 3, but showing video of the real world in front of the user with virtual graphics superimposed on it.
- Monitor-based AV system: showing 3D graphics on a monitor with superimposed video.
- Immersive or partially immersive AV: showing 3D graphics in an immersive display with video superimposed on it.
- Partially immersive AV systems: AV systems allow additional real-object interactions, such as interacting with one’s own (real) hand.
Microsoft’s HoloLens is an optical-see-through display that allows virtual information to appear in the real world and so is a type 3 MR display.
Similarly, a large screen that shows virtual characters interacting with real people on a camera feed is a type 1 MR display.
Applications of mixed reality in healthcare
1. Instant diagnoses
MR is built with a lot of processing power that realistically visualizes a patient’s anatomy, juxtaposing it with normal anatomy while factoring in the patient’s clinical information to arrive at a diagnosis of the patient’s condition with safe accuracy.
2. Medical training
Students in the medical profession are taught about the human body by physically dissecting a dead body cadaver.
MR presents a far better way of teaching students with a 3D simulation of the human body – students get to operate on these virtual human bodies in an environment that replicates what it’s like to perform surgery on a live patient.
3. Surgical assistance
Surgical procedures are guided by clinical information and surgeons often have to pause their procedures while they review some of these patient details e.g. imaging, tumor location, extent, etc.
An MR headset conveniently displays these details, without interrupting the procedure or posing harm, serving as a precise visual guide for the surgeon while operating.
4. Telemedicine
Telemedicine has to do with consultations with a physician from a remote location.
MR presents a means that allows consultation and treatment with physicians from a remote location, as it employs techniques such as holographic overlaying of images and data onto real life.
