Mixed Reality

VIROO Studio provides a way to feed Mixed Reality content to HTC Elite XR headset using a new feature of the VIVE Business Streaming (VBS).

Steps to include Mixed Reality in a VIROO Application

1. Configure VIROO Studio Project with Packages from HTC.
2. Add Mixed Reality Components to allow PassThrough Content.

Configure VIROO Studio Project with Packages from HTC

To add the necessary VIVE packages, you have to edit the manifest.json file. You can find the project manifest file in the Packages folder under the root folder of your Unity project.

Add the following packages to the dependencies section. To find out which version to use, go here and get the latest available version 5 (for example: 5.6.0-r.10.2):

• com.htc.upm.wave.native
• com.htc.upm.wave.xrsdk

And the VIVE registry to the scopedRegistries section:

OpenUPM

Name: VIVE
Url: https://npm-registry.vive.com
Scopes:

• com.htc.upm

The result should be the following, using the latest version of the packages:

{
  "dependencies": {
    /* ... other dependencies ... */
    "com.htc.upm.wave.native": "5.6.0-r.10.2",
    "com.htc.upm.wave.xrsdk": "5.6.0-r.10.2",
  },
  "scopedRegistries": [
    /* ... other scoped registries ... */
    {
      "name": "VIVE",
      "url": "https://npm-registry.vive.com/",
      "scopes": [
        "com.htc.upm"
      ]
    }
  ]
}

Add Mixed reality components

To generate a Mixed Reality content, we are going to make a composition of our 3D content over the Passthrough image provided by the Headset cameras. Our 3D generated frame buffer will be multiplied with the camera image using Alpha channel. The image from the cameras will be visible in areas where our Unity generated 3D frame buffer is transparent, and it will be hidden by 3D objects where they are opaque.

So the trick is to use a camera which renders a transparent background using Full Black and Transparent Color (0,0,0,0) as Clearing method (instead of a Skybox).

MR_Passthrough.cs component (listed below) provides a simple implementation to allow switching between Mixed and Virtual Reality with three methods.

Drag it to any GameObject to be present in the scene, and then:

• Make a call to ToggleMixedReality() to switch between mode.
• Make a call to SetMixedReality() to enter MR.
• Make a call to SetFullImmersiveVR() to enter MR.

Note: This script looks for GameObjects in the scene tagged with the HideInMR Component, those decorative GameObjects containing HideInMR (like walls, ceiling, ...) will be deactivated automatically in Mixed Reality. This is useful to allow the same VIROO Application to be enjoyed simultaneously in both Mixed Reality and Virtual Reality for several users, allowing them to decide the render mode in realtime.

HideInMR.cs Code listing.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class HideInMR : MonoBehaviour{}

In the example code provided there are public methods to enable/disable the MR or toggle between the two. You can call these methods from VIROO Actions.

In the example below it is being added to a menu with a RayInteraction and UnityEventNonBroadcastAction, since this should only happen on the local user's instance, but you could use other Actions or your custom code.

MR_Passthrough.cs Code listing.

using System.Collections;
using UnityEngine;
using UnityEngine.Rendering.PostProcessing;
using Viroo.Interaction.XRInteractionToolkit;
using Wave.Native;

public class MR_Passthrough : MonoBehaviour
{
    public bool inMR = false;

    private Camera cam;
    private Color savedColor;
    private CameraClearFlags savedClearFlags;

    // Toggle between modes
    public void ToggleMixedReality()
    {
        if (inMR)
        {
            SetFullImmersiveVR();
        }
        else
        {
            SetMixedReality();
        }
    }

    public void SetMixedReality()
    {
        ShowPassthroughUnderlay();
    }

    public void SetFullImmersiveVR()
    {
        HidePassthroughUnderlay();
    }

    // Unity Start Event Initialization
    IEnumerator Start()
    {
        cam = Camera.main;
        savedColor = cam.backgroundColor;
        savedClearFlags = cam.clearFlags;

        yield return null;

        if (inMR)
            ShowPassthroughUnderlay();
    }

    // internal private implementations

    void ShowPassthroughUnderlay()
    {
        cam.clearFlags = CameraClearFlags.SolidColor;
        cam.backgroundColor = Color.white * 0;

        // HTC provided methods to handle Passthrough functionality
        Interop.WVR_SetPassthroughImageQuality(WVR_PassthroughImageQuality.PerformanceMode);
        Interop.WVR_ShowPassthroughUnderlay(true);

        HideInMR[] hideInMRs = GameObject.FindObjectsOfType<HideInMR>(true);
        foreach (var hideInMR in hideInMRs)
        {
            hideInMR.gameObject.SetActive(false);
        }

        inMR = true;
    }

    void HidePassthroughUnderlay()
    {
        cam.clearFlags = savedClearFlags;
        cam.backgroundColor = savedColor;

        Interop.WVR_ShowPassthroughUnderlay(false); //Hide Passthrough Underlay

        HideInMR[] hideInMRs = GameObject.FindObjectsOfType<HideInMR>(true);
        foreach (var hideInMR in hideInMRs)
        {
            hideInMR.gameObject.SetActive(true);
        }
        inMR = false;
    }
}