Category Archives: Mixed reality

YAXRH – this time a Samsung Odyssey Windows MR headset

This Samsung Odyssey Windows MR headset just arrived and it is really quite good. The earlier developer’s HP headset didn’t have the motion controllers so a HoloLens clicker (or Xbox controller) had to be repurposed for meaningful interaction. The motion controllers are really kind of fun and it’s totally spooky to watch the virtual joysticks move all by themselves when you adjust the real joysticks. The built in sound is another great advantage. It makes the headset somewhat bulky but the benefit is great spatial sound. The images are pretty good too although you do have to get the headset positioned correctly for optimum quality. Once you do, there’s not too much chromatic aberration in a fairly reasonable central area. The distance between the lenses is also adjustable which is another assist in getting good visual quality. The display certainly has the usual screen door effect but it isn’t really very offensive and resolution seems very acceptable. On the negative side, the display does not flip up (well it does once if you push hard enough 🙂 ) which is a bit of a negative while developing software where it is sometimes handy to go back and forth to a desktop display.

It’s kind of fun to open up the desktop and look at the MR Portal there so you can get the classic video feedback effect. I tried watching some movie trailers – not too bad. I then tried a game called Rock and Rails. Yes, well, that didn’t last too long. Instant vertigo and motion sensitivity – these things are just too immersive!

Anyway, a worthy addition to to the growing pile of headsets here.


My day with Windows Insider Preview, Unity 2017.2.0b8 and Windows Mixed Reality Headset

Yes, I am drinking a beer right now – it has been a long day. Mostly I seemed to spend it nursing Windows through its upgrade to the latest Insider Preview (16257) and begging the Insider Preview website to allow me to download the Insider Preview SDK which seemed to require all kinds of things done right and the wind blowing in the right direction at the same time.

The somewhat bizarre screen capture above is from a scene I created in the default room. The hologram figures are animated incidentally. What I mostly failed to do was to get existing HoloLens apps to run on the MR headset as Unity kept on reporting errors when generating the Visual Studio project for the apps, after having performed every other stage of the build process correctly. Very odd. I did manage to get a very simple scene with a single cube working ok, however.

Then I went back to the production version of Windows (15063) and tried things there. Ironically, my HoloLens app worked (apart from interaction) on the MR headset using Unity 5.6.2.

Clearly this particular Rome wasn’t built in a day – a lot more investigation is needed.

Spatial maps – the next valuable currency

Fascinating and thought provoking article here about iRobot’s reported plan to monetize the spatial maps created by Roombas. Time and time again in my career (including right now) there has been a need for accurate spatial maps. Once only accessible to high-end robots outfitted with Lidars, now almost anything that moves is capable of generating and refining spatial maps.

This fits very nicely with the idea that mixed reality glasses will become ubiquitous. Imagine walking into a new space and getting a spatial map automatically downloaded from the cloud. No need to ask where the restrooms are any more! This kind of capability would be of benefit to almost any enterprise. For example, check into a hotel and the spatial map with directions to your room gets downloaded to your glasses.

There are three parts to this puzzle – mapping, storage and delivery. Once all these become ubiquitous, not having access to this data or MR glasses will seem very odd indeed. Of course, selling data about private houses is not something that should be allowed without the owner’s explicit permission but making the data available to the owner would have tremendous value. There’s going to be a whole new type of specialist – the virtual interior designer. Unless you need to interact with something physically, why bother having the real object rather than a virtual version of it?

Of course there’s always the chance that some company gets the data and has some software that can detect if your floor plan has space for one of their products. In some kind of bizarre world the product could appear virtually in the space with a link to where you could buy it. This would be real/virtual product placement! What a ghastly prospect :-(.

Second version of HoloLens HPU – separating mixed reality from the cloud

Some information from Microsoft here about the next generation of HoloLens. I am a great fan of only using the cloud to enhance functionality when there’s no other choice. This is especially relevant to MR devices where internet connectivity might be dodgy at best or entirely non-existent depending on the location. Putting some AI inference capability right on the device means that it can be far more capable in stand-alone mode.

There seems to be the start of a movement to towards putting serious but low power-consuming AI capability in wearable devices. The Movidius VPU is a good example of this kind of technology and probably every CPU manufacturer is on a path to include inference engines in future generations.

While the HoloLens could certainly use updating in many areas (WiFi capability, adding cellular communications, more general purpose processing power, supporting real-time occlusion), adding an inference engine is certainly extremely interesting.

Mixed Reality and the missing fourth dimension

The screen capture above is a scene from a HoloLens via mixed reality capture (MRC) showing four virtual rings with different levels or brightness. The top left is 100% red, the bottom right black and the other two are intermediate levels of brightness.

The photograph above was shot through a HoloLens and is a reasonable representation of what the wearer actually sees. Unsurprisingly, since all see-through MR headsets work by overlaying light on the real scene, the black ring has vanished and the intermediate brightness rings become transparent to some degree based on the relative brightness to the real world scene.

This is a considerable obstacle for inserting realistic virtual objects into the real world – if they are dark, they will be almost transparent. And while indoors it is possible to control ambient lighting, the same is certainly not true outdoors.

What is needed is not just support for RGB but RGBA where A is the fourth dimension of color in this case. The A (alpha) value specifies the required transparency. The Unity app running on the HoloLens does of course understand transparency and can generate the required data but the HoloLens has no way to enforce it. One way to do this would be to supplement the display with an LCD that acts as a controllable matte. The LCD controls the extent to which the real world is visible at each display pixel while the existing display controls the color and intensity of the virtual object. No doubt there are significant challenges to implementation but this may be the only way to make see-through MR headsets work properly outdoors.