Category Archives: HoloLens

Using a webcam with a HoloLens SpectatorView rig

Following on from my previous post regarding HoloLens SpectatorView, I had been wondering if it was possible to use a webcam instead of a DSLR. It changes the mounting concepts but, just for testing, it wasn’t hard to place a Logitech C920 webcam on top of the HoloLens and get it aligned enough physically so that the calibration data numbers looked reasonable.

An immediate problem was that the code was not setting the webcam’s frame size. A quick look at OpenCVFrameProvider.cpp showed the problem. The code was trying to set the frame width and height before opening the capture object which doesn’t work. This is the original:

The fix is put line 44 before line 41. Then it works fine. The preview window in the calibration code has red and blue swapped but the processed images are correct. Once it was calibrated, I could go on and run the Unity app and look at the composite output – now pretty decent 1080p video.

Using a webcam like the C920 is far from perfect however. The field of view was measured at 75 degrees by the calibration software which really isn’t enough to be useful. Another problem is the autofocus which causes frequent focus breathing. And then there’s the challenge of proper mounting but at least the C920 does have a 1/4 inch thread so there are possibilities.

A decent DSLR (this would be my choice as it can output clean 4k 4:2:2 video over HDMI apparently at a decent price and I have all the lenses ­čÖé ) is going to give better results for sure. On the other hand, there may be many applications where a webcam is just fine and you can’t argue with the price.

Putting together a HoloLens SpectatorView rig

Having sorted out a way of mounting the HoloLens on a camera using the alternate rig described here, it was then time to put the rest of the system together. First thing was the Blackmagic Intensity Pro 4K capture card. Hardware and software installation was very straightforward and nicely captured video from the camera’s HDMI port. Next up was the SpectatorView software itself.

The first step is to get the calibration software working – instructions are here. The OpenCV link doesn’t work – use this instead. I am actually using VS2017 but had no problems apart from being asked if it was ok to upgrade things.

To calibrate the rig, a pattern is needed. This is my attempt:

Seemed to work ok. The next thing is to build the Compositor. It needs to be built for x86 and x64 in Release mode (x86 for the HoloLens app, x64 for the Unity Editor app). I think I had to force it to build SpatialPerceptionHelper in x86 mode. Anyway, once all that’s done, the DLLs need to be copied into the sample app (I was using the sample which is the Shared Holograms tutorial code).

It took me a while to realize that the CopyDLL.cmd needs parameters for it to work with the sample app. Comments in the code tell you what to do but it is basically this:

CopyDLL "%~dp0\Samples\SharedHolograms\Assets\"

Time to fire up Unity using the sample app.┬áDouble click on the Sharing scene to kick it in. Then, click on the SpectatorViewManager object and look at the inspector. The Spectator View IP address needs to be set to the IP address of the HoloLens in the SpectatorView rig. Took me a while to work that out :-(. The Sharing Service IP field needs to be set to the address of the machine running the sharing server. The sharing server can be kicked off from Unity using the menu bar with┬áHoloToolkit->SharingServer->Launch Sharing Service. The Sharing prefab also needs to be configured with the address of the sharing server. Once that’s done, it’s pretty much ready to deploy to the SpectatorView HoloLens and any others in the system.

The app needs to be run in the Unity Editor and then, using the menu bar again, kick off Spectator View->Compositor. This will shows a window with the combined live video from the camera and the virtual objects mixed in. This window also provides buttons to save video and snapshots.

Unfortunately, I only have one HoloLens to hand so I couldn’t really test the system. I did build a little test app that seemed to work ok as well as far as I could test it.

The biggest issue was my inadequate camera. I was hoping to find a way to use my Canon 6D for this, even though it does not fill the 1920 x 1080 output frame via its live HDMI port. I figured an OpenCV hack could deal with that. The bigger problem is that output is interlaced and causes horrible horizontal tearing in the composed video if anything in the scene is moving. I think it’s the end of the line for the 6D and SpectatorView.

Time for a proper 1080p/4K camera.

Even simpler mount for HoloLens SpectatorView

One of the most fascinating uses for the HoloLens is as part of a SpectatorView rig, which allows recording of very high quality mixed reality videos. The original mount was complicated but there is also a design for an alternate mount that is much simpler and does not require taking the HoloLens apart. My version is actually slightly simpler than that in some ways.

The original instructions suggest taking the thumbscrews out of the horizontal bracket but I left them in. This requires drilling out the center hole of the two clamps to 0.25 inches and using a nut but that seemed easier than changing everything. BTW the EOS 60D is not the camera I plan to use – it is just a mule for the photos. I was using the real camera to take the photos :-).

There’s actually another useful mode and that’s without the camera so that the tripod bolt goes straight into the horizontal bracket. This is very handy for capturing stable mixed reality from the HoloLens itself rather than the much more complicated true SpectatorView setup.

These are the parts that I used:

Add a tripod and HoloLens and you are good to go.

Smart spaces and IoT data – the challenge is what to do with it

A while back I built some add-on cards for Raspberry Pis to do some environmental monitoring around the house. This is one of them.

The project starting collecting dust when I couldn’t really think of good ways of using the data, beyond triggering an alarm under some conditions or something. However, it’s often interesting just to see what’s going on around the place so I have revived the sensors (a good use for old first generation Pis). The screen capture shows a simple but actually quite effective way of using the data that’s being generated, providing a display that’s adjacent to the camera feed from a webcam on the same Pi. Between the two streams, you can get good confidence on what’s happening in the smart space.

One day, I’d like to get the HoloLens integrated with this so that I can see the data when I am in the smart space. That would be even more fun.

Why HoloLens is like Aibo…except hopefully it isn’t

aibohololensLooks like Aibo has got hold of my HoloLens again. So why is HoloLens like Aibo? Well Aibo was an absolutely fantastic piece of engineering and way ahead of its time. Sony managed to make a viable consumer robot that didn’t do anything practical but nevertheless was highly entertaining. Some of the tricks it can do with its ball are very impressive to say the least! The skill in building robots is to bring together a large number of disparate technologies and integrate them into a consistent product. Aibo is a great example of doing this in a very successful way.

HoloLens similarly brings a raft of disparate technologies into a very well engineered and complete device that seems to stand alone in terms of the totality of its capabilities for Mixed Reality. It really does remind me of Aibo in this regard.

Just one thing. Sony killed off the entire robotics effort because it wasn’t making enough cash in the short term, a wonderful example of myopia in my opinion. I am hoping that Microsoft don’t fall into the same trap with HoloLens. This piece suggests that HoloLens won’t suffer a similar fate which is fantastic. The AR and MR market is going to be driven by continuing new developments in devices that make them smaller, lighter and have longer battery life so that, one day, people will wear them all day and leave their smartphones gathering dust in a drawer. I look forward to seeing and using many future generations of HoloLens!