Another interesting New York Times piece on my favorite subject.
Related story here. The precision and stability is very impressive.
Having sorted out a way of mounting the HoloLens on a camera using the alternate rig described here, it was then time to put the rest of the system together. First thing was the Blackmagic Intensity Pro 4K capture card. Hardware and software installation was very straightforward and nicely captured video from the camera’s HDMI port. Next up was the SpectatorView software itself.
The first step is to get the calibration software working – instructions are here. The OpenCV link doesn’t work – use this instead. I am actually using VS2017 but had no problems apart from being asked if it was ok to upgrade things.
To calibrate the rig, a pattern is needed. This is my attempt:
Seemed to work ok. The next thing is to build the Compositor. It needs to be built for x86 and x64 in Release mode (x86 for the HoloLens app, x64 for the Unity Editor app). I think I had to force it to build SpatialPerceptionHelper in x86 mode. Anyway, once all that’s done, the DLLs need to be copied into the sample app (I was using the sample which is the Shared Holograms tutorial code).
It took me a while to realize that the CopyDLL.cmd needs parameters for it to work with the sample app. Comments in the code tell you what to do but it is basically this:
Time to fire up Unity using the sample app. Double click on the Sharing scene to kick it in. Then, click on the SpectatorViewManager object and look at the inspector. The Spectator View IP address needs to be set to the IP address of the HoloLens in the SpectatorView rig. Took me a while to work that out :-(. The Sharing Service IP field needs to be set to the address of the machine running the sharing server. The sharing server can be kicked off from Unity using the menu bar with HoloToolkit->SharingServer->Launch Sharing Service. The Sharing prefab also needs to be configured with the address of the sharing server. Once that’s done, it’s pretty much ready to deploy to the SpectatorView HoloLens and any others in the system.
The app needs to be run in the Unity Editor and then, using the menu bar again, kick off Spectator View->Compositor. This will shows a window with the combined live video from the camera and the virtual objects mixed in. This window also provides buttons to save video and snapshots.
Unfortunately, I only have one HoloLens to hand so I couldn’t really test the system. I did build a little test app that seemed to work ok as well as far as I could test it.
The biggest issue was my inadequate camera. I was hoping to find a way to use my Canon 6D for this, even though it does not fill the 1920 x 1080 output frame via its live HDMI port. I figured an OpenCV hack could deal with that. The bigger problem is that output is interlaced and causes horrible horizontal tearing in the composed video if anything in the scene is moving. I think it’s the end of the line for the 6D and SpectatorView.
Time for a proper 1080p/4K camera.
One of the most fascinating uses for the HoloLens is as part of a SpectatorView rig, which allows recording of very high quality mixed reality videos. The original mount was complicated but there is also a design for an alternate mount that is much simpler and does not require taking the HoloLens apart. My version is actually slightly simpler than that in some ways.
The original instructions suggest taking the thumbscrews out of the horizontal bracket but I left them in. This requires drilling out the center hole of the two clamps to 0.25 inches and using a nut but that seemed easier than changing everything. BTW the EOS 60D is not the camera I plan to use – it is just a mule for the photos. I was using the real camera to take the photos :-).
There’s actually another useful mode and that’s without the camera so that the tripod bolt goes straight into the horizontal bracket. This is very handy for capturing stable mixed reality from the HoloLens itself rather than the much more complicated true SpectatorView setup.
These are the parts that I used:
- 2 x clamps.
- 1 x dual flash bracket.
- 1 x tripod screw to hot shoe adaptor (pack of 2 but much the same price as one).
Add a tripod and HoloLens and you are good to go.
The earlier Smart space post got me thinking about other related projects and I came across these old screen captures from the AwareSpace project. This was a much more serious attempt to make use of ubiquitous sensor data. It worked fine, giving easy access to real time and historic data from sensors. There was even web access to the system. Like many projects, it was never really finished and needed a lot more work to do everything that I wanted. One day…
Another piece in the New York Times arguing against non-competes that highlights California’s enlightened approach. How is it that only one state (and actually North Dakota is not too bad) has seen the light on this?