Having sorted out a way of mounting the HoloLens on a camera using the alternate rig described here, it was then time to put the rest of the system together. First thing was the Blackmagic Intensity Pro 4K capture card. Hardware and software installation was very straightforward and nicely captured video from the camera’s HDMI port. Next up was the SpectatorView software itself.
The first step is to get the calibration software working – instructions are here. The OpenCV link doesn’t work – use this instead. I am actually using VS2017 but had no problems apart from being asked if it was ok to upgrade things.
To calibrate the rig, a pattern is needed. This is my attempt:
Seemed to work ok. The next thing is to build the Compositor. It needs to be built for x86 and x64 in Release mode (x86 for the HoloLens app, x64 for the Unity Editor app). I think I had to force it to build SpatialPerceptionHelper in x86 mode. Anyway, once all that’s done, the DLLs need to be copied into the sample app (I was using the sample which is the Shared Holograms tutorial code).
It took me a while to realize that the CopyDLL.cmd needs parameters for it to work with the sample app. Comments in the code tell you what to do but it is basically this:
Time to fire up Unity using the sample app. Double click on the Sharing scene to kick it in. Then, click on the SpectatorViewManager object and look at the inspector. The Spectator View IP address needs to be set to the IP address of the HoloLens in the SpectatorView rig. Took me a while to work that out :-(. The Sharing Service IP field needs to be set to the address of the machine running the sharing server. The sharing server can be kicked off from Unity using the menu bar with HoloToolkit->SharingServer->Launch Sharing Service. The Sharing prefab also needs to be configured with the address of the sharing server. Once that’s done, it’s pretty much ready to deploy to the SpectatorView HoloLens and any others in the system.
The app needs to be run in the Unity Editor and then, using the menu bar again, kick off Spectator View->Compositor. This will shows a window with the combined live video from the camera and the virtual objects mixed in. This window also provides buttons to save video and snapshots.
Unfortunately, I only have one HoloLens to hand so I couldn’t really test the system. I did build a little test app that seemed to work ok as well as far as I could test it.
The biggest issue was my inadequate camera. I was hoping to find a way to use my Canon 6D for this, even though it does not fill the 1920 x 1080 output frame via its live HDMI port. I figured an OpenCV hack could deal with that. The bigger problem is that output is interlaced and causes horrible horizontal tearing in the composed video if anything in the scene is moving. I think it’s the end of the line for the 6D and SpectatorView.
Time for a proper 1080p/4K camera.