Absolutely fascinating paper here from Microsoft Research describing the design of a holographic display technology that can achieve 80 degrees field of view or more. I remember sitting in a bar in London circa 1980 with a colleague discussing how to produce custom wavefronts for CGI applications. We went down a black hole fast but this kind of tech is exactly what we would have needed.
Following on from my previous post regarding HoloLens SpectatorView, I had been wondering if it was possible to use a webcam instead of a DSLR. It changes the mounting concepts but, just for testing, it wasn’t hard to place a Logitech C920 webcam on top of the HoloLens and get it aligned enough physically so that the calibration data numbers looked reasonable.
An immediate problem was that the code was not setting the webcam’s frame size. A quick look at OpenCVFrameProvider.cpp showed the problem. The code was trying to set the frame width and height before opening the capture object which doesn’t work. This is the original:
The fix is put line 44 before line 41. Then it works fine. The preview window in the calibration code has red and blue swapped but the processed images are correct. Once it was calibrated, I could go on and run the Unity app and look at the composite output – now pretty decent 1080p video.
Using a webcam like the C920 is far from perfect however. The field of view was measured at 75 degrees by the calibration software which really isn’t enough to be useful. Another problem is the autofocus which causes frequent focus breathing. And then there’s the challenge of proper mounting but at least the C920 does have a 1/4 inch thread so there are possibilities.
A decent DSLR (this would be my choice as it can output clean 4k 4:2:2 video over HDMI apparently at a decent price and I have all the lenses 🙂 ) is going to give better results for sure. On the other hand, there may be many applications where a webcam is just fine and you can’t argue with the price.
Interesting article here about eliminating null pointer crashes at the language level. I was intrigued by this since the other day I fixed an incorrect and unsafe cross-thread call situation that resulted in…a null pointer crash.
Another interesting New York Times piece on my favorite subject.
Related story here. The precision and stability is very impressive.
Having sorted out a way of mounting the HoloLens on a camera using the alternate rig described here, it was then time to put the rest of the system together. First thing was the Blackmagic Intensity Pro 4K capture card. Hardware and software installation was very straightforward and nicely captured video from the camera’s HDMI port. Next up was the SpectatorView software itself.
The first step is to get the calibration software working – instructions are here. The OpenCV link doesn’t work – use this instead. I am actually using VS2017 but had no problems apart from being asked if it was ok to upgrade things.
To calibrate the rig, a pattern is needed. This is my attempt:
Seemed to work ok. The next thing is to build the Compositor. It needs to be built for x86 and x64 in Release mode (x86 for the HoloLens app, x64 for the Unity Editor app). I think I had to force it to build SpatialPerceptionHelper in x86 mode. Anyway, once all that’s done, the DLLs need to be copied into the sample app (I was using the sample which is the Shared Holograms tutorial code).
It took me a while to realize that the CopyDLL.cmd needs parameters for it to work with the sample app. Comments in the code tell you what to do but it is basically this:
Time to fire up Unity using the sample app. Double click on the Sharing scene to kick it in. Then, click on the SpectatorViewManager object and look at the inspector. The Spectator View IP address needs to be set to the IP address of the HoloLens in the SpectatorView rig. Took me a while to work that out :-(. The Sharing Service IP field needs to be set to the address of the machine running the sharing server. The sharing server can be kicked off from Unity using the menu bar with HoloToolkit->SharingServer->Launch Sharing Service. The Sharing prefab also needs to be configured with the address of the sharing server. Once that’s done, it’s pretty much ready to deploy to the SpectatorView HoloLens and any others in the system.
The app needs to be run in the Unity Editor and then, using the menu bar again, kick off Spectator View->Compositor. This will shows a window with the combined live video from the camera and the virtual objects mixed in. This window also provides buttons to save video and snapshots.
Unfortunately, I only have one HoloLens to hand so I couldn’t really test the system. I did build a little test app that seemed to work ok as well as far as I could test it.
The biggest issue was my inadequate camera. I was hoping to find a way to use my Canon 6D for this, even though it does not fill the 1920 x 1080 output frame via its live HDMI port. I figured an OpenCV hack could deal with that. The bigger problem is that output is interlaced and causes horrible horizontal tearing in the composed video if anything in the scene is moving. I think it’s the end of the line for the 6D and SpectatorView.
Time for a proper 1080p/4K camera.
One of the most fascinating uses for the HoloLens is as part of a SpectatorView rig, which allows recording of very high quality mixed reality videos. The original mount was complicated but there is also a design for an alternate mount that is much simpler and does not require taking the HoloLens apart. My version is actually slightly simpler than that in some ways.
The original instructions suggest taking the thumbscrews out of the horizontal bracket but I left them in. This requires drilling out the center hole of the two clamps to 0.25 inches and using a nut but that seemed easier than changing everything. BTW the EOS 60D is not the camera I plan to use – it is just a mule for the photos. I was using the real camera to take the photos :-).
There’s actually another useful mode and that’s without the camera so that the tripod bolt goes straight into the horizontal bracket. This is very handy for capturing stable mixed reality from the HoloLens itself rather than the much more complicated true SpectatorView setup.
These are the parts that I used:
- 2 x clamps.
- 1 x dual flash bracket.
- 1 x tripod screw to hot shoe adaptor (pack of 2 but much the same price as one).
Add a tripod and HoloLens and you are good to go.