Hand Tracking for SteamVR + VRidge

Cameron Swisher wrote a guide on how to use Leap Motion as Vive Controllers. According to Cameron it works with VRidge so we thought we should share it since it allows to play games like The Lab (some tests are broken) which before were inaccessible.

Warning: This program is still very new and prone to be buggy and glitchy, so expect bugs and for it to not be 100% perfect. Will make a guide later on how to compile this manually for unreleased builds.

1. Make sure you have these before installing the SteamVR Leap Motion Driver:

• Install "Leap Motion Orion SDK". (Click on the Orion Beta (VR) button), Account is required to download the software
• Install SteamVR (Most of you should already have it)
• Install the Visual C++ 2015 Update 2 redistributables (32 and 64 bits) https://www.microsoft.com/en-us/download/details.aspx?id=51682

2. Download the file called "Steamvr_leap_driver_(version name_.zip" (created by cbuchner1 @ Github) (If you have installed Alpha One version before, make sure to uninstall Alpha One and install Alpha Two).

3. Extract zip file and run setup.exe.

4. Go through the installation process (just keep clicking "Next").

5. Once the installation process is done just close the window.

6. If everything worked, controller icons in SteamVR window should lit up green now (try restarting SteamVR if it doesn't for some reasons).

What about controls?

Currently following gestures are possible (subject to change a lot, so visit here to keep up to date):

• Bending of the index finger maps to the trigger button, like you would fire a gun
• Clenching the middle, ring, pinky finger to a fist maps to the grabbing buttons
• The thumbpress gesture (just point the thumb in the direction of your palm) touches and clicks the trackpad
• Flat hand held in front of you, palm towards your face is used for application menu button
• Flat hand held in front of you, palm away from you is used for system menu button

Leap Motion in action

Note: The Lab is working, but some tests are broken currently.

Note: Minecraft does indeed work with both, the program and VRidge, see the link for details. Very buggy since the hand tracking program is still new, but it will get better as time goes on. The Minecrift program (linked above) is still kinda new and some bugs need to be worked out. The main movement method is teleportation.


You can find the list of currently known issues/bugs here.


Cameron will be updating his guide and you can send him your feedback in the google docs here.

Dev update #8

This week we are not deploying desktop client patch - Android is getting small update with version 0.8.3. We optimized several things in mobile app but most of the week was spent on improving our SteamVR driver (new version is not ready yet) and searching for Android artifacting solution.

Small QOL changes

Several people suggested to hide the vertical white bar because it was visible in the viewport. It will now automatically hide after 10 seconds. Settings button will now automatically dim by 70% too.

If you want to align your cardboard to the screen, tap anywhere to show the vertical bar again (it will disappear after 30 seconds after manual toggle).

We also added "Never ask me again" option for bluetooth prompt so it won't annoy you anymore. We're leaving a small note at the bottom if it's online.

There were some other changes, optimizations mostly but nothing worth writing about. Some bytes were spared. Some bytes move a little faster.

SteamVR driver progress

As some of you noticed you had to reinstall our SteamVR driver after yesterday's Steam update. Next version of our driver will be properly registered with Steam API and won't require reinstallation anymore. Actually it won't even require manual installation, it will be done automatically.

We are not entirely sure but after some changes in our pipeline it seems that SteamVR latency is even lower now. It's hard to measure but it all seems more fluent.

New version will also allow better context switching between VR apps. More on this when it's out.

The ongoing battle with artifacts - frame loss / latency @ decoder

This is very annoying problem because whenever we finally think "Okay, that was it, fixed" something new comes up. This is the issue that can be identified by tapping top-left part of the streaming screen and seeing high (above 0.1 ms) decoder latency and constantly raising Frames lost @ decoder counter.

We had ~30 reports of this happening and we can now share some data - this is top 8 of reported problems.

Phone name	Reported cases	Install count	%	Chip
Asus Zenfone 2	5	89	5.62%	Intel® Atom™ Quad Core Z3580
Galaxy S4	5	84	5.95%	Exynos 5410 | Snapdragon 600
Galaxy S7	2	125	1.60%	Exynos 8890 | Snapdragon 820
Galaxy S7 Edge	2	339	0.59%	Exynos 8890 | Snapdragon 820
Redmi Note 2	2	60	3.33%	MediaTek Helio X10
Nexus 6	1	110	0.91%	Snapdragon 805
HUAWEI Honor 6	1	8	12.50%	Hisilicon Kirin 910 Octa Core
Xperia Z C6603	1	28	3.57%	Snapdragon S4

Galaxy S4 has the most problems. There were 2 users reporting hard device crashes. 

Zenfone 2 also has constant artifacting. No crashes were reported but it is still unusable if it's turning into blocky pixel mess after several seconds.

Galaxy S7/Edge seems to have something special about its decoder too.

These numbers are high because when we look at top 10 most popular devices of our users (S6 & OnePlus One leading the pack) there are 2000+ installs shared between them and 0.1% error report rate. Also, the reported errors are probably caused by something else because there were more issues than artifacting there.

One thing I would like to know. Is there anyone that got VRidge to work successfuly with bolded phones in the table? 

Please let us know if you got it to work. I know about one Zenfone 2 user that fixed it by changing CPU governor to interactive but the same didn't work for other phones. Send us an e-mail, PM @ reddit or use any other contact channel listed at the right column, at the top of the page.

Dev update #7 - The one with SteamVR

Before we start, we want to say one thing:

It's a beta! Please do not buy SteamVR games specifically for VRidge.

This technology in current state relies on many third party components and compatibility cannot be guaranteed. You can test SteamVR compatibility with many free SteamVR games available on Steam.

We've been working on SteamVR integration for quite some time and we finally can release a version that should work more often than not. I'm cross-posting these notes to reddit because many of you already have accounts there and it's an easy way to discuss and troubleshoot as a community.

https://www.reddit.com/r/vridge/comments/4fu4sn/dev_update_7_the_one_with_steamvr/

SteamVR

"It's happening!"

Please keep in mind that this tech is still in experimental / unstable stages. Save your games often because crashes can and will happen. Not all games work and compatibility may come and go with game updates. We tried some most-wanted titles and here are quick summaries what you can expect:

• Euro Truck Simulator 2
• You need to go to Steam properties of ETS and use beta branch named "Oculus". Then you need to add -openvr in launch arguments. Disable Steam Desktop Theater for this game so it won't launch in flat mode.
• Elite: Dangerous
• It works but has issues with blurry text. This problem is common even on real HTC Vive devices. You can read some research here.
• We recommend using 1920x1080 for stream resolution. Make sure your phone has 1080p display as well.
• If you are getting black screen, see this post by@adrensnyder.
• Virtual Desktop / Steam Desktop Theater
• Those apps have extra lag and capture problems because essentialy everything is captured and rendered twice as many times as native VR games. We are trying to figure out ways to reduce the latency in those cases.
• We recommend using 1920x1080 for stream resolution. Make sure your phone has 1080p display as well.

There are few steps you need to follow before SteamVR games can work.

1. Install SteamVR runtime in your Steam.

2. Install VRidge SteamVR driver. SteamVR path should be filled automatically. If it's empty, manually point it to vrpathreg.exe, which is located somewhere in your SteamVR location. This can be done after connecting your mobile app to the client. 

3. Launch SteamVR. You need to launch SteamVR while your mobile app is connected to RiftCat. If you launch SteamVR without mobile app paired, close it and start it again after connecting mobile app.

If you can't see this button, start SteamVR from your library (it's a regular game)

If you are asked to run "Room setup" please choose standing experience and enter your height when you are asked to. Otherwise your camera may be too close to the ground for comfortable use.

SteamVR setup/tutorial VR mode is upside down and aligned incorrectly. It won't affect other games.

4. Small SteamVR window will open. If your mobile device was connected when you started SteamVR, it should display Connected and SteamVR games will show Play in VR as an option.

That's it! You should be able to play a lot of amazing SteamVR games now. If your mobile device drops connection you can reconnect VRidge by reopening mobile app. It shouldn't require game/SteamVR restart.

If you are stuck with Headset not detected message, close SteamVR, make sure your mobile device is connected to Riftcat client and start SteamVR again.

If you have problems, please reach out to us through our help center (we will keep adding known issues with SteamVR there soon) or reddit (so we can share workaround and solutions as a community):

https://www.reddit.com/r/vridge/comments/4fu4sn/dev_update_7_the_one_with_steamvr/

Android CPU usage optimized == a lot less pixel artifacts

We solved one very common cause of pixel artifacts happening due to "Frame loss @ decoder". We optimized some of our algorithms and the overall CPU usage dropped by ~50%. This is important because Android devices will generate less heat and therefore Android OS won't downclock CPU to counter the heat.

Core frequency downclocking due to high heat (thermal throttling) fixed pixel aritfacts for nearly all of our devices. There is still room for more optimizations so if you are still experiencing high frame loss @ decoder, don't worry - we will keep trying to optimize and find all the problems.

We have a request - can anyone with Asus Zenfone 2 reply in comments if they got VRidge working correctly? It seems that everyone with this specific phone model is getting constant artifacts.

Memory leaks fixed in NVENC and Media Foundation encoder modules

Some of you reported that staying in-VR with VRidge for longer than 30-40 minutes resulted in VRidge crashing due to lack of memory. We fixed two memory leaks that caused RAM usage to grow 30-100 MB every minute. Sorry for the crashes, totally our fault.

Dev update #6 - Win 7, 8, 8.1 support for NVIDIA users

Update for NVIDIA users:

Windows 7 & 8 is still having some issues (infinite awaiting VR stream/crash) due to missing .DLL. Windows 8.1/10 should work. Possible workaround here.

---

The numbers in the title aligned quite nicely. This week we focused on NVIDIA support for Windows 7+. Next focus is solving artifacts @ Android's video decoder.

NVENC / Windows 7+ support for NVIDIAs

Note: If this update causes more artifacts for you, go to stream quality settings and disable NVENC. It's still highly experimental feature but it's enabled by default for all NVIDIA users.

We implemented NVENC. Until now we used Media Foundation to handle video encoding but unfortunately it requires Windows 8.1+ for AMD or Windows 10 for NVIDIA. It also has some undefined issues ("Not Enough Storage" errors and some other crashes) with AMD video cards, especially after WIndows 8.1->Windows 10 upgrade instead of clean installation.

NVENC is alternative encoding layer supported by NVIDIA driver. It has some extra features for low-latency streaming and anti-lag measures (error concealment). They are not all enabled yet because they amplify Android decoder problems (read about it later in this post) but we already tested some techniques and you can expect less artifacts and even lower latency in the future thanks to NVENC capabilities.

AMD users might wonder "Can we get something like this too?" The answer is yes (via direct VCE implementation) but we don't have any ETAs yet. This might solve some crashes for AMD users, especially those with mobile Radeons.

The ongoing battle with artifacts

We can finally reproduce one major problem reported by many users -  frame loss @ video decoder. 

For some people it starts very shortly after start of the stream (basically preventing VRidge usage because it artifacts all the time). Other people experience it after several mintues (the higher quality settings, the sooner it happens). 

Now that we can finally reproduce it - it shouldn't take too long to finally solve this problem.

When can I play new VR games (like Elite: Dangerous, ETS, ATS, Virtual Desktop, etc.) with Riftcat? 

This is the question asked the most. One of our devs has been working for several weeks to get them working with VRidge. Expect more news on this matter next week.

Dev update #5

This week we've been working on some of the new VR SDKs because we really want to bring the most recent VR products (like Virtual Desktop) to VRidge. We have nothing to announce yet but things are going pretty good.

This update is focused on bugfixing and diagnostic tools.

Nexus 5X and 6P no longer get stuck in reconnecting loop

What a long strange trip it's been. Eventually it turned out that it was very simple but odds of it happening were like 1 to 65 thousand so we did not bother checking it at first.

We chose UDP port that was actively in use on these devices and VRidge could not bind to it. All bug reports were useful but we'd like to especially thank to following people who were patient enough to test countless APKs: Tailslol, Brad Barnes, WACOMalt. 

Nexus 5X joins testing family.

Fixed head tracking not starting properly

Some people reported head tracking not working. Screen was slightly shaking when you rotated your phone but view did not actually rotate. It should be fixed now.

Screen is no longer skewed when rolling your head

In update #3 we changed our viewport calculation to maximize available screen space. In update #4 we added extra Scale/IPD options because viewport was too big (pixels were outside visible space, reducing effective FOV). At some point we broke our aspect ratio calculation resuliting in jello effect when you rolled your head in Z axis. We're fixing it now as seen on picture below.

@kill_dano - thanks for reporting this.

You can see that the left rectangle does not fit black void in the distance.

Ctrl + Alt + Shift + R resets your head orientation

Some games already allow you to reset head tracking to original position but we are also adding our own hotkey for games that don't have this built-in. Look straight ahead and press Ctrl+Alt+Shift+R. 

Using both game-implemented tracking reset key and VRidge tracking reset key can result in undefined behaviour. Use only one, don't mix them.

We'll add some UI tooltip about this hotkey when we're sure it works correctly. 

The ongoing battle with artifacts

Some people still experience artifacts. It's not easy to fix because there are so many variables that can affect stream quality - encoder, antennas, cables, wifi interference, usb controller, decoder, etc. Even single component not performing good enough can break the whole chain and cause artifacts. 

We are lowering preset bitrates from (Low/Medium/High) 6/12/16 Mbps to 6/10/13 Mbps (you can still manually set whatever you want). We're also adding a note in UI what bitrate actually means. 

We are also adding optional diagnostics screen. If you experience constant artifacting with USB please tap the top-left part of the streaming screen and report back wether it's decoder or network dropping your frames.

I love numbers and stats. Maybe we'll add graphs someday too.

Simplified connection again

It will be less likely to get stuck in "restoring connection" loop. We probably broke something else with that because changing one thing can cause a chain reaction with some other stuff so please report if anything new breaks for you.

Added storage permission prompt for Android 6.0

Scanning Cardboard QR code with Android 6.0 had no effect because Cardboard settings are stored on device storage. Android 6.0 was denying access to device storage by default. We're adding storage permission prompt so VRidge can access and modify Cardboard settings.

More news on post-release VR SDKs soon

We're not gonna promise anything until we are absolutely sure we can deliver. More news in 2nd half of April.