Dev update #34

In short

VRidge 2.0 will be released to beta channel in few weeks. NOLO update won't require hardware firmware update but will be delayed until 2.0.

---

Full version

NOLO update

Overview

When we released NOLO wireless update we estimated that we will be able to release another update in September that will contain rest of the features that were missing from the initial update. As you already know, we missed this deadline by two months. The following post is not an excuse because we definitely could have handled this better. Here's the story behind the delays.

We originally designed VRidge-API in early 2017 as a part of our shared vision to make NOLO work with VRidge seamlessly. Basically, it's an open interface that allows anyone (with some basic coding skills) to seamlessly intercept and manipulate VRidge tracking in any way. You can override rotations, positions, offsets, correct drifting, send controller data, etc. Everything will be automatically combined with whatever is sent from mobile phone in a non-conflicting way.

After it was ready, NOLO developers decided to take a new approach. They designed their own SteamVR driver that sends some of data directly to SteamVR (controllers) and some to VRidge (head data). This was released as NOLO windows driver delivered as their software installer package. This had some benefits and drawbacks. One benefit is making their hardware compatible with other SteamVR HMDs. The major drawback is that we had pretty much zero control over problems happening in this scenario. We were trying to support this mode both with support advice and some patches on our side but in the end - major part of this codebase was simply not accessible.

After some time, we decided to develop “NOLO wireless mode”. We changed VRidge API so it can be accessed over Wi-Fi. We then decided to create full NOLO windows driver equivalent on the Android side so we can control all data flow. This was supposed to allow us to fix bugs in a less hacky way. Open specification of VRidge API allows anyone (including NOLO or any developer) to connect ANY hardware connected to Android with VRidge. Our implementation was released with 1.5 for both Gear VR and Android.

Nevertheless, even with rewriting the driver on our side, we encountered quite a few challenges. The only way to communicate with NOLO hardware on the Android side is the NOLO Android SDK. We decided to use it but this has resulted in few minor and major challenges.

Challenges

The initial release wasn't working at all for nearly all of the people

It turned out that we were developing using a version that was behaving in a different way than consumer sent units. After we reported this we were sent another hardware kit with CV specs.

Our fix: We corrected the comms shortly after it was reported thanks to few helpful users that sent us diagnostic data from their CV units before our new hardware kit was shipped.

Random controller button press events are triggered every now and then

This is caused by NOLO Android SDK returning data that is randomly simply wrong. Based on our experience - the longer the devices are turned on and the more of them are communicating at the same time, the more likely it is for SDK to send random button presses. It can manifest itself usually with random SteamVR screenshots being triggered. All button states are stored into single number encoded as a 32-bit bitmask. This bitmask turns into something like 10011011101101010111011001111001 (some random buttons pressed) instead of 00000000000000000000000000000000 (no buttons pressed). Screenshot combo requires only two bits to be set, so it's highly likely that this random number will land a screenshot combo.

Our fix: We added filtering that requires the same bitmask to be set for at least two consecutive frames. This added extra frame of delay but filtered nearly all of those random bit flips.

Our fix #2*: "Nearly all" was not enough because for some users it was still happening (silicon lottery – not all units work equally good). We decided to add a configurable filtering window size (3-10) frames to de-noise the button input.

Wireless mode wasn't working with ceiling mounts properly.

This was caused by lack of any configuration/setup with NOLO hardware. With Valve Lighthouse there's a setup stage where all Lighthouse devices communicate with each other to automagically prepare 3D representation of the room. With NOLO Windows driver you had to swap a DLL to trigger ceiling mode because there was no configuration step. You had to physically alter the code (the DLL) to switch it every time you made changes to your NOLO base station position.

Our fix*: We added a checkbox in mobile settings that triggers the celling mode by transforming the room coordinates from front-facing scenario to top-down scenario.

NOLO hardware is hard to use with multiple apps installed that can use NOLO hardware on the Android.

This is caused by Android SDK architecture. Most hardware products are split into driver and whatever uses the driver. For example, Vive has a runtime (SteamVR) and API (OpenVR). Multiple OpenVR programs can use hardware or act as games while SteamVR resolves all conflicts. The exact same scenario applies to Oculus Runtime/Home and Oculus SDK or even VRidge.exe and VRidge-API.

NOLO works in a different way. Driver/runtime and client/access layer is mixed into single Android binary (it's even more complicated on the PC side but that's another story). With Android it caused conflicts because you had to always ask Android OS to grant USB permission for NOLO hardware. It became quite annoying during development because clicking on the same box every single restart is not the best UX. We then decided to make use of "Remember this decision" checkbox so once VRidge was granted NOLO hardware access, it was always auto-started when it was connected with permissions.

This turned out to create another problem because Android doesn't allow two apps (for example VRidge and some native NOLO android app) to both have access to the device. You now have to go to app settings and tap "Clear Defaults when you wanted to switch NOLO-using apps (even VRidge and VRidge for Gear VR). This could have been resolved by some sort of authoritative NOLO runtime/service on the Android side and we hope that something like this will be created in the future to prevent app conflicts.

Throwing motion does not work in some games.

Throwing can be implemented in several ways. The recommended way for SteamVR games is to ask controller for its velocity and acceleration and transfer the values to thrown objects while detaching them from the controller's grasp. That's kind of how physics work and it is the most accurate way to create a throwing motion. Unfortunately, the NOLO controllers were not providing acceleration and velocity at all and therefore - throwing in some games resulted in grenades dropping right down your feet.

We won't go into details of the long story of multiple NOLO SDKs and firmware updates taking over a month of trial and error changes to make it work at the hardware level. Adapting to every set of changes took a lot of time and eventually - none of these changes were good enough to be released.

The technical root cause of most of the problems in this story is the fact that providing acceleration and velocity triggers SteamVR positional interpolation that makes the controllers movement smoother. Unfortunately, those values need to be perfect - otherwise it causes a lot of terrible jitter of SteamVR trying to adjust position and velocity that are not in sync with each other. As you may have noticed with all those delays - this did not work out.

The throwing motion was the most time-consuming thing that delayed the whole update. This has also delayed all other fixes marked with * because we were always switching SDKs and firmware and it always seemed that the final working version is just around the corner. Now that we still can't define how far “the corner” actually is, we decided to go another way. We are dropping the idea of relying on firmware update that adds velocity hardware data and do it our own way. It might be not 100% physics accurate but at least it will throw your grenades in the correct direction.

The challenges listed above are only several of the technical obstacles that we had to overcome.

NOLO changes release date

This is basically "long story short" version of the above paragraph in case you skipped over the explanation and only want the most important info from your perspective. The new approach to fix throwing problems will delay the nolo update until 2.0 update. We could have taken few days to reimplement it now but this will delay 2.0 version which is a priority since only few percent of VRidge users have NOLO devices.

VRidge 2.0

When?

As explained before, the whole NOLO integration thing took way longer than we expected. It pushed back 2.0 ETA from Q2 to Q4 – that's a lot. Fortunately, we weren't focused 100% on NOLO. There were multiple periods where we waited for new 3rd party update and we took the time to continue working on 2.0. The current scheduled beta release date is late November, maybe early December. We still have few external events that might divert our attention for few days but we should be able to make it in late November.

Why?

2.0 is mostly about stability and creating a better groundwork for future improvements. RiftCat was initially developed as a VR downloader, "Steam for VR" of sorts. This was before SteamVR/OpenVR was even a thing so we had quite a few people using it and over 100+ games available for DK1/DK2. VRidge was created as an extra tool for people without real HMDs to experience those games. There was no positional tracking and no controllers back then, when we started working on it. It was pretty good plug and play solution that allowed you select Oculus game and just launch it.

Then things started changing, SteamVR became a major player in the VR industry, positional tracking became a standard, runtimes became much more complex, motion controllers were shipped. This caused a lot of refactoring on the VRidge side to adapt to the industry that was developing in a lightning speed. There are still hundreds of games that were great for DK1/DK2 but they are unusable now because they were not adapted to breaking changes of VR runtimes.

This caused a great deal of dirty hacks caused by quick refactoring. Those kinds of quick fixes stack up and slowly become a monster that is hard to maintain.

What?

No more cascading crashes

We want to solve those problems by decoupling a lot of things that are dependent on each other now. We know that with current VRidge version one thing going wrong can cause entire chain of crashes resulting in a need to restart of VRidge, RiftCat, SteamVR, game that you have been playing.

Settings adjustable on the fly

Some settings will be changeable on the fly. Not all of the settings can be changed dynamically but some of them, like bitrate, can be adjusted easily without restating a thing. This will allow greater ability to find your best settings.

New Android app

Completely redone with new unified renderer designed for Daydream, Google Cardboard and Gear VR. One codebase that will automatically switch backends depending on your phone capabilities. We still need to test it on wider set of devices but generally the whole thing seems to run smoother than 1.x versions.

All settings on the PC

You won't need to switch between PC and mobile apps back and forth to adjust settings – all of them will be stored on your PC and sent to the mobile along with the stream.

Desktop mirror and new latency timings

This is a groundwork for time warp and “fake 3D” and latency optimizations. We're creating 2.0 as an expandable platform that we can continue improving with post 2.0 update. We created a desktop mirroring mode to test it with some high-framerate cameras. This mode will be disabled by default as it's currently serving debug and diagnostic purposes but we might leave an option to enable it if you want to test streaming only.

We also might share some of the 240 FPS recordings in the next posts that will come in few weeks because seeing “instant” screen refresh going slowly from the top to the bottom of the screen is very interesting.

Moonlight mode

We mentioned decoupling modules in 2.0 and this is one of the major features that will benefit from it. You will be able to run VRidge as tracking-only service and start NVIDIA GameStream that will stream properly formatted window to your phone running Moonlight.

HDMI support

The window mentioned in paragraph above will also allow you to send properly formatted stream to any display that you have attached to your GPU, including DIY headsets with HDMI passthrough. Tracking will have to rely on VRidge-API or VRidge running in tracking only mode the background.

New RifCat client

We're removing all the things that are very rarely used like game library/downloads. These games mostly serve as demos and a quick way to test. We still leave an option to run non-SteamVR old Oculus SDK games because there are some experiences that weren't ported to new runtimes and some people still ask for help with them through our support channel.

New client will automatically start VRidge in the background and begin pairing/connecting to be ready for the moment SteamVR is started. It will remember your phone, your settings and will take a lot less clicks to get VR running.

The client will also allow for an easier integration of “plugins”. For example, PS Move could be one of those plugins. We already have few of them in late stage of development, so you will hear more about those integrations post 2.0 release. Some of them might turn out as a great addition.

No registration required

Less registration forms and formalities. Just download it and start testing VRidge.

…and more

Those are the features that are scheduled to be released in 2.0 update. Most likely all of them will make it to beta release in few weeks unless we find some game-breaking bugs. Of course, as with every new release, new bugs will arise but we will continue directly updating beta app to make it stable ASAP (by the end of the year) and start working on the 2.1 in December/January.

Post 2.0

There will be two major focus points post 2.0 – time warp / latency optimizations and iOS. Both of them will greatly benefit from more adaptable codebase of 2.0 release so that’s why we wanted to finish it first.

We also want to look into ARCore. It’s available as preview SDK by Google and it seems that it has great inside-out positional tracking capabilities. We really want to know if camera feed decoding & processing will leave enough room for our VR stream decoding & processing. If it’s viable, it might greatly enhance mobile VR immersion. Moving in VR really adds a lot to the experience.

We still have fake 3D on our task board but optimization and iOS will be a higher priority.

---

We are really sorry that it takes us so long to bring you an update. Please, just give us a few weeks more and we will deliver. This will be the biggest update that we ever shipped for VRidge.