Friday, April 21, 2017

Dev update #29 - an update on our progress



This is text update only, we want to explain what we are doing and why we think it's a best way to do currently. 
---

VRidge core improvements 

Moonlight, stability & launch flow improvements, fake VR groundwork

Reasons

VRidge was first designed as Oculus proxy interceptor that acted like Oculus DK2. Its work flow was quite simple because VR was simple back then. There were no controllers, no always-on services, no Oculus Home / SteamVR dashboard like environments. You started a game by running its executable and VRidge process lifetime was limited to game process lifetime.

We added SteamVR support exactly year ago, on 21th April 2016. It was first SteamVR-mobile HMD bridge back then and it helped us grew a lot. Then we continued adding more and more features and changes like UI slider for supersampling (which was later added by SteamVR itself), better support for controllers, FreeTrack, 1440p, Intel and Software encoders, Gear VR, partial Daydream support and lately - VRidge Tracking API.

The shape of VR has changed from headsets being simple 3D glasses to full room scale systems running multiple services in OS background. The problem is that the core of the VRidge remained mostly the same and if you ever worked on a big project, you know that when goals and requirements shift, the project maintainability suffers. The VRidge was designed with pre-release Oculus headsets in mind when there were no controllers or room scale lighthouse systems. And those legacy foundations slow us down a lot.

Core rework

That's why we are currently doing major rewrite of the VRidge core. It's actually going pretty good and we already made a lot of progress. In our development version VRidge was decoupled from SteamVR launch flow. We dropped that annoying chain of dependencies that required restarting all components (mobile, SteamVR, VRidge, Riftcat) when any of the mentioned components ran into some sort of problem. We can start and stop SteamVR/VRidge at any time and in any order - they will connect without problems. 

Another thing that rework enables is modular architecture. We can now substitute capture, encoding and output components during runtime. When we detect SteamVR connection being dropped, the stream automatically switches off to screen duplication stream. While it's not very useful at this moment (because VRidge doesn't have any Remote Desktop functions) it lays a solid foundation for future Fake VR. It can capture any window with very little overhead (<1% CPU usage, sub-frame latency, duplication tech stack very similar to Virtual Desktop, Bigscreen or OBS).

Another great thing that we already have working is a plug and play Moonlight support. It works as easy as possible. We have special VridgeView.exe that can be added to GameStream library in GeForce Experience panel. Then we simply start the VridgeView through Moonlight and it automatically connects to VRidge runtime and switches VRidge to fullscreen SBS output that is handled by Moonlight. And it works quite amazingly since as we said before - we think that NVIDIA GameStream + Moonlight pair is a "relationship goal" for any streaming software.

This also allows HDMI (or any video signal cable) output. You can use the VRidgeView.exe which will find its way to your chosen display and show VR on your device's display with no added latency. This is great thing for DIY fans that enjoy building their own HMDs.

We also want to unify Android rendering. Currently Gear VR and Android use completely different rendering engines. This makes it complicated because Daydream's immersive mode requires another VR renderer. We have a way to combine them all into one which would finally allow proper implementation of Daydream's immersive mode. We didn't want to write separate renderer for Daydream only because writing three things for three platforms that do essentially the same thing is a hell to maintain. With unified renderer, it should be a much less bumpy road.

We are also moving all settings to desktop app since adjusting some settings on the mobile makes absolutely no sense. Trying to adjust FOV with your headset off is an annoying guessing game.

Let's not forget iOS and possibly other mobile clients. This modular rework will remove a lot of spaghetti netcode which will then make it easier to port the mobile app over to iOS.

To sum things up, here's our current progress of the most important components:

☑ SteamVR driver connectivity rework (plug and play startup).
☑ Moonlight streaming support.
☑ HDMI output.
☑ High performance desktop capture engine for the future.
☑ Encoder quality settings adjustable on the fly.
☑ Encoder migration: NVENC.
☐ Encoder migration: MF (including AMD, Intel QS, Software).
☑ Network output
☐ Android network connectivity.
☑ Android decoder (not changed). 
☐ Android unified renderer. 
☐ Android background mode (for Moonlight mode tracking).

Scope of the work above is enough to call the reworked release VRidge 2.0 but those are only numbers without any other meaning than a label (no price changes or anything, everyone will get this update).

Since it's a major rework we cannot provide a testable version because Android connectivity wasn't migrated yet. Nonetheless we hope that this will allow us smoother development after most of legacy code is cleaned. During last year VR matured with a release of SteamVR and Oculus CVs. Controller interfaces are pretty well documented. VR environment APIs are pretty stable. With a more solid set of VR standards we are confident that we won't repeat the story pictured below:
by: Manu Comet

Sound progress

We planned to release it faster but we ran into some problems during testing on other machines. As it turns out - different sound chips use different audio formats so we are still testing and adapting to new motherboards/sound cards that we can find. After it's released to beta channel we will probably find some more bugs with exotic sound devices but that's what the beta channel is for.

SideloadVR & ConstructVR

Apparently SideloadVR is still online so we are going to update our Gear VR app on SideloadVR to match ConstructVR's latest version. It should be updated soon.

VRidge API changes

We are making some changes to VRidge API for next API release.
1. API call to re-center view.
2. Haptic feedback response for controllers.
3. A way to mark HMD trackers state as "out of tracking range".

Those changes were requested by 3rd party device manufacturers. The changes will be reflected in next week's API release on VRidge API GitHub.

Update release schedule

  • VRidge API will be updated by the end of next week.
  • Soon everything in current beta channel will move to stable channel. We want our stable version to be compatible with NOLO VR hardware when Kickstarter backers receive their units. 
  • Sound and protocol changes will be deployed to beta as soon as they are release-ready. Most likely it will be released in early May.
  • Core rework is estimated to be released to beta by the end of Q2, hopefully. It's our biggest change since release, but for a good reason.

16 comments:

  1. Wow! Love your work and your commitment! VRidge is getting better and better.

    ReplyDelete
    Replies
    1. Thanks for the kind words!

      I know that refactors and reworks are not really exciting for everyone to read about but it's something that really needs to be done. The technical debt was getting bigger and bigger over time and it continued to slow development down.

      Moonlight mode and fake VR are relatively easy things to implement but they were postponed over and over because we couldn't fit them correclty in VRidge's process lifecycle which was tightly linked to VR executable. :/

      Delete
  2. Does this mean you will be taking advantage of the daydream tracking API soon? Drift is my single complaint with vridge.
    I do not have problems with drift in daydream api applications at all. Vridge is absolutely amazing, otherwise.

    ReplyDelete
    Replies
    1. Waaait, the tracking should be driftless on Daydream enabled devices and it seems to be working that way on our Daydream ready Pixel XL. Are you using the latest Android beta version? Make sure that you have tracking mode: Google VR selected in Android's app settings.

      Delete
  3. guys you are awesome , keep up the great work

    i want to ask 2 things :

    1:-is there any plan on implanting trackir with vridge?

    2:-is there any plan on adding support for HDMI connectivity with the phone (hardware and software)

    other than that i am so happy with the program.

    thank you so much

    ReplyDelete
    Replies
    1. 1. We need to check out licensing for TrackIR. We currently are open to anything that works with FreeTrack 2.0/OpenTrack but we didn't do much research about TrackIR.

      2. Outputting VR stream to HDMI-connected display will be available after the rework is complete but I'm not sure if I understood your question correctly.

      Delete
  4. @Galal Rashid Sallam:

    1. The software of TrackIR isn't open source, so I guess that a native support won't happen. But you could use freetrack with your TrackIR equipment. Follow the steps on this site for further instructions: http://naturalpointofview.blogspot.de/p/trackir-cameras-with-freetrack.html
    Since VRidge supports Freetrack, this solution should work for you.

    2. As far as I know, there are no mobile phones out there, that support HDMI input. The MHL (USB to HDMI adaptor) thing just works as an output from the phone to the TV.


    @DEVS:

    Wouldn't it be easier to implement the soundstreaming via the soundchip that sits on the GPU? So you would just have to handle with 3 different types of hardware (AMD, NV and Intel) instead of a dozen more. I'm no expert, so I don't know if this is technically doable or just creates a bunch of new problems. But since every GPU above the minimal requirements of VRidge has a soundchip for HDMI output, this is something that should be considered.


    Thanks for working so hard on this great software.

    ReplyDelete
    Replies
    1. We're not interacting directly with hardware when it comes to sound capture but with an abstraction of sound device given by OS. Underlying device can be anything (mobo sound chip, discrete sound card, USB audio stick, etc.). We just need to make sure we are querying all relevant parameters so the stream is formatted properly.

      We found few bugs related to unexpected formats and they were fixed so it's almost ready to be released.

      Delete
  5. Hi,

    There is still this rather big problem for Gear Vr users who use Vridge for Gear Vr the thing is I can't use it, in my netowrk

    It doesn't detect my desktop app running Riftcat is launched on computer that is connected trough Ethernet to Router (Orange Neostrada Livebox ;p) and I'm connecting my phone to the same network using wi-fi but it doesn't connect it keeps displaying "searching for desktop app"

    I tried normal cardboard and in normal cardboard app Vridge I can specify the IP and than it work but it doesn't work if I use Vridge for GearVR I would like to have option to enter the IP of my network manually as it is the only way I can connect

    ReplyDelete
    Replies
    1. Aye, we know about the Gear VR connection problems when automatic discovery fails. This is one of the reasons we really want to combine all 3 (Daydream, GearVR, Cardboard) of our VR handlers and glue them together with one abstraction layer.

      This will allow to keep features of all versions at the same level, including manual connection by IP address.

      If I recall correctly some people were able to resolve this automatic connection problem with router config. They found that enabling SIP would allow automatic discovery to work. I'm not sure if Neostrada's Livebox has some sort of SIP toggle but it's worth a shot to check it.

      Found here: http://blog.riftcat.com/2016/09/dev-update-22-gear-vr-first-release.html?showComment=1473609550105#c1120359846002072820

      (see comment by Mectron Noodle)

      Delete
  6. the most important thing is, smothness when you turn your head and grafic quality... please work on it

    ReplyDelete
    Replies
    1. Moonlight mode is an answer (at least partial) to both of these problems. After the rework is complete we'll most likely start working on "time warp" but this will be uncharted territory riddled with algorithmic dragons. It's hard to estimate how much of improvement it will be.

      Delete
  7. In the next update. Will it be possible to enter in the IP of the computer using Gear VR version of Riftcat?

    It takes forever to scan detect my device sometimes.

    ReplyDelete
  8. I am one of the 'diy fan' and I can't thank you enough for the HDMI option.

    ReplyDelete
  9. Will you ever be able to use the built-in tracking of the Gear-Vr? I'm thinking it will improve the slight judder with head movement? Otherwise will something like Edtrack improve the head motion tracking? I'm using a Galaxy S6 via USB. Graphics card is GTX 1060

    ReplyDelete
  10. Adding support for moonlight for nvidia users seems like a neat idea :), can you please also consider adding amd amf support? (https://github.com/GPUOpen-LibrariesAndSDKs) it has the potential to help encoding performance on AMD GPUs quite a bit compared to media foundation.

    ReplyDelete