Friday, April 21, 2017

Dev update #29 - an update on our progress



This is text update only, we want to explain what we are doing and why we think it's a best way to do currently. 
---

VRidge core improvements 

Moonlight, stability & launch flow improvements, fake VR groundwork

Reasons

VRidge was first designed as Oculus proxy interceptor that acted like Oculus DK2. Its work flow was quite simple because VR was simple back then. There were no controllers, no always-on services, no Oculus Home / SteamVR dashboard like environments. You started a game by running its executable and VRidge process lifetime was limited to game process lifetime.

We added SteamVR support exactly year ago, on 21th April 2016. It was first SteamVR-mobile HMD bridge back then and it helped us grew a lot. Then we continued adding more and more features and changes like UI slider for supersampling (which was later added by SteamVR itself), better support for controllers, FreeTrack, 1440p, Intel and Software encoders, Gear VR, partial Daydream support and lately - VRidge Tracking API.

The shape of VR has changed from headsets being simple 3D glasses to full room scale systems running multiple services in OS background. The problem is that the core of the VRidge remained mostly the same and if you ever worked on a big project, you know that when goals and requirements shift, the project maintainability suffers. The VRidge was designed with pre-release Oculus headsets in mind when there were no controllers or room scale lighthouse systems. And those legacy foundations slow us down a lot.

Core rework

That's why we are currently doing major rewrite of the VRidge core. It's actually going pretty good and we already made a lot of progress. In our development version VRidge was decoupled from SteamVR launch flow. We dropped that annoying chain of dependencies that required restarting all components (mobile, SteamVR, VRidge, Riftcat) when any of the mentioned components ran into some sort of problem. We can start and stop SteamVR/VRidge at any time and in any order - they will connect without problems. 

Another thing that rework enables is modular architecture. We can now substitute capture, encoding and output components during runtime. When we detect SteamVR connection being dropped, the stream automatically switches off to screen duplication stream. While it's not very useful at this moment (because VRidge doesn't have any Remote Desktop functions) it lays a solid foundation for future Fake VR. It can capture any window with very little overhead (<1% CPU usage, sub-frame latency, duplication tech stack very similar to Virtual Desktop, Bigscreen or OBS).

Another great thing that we already have working is a plug and play Moonlight support. It works as easy as possible. We have special VridgeView.exe that can be added to GameStream library in GeForce Experience panel. Then we simply start the VridgeView through Moonlight and it automatically connects to VRidge runtime and switches VRidge to fullscreen SBS output that is handled by Moonlight. And it works quite amazingly since as we said before - we think that NVIDIA GameStream + Moonlight pair is a "relationship goal" for any streaming software.

This also allows HDMI (or any video signal cable) output. You can use the VRidgeView.exe which will find its way to your chosen display and show VR on your device's display with no added latency. This is great thing for DIY fans that enjoy building their own HMDs.

We also want to unify Android rendering. Currently Gear VR and Android use completely different rendering engines. This makes it complicated because Daydream's immersive mode requires another VR renderer. We have a way to combine them all into one which would finally allow proper implementation of Daydream's immersive mode. We didn't want to write separate renderer for Daydream only because writing three things for three platforms that do essentially the same thing is a hell to maintain. With unified renderer, it should be a much less bumpy road.

We are also moving all settings to desktop app since adjusting some settings on the mobile makes absolutely no sense. Trying to adjust FOV with your headset off is an annoying guessing game.

Let's not forget iOS and possibly other mobile clients. This modular rework will remove a lot of spaghetti netcode which will then make it easier to port the mobile app over to iOS.

To sum things up, here's our current progress of the most important components:

☑ SteamVR driver connectivity rework (plug and play startup).
☑ Moonlight streaming support.
☑ HDMI output.
☑ High performance desktop capture engine for the future.
☑ Encoder quality settings adjustable on the fly.
☑ Encoder migration: NVENC.
☐ Encoder migration: MF (including AMD, Intel QS, Software).
☑ Network output
☐ Android network connectivity.
☑ Android decoder (not changed). 
☐ Android unified renderer. 
☐ Android background mode (for Moonlight mode tracking).

Scope of the work above is enough to call the reworked release VRidge 2.0 but those are only numbers without any other meaning than a label (no price changes or anything, everyone will get this update).

Since it's a major rework we cannot provide a testable version because Android connectivity wasn't migrated yet. Nonetheless we hope that this will allow us smoother development after most of legacy code is cleaned. During last year VR matured with a release of SteamVR and Oculus CVs. Controller interfaces are pretty well documented. VR environment APIs are pretty stable. With a more solid set of VR standards we are confident that we won't repeat the story pictured below:
by: Manu Comet

Sound progress

We planned to release it faster but we ran into some problems during testing on other machines. As it turns out - different sound chips use different audio formats so we are still testing and adapting to new motherboards/sound cards that we can find. After it's released to beta channel we will probably find some more bugs with exotic sound devices but that's what the beta channel is for.

SideloadVR & ConstructVR

Apparently SideloadVR is still online so we are going to update our Gear VR app on SideloadVR to match ConstructVR's latest version. It should be updated soon.

VRidge API changes

We are making some changes to VRidge API for next API release.
1. API call to re-center view.
2. Haptic feedback response for controllers.
3. A way to mark HMD trackers state as "out of tracking range".

Those changes were requested by 3rd party device manufacturers. The changes will be reflected in next week's API release on VRidge API GitHub.

Update release schedule

  • VRidge API will be updated by the end of next week.
  • Soon everything in current beta channel will move to stable channel. We want our stable version to be compatible with NOLO VR hardware when Kickstarter backers receive their units. 
  • Sound and protocol changes will be deployed to beta as soon as they are release-ready. Most likely it will be released in early May.
  • Core rework is estimated to be released to beta by the end of Q2, hopefully. It's our biggest change since release, but for a good reason.

48 comments:

  1. Wow! Love your work and your commitment! VRidge is getting better and better.

    ReplyDelete
    Replies
    1. Thanks for the kind words!

      I know that refactors and reworks are not really exciting for everyone to read about but it's something that really needs to be done. The technical debt was getting bigger and bigger over time and it continued to slow development down.

      Moonlight mode and fake VR are relatively easy things to implement but they were postponed over and over because we couldn't fit them correclty in VRidge's process lifecycle which was tightly linked to VR executable. :/

      Delete
  2. Does this mean you will be taking advantage of the daydream tracking API soon? Drift is my single complaint with vridge.
    I do not have problems with drift in daydream api applications at all. Vridge is absolutely amazing, otherwise.

    ReplyDelete
    Replies
    1. Waaait, the tracking should be driftless on Daydream enabled devices and it seems to be working that way on our Daydream ready Pixel XL. Are you using the latest Android beta version? Make sure that you have tracking mode: Google VR selected in Android's app settings.

      Delete
    2. I have since signed up for the beta program, and will be testing again as soon as I have some free time. I too, have a Pixel XL. I'll let you guys know once I've tested. :)

      Delete
    3. Yeah, after updating to the latest on the pc and the android phone in the beta channel, I can connect, the screen flickers, then it throws a modal
      "Incompatible app
      This cardboard application is not compatible with Daydream headsets."

      I have the option to pair viewer, but that just launches the cardboard QR scanning app, and being a daydream, there is no QR to scan?

      Delete
    4. You can scan any daydream profiled QR code
      https://www.reddit.com/r/daydream/comments/5cfkvm/daydream_view_qrcode/

      This should enable compatibility mode and work with many daydream features enabled (reprojection, low persistence). The immersive VR home and in-VR app switching won't work but the VR itself should be mostly the same.

      Delete
  3. guys you are awesome , keep up the great work

    i want to ask 2 things :

    1:-is there any plan on implanting trackir with vridge?

    2:-is there any plan on adding support for HDMI connectivity with the phone (hardware and software)

    other than that i am so happy with the program.

    thank you so much

    ReplyDelete
    Replies
    1. 1. We need to check out licensing for TrackIR. We currently are open to anything that works with FreeTrack 2.0/OpenTrack but we didn't do much research about TrackIR.

      2. Outputting VR stream to HDMI-connected display will be available after the rework is complete but I'm not sure if I understood your question correctly.

      Delete
    2. This comment has been removed by the author.

      Delete
    3. But FreeTrack Axis problem still unfixed (it reported 5 month ago in Request #11641) =(
      Now with video example (use subtitles)
      https://www.youtube.com/watch?v=j2nDgRiHCPI

      Delete
  4. @Galal Rashid Sallam:

    1. The software of TrackIR isn't open source, so I guess that a native support won't happen. But you could use freetrack with your TrackIR equipment. Follow the steps on this site for further instructions: http://naturalpointofview.blogspot.de/p/trackir-cameras-with-freetrack.html
    Since VRidge supports Freetrack, this solution should work for you.

    2. As far as I know, there are no mobile phones out there, that support HDMI input. The MHL (USB to HDMI adaptor) thing just works as an output from the phone to the TV.


    @DEVS:

    Wouldn't it be easier to implement the soundstreaming via the soundchip that sits on the GPU? So you would just have to handle with 3 different types of hardware (AMD, NV and Intel) instead of a dozen more. I'm no expert, so I don't know if this is technically doable or just creates a bunch of new problems. But since every GPU above the minimal requirements of VRidge has a soundchip for HDMI output, this is something that should be considered.


    Thanks for working so hard on this great software.

    ReplyDelete
    Replies
    1. We're not interacting directly with hardware when it comes to sound capture but with an abstraction of sound device given by OS. Underlying device can be anything (mobo sound chip, discrete sound card, USB audio stick, etc.). We just need to make sure we are querying all relevant parameters so the stream is formatted properly.

      We found few bugs related to unexpected formats and they were fixed so it's almost ready to be released.

      Delete
  5. Hi,

    There is still this rather big problem for Gear Vr users who use Vridge for Gear Vr the thing is I can't use it, in my netowrk

    It doesn't detect my desktop app running Riftcat is launched on computer that is connected trough Ethernet to Router (Orange Neostrada Livebox ;p) and I'm connecting my phone to the same network using wi-fi but it doesn't connect it keeps displaying "searching for desktop app"

    I tried normal cardboard and in normal cardboard app Vridge I can specify the IP and than it work but it doesn't work if I use Vridge for GearVR I would like to have option to enter the IP of my network manually as it is the only way I can connect

    ReplyDelete
    Replies
    1. Aye, we know about the Gear VR connection problems when automatic discovery fails. This is one of the reasons we really want to combine all 3 (Daydream, GearVR, Cardboard) of our VR handlers and glue them together with one abstraction layer.

      This will allow to keep features of all versions at the same level, including manual connection by IP address.

      If I recall correctly some people were able to resolve this automatic connection problem with router config. They found that enabling SIP would allow automatic discovery to work. I'm not sure if Neostrada's Livebox has some sort of SIP toggle but it's worth a shot to check it.

      Found here: http://blog.riftcat.com/2016/09/dev-update-22-gear-vr-first-release.html?showComment=1473609550105#c1120359846002072820

      (see comment by Mectron Noodle)

      Delete
  6. the most important thing is, smothness when you turn your head and grafic quality... please work on it

    ReplyDelete
    Replies
    1. Moonlight mode is an answer (at least partial) to both of these problems. After the rework is complete we'll most likely start working on "time warp" but this will be uncharted territory riddled with algorithmic dragons. It's hard to estimate how much of improvement it will be.

      Delete
  7. In the next update. Will it be possible to enter in the IP of the computer using Gear VR version of Riftcat?

    It takes forever to scan detect my device sometimes.

    ReplyDelete
    Replies
    1. I had the same problem especially when I wanted to use moonlight over USB. The thing that I do was to set an IP address instead of automatic IP.
      Now it detects faster.

      Delete
  8. I am one of the 'diy fan' and I can't thank you enough for the HDMI option.

    ReplyDelete
  9. Will you ever be able to use the built-in tracking of the Gear-Vr? I'm thinking it will improve the slight judder with head movement? Otherwise will something like Edtrack improve the head motion tracking? I'm using a Galaxy S6 via USB. Graphics card is GTX 1060

    ReplyDelete
    Replies
    1. Wireless utilizes the gearvr headtracking.

      Delete
    2. Er there's already a GVR version........

      Delete
    3. Yeah it's here:
      https://www.constructvr.io/gearvr/r14JZZIdx

      As for the external trackers we are currently working on VRidge API that can be used to plug existing tracking solutions and mix the data to improve the experience.

      https://github.com/RiftCat/vridge-api

      Delete
  10. Adding support for moonlight for nvidia users seems like a neat idea :), can you please also consider adding amd amf support? (https://github.com/GPUOpen-LibrariesAndSDKs) it has the potential to help encoding performance on AMD GPUs quite a bit compared to media foundation.

    ReplyDelete
    Replies
    1. While going through Media Foundation on AMD GPUs isn't as dreadful as on nVidia cards, a more direct access to VCE via their Advanced Media Framework should improve performance and image quality by a lot. It sure helped a lot in OBS studio.

      Delete
    2. This is planned to be implemented eventually but it's always pushed back by other stuff because NVIDIA users are overwhelming majority (IIRC 80+%) so we wanted to give NVIDIA something first. Hopefully we can implement VCE sooner than later.

      Delete
  11. Hey! Currently using vridge with my phone and PSMove, but i see that you plan to add HDMI support, would this work with some "ALL IN ONE" VR Headset that have HDMI Input (OS is Android) ? I have one and the quality is way better than my phone.

    ReplyDelete
    Replies
    1. If it can act as a HDMI connected display and can still run some app in the background to keep sending tracking data - most likely it will work.

      Delete
  12. GearVR s8 user here. There seems to be a bug related to the longer screen on the s8 and s8+, caused by the video being stretched over the extra pixels rather than centered dot-for-dot. This causes extreme blurriness and aliasing. Should be easily fixable by either rendering for the phone's extra pixels in the desktop app/stream, or updating the GearVR app for s8/+ users.

    Note: this issue is across ALL games and does not affect the non-GearVR version of VRidge on my s8.

    Do you have any eta on a fix for this? Otherwise the app is amazing, and I'm excited for the future updates. Thanks!

    ReplyDelete
    Replies
    1. We have the "one blurry eye in GearVR" bug pinned on our task board but it's waiting for the sound to be finally pushed out so we can assign someone to look into it. Hopefully in May.

      Delete
  13. This comment has been removed by the author.

    ReplyDelete
  14. Wow these are GREAT news! Anyway should be possible to release a small update just with the moonlight support? Using riftcat codec (even NVENC) my phone become very hot, instead when I use moonlight for streaming, the temperature are quite low so I can play for longer time.
    Thanks

    ReplyDelete
    Replies
    1. I wish, Moonlight mode is the main reason behind this rework. There was no simple way to keep current execution flow with Moonlight mode so we decided to rework the core to be more modular.

      Delete
    2. thank you for your reply. The thing that I don't understand is that you use the NVEC stream by NVIDIA but isn't the same that use moonlight??

      Delete
    3. It's related.

      - NVENC creates the video stream from series of pictures (encoding process).
      - GameStream (server) + Moonlight (client) both create the video stream (encoding) and send it over network (transport).

      GameStream handles the encoding and sending. Moonlight configures options and handles receiving, displaying, forwards input + some extra work.

      We currently use our own transport layer because GameStream + Moonlight is NVIDIA only and we wanted to have something usable by both NVIDIA and AMD.

      Delete
  15. This comment has been removed by the author.

    ReplyDelete
  16. Is there a way to use the GearVR controller for SteamVR? Is that even something that can be added via the current API or does it have to be baked directly into VRidge?

    ReplyDelete
    Replies
    1. It's not built-in, we want to add it eventually. Our API is available to everyone but I'm not sure if third party developer can access Gear VR controller data while VRidge is in the foreground.

      Delete
    2. Yeah, I had a look at their API and it doesn't seem likely that I could access the controller in the background.

      Oh well, I should have my Nolo controllers soon enough... I'm just itching for some form of controller support in the meantime. XBox controller + FreePie isn't cutting it.

      Delete
  17. It come up with an error when clicking the PlaySteamVR button on RiftCat. Steam pops up and says "Steam VR failed to initialized for unknown reasons. (Error:Shared IPC Compositor Invaild Connect Response (307))" On the SteamVR window that also pops up, it says "A key component of SteamVR isn't working properly. (306)"

    Any solutions?

    ReplyDelete
    Replies
    1. There is a problem with newer NVIDIA GPU driver (382.05+), Kepler-based GPU and old Maxwells. Unfortunately this issue also affects some Vive users and both Steam and Nvidia know about this problem but we haven't gotten a fix yet, not sure how long it's gonna take.

      Currently the only solution is to downgrade GPU driver to version 381.89. Make sure you uninstall your current driver first, as it's described in this link:

      http://www.nvidia.pl/object/IO_13955.html

      and then install the 381.89 version. You can find older driver versions here

      http://www.nvidia.com/download/find.aspx.

      Delete
  18. Is there a problem with Nvidia drivers 375.63?

    ReplyDelete
  19. Great job on the update. But currently, I am unable to find "VRidgeView.exe". I've tried going on both the stable and beta branch, but the exe simply doesn't exist. Did I misunderstand that it would be included in the already public version or am I missing something?

    ReplyDelete
  20. If I understand well, with "VRidgeView.exe", the HDMI Output could be send to a PlaystationVR headset (or Pimax or OSVR...) and then we will just need a Tracking Method like NoloVR (or PSMoveService) to have a complete VR bundle ?

    ReplyDelete
    Replies
    1. The PSVR tracking solution is something that I'm playing with, see example folders in:
      https://github.com/mungewell/pyPSVR

      Mostly as fun, learning project. The idea is that user would stick a 'April Tag' to the front of the PSVR and use a camera (such as PS3Eye) to track it, work out the position headset and compensate for drift.

      https://www.youtube.com/edit?o=U&video_id=fcf8Qy-Vcgo

      Would be seated experience (rather than roomscale), but since I like driving games that's good for me. ;-)


      @VRidge Devs
      Very excited to test out HDMI output when it's available.

      Will the lens parameters allow for chromatic correction? ie:
      https://www.flickr.com/photos/24244464@N03/33627256153/in/dateposted-public/

      Delete
    2. We checked and google VR cardboard doesn't have chromatic aberration correction feature, which is a bit weird since GearVR does have one. So no, these parameters won't help. We'll put it on our todo list.

      Delete