Thursday, January 12, 2017

Dev update #26

This update took us much longer than expected but we finally believe that we made an improvement that is noticeable and worth giving a shot if you experienced terrible stuttering on your Galaxy S6-7 and some other phones.

We also bring some long term bug fixes like 30 and 60 FPS staying at the actually 30/60FPS, some Redmi Note and Zenfone fixes and several SteamVR improvements. There's an experimental version of HEVC too.

This update is released as version 1.2 (both mobile and desktop) through beta channels. Go to Riftcat desktop settings > Updates and get beta version there if you want to try it out.

Gear VR edition will be updated early next week if no major bugs are reported. We don't have beta channel on Sideload VR so we want to test it for few days on regular Android devices because it's much easier to rollback to stable version with Google Play.


---

Jitter / stutter improvements

TL;DR: It should be better now, more improvements will come in the future.

This took us a while. The problem first appeared when we released Gear VR edition in September. Some users were reporting various levels of stuttering and frame skips. We wanted to fix it quickly but after we investigated the problem it turned out that the underlying cause was really complex.

We first suspected frames being out of order but after high framerate recording it turned out that it wasn't true. We then investigated frames lost on the network but after adding some extra metrics we found that it wasn't packet loss. All frames were arriving on the phones yet the overall feel was very jittery.

We then started adding metrics and timings for every part of the virtual HMD pipeline. There are multiple places where VRidge needs to wait for something (game, steamvr, encoder, network transfer, decoder, Android vsync, etc.). If any of these components lags for even few miliseconds - it throws the whole thing out of sync.

We found that most of the microstutters came from network and Android decoder. Network micro lags are reasonable because that's how networks work. Android decoder optimization is a tougher nut to crack. It's performance is dependant on installed hardware and there are hundreds of chipset variants. The most annoying thing about is its inconsistence when it comes to decode times. For everything to be fluid, everything needs to take roughly the same amount of time every frame. Decoding time was varying greatly from few miliseconds to 12ms (with random spikes as high as 40). This kind of randomness is very bad for fluid playback.

Usually it's countered by buffering but it's not an option when we are trying to achieve VR-bearable levels of latency. We can't buffer even a second ahead to counter random spikes of interference.

Previously our FPS was stuck at almost 58 FPS. Some people suggested to turn it up to 60 and see if it fixes the stuttering. It didn't because the problem wasn't amount of frames per second. Generally to achieve fluid motion you need both high FPS and proper synchronization and intervals between them. An average of 60 FPS isn't good if sometimes the frames comes in burst of 2-3 at the same time with 30-40ms breaks after.

We decided to rework most of the timing and synchronization that was controlling capture, encoding and decoding. New frame rate is using better methods that combine multiple methods of timing and synchronization into one loop that is now much more accurate. Previously we had an average of 58.582 FPS in our capture and encoding module. Now it's running at 60.073 FPS with proper and stable intervals. This alone should lower deviation from 60 FPS from 2.5% to 0.1%.

Then we decided to rework Android decoding & rendering. Previously if no new decoded frame was ready at the beginning of the render cycle it would skip a frame. We now wait as much as possible for decoder to finish its work before giving up on the frame. You can control it by going to settings and changing frame timing:

  • Prefer low latency (new default) will try to keep the latency minimum but still wait as much as possible for decoder to finish its work.
  • Balanced will wait and buffer up to 1 frame if inconsistent decoding is detected.
  • Prefer smoothness will buffer up to 2 frames.

There are 2 legacy timing modes that are generally inferior to Prefer low latency but we are leaving them because there are 500+ different phone models running VRidge (according to Play Store data) and new modes might not work for someone.

We might make this even more configurable but please give us feedback if you feel the difference between those new timing modes and old ones.

With new timing we also improved 30 FPS mode for low end devices. Previously it would render the game at 60 (58.5 actually) FPS and skip every other frame during encoding. Now 30 FPS mode will slow the game down to 30 FPS. This saves up some CPU and GPU load and allows game to run in a smoother fashion instead of rendering the frames that were being dropped anyway.

This problem was happening across all devices but it was mostly noticeable on Gear VR edition and Google Pixel which use low persistence displays. It didn't look too bad on regular screens because it was hidden by micro blurs and smudges which are common among the regual 60 Hz displays.

This is not the end of our work to counter the stuttering. There are still frame skips and some of them are not avoidable. We cannot control a game that might drop a frame, we can't force network to give 100% of its resource to VRidge and we cannot make background Android apps drop all of its work and give 100% of device power to VRidge. Vive counters this with async reprojection. Oculus has great async timewarp and spacewarp. These technologies increase smoothness greatly and this is something we eventually want to do too. It's not an easy task and it's especially harder because we have to account for variable network latency but this is our goal. It won't come this month or the next one but our goal is perfect smoothness and we want to comes as close to it as possible.

Previous version
New version:

It might be hard to notice on the recording (you can click samples above and slow it down with gfycat speed controls to emphasize frame skips) but you should be able to feel the difference in VR.

VRidge for Gear VR needs an update for those improvements to take effect (sometime early next week).

Google VR SDK update to 1.1

We updated VR backend on Android to the most recent version. It enables some new features but it has some changes you might not like. Google decided to force its UI to be always on so we cannot dim the buttons and middle-screen divider anymore. On the bright side it's a bit dimmer now and it shouldn't be visible through lenses anyway.

New 1.1 SDK brings some new features too. Daydream-ready devices should take adventage of new async reprojection. It's not accounting for network latency but it should lower the perceivable latency by a small amount. Every little bit helps so we are leaving this enabled by default but you might turn it off in app settings by toggling display stabilization on/off. For non-daydream devices this option enables old "shaky" stabilization which works in a different way but it also may lower the perceivable latency too.

Daydream phones will also enter into low persistence VR mode with VRidge increasing smoothness greatly.

Daydream view and controller support is still not available because it requires much more work. We received our Daydream View only few days ago so we didn't have much time to integrate new features. Stay tuned for more Daydream news in the future.

SteamVR improvements

Sometimes we get e-mails about certain SteamVR games not working with VRidge. We always investigate those reports and try to bring compatibility to most titles available on the market. Some people reported Serious Sam VR having terrible framerate which was result of a interaction between the game itself and SteamVR which was happening on 3rd party HMD drivers. We reported those issues to SteamVR and Serious Sam VR devs and the problem was resolved shortly in SteamVR beta and now is also fixed in SteamVR live channel. This also solves Unity 5.5 FPS issues with VRidge. Again, we cannot be grateful enough to SteamVR devs taking all reports and suggestions from both game and HMD developers seriously and solving them very quickly.

Sometimes it's up to us to fix our bugs and add missing features to bring better compatibility. This update brings virtual presence sensor which should prevent some games from pausing thinking that HMD was taken off. We also add opt-in virtual positional sensor. Some games were simply refusing to work without some sort of positional tracking so you can now enter Riftcat desktop settings and turn "SteamVR Fake Sensor" on.

We also updated our HMD driver to the neweset OpenVR standards and fixed one minor memory leak which could end up increasing RAM usage over few hours to gigabytes.

If one of the games below wasn't working for you with VRidge, you can try again today with new beta version: Serious Sam VR, Edge of Nowhere, Herobound and Pinball FX 2 VR.



High Efficiency Video Coding

We're also adding HEVC as an opt-in feature. It's still highly unstable on certain devices so it's not enabled by default. It's currently avaialble only for Media Foundation encoder and GeForce 960 or newer GPUs. It also requires relatively new phone. It works alright on our Galaxy S6 (Exynos chipset).

It doesn't work correctly on our Pixel XL. When the phone receives HEVC frames it simply crashes the phone completely causing a full device reboot. It seems that there's a bug in the hardware codec because no software should be able to cause a hard crash like that. We wlll investigate this further and report it to as a bug if it's not fixable on our side and no workarounds can be developed.

VRidge for Gear VR needs an update for HEVC to work so wait until next week if you want to try it.



Redmi Note 2 and Asus Zenfone 2 fixes

We found a bug that was crashing decoder on Redmi Note 2. It should no longer crash with stream fixes enabled. We also applied workaround for Zenfone 2 suggested by Moonlight dev (thanks!). Intel chipsets should now work correctly on both Android 5.0 and 6.0 without 3s lag.

Minor changes


  • Disabled stream adjustments for Google Nexus 5 and Samsung Galaxy S5. It was sometimes crashing the app.
  • Display stabilization is now toggleable in mobile options. It's a matter of preference so you can now turn it on or off. It's on by default (it was always on before).
  • Changed IPD & Scale adjustment settings icon. New Google VR UI icons cannot be disabled so we had to use something else.
  • Increased maximum IPD by 50%.

What's next?

We are still working on sound streaming and we want to get back to "Fake VR" so you can stream regular games into your Android phone. We suspended work on both to bring improvements described above but we will now be able to come back to new features.

We also have a version that works on some Android 4.x  devices. We asked you to e-mail us if you want to test this version and we have a group of people willing to test it before making it avaialble for everyone. If you sent us an e-mail about it - sorry for not getting back to you earlier. Android 4.x was delayed by this update unfortunately. We will reply to you with a testable version link as soon as we are done making this update "stable enough".

We want to make above changes stable first so please give us your feedback and send bug reports. We have several of the most popular phones among our users and we test all changes on them but it's impossible to predict behaviors on all phones on the market. You can send e-mails to support@riftcat.com or send us a message or comment through any of the social media channels (Facebook/Twitter/reddit). We always read everything and while it may take some time to respond to everyone - we are not ignoring any of the comments.

---

Post edit history:
13 Jan 00:00 UTC - Added clarifications about VRidge for Gear VR requiring an update for HEVC and jitter improvements to work.

60 comments:

  1. Awesome updates!Greatly appreciated!

    ReplyDelete
  2. Many Thanks for the update! Keep up the good work :)

    ReplyDelete
  3. A question. How do I update the Gear VR version of the app?

    ReplyDelete
    Replies
    1. As of this moment, it looks like Gear VR version. hasn't been updated yet.
      "Gear VR edition will be updated early next week..." 4th paragraph. YW.

      Delete
  4. Looking amazing! Can't wait to see some sort of timewarp! I really appreciate your work. One thing I'd like is some better drift correction since my Moto G4 has Terribke drift. I now this might be difficult but maybe you could do a drift calibration so the game knows the drift bias and van correct it. IDK exactly, anyway I love your work. Keep it up!

    ReplyDelete
  5. I had a play with this update on my wireless N network to my phone (but Gig ethernet to my PC) with GTX 1080 and Galaxy S7 on the latest sideload Gear VR app. Riftcat has gotten better! Less jitter and image smudging, it just feels more polished compared to the previous version even though there is a way to go for this still, but us enthusiasts like to live on the edge :>

    I have really been looking forward to experimenting with HEVC for the significant potential it can bring but alas despite restarts, trying different steam games and adjusting the mbits I get a black screen, do we need to wait for the updated app on sideload in order for HEVC to work? Hardware wise I should be meeting the requirements, I hope that the black screen issue is an easy one to resolve!

    Is anyone else having this HEVC issue or is this maybe limited to my setup?

    Thanks for building something really great!

    ReplyDelete
    Replies
    1. > do we need to wait for the updated app on sideload in order for HEVC to work?

      Yes, HEVC needs Android side update too for it too work. The Gear VR update is also almost ready to go.

      Most of the codebase is shared between Gear VR and Cardboard/Daydream apps so if no major bugs surface over the weekend you can expect SideloadVR version to be updated early next week.

      Delete
    2. Thanks Kogi, holding thumbs on the QA, i'm sure it'll be a smooth deployment, a shared code base does make life easier!

      Delete
    3. Using it on my GearVR and s7e on ac wifi it works amazingly, i can actually play stuff like project cars really well now

      (almost no jitter or lag on wifi, or connection issues)
      Running 2560x1440p, 60fps, media foundation, HEVC, 200%scaling. Don't remember the bitrate settings

      Delete
  6. Yay! Gonna try edge of nowhere again

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. Thanks guys a lot for your hard work you put into this!
    Can you please advise what causes the less smooth picture when rotating head around on Samsung Galaxy note 4 (Snapdraggon version)? It has no problem on some native android vr games, but it does on some, as well as on Vridge. The vr games on pc screen look smooth but at the same time it's not that smooth on the phone if I try to look around. It feels like there are many frame skips and phone tries to fill the gap between those frames with some content, which does not really help. Idk if I explained well.
    Thanks

    ReplyDelete
    Replies
    1. 1. How often do those frame skips happen? It it multiple per second or once every few seconds?
      2. Is it Gear VR or Cardboard version?
      3. What connection method do you use (USB/wifi (what kind of router)).
      4. What quality settings do you use (configured in Riftcat desktop app)?

      Delete
  9. Another quick question here: is there any affordable controller alternative (on the market, or even an emulator software) that can work with many controller required VR games? (I've seen some work done with PS3 move and leap motion, but any easier setups or products?)
    Any suggestions will be appreciated.

    ReplyDelete
    Replies
    1. Wait for 'Nolo' its like the vive controllers and a tacker on a budget. Should be here mid this year. Worth the wait. All other tracking options suck, trust me.

      Delete
    2. I'm currently using PS move controllers, and it's working great. Once the setup is done, it works pretty well and can give you a full 360 degree head tracking and full vive controller emulation.

      Delete
    3. There is a kinect positional tracking project. It's here: https://github.com/sheepsskullcity/opentrackkinect/issues

      I think a blutooth controller tracking with Leap Motion would be cool, but I have not done it.

      Delete
  10. Hellooo
    I have a samsung S7, the new HEVC codec doesnt work for me only black screen, i only can play with nvenc, is this normall?¿?¿

    ReplyDelete
  11. same problem, only black screen (galaxy s7 using app mode cardboard)

    ReplyDelete
    Replies
    1. Can you send both Android and desktop logs so we can diagnose the problem?

      See two articles below to find out how to send those logs:

      https://support.riftcat.com/hc/en-us/articles/208781529

      https://support.riftcat.com/hc/en-us/articles/211601805-How-to-gather-diagnostic-data-on-Android-

      Delete
    2. I was getting a black screen(with HEVC checked in riftcat options) until i joined the beta and updated the android app

      Delete
  12. First I want to thank you for the work!
    I want to share my impressions and leave my feedback:
    I have a galaxy s7 with gear vr. I tested the Assetto Corsa, Elite Dangerous, Euro Truck, NoLimits 2 and The Vanishing of Ethan Carter.

    1 - Jittler: Really improved considerably, not reach 100%, but went from a level where I could not play to a level where I can play. I would say it went from a level of 40% to 75% of quality.

    2 - FPS: Much improved, the game flows with much more naturalness and fluidity, I do not have what to point in this question.

    3 - Sensors: The sensors of the cell phone are bad for everything I've tested so far, so I think the problem is not the rifcat, but the cell phone, but it's worth noting that the best set up in my case was "Prefer low latency" .

    4 - It seems like my cell phone is getting warmer.

    In summary all games have had a considerable improvement in gameplay.

    I'm looking forward to the version of the gear vr sideload, because I think part of the jittler that I'm still noticing is the sensors fault.

    I already tested the PS VR once, and the galaxy s7 screen is much better. I really believe that if this program has another update with jittler reduction, the Gear Vr + S7 + Rifcat, will be better than PS VR. Due to the quality of the images.

    I'm sending my log.

    Congratulations again for the work. Joao Paulo.

    ReplyDelete
  13. I only update the beta via pc, in the s7 I did not have update, I continue with the black screen, the delay is reduced via HEVC to part of the upload of frames ?, the delay is one of the most uncomfortable things compared to the HMD like vive or oculus.

    ReplyDelete
  14. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. This comment has been removed by the author.

      Delete
  15. How do you get the beta app? I joined the beta on my PC (ticked the beta program box) how do i get the beta android app?

    ReplyDelete
    Replies
    1. Scroll down in the play store app and there is a green button that says join betA or something like that , wait a few mins and update will be available. Hope that helps matey

      Delete
  16. Hey everyone.

    Some great improvements here. Keep up the great work Riftcat team!

    I've tried out the current beta with the new gearVR version and here are my findings. Settings - S6 with GearVR - 120%, 20mps, 2560x1440, 60fps, hevc.

    On Balanced - Frame jitter seems better. More consistent at least. Turning my head slowly seems to give a consistent turning effect.
    Latency seems to be about 100ms but thats a complete guess based on how it looks.
    There's still quite large jumps in rotation when turning at a reasonable pace but thats got to be the 60fps limit combined with no re-projection I'd have thought. Re-projection would also greatly hide the latency issue as well? assuming I understand it correctly?
    I did notice that the beta notes of the vridge android app notes mention that re-projection is in for some stuff but the blog entry says its not due for a few months. Have I misread something?

    On Smooth mode the turning jitter is all but gone which is great but the latency feels mush worse. Maybe 200ms? Sadly even 100ms made me feel a bit odd so 200 exacerbated that.

    I didnt try the two legacy modes

    Cheers

    Paul

    P.s If you need logs I can probably generate some tomorrow (That possible with the gearvr version I assume?). Let me know

    P.s.s Why the 60fps limit? Can the s6 not do higher? Or is it a safe limit to ensure compatibility with most phones?

    ReplyDelete
  17. my god it workss!!!!!! i download vridge gear vr from sideload, 0 delay, 0 artifacts, i cant belive it!!!!

    ReplyDelete
  18. I'm on Windows 7, and use an AMD card. I've been using NVENC, but haven't really done anything lately due to the S7 issues.

    What exactly am I supposed to be using?

    ReplyDelete
    Replies
    1. NVENC works with NVIDIA cards only. We don't support Windows 7 combined with AMD. It might work but it will probably be very laggy due to lack of low latency mode in MF encoding engine in Win 7.

      Delete
  19. Let's see, if you notice more fluency of course and I do not notice delay the response time to decreased a lot, the problem is the bitrate, I'm using the VRC of gear VR, to 1440p many artifacts and many jumps in the image, to 1080p with a bitrate less than 50 mb the thing goes better but the graphic quality is a bit unfortunate. I suppose it will improve the quality of data transmission, HEVC will be able to increase the performance more. I use gear vr with galaxy s7 edge

    ReplyDelete
  20. vridge cradboard dooes not work yet, fix it please.

    ReplyDelete
  21. A lot of people has a little question. When we can use VRidge in our Phones(iOS) :(

    ReplyDelete
  22. never jajajaja!!! android is the boss!! jajajaj

    ReplyDelete
  23. Great update !
    HEVC grealty increased image quality in 1920x1080 but it's buggy in 2560x1440 : i get the image during a few seconds, but it seams dithered, tinted blue with artifacts.

    TL;DR :
    NVENC 2560x1440 : OK
    HEVC 1920x1080 : OK
    HEVC 2560x1440: Buggy

    ReplyDelete
    Replies
    1. How much do you put the bitrate and the super sampling? How do you connect the mobile phone? Cable or wifi?

      Delete
    2. I have the same result using Wifi 5Ghz or USB.
      Super Sampling off (100%)
      For the bitrate : NVENC works fine @50Mbps
      HEVC 1080 work the same
      But HEVC 1440 is buggy even @5MBps
      So i don't think this is a network problem.

      Delete
  24. i cant seem to find where to join the beta on the android store?

    ReplyDelete
  25. I only work with Vridge gearVR, which of the two versions gives more performance? Can I upgrade to the future with HEVC? Is it possible to upgrade the stream, I also connect via WiFi to 866mb, is it better or USB cable?

    ReplyDelete
  26. Sadly I get some strange screen refresh rate on Nexus 6P with VRidge beta. I can even directly see screen mode changed when connection is established: it dims a little and goes flashing a little bit :(

    ReplyDelete
  27. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. Many Thanks for Your Work. Since one week i own a S7Edge and a Gear VR. Results are much better than with other VR Sets, there is no more Drift noticeable as the Gear VR got its own Sensors. I would like it if the GearVR had a USB Port for USB Thetering, but the Framerates with WLAN are good enough to play. I have to load the Beta Version and give it a try again. Thanks for increasing the IPD, was one of my biggest Problems!

      Delete
  28. hello how can i configure my router for best streaming quality¿?¿?¿

    ReplyDelete
  29. Any advice in general, i connect my s7 edge via wifi ac 866mb

    ReplyDelete
  30. Sideload VR not see device id my phone (s7 android 7.0 official)

    ReplyDelete
  31. Steam VR gave me the "VR server has stopped working" whenever I try to run any steam VR games, any ideas why guys? I've tried uninstall and reinstall both vridge and steam VR app :(

    ReplyDelete
    Replies
    1. I'm getting the same error, even when trying to launch the Room Setup or VR Tutorial. I went back to the Stable version and that works fine except it doesn't contain the fixes for Asus Zenfone 2 so I guess I'm stuck waiting for a fix again.

      Delete
  32. When I choose HEVC encoding It always shows this message when opening Steam VR

    Could't find compatible códec. The safest combo is H264 and software encoder.

    I'm using Vridge for Gear VR 1.2 and a S7

    ReplyDelete
    Replies
    1. I am having the same problem... on a s6 edge plus...

      Delete
  33. Hello,

    I've tested new version yesterday. Sadly HEVC don't work for me despite having GPU that support it (R9 Fury). Do you plan to implement it for AMD cards?

    ReplyDelete
    Replies
    1. I'm running s7e on win 8.1 R9 280x gpu and I only get a loading screen then immediate crash back to main screen of vridge and riftcat..

      Delete
  34. The installer crashes as soon as it's launched.

    ReplyDelete