Tuesday, April 28, 2020

Dev Update #58 - RiftCat 2.7 Beta - Reprojection for Everyone


RiftCat 2.7 Beta - Reprojection for Everyone
This was originally planned to be a part 2 of the VRidge 2.6 Streaming Enhanced update but we had to postpone it due to various reasons. Now it's here as its own update!

This update will improve head turning motions on most mobile phones (even those not designed with VR in mind) by utilizing reprojection (also known as TimeWarp in Oculus products).

To test the Beta features, learn how to join Beta channels by reading THIS guide.



Reprojection and reasons behind it.
As some of you might have read, Google killed another project of theirs which is Daydream. This means than there is a constantly rising chance the next mobile devices generation will not be able to launch Daydream apps on which VRidge is based a lot.

We could not allow our users to lose the ability to play PC VR games with their Android devices so we created our own reprojection within VRidge so capable devices can use it even without being Daydream-certified.

What this means is that if your mobile phone supports some specific GPU capabilities it will be able to utilize reprojection - it will provide a completely smooth experience when moving your head in your Cardboard device!

VRidge will now ask for the camera and file read/write privileges on your device (to be able to read and store your VR viewer QR code) as previously Daydream SDK handled that.



Will it work on my device?
As of this Beta release, this feature will work only for Android devices (iOS is in progress). We will deploy support for next devices to the Beta channel as soon as they'll be ready.

Our reprojection implementation has 3 modes, best mode will be selected automatically, depending on GPU capabilities, OS version and so on. Those levels are:
  1. Synchronous reprojection - Most simple implementation. All rendering work is done on one GPU thread. After app finishes frame, that frame is being warped to screen with offset to cover difference between pc render orientation and current mobile orientation. In this mode frames should be placed correctly, however motion might not feel silky smooth.
  2. Asynchronous reprojection - This mode separates app rendering loop from VR rendering loop. Because of that we can achieve stable framerate and smooth motion, even if some frames are dropped in transport from PC.
  3. Asynchronous reprojection with direct rendering - Just like in previous mode, this one preserves constant framerate and smooth motion. However because of rendering directly to screen, latency can be significantly lowered. Typically when frame is rendered, it goes to Android Surface Flinger, which is responsible for composing final image that is displayed on the screen. Because of that composing step and double buffering, in default rendering mode frame is displayed 1-2 vertical syncs after it is rendered. That means about 17-33 milliseconds of latency on standard 60Hz display. With direct mode we are able to render directly to front buffer of display. That allows to skip all composing steps for VR layer, significantly lowering latency. Our current implementation works at 8ms latency on 60Hz display. This mode however requires a lot of modern GPU extensions and is mainly designed for newer phones (we mainly tested it on Samsung Galaxy S20, Samsung Note 10 and Razer Phone).
To check out with which mode your reprojection is working (or not working at all), launch VRidge (Diagnostics). It will display the appropriate information.



How to help us with testing?
When you'll test this Beta release and encounter any issues then please send us detailed logcat to support@riftcat.com with a description of your phone model. You can read about logcat here.

If your issue is directly visible on the screen, then simply send us the screenshots. Any additional info is highly valuable.

Your personal opinion also interests us! What do you think about the update? Give us all your thoughts.

Tuesday, April 14, 2020

Dev Update #57 - Streaming Enhanced release.


RiftCat 2.6 Release
Thanks to the Beta tests, we were able to catch most of the bugs within the new version and today it is becoming available for everyone.

To be sure that your update goes smoothly:

Platform What should happen?
PC RiftCat client should update automatically. Make sure to switch to stable channel in RiftCat Settings -> Update tab.
VRidge from Google Play It should update automatically.
VRidge from iOS store It should update automatically.
Oculus Quest Re-do the installation process if installed via script. SideQuest version will update automatically.
Oculus Go / Gear VRIt should update automatically.
Daydream It should update automatically.
Viveport It should update automatically.

Streaming Enhanced brings vastly improved streaming quality, especially on NVidia GPUs.

Initially, we planned Streaming Enhanced to be a two part update, but due to certain circumstances we decided to postpone Part 2 to a separate 2.7 update that will be available in Beta soon. For now, just enjoy VRidge with a much better quality.

There are three categories where VRidge got upgraded.

Image Quality
Some time ago, we found out that streamed image by VRidge has a bit oversaturated color pallete. At the time we thought that it made colors a bit more dynamic and alive. Turns out, it causes visibility issues in dark environments - which is especially visible in Skyrim VR.

The issue is now resolved. We have improved color accuracy to be 1-1 between the generated image on the computer and the receiving mobile device. Everything should now be clearly visible.

Below image shows the color comparison between VRidge 2.5 and the newly released version 2.6.

Click image to enlarge

The 2.6 frame may seem to look a bit more washed out but it is nearly perfectly color-accurate with frame rendered by SteamVR. Mobile VR displays often use low persistence displays where 2.5 frame was displayed too dimly, resulting in lost detail in darker scenes. You can also manually adjust frame brightness with gamma slider in PC app settings.

Latency

An endless subject of latency is often asked whenever PC VR streaming is discussed. We did a fairly good job at keeping the delay of streamed image at bay but we found out a way to squeeze even more performance.

Stream latency
If you have an Nvidia graphics card you can expect a quicker image response. Thanks to the optimizations that we made you should expect latency reduced by additional 5 to 10 ms.

The image below, shows the sampled latency before VRidge optimizations.

Click image to enlarge

Next image shows the sampled latency after new changes got applied.

Click image to enlarge

Rendered at 1080p@60FPS, encoded with GTX 970 @ 17 Mbps, transported with 5 GHz Wi-Fi.

Pictured latency is measured as a time between first and last event of this process:
  1. Getting fresh head and controller pose data from device sensor. Usually sampled at 200 Hz, depends on mobile device.
  2. Sending sensor data to your PC.
  3. Submitting sensor data to SteamVR.
  4. Rendering game frame with the tracking data from phone.
  5. Encoding pixels by NVENC with H264 or HEVC.
  6. Sending encoded frame back to your mobile device.
  7. Decoding frame from H264 or HEVC to GPU-understandable RGB texture.
  8. Binding texture to target VR environment.
At this point the frame is out of our control, it is going to be displayed ASAP with reprojection appropriate to target VR runtime.

In 2.6's NVENC update we improved CPU, GPU and memory transfer delays that were occurring between steps 4 and 6 which should result in 7-8ms better latency.

Controller latency
Additionally to the stream speed improvements, we have used better prediction algorithms for rendering and sampling of the controllers.

With this, you can expect a much faster response whenever you'll move your hand in virtual reality - no matter if you are using VRidge Controller, Oculus Quest Touch or Daydream View 3-DOF controller. All of them should benefit from this new improvement.

This change brings the controllers latency to near native level, making a pleasant gameplay experience. With newly released Half Life: Alyx - Quest users have an awesome gaming time ahead of them :)

Stability

Some of VRidge users had issues with artifacting. It could happen from time to time or happen constantly depending on the user hardware configuration. Especially on 2.4 GHz WiFi networks.

With this upgraded release we re introducing an error correction system. It will compensate for lost packets during the stream allowing for more stable video streaming.

This should allow VRidge to push for higher bitrates as the stream won't get broken with even the smallest packet drop. It will increase the picture quality and get rid of ugly artifacts.

On default, VRidge stream will now add 5% additional data with each frame for the error correction. The system is adaptive, when it will discover that the stream is losing more data than usual, it will adjust the amount of error correction data to provide the best experience possible in granted conditions. If you want to manually tinker with this system, you can override all values as described in our previous blog post (link).

In addition to that, VRidge should now detect lost frames much faster than before to make the error correction system response quicker. With this, you should expect basically no image artifacts during your playthroughs.

If you have Nvidia video card, we did some encoder tweaks as well which should also provide additional improvements in this area.



What's next?
We still have before mentioned Part 2 of this update in preparation. It is related to Daydream cancellation and it will be a game changer for many VRidge users that use regular phones with it. The Beta should be ready soon - we'll let you know about it as soon as it will be published.

We hope that VRidge Streaming Enhanced update will greatly improve your experience. If you like the new changes, let us know about it. If you don't, also provide us feedback so we could fix any issues that might have appeared. Your voice can always be heard at support@riftcat.com

If you like VRidge and the new update - spread the news. Our software lives thanks to your engagement in VRidge community. Any social media posts, articles, videos with VRidge help us tremendously.

Have fun in VR!