Friday, October 7, 2016

Dev update #23 - After the storm

What happened 

On October 4th an update was released to SteamVR. It included changes in direct display driver component which is utilized by VRidge. The changes weren't documented/reflected in OpenVR interface. We've seen those changes in SteamVR's beta channel and after it started flickering there (few weeks ago) we posted the issue to SteamVR forums. An unfortunate series of events caused this changes to go into live SteamVR channel anyway. 

The result was preeeeetty bad. Everyone experienced flickering in random intervals making VRidge totally unusable with SteamVR. Less than 1 hour after the changes went live we contacted SteamVR team through more channels to ask for help.

The SteamVR team was very responsive but it takes some time to update the live system. We were amazed that they pushed the update with a fix into their live stable channel on such short notice (everything was fixed in less than 24 hours). Some companies attempt to wall themselves off from 3rd party support and modding but Valve is doing the complete opposite with this kind of open attitude. We always recommended OpenVR/SteamVR for aspiring VR developers when they asked us how to start VR gamedev and we will continue to do so. It's nice that some companies care about VR's future overall. 

What we learned

It's hard to have a system working 100% of the time when it relies on multiple 3rd party components (SteamVR, GPU drivers, Windows version, OS components, router config & rules, Android OS flavors, phone chipsets, Direct3D, etc). It is difficult, but there are still ways to mitigate the critical situations like this.

We will add some sort of emergency alert to the desktop client so we can notify about the possible compatibility issues. Sometimes all you need is to update some 3rd party components (like video driver) and everything starts working again. We can write this kind of important updates on our blog/facebook/twitter but over 90% of our users stay in client without checking the social media. A simple way to push a message to the client from our servers with a link to a detailed article can do wonders. Many other companies already have something like this ( or LoL launchers come to my mind). 

We will also follow SteamVR's beta updates more closely now. If there's a chance of this kind of stuff happening again we'll start preparing alternative fallback ways. There is a way to capture SteamVR game surface with a slightly increased latency that could have worked even after the Oct 4th changes. Degraded performance while we are working to fix it "the right way" is better than app not working at all.

What we were doing in September

We've been working on behind-the-scenes modules and refactoring some code. This is in preparation for future Android 4.x version and iOS. This also allows to see clearer performance picture which is very helpful to locate Android performance problems (Samsung jitter mostly).

We were updating part of the platform so we can add more servers. Currently all our online service are running on single machine which works well enough but it sometimes need maintenance/updates or simply just fails due to some unforeseen event. Having one more gives us some more breathing space. 

What's next

It's been a while without a feature update. There are a few unsolved problems that we are prioritizing now - Samsung jitter issue and motion controller black screen. Planned features are fake VR (playing regular flat games in VR), sound streaming and iOS version. We want to squeeze in an updated video encoder (H265/HEVC, double the performance of current one) somewhere in the meantime.

We know that some of you really want to know when we'll add those but giving ETAs often causes devs to go out of our way to meet the deadlines instead of doing things "the right way". This can work short time but over a longer period it leads to spaghetti code and long terms bugs that are hard to fix. All we can say is that our current short term priority is Samsung jitter & motion controller black screen and we really hope to see progress on these in October.

Wednesday, October 5, 2016

[Resolved] SteamVR broken after Oct 4 SteamVR update

Problem is fixed

Many thanks to SteamVR devs who were very helpful and responsive during past 24 hours.

All you need to do is to update your SteamVR to the latest beta. Right click SteamVR in Steam library -> Properties -> Beta -> Select "SteamVR beta" in the dropdown box.

The fix is live in stable SteamVR update channel! Let your SteamVR install any pending updates and you're good to go. You might need to restart Steam to get latest updates.

After latest SteamVR update an important component stopped working in SteamVR. New SteamVR version is using undocumented closed code which broke VRidge.

We are surprised that they pushed the beta runtime into live channel without updating their documentation first even after multiple developers* reported the problem to Valve as compatbility breaking.

It's hard to fix a problem without proper documentation or at least an explanation what changed inside the black box. We'll continue to search for a quick workaround. We'll also try to reach out to SteamVR team through more channels than we already did.

It's middle of the night here and I don't have all the information yet.  This post will be updated with more news as soon as we know more.


Past updates:

Update @T+30mins:
We received response. SteamVR beta update with a fix should be available this afternoon or tomorrow (Seattle time). Switch to SteamVR's beta channel to make sure update is installed as soon as it's available.

Update @T+60mins:
SteamVR beta update is deployed. We are investigating remaining compatibility problems.

Update @T+80mins:
The problem still occurs but the characteristics changed. We are trying to find out why the new version still has this issue.

Update @T+120mins:
Many thanks to Valve for quick updates. They are investigating update changes further.

Update @T+12 hrs:
We're eagerly awaiting Seattle office hours to hear back from Valve. In the meantime we're trying to develop an alternative capture method that doesn't rely on direct mode driver compatibility.

Update @T+18 hrs (Oct 5 - 10:00 AM PDT, 1:00 PM EDT, 19:00 CEST):
The problem has been escalated further by SteamVR devs. We are thinking about creating temporary SteamVR driver rewrite but we're still estimating the scope of work needed to do so. We updated our main site with a site-wide notification about this problem.

Update @T+20 hrs (Oct 5 - 12:00 PM PDT, 3:00 PM EDT, 21:00 CEST):
An update has been released to SteamVR's beta channel. After updating SteamVR the issue should be resolved. Both Riftcat beta and stable channels seem to work correctly. We're running some more tests to make sure everything else works fine.

Update @T+22 hrs (Oct 5 - 2:00 PM PDT, 5:00 PM EDT, 23:00 CEST):
The fix is live in stable SteamVR update channel! Let your SteamVR install any pending updates and you're good to go. You might need to restart Steam to get latest updates.