To me this was the least interesting part of the Stadia reveal. State share was the real innovation imo. Would any serious game studio use style transfer ml to actually replace their visual artists? Didn't seem like the quality was there.
That said, it is an awesome PoC. Just not something I see being practically applicable.
> Would any serious game studio use style transfer ml to actually replace their visual artists?
No, a serious (AAA) studio would not. OTOH, the platform might intend to support producers that aren't AAA studios, and whose limited pre-release resources are focussed on gameplay more than art assets, and those producers might view it very differently than AAA studios.
You still have to do all the work of building the geometries for the art assets, are you really saving that much on not having to color/texture those assets?
Maybe? One of the hacks of adding more assets without having to do much work is just recycling existing assets by assigning different color schemes/textures.
Come on, think bigger. If they're doing this for art styles you can bet your bottom dollar they'll be doing something using Google images. No longer having to actually create 3D meshes - just a Google image search, click a button and have the geometries automatically created for you. Then just add an automatic style.
Actually Google is just seeing the wealth of innovation in procedural generation and in computer creativity, knowing where this is going, and just want in on the action.
And since as a cloud gaming provider, they stand to make much more money than tool vendors, their incentive is much bigger.
Imagine a game where you step into a portrait and are suddenly in the art style of the image. You suddenly walk into a countryside that feels american-gothic style, and then time-jump to 1940s New York in the style of Nighthawks.
(different) Stylization as a medium in games isn't that explored, and the opportunity for indie devs to play with that is, IMO, really cool.
Would it help in game development? For example giving developers better looking playable prototypes while the artists are still working out their designs?
Hypothesis: the rough and relatively unpredictable results are not so much a good fit for finished products as it is for _rapid visual prototyping_. A good art director could use this to quickly settle on a rough goal around which the rest of the game would be targeted. I'm aware the pipeline usually goes from art director -> development, but this could help, say, a competent dev team with a gameplay direction but no artist yet, or a long-standing developer that wants to use its existing engines and mechanics to draft new, distinctive franchises.
I think the more interesting applications will come when you get tools like this used by visual artists, complementing handmade design. The demo is cool but uses style transfer very bluntly, applying one style to the whole image with high intensity. I imagine the future of this is something more like neural "shaders", able to apply real-time pixel alterations to scenes or objects within scenes. Used carefully, they could increase the range of creative choices a visual artist can make.
I have been thinking about a process where a procedurally generated template is completed by artists and subsequently used as a base for training an AI to apply to further procedural generations.
I also wonder if you can do micro style adjustments to unify artistic style, train a net on one particular artists work, the rest of the team copy the general style but use a final pass of a style transfer to make it a closer match.
Honestly seems more like a user-end personalization than something a developer would use. In a game settings menu, you choose an image that maybe you uploaded to use as the game style? Definitely something people would use. Especially if they figure out how to make it selectively apply to only certain textures/objects or parts of the screen, which seems likely.
Street Fighter 4 had a feature kinda like that (with predefined selectable styles). AFAIK it was not popular. Sometimes my friends would turn it on as a joke.
Actually, I wonder if there's potential in something like this for accessibility purposes. If you have one of the many variations of color-blindness, for instance, could such a restyling be helpful? Could other neurological disabilities be similarly aided by dynamically restyling visuals?
Yes, but they also require active participation from the developer and many developers are unwilling or unable to spend the time to implement them, especially when there's a rather large range of disabilities to account for.
It's possible that tech like this could offer a user-definable solution to these issues.
One incredible use is actually in network training for things like AR and autonomous driving. You want a slightly fake world and you could easily try out various network approaches using a free fake world found in game engines. Microsoft has something for this but cannot find a link or the name right now.
I would have been much more interested in using deep learning to make human faces more realistic. Somebody was experimenting with it in the past quite successfully IIRC, but I'd love to try it on a real game.
Maybe used as a player entering dream like equivalent after eating magic mushrooms. Other than that you’d be hard pressed to see this in any game in memory
I was baffled with the style transfer announcement. It seems arbitrary, random, and out-of-place. Style transfer (applied to art) is really neat, but I can't see it being used to re-clothe games for production.
Before I get downvoted, yes I do think it could be useful for prototyping and conceptualization during development, but I doubt anyone would actually ship a complete game with the entire camera view transformed by style transfer.
A neat tech demo, but as with so many things coming out of The Valley these days, I don't think anybody actually wants this. No actual game is so bland that you'd want the ability to overlay your own "themes" on it after the fact. AI has lots of potential for assisting artists, and maybe part of that will include post-processing effects, but putting such things in the player's hands is pointless.
this is certainly just a neat gimmick, but it's interesting to me for a particular reason: the Stadia box uses an AMD GPU, and per the Google announcement, all the style transfer is done in real time.
this suggests that we may soon have better AMD support in TensorFlow.
Style transfer is most likely not running on the GPU in this case. It's probably running on Google TPUs, given that the GPUs are likely already under high load in order to play AAA games at 4K HDR 60fps.
this is like bad selection bias though. people never write issues to say things are going well. the general Tensorflow issues are also about only bad things, by definition
Consensus in this thread seems to be that this is a neat gimmick but nothing else.
Maybe because I have a friend working on a video game as a solo developer, and putting a /lot/ of energy/money/time into art--I really see the potential in this (type of thing).
This is not really for the player's benefit to put on whatever mods they want.
This is for fast iteration for artists and creators to automate a huge (sometimes necessary evil) burden.
> Maybe because I have a friend working on a video game as a solo developer, and putting a /lot/ of energy/money/time into art--I really see the potential in this (type of thing).
That gets you usable texture assets super quickly, which is probably what your solo developer friend spends the most time on. That or models. Neither of which Stadia's style transfer helps with. They probably aren't spending much time tweaking a shader-applied post-process effect, but hey, maybe they are, and this style transfer ML thing is useful. Very, very unlikely, but maybe.
> But it doesn't seem suited for final asset generation.
Well yes, of course, and neither is Google's Style Transfer ML. These are both just prototyping tools, not a magic wand to make production-quality assets.
It could be that they are throwing it out so that the community try to find a practical application of it. It is obvious that they understand that what they are demoing is not the end product so to speak, but just the best way to represent general idea.
I'm already excited to see what Microsoft will show at E3 with Xcloud streaming service that they have been working on for a couple of years- lots of great progress in that area right now.
Impressive sense of realism in level of detail variety..show the potential...BUT consistency, sharpness and realism of object boundaries would not survive close inspection.
This is not a consumer product at all. I don't know how useful it will be but it's designed to be development tool for early art direction visualization.
Even for 2d visualization it could be very useful if you could drop images into your website mockup files and have it intelligently apply color palette and texture information to your existing content.
It won't look good, but it might look good enough to save you from having to manually tweak the design 20-30 times to explore a range of styles.
I don't think there will be heavy users of style transfer among high budget games anytime soon, but the important part is that it's done in real time. With an assumption that it has a reasonable level of latency, probably AA, super resolution or any other neural net based quality improvement can be a real thing.
This is really cool to think about. In practice though I would imagine every incorrect prediction to be incredibly jarring. Additionally my intuition is that latency is easier to decrease than predicting the next 30ms. If there is some level of latency that is very difficult to reduce (more so than training an AI to predict the next 30ms with very high accuracy), I'd be curious to know why.
Between e.g. central and west US you'd have a physical ping of ~10ms so 20 round trip just based on the speed of light (i.e. very hard to get rid of).
Meanwhile the input lag you want is one or two frames so on the same order. But this lag is additive, when the game receives the new state it has to compute and render the world, the display has its lag before it shows up etc. So the new frame has to start computing as soon as the input is registered in order to hit reasonable input lag.
Predicting 2-3 frames should't beyond what we can do at least roughly. Especially for some important inputs such as turning in first person view it should basically just start moving the screen. I belive John Carmack tried some kind of rotational prediction (Not based on AI) for Oculus.
I am not a game developer, but couldn't you make a multiplayer game more fair in this situation by adding artificial lag into everyone's controls to match the highest lag person? Just limit it to a certain amount, like no more than 200-300ms.
I wasn’t considering fairness here (that’s a whole different can of worms). Even for a game where everyone’s lag is the same, X milliseconds of input lag and Y milliseconds of network latency is painful for some values of X and Y. For fast paced games I’d say X is probably just a couple of frames (15-30ms) and Y is about 100ms.
Silent hill concepts could be fun with this. Otherwise eh. I would like to see Jojo palette swaps occur to highlight tension but I don’t see that being implemented with something like this.
Onlive - required other people's infrastructure and a hardware play, and a software client.
Playstation Now - required Sony to buy/use a cloud service, maintain it, and expensive Sony hardware just as a force of habit.
Google - negligible instance running on their existing infrastructure, maintained under existing processes, playable on chromecasts already in hotels and homes everywhere, alongside browsers and mobile devices.
If Google doesn't shut it down within the 5 years for people to notice this is even an option, they could focus more on titles and developers more than any other entrant to this market.
And perhaps the gaming is just the application to get people's attention.
Are you implying Google doesn’t need to maintain an expensive cloud infrastructure for it as opposed to Playstation?
The only reason they can do that today without specialised hardware a-la OnLive is because Nvidia and co. now have specialised GPU circuits allowing one to stream the encoded video output to memory.
When OnLive started they had to do it themselves.
I don’t believe you have any idea what you are speaking about.
> Nvidia and co. now have specialised GPU circuits allowing one to stream the encoded video output to memory.
Could you clarify what you (and the comments mentioning NVENC below) are talking about ?
As far as I understand it, NVENC and the ilk are solutions that capture video output and encode it to H264. So they are x264 encoding accelerators, nothing more.
If you were running a datacenter this way, you'd be much better of having a bank of Matrox capture cards (which support multiple simultaneous inputs) in dedicated hardware converting your video game output to x264 streams for broadcast. They even support capturing, x264 encoding and streaming to IP addresses on your network for further distribution.
What would be interesting is if they bypass the video display output phase completely and render the finished framebuffers to memory (or an encoder chip via DMA over PCI) instead. I assume this would cut down manufacturing (+licensing for HDMI ?) costs a bit.
I know AMD has DirectGMA that allows other devices on the PCI bus access to limited chunks of GPU memory. There are signalling mechanisms in DirectGMA so that devices can basically implement producer-consumer pairs.
As far as I know this doesn't exist in consumer chips though. You need "workstation" GPUs. Which might explain Google's particular GPU choice, now that I think about it.
>> As far as I understand it, NVENC and the ilk are solutions that capture video output and encode it to H264. So they are x264 encoding accelerators, nothing more.
Me:
>> allowing one to stream the encoded video output to memory.
You're being pedantic. What I meant is that OnLive actually needed specialised hardware because they could not provide their service without it (as in: there was no capability in hardware, without changes to the codebase - to render not to an external display). You, on the other hand - are speaking about the conceptualised "perfect" implementation.
Google doesn't need that hardware in order to provide that service because by now there's a hardware capability that allows you to capture the screen of the device you are rendering to. They could try to be more efficient by using other hardware, but that's not a prerequisite for their Gaming Service.
>What would be interesting is if they bypass the video display output phase completely and render the finished framebuffers to memory...
That's exactly what NVENC, Capture SDK, AMD's AMF, and Intel's QuickSync do. Just encode the framebuffer output of whatever game engine you're using with one of those API's. (And remember to make sure your game engine license allows you to use it that way. As always, if you use Unity, you're out of luck, they specifically forbid you from doing this.)
I'm assuming you mean how can you encode the output of the nvidia card to a memory buffer using an API?
The NVENC API allows you to do this on your commodity NVidia card. If you have the good stuff from NVidia, you can likely use the Capture SDK, which is a bit more scalable. If you use AMD there is AMF. And if you're cheap, you can just use QuickSync from Intel. All of the encoding API's will likely be unified with an industry wide API a la opengl at some point in the future, but it won't be implemented by everyone in a timely fashion, so you have to pick your poison right now.
One gotcha though, you can only do this if your game engine's license allows. Which means anyone who uses Unity is out of luck. Unreal and other engines? I don't know, I think you can talk to the companies and they'll probably? give you permission? (At least I'd hope they would give you permission until they go into the cloud gaming business.)
Your best shot though is just to take an open source engine, like Godot, and "NVENC it up" so to speak. It'll take maybe a couple of hours and it'll save you a lot of headache.
Nvidia's is called GameStream, and has been in consumer cards for about 4-5 years now. Moonlight[1] is an open-source client that lets you stream your PC games to most any device.
They may be referring to ShadowPlay, a feature that keeps a buffer of the output so you can save something that just happened. Again, available for 4-5 years.
> I don’t believe you have any idea what you are speaking about.
You are gatekeeping based on irrelevant details when my point is that prior entrants had higher or untenable costs and Google might not, which I think you agreed on.
Google's advantage is that it doesn't need a dedicated hardware play, has saturation in relevant markets in their existing hardware plays, and has existing capacity in their large compute infrastructure. Whether consumers care is another story, but they can throw more money at things to make them care without having other untenable overhead costs as a distraction.
Where are you getting that it’ll only require a negligible instance on existing infrastructure? Running a AAA game on 4K takes a pretty beefy machine. I doubt it’s a workload that you can throw on some small VM without making any changes to your hardware infrastructure.
The hotels I've seen them in (in the US) have been Hampton, Hyatt, and Mariott. I might've seen one in a Sheraton, but don't quote me on that (and if so, only once). Definitely seems to be per-hotel rather than brand-wide, though.
Yeah its a game changer. Marriott brand is rolling them out, and Hyatt has them too. These are massive portfolios now, and will save them from getting screwed over continually by Apple changing the docking station ports.
For now Chromecast just being there and part of the TV will let users do what they want with music and now this.
The market is dead because Nvidia only want their high tier graphics cards to be deployed... can you imagine if instead of deploying 30,000 dollar GPUs, you could drop quad Titan X's or Titan V's. The datacenter license is cancer!
100ms+ feels worse in this kind of thing compared to 100ms ping to a multiplayer server because even if you have bad ping to a server the game still records your inputs locally and feels responsive. More realistically it will be more like 180ms+ on a home connection.
I was an OnLive user when it existed earlier in the decade and I would say that it felt generally fine in games with forgiving timers that were not all that hard. In any game that is challenging, this kind of lag even in single player games can be very frustrating, and you might as well not even think about multiplayer because it will be a much worse experience than playing on a 56K modem in the late 1990s.
The old Assassin's Creed games were a great fit for the technology because nothing required a reaction time of under 1000 ms and movement did not require a lot of fine adjustments. Anything turn based is also a great fit for this kind of technology.
Even Steam Link wireless in the same house over a good router has annoying input lag that may not be immediately noticeable until you compare it to a wired connection or using the PC itself.
Another big issue with this in the US is that only rich people are going to have the quality of connection you would need to enjoy this at good quality. And if you are rich, spending like $1k on a budget gaming PC or a few hundred bucks on a console are going to be pretty reasonable. No one would be dumb enough to spend $300/month+ on a high bandwidth connection so that they can stream games to potato hardware.
On the other hand, if high bandwidth connections were ubiquitous in the US as they are in some other developed countries at an affordable price, the calculus would be totally different.
I used a game streaming service called Shadow, and it works amazing for Sim City, Frost, and Civilization type games.
I really hope Google focused on putting out quality titles in the city builder genres. Hell it might even make sense if they outright bought the Civilization franchise.
I would love to play a turn-based strategy like Rome Total War using this. Games like that are perfect for 20 minute chunks of gameplay on a bus/train, in the airport, waiting somewhere.
> No one would be dumb enough to spend $300/month+ on a high bandwidth connection so that they can stream games to potato hardware.
I'm paying $80/mo for nominally gigabit (usually tests out at 700+Mbit from the router, a bit lower for WiFi clients) with no bandwidth cap, and not in a market with much (or any) competition for really high-speed connections. What are people paying $300+/mo for, and is even remotely needed for something like this?
That's great for you and that is what it should cost but for a large portion of people that kind of bandwidth is not attainable for an affordable price.
Style Transfer might be a way to paper over lag for some connections and some types of games? It might be better to have local ML-interpreted "pretty" screens than the usual rain fade/digital tears of struggling-to-keep-up-with-the-action video codec?
A while ago PUBG plastered ads for their Berlin Invitational. While they were ads related to the game, the incrementally added them EVERYWHERE!
First they're on a bus-stop wall, that's thematic okay. Then they're on random walls - getting annoying. Then the loading screen and the parachutes, nows it's annoying. Like, guys I'm from the US and I don't want to go to your stupid competition!
Is anyone else concerned this will be yet another Google product that goes by the wayside in a few years? This has essentially become my biggest fear now with any new product they announce/release.
I understand that it's inconvenient when services shut down but I see 3 options.
1) Wait until it seems there's enough comittment from Google before using it.
2) Start using it right away knowing that it might not be around forever.
3) Don't use it because you don't trust Google to maintain it.
Given these 3 options, does Google launching a service really have that much impact on consumer's lives? I'm a huge Gmail user and have been for over a decade. If they shut it down it will suck but I'll find another way to email people.
I get the philosophical arguments but curious why some people have the level of concern or angst when Google announces a product.
(for the sake of this comment let's forget about competition and monopolies)
It's pretty much a certainty with this project, in my opinion. Until everybody somehow has gigabit lines to their home and to their phones, I don't see this becoming popular. Like most Google projects, their actual success period is most likely 10+ years away, and the project will be axed long before then.
5G deployments have already started rolling out. 5G has average latency in the range of 1-2ms. Google controls a huge swath of the backbone after that. They can realistically achieve < 5ms latency on 5G devices in the next 1-2 years.
IMO, a more valid argument against its success would be that any consumer who is in a position to ensure they have fast enough internet for this service, is also in a position to buy a gaming console. But the counterargument to that is that there is no reason someone cannot have a gaming console and also use Stadia.
I'm normally very skeptical of Google's motivations, respectful of their cloud technology, and doubtful about many of their products. But I think this one will be a success, and shortly followed by competing products from Amazon and Microsoft.
Seeing as I live just on the outskirts of a capital city (Cardiff), and I can't even get 3G signal inside my house, and no more than 80/20 broadband; I'll remain skeptical about the possibility of success for this outside of very specific cities. There's also a lot to be said for reliability and the requirement to always have a fast connection: one of the benefits to many videogames is the fact that they are not online.
Isn't 5G on a wavelength that has poorer penetration than 4G? It might work in the USA but signal is a constant issue in stone housing here in the UK (though it may be better where you are based). I had to get a femtocell from EE in order to get signal in my house :\
I doubt this is going anywhere. As a PC gamer I have no need for this, don't want to deal with lag, and don't trust Google to operate the platform in a consumer friendly way.
Look at how angry people are at epic right now. Taking a brand that already has a bad rap and trying to start a new market (that has already failed several times over) is not a great angle, especially with an audience as skeptical and anal retentive as gamers.
Additionally, who is the audience for this? Casual gamers use consoles, and those more invested use PCs. This is too out of the box for the former and completely useless to the latter. I could also see Google trying to get exclusives which would be a PR disaster.
Sure, it's a novel idea and looks like a decent execution, but its by no means a new one. Onlive did this 10 years ago and couldn't get market share. PSnow is available right now and its completely unusable even with a great connection because they dont have enough servers. The game selection is also terrible considering they should be able to load up any game in their ecosystem. I cant play MGS4, the PS4 remaster of last of us, nor horizon as they want me to buy a console to play it. So what's the point?
That said, it is an awesome PoC. Just not something I see being practically applicable.