Multiplayer Doom on Cloudflare Workers

Hey welcome Celso, good morning. Good morning. Welcome everybody watching from we're here in Lisbon and Celso Martinho is engineering director in Cloudflare who works for me and I'm Cloudflare CTO if you didn't know that already and what we're going to talk about in the next 30 minutes is some work that Celso did porting a multiplayer version of the classic first -person shooter Doom to work on Cloudflare workers and also to use a thing called durable objects which is being used for coordination between the different players but Celso before we get into how this works do you want to, should we just play it? How do we do it? Do I want to what sorry? Did you want to play the game? Sure why not let's do it I'll just start screencasting my screen so I'm gonna send you the link go on and send me the link let's try that I'll start the game do you want the deathmatch or you know I want deathmatch right yes I do so here's the link all right so let's see where did you send me the link and I'm waiting for you where did you send me the link in chat yeah okay got it all right all right I'm in all right okay I'm in so I see you let's start it's connected to web sockets and now I'm in here so and I can hear things okay wait a minute whoa okay so I'm dead already well you know just a little bit so all right so what's going on here while we while we play the game here what what's actually happening so what's happening here is there's a web assembly version port of doom running on the browser this particular port has been adapted to use web sockets and durable objects instead of UDP and it's connected to the cold flat edge so it's using a web socket web socket connection and then it uses the durable object instance to coordinate the communications between the the doom clients in this gaming session and basically we have a message router running on Cloudflare edge who's receiving messages from from all the clients in this game session and making sure that the corresponding clients get their their messages and the game works um this is this has been a bit of an act uh as as we wrote in the blog post but it works and it proves that you know real-time multiplayer gaming is possible on top of uh on top of Cloudflare Workers yeah it's pretty amazing actually because you sort of run around in here and I'm just uh getting getting my health back a little bit here because I'm a bit of a mess and um it's interesting that uh you know we can just play this man how how much work was this to get this actually working tell me a bit about that sure so um let me just maybe we should stop playing we should just talk for a bit because I can't really concentrate I'm absorbed by the game trying to find where the hell you are and kill you so all right sure sure let's go let's go back to the so this this was uh this was actually first of all very a lot of fun to do um and uh the first requirement we had in mind is that it would it would need to to be very simple to to use um and so and so we needed to have everything working on the browser um and uh and the first part of the job was to make sure that doom could work uh on a regular modern browser which which meant that I needed to port doom to WebAssembly for those who aren't familiar with WebAssembly it allows it allows you to run uh typical applications um natively on a lot of environments including including a regular browser so you can you can take a classical C application like doom compile it with a framework like um like Emscripten um and it will target the binary to WebAssembly um and then it will basically run in the browser uh this all seems simple but in practice uh it has a few details um and and that and that was the first the first part of the project uh first detail is uh you know typical old C applications especially games rely on the fact that you need to have a main loop um and and that main loop basically runs the game and it makes sure that you know every frame is designed uh is rendered um every time every time the main the main loop uh iterates but on the browser you can't have a a main loop or a loop running forever as as I think most people know uh the browser environment is event driven which means that the browser needs to be free all times to uh handle events from from multiple things he he it has to do uh while it's running and if you have a main loop you you basically block the page you block the browser and that's not possible so the first the first thing I had to do is is was basically to solve this problem luckily uh Emscripten provides a function to replace the main loop um with something else um which takes care of the problem and make sure that the browser is always free to keep running other things besides the web assembly the web assembly application um the second problem uh we had is so we wanted to use doom for the demo doom is kind of it's kind of like the new hello world in computing uh everyone everybody wants to use doom to prove something um and that's been the case for the last 20 years or so um but doom uses udp um to implement multiplayer uh in fact the original version of doom used something else which which was uh ipx uh yeah you know i'm pretty sure you know what that is right yes yes that was my first job my first job was writing novel ipx spx software yeah very good well it's great use now did it broadcast on the local land is that we assumed you were on a land as well yes yes so originally doom was supposed to run on a land environment uh and and didn't have to deal with you know stuff like latency or um or slow connections um and udp is great for gaming because um it's very fast um it's stateless um and and that's and that's good but there's a trade-off with that because it's stateless then the application needs to needs to make sure that needs to be aware that if if there there was packet loss um or or the packets or the packets are in order um so there are trade-offs but usually udp is the protocol that multiplayer games use um even even when when they use the Internet instead of instead of land and environments um and we wanted to use web sockets so web sockets are persistent connections that you can use in the browser but they use tcp um so so i guess so we we need we we had to adapt that so we we basically built a couple of things on the on top of the original doom source and we played around with some buffers and we wrote a new network driver to use web sockets instead of udp so so you modified the networking portion of chocolate doom to to work with a web sockets layer is that what happened yes uh luckily chocolate so chocolate doom is a modern port of doom uh it uses the original source code but it's been cleaned up um it's uh it's very readable uh it's modular and one of the things it does well is it abstracts the network layer too um and so i i took a skeleton of of a local host network driver that chocolate doom provides uh and i i built the web socket support on top of that um and in order to mitigate the the the problems with using tcp and web sockets instead of udp i played around with a couple of buffers um and you know because doom is an old game uh the the multiplayer protocol is kind of chatty but it's not uh as demanding as modern games um it actually works quite well um um it's not the ideal situation this was a proof of concept but it works quite well but we also have a separate blog post which is about using unity 3d in a similar way to actually create multiplayer modern multiplayer games as well so whether it's old or new it's possible on on the workers and durable objects platform yeah i saw that um good so okay so you so okay so you let's let's go through it you take chocolate doom you get it to compile to web assembly and there's obviously some magic going on here because you've got sound and you've got um you know obviously writing to some sort of frame buffer because it appears in the browser right it looks like so there's obviously some layer that's happening there the interface with the browser yeah so so so emscripten one of the nice things that emscripten provides is a couple of wrappers um so one of the things emscripten supports is stl um and um and doom uses stl to uh in fact it uses stl for the frame buffer and the udp network driver which which we replace it completely but uh but sounds and uh and the frame buffer worked instantly as soon as i got a hold of the correct flags um and a couple of other details but it was quite magical to uh at the end you know press make compile doom and it was working on a on a on a browser page and so okay so to get that bit working you rip out the novel ipx layer replace it with a thing that talks to talks web sockets which is then going to talk to Cloudflare to do the coordination so when we were playing the game at the beginning you created a game and you shared a link with me um now what happened with that link so how did the sort of the connection between you and i get made because previously we would have been on the same LAN and it would literally just been broadcasting right it would have worked yeah so how does that rendezvous and how does the message exchange happen yeah so maybe i can actually put something on the screen which uh which can help by the way this is all open source right anyone who wants to look at how this all works they can literally go to Cloudflare's github and just download it build their own it's all open source so we've we've put both the web assembly port and the workers router and the workers pages for the website on our github um so what happens when i start the game session is so typically the way that the original doom works and chocolate doom works is you're you're expected to be in a LAN environment the first player starts the game and internally doom calls that player the server and then the other clients that start doom on the same LAN environment become clients and they basically have a discovery mechanism and there's that initial screen where you see the players joining in the game session and then the server decides to start the game okay and by the way when once the game starts um then nobody else can join the the game session that that's the way doom works right um and and and then there's the protocol itself so on a LAN environment everyone sends messages to the server and the server is the one that receives all the traffic um and make sure that it replies to the correspondent client when it needs to and by the way the doom protocol is quite simple um it basically broadcasts the inputs uh from uh from all clients so if you if you press the key to move forward um it it basically broadcasts that event in the network protocol I see so it's literally keystrokes going around and then it's like everybody is everybody is getting the same keystrokes so um that was new for me because I've never seen a multiplayer game protocol before and and initially I thought that you know there were uh graphics and assets moving from one place to the other and that's not the case the only thing that's being transmitted between clients is keystrokes um and so every every client has a full copy of all the keystrokes from everybody during the game session um and the protocol needs to be um aware of this and uh and if someone misses a keystroke there's actually a mechanism internally to rebroadcast that keystroke to make sure that the clients have everything they need okay um so in our case we we couldn't have this network topology because we wanted to use uh um Cloudflare as the message router instead of relying on a single server um also because we didn't want to deal with incoming connections and uh you know opening uh opening uh firewall rules and right reports that this was something that we didn't want to do uh so so I basically had to change all of this um and what happens is the when the first player starts the game it becomes the server and it starts connecting with the with the message router in in Cloudflare Workers uh and then I basically generate a link that anyone can take put on the brow on the browser and that and that link basically adjusts the the starting options um the starting options of the doom application the web assembly the web assembly doom application and it tells you if you're a client and um and if you're a client it also tells you where to connect to okay it it tells you it tells you a couple of things it tells you the address of the server and it tells you what your doom name is it basically adjusts your input uh command line parameters um on the doom assembly application when it starts running on on your browser um but anyhow everyone connects to the same message router there's uh there's a gaming session id that is common to both the server and all clients and that's what Cloudflare workers uses to make sure that there's a single durable object doing the coordination for that single gaming session okay so there's so just to enter the workers implementation of this there's a single durable object created for a set of players playing the same game yes so every game is a is a new durable object that's responsible to do uh to keep state basically to keep state between uh everyone on the same on the same game um we we're doing everything real time so this means that we're not using everything that a durable object allows you to do for instance one of the things you can do with the durable object is use is to use storage uh but we're not using storage at all um uh this make makes things much faster and much more reliable for for this use case and it actually simplifies it simplifies things but if but if you had wanted to freeze game state or something you presumably potentially could have done because durable objects provides a mechanism for keeping yes yes if instead of a proof of concept this was to be a you know full-blown production environment yeah we would probably need to make sure that we have a copy of the state on storage in case for instance uh some client drops right and reconnects um or in the case of the worker isolates um terminates right i i don't think we want to go into this complexity uh right now but uh and and if anyone is interested that they can read about the Cloudflare workers architecture there's a pretty good blog post about that on our blog yeah um so yes in in if we wanted this to be completely perfect we would need to use states to use storage to to to to keep the states uh reliable um on on a few things that can happen during the gaming gaming section one thing is that i looked at the the code for the durable object side of things i think one thing that was perhaps the surprising was how little code is uh yes it's um i actually have the the code here so uh the router is basically one single uh javascript file um and you have a couple of functions one function to create a doom a gaming session id another one to verify if if the if that id is valid or not um and then it's let me see how many lines we're talking about 174 lines of code yeah less than 200 lines to do the whole thing yeah it's pretty simple uh and the router the router is basically just this oops yeah this is the wrong yeah so this is the thing that when the game is actually running this is all the code that's needed yes actually yes fly stuff back and forth between the doom sessions yes the the rest of the code is just you know skeleton and uh creating the the the game room and and validation of that nothing else yeah yeah it's pretty amazing i thought that was pretty fascinating it's how easy it was and i think we see that in the other the blog post using unity as well it's like how easy it is to get up these multiplayer games and how simple the the durable object stuff is in fact especially even with web sockets it's like yeah just hook it up here and it just all connects together so yes uh it's impressively simple to to do i this was your first go at writing something for durable objects right it was i i i did use workers in the past but never never the durable objects uh so first i did an implementation in node.js um it's also in the github uh this is my local implementation of the router um but i didn't know what to expect in practice when i when i started using the durable objects and i was impressed that in a couple of it took me about one hour to fully port the node.js implementation to durable objects and uh and it started working um yeah not bad for a first go at writing something on durable objects nice job nice job thank you if people want to play this what do they do so there's three things i'd suggest to anyone curious about our doom implementation one is read our blog post right uh which explains the whole project the whole process what we've done then there's the github repos which you mentioned uh so if you go to github .com slash Cloudflare there's i can show this there's there's the doom web assembly repo and there's also there's also the the workers implementation okay right and if you want to play the game we've put up a uh public url which is called silent space marine by the way this was john's idea uh and uh and everything's working and running in production in this url and you can just go there start a multiplayer game uh pick your name press go you can both play in deathmatch or cooperative uh modes you copy the permalink to share with your friends you press next and then all you need to do i can actually do it locally let's just use the link i'm celso number two and here we are yeah look at that okay and by the way this also works on mobile uh and it uses uh touch gestures with i was surprised about that that that suggestion came up internally and it seemed like about five minutes later you said oh yeah it works on mobile now what did it take to get it working well on mobile um i basically used an open source library to implement the virtual game gamepads and then i had to inject the key events into the uh web assembly application uh it wasn't so hard to be honest i i think i struggled much more with making sure that the css was properly working than the then the code itself to be honest that's that's an amazing indictment of css it was like porting this gigantic thing to web assembly was the easy part getting css to align something correctly in a div yes yes i i dare you to uh center a div in css and make it work first time just use tables for layout and yeah let's go 90s and use tables once they work understandable by humans it was my plan b should have been your plan a i mean anyway let's not talk about css too much so when are you doing quake um i don't know john uh whenever i'm doing quick um i guess you have other things to work on i guess i have but actually quake solves some of the problems we had with doom uh the protocol is a little simpler um and uh maybe we have a go at some point why not we'll see yeah all right we'll see well i don't know this was brilliant thank you for working on this project it was fun to see it i really you know as i said in the blog post one of the reasons why i nerd sniped you into doing this was i wanted to demonstrate to people how easy it is to build a multiplayer game on on the geroblogix and workers platform and i think this demonstrated it pretty clearly that you can you can build a real thing that's usable at Internet scale and just just deploy it without too much trouble and if anyone looks at celso's blog post and then the follow-up blog post about using unity 3d to make a modern game you see the same kind of power uh in in the entire platform so okay if it's not going to be quake is there something else you dream of building on cloud workers um i've been thinking a lot of i mean what's funny about doing this is that once you see it working you start having a lot of ideas of things that are possible uh and that was the whole point of the blog post right right uh so yes i i do have a few ideas i don't think we have time to explore them all today but thank you so much for asking me to do this proof of concept it was a lot of fun good and hopefully it demonstrates people the power of the platform and they can go and build something that isn't a port of an old game yeah so it looks like we're running out of time thanks to the demo if anybody wants to play at silent space marine.com and also you can click through from there into the blog post and then the github and learn exactly how to do it so cheers celso bye john thank you