Ask a Solutions Engineer

Right, hello everyone and welcome to our first session of Ask NSC Live. My name is Michael Tremante, I'm a Solution Engineer at Cloudflare.

I'm based in San Francisco and I'm joined by also Matt Bullock.

Matt Bullock, do you want to introduce yourself?

Yes, so I'm a Solution Engineer based out in London and so I do exactly what you do just in a different part of the world that is not as nice and it's not as sunny as San Francisco.

Great, great. So I guess before we jump into everything I want to give a little introduction of why we're doing this and the concept behind it.

As Solutions Engineers at Cloudflare we essentially work a lot with customers and we try to get as much of a wide scope of the product as we can.

So we are generalists but we have a pretty good view of all the different components that make Cloudflare what it is today and we tend to advise and help customers in a lot of different scenarios.

And we also, you know, over time we've actually become a sort of a reference point even internally to help some of our colleagues basically answer and understand some of the Cloudflare features.

So about I would say four years ago we started the chat channel inside of the company which is literally called Ask an SE and it has grown to be pretty popular over time and we have more than half of the company in it today.

So SEs are constantly being pinged with all sorts of different questions from all sorts of different categories, product questions, technical questions, etc.

So we said why not, why don't, why shouldn't we try opening this up to the wider audience and maybe allow everyone, our customers, our prospects or, you know, the casual Cloudflare user to send us some questions and maybe we can help them out and also answer.

And before we jump in a little bit a few housekeeping items.

So first of all we are answering your questions. That means that we may not necessarily know the answer to your questions but we'll give it our best shot.

If you want to send us a question there's several ways you can submit it.

The easiest way is to send an email at livestudio at Cloudflare.tv. So that email address is constantly monitored so feel free to send an email there with your question.

If we don't get to it today and if things work out today we'll hopefully be doing more sessions and we'll be able to get to it next time around.

The other few housekeeping items is we are a little bit spoiled as solutions engineers and we tend to work with our enterprise customers.

Of course we know that our audience here is a lot more varied and we have a lot of customers using our self -service plans.

We will try to call out when features are available in the various plans and but if you see something in the dashboard whilst we're demoing or maybe answering a question that you do not see it's likely simply because it's available on some of the higher plans and not on maybe on some of the lower plans.

The other part is again we will make some mistakes so excuse us in advance from for that but again if we do make a mistake we'll try to correct ourselves and I guess we'll just see we'll have some fun and see how it goes.

With that in mind I guess we can kick this off.

So we do have we have received number of questions from from you from everyone so we do have a list to get us going and then we'll see how this evolves over time.

So let's begin I guess. The first question we received is from Chad and he basically and Matthew maybe we can take it in turns and answering the questions and he asked us does Cloudflare protect TCP applications?

So this is a question we get pretty often actually as people think of us as protecting websites but I guess Matthew do we protect TCP applications?

Yeah and I think it sort of stems from what you said.

So when Cloudflare had this mission to help build a better Internet we would just protect HTTP HTTPS and so we'd go into customer meetings and go oh so you do all this what about other things what about mail servers what about SSH and we would do something that we call in the SE term called razzle -dazzle which is look over here and not over here but obviously one of the things we built was a feature called spectrum.

So I'm just going to share my screen and hope this works.

So within the Cloudflare dashboard and spectrum allows us to do Cloudflare magic but at a layer 4 level.

So our normal HTTP HTTPS pipeline works at the top of the stack which is layer 7 and then at layer 4 and this is where spectrum operates.

So you are able to get the same DDoS protection that you would expect at the layers but also allow to proxy and the actual TCP traffic terminates at the closest Cloudflare pop.

So it's really actually really simple and it's again provisioned by DNS that when you enter the DNS for your zone so if I wanted to protect SSH I could then just deploy it at the edge port of 22 and then I can pipe that back to the IP address of my origin and even do port translation there as well.

So if I didn't want it to go back to 22 and I wanted it some weird port I'm running SSH on just to be obscure I could do that as well.

So you're almost looking at a remap of TCP traffic if you like and then we also have a few things which is do you want to terminate TLS and if you don't we can then do pass-through so some customers like this not only for TCP protection but also if they don't want SSL to be decrypted at Cloudflare's edge and we also support proxy protocols which allows us to understand the actual client IP that is connecting from the application and not the Cloudflare IP because that's not great for your logs or if you're doing things with email traffic and protecting MX then you want to see who's actually sending the email.

So this is what we built and it's been sort of successful and it actually a big part of this product a lot of Minecraft servers have been protected by this because for some reason I don't play it seems like it's something Michael would probably play his Minecraft in his evenings but they protect given DDoS protection to those because people like taking each other out definitely sort of opened up to a whole different world of Cloudflare so again provisioned at our edge in the same data centers and there is also Argo which we may talk about depending on questions later which can actually accelerate the TCP traffic so people are actually getting performance benefits so if you are connecting to Minecraft servers legitimately then you're actually getting the improved performance against just going directly to wherever that origin is globally so yes the answer is we do protect TCP and I know this is sort of the enterprise looking UI and we have actually rolled this down plans so this is available I think on the pro and business again I may be sort of wrong on that and I look forward to feedback but it's definitely something that I guess the the other of this question that is okay we protect TCP now does that include UDP as well so again from this we do protect UDP and so we can't accelerate this with Argo at the moment a keel rpm hopefully one day we'll find a way to integrate this but yeah we can do UDP to sort of protect as well so really any protocol there is caveats to this and I won't go into them because yeah I don't know sort of make everyone sleep on the other side of the screen but most protocols we can actually protect through Cloudflare I mean the most common ones we have is SSH and SFTP FTPs and things like that I see we've added an actual option for Minecraft down there at the bottom I think this was actually of the self-serve so when you actually click into those plans you see Minecraft and you don't have to work out what port I think that gives you the standard Minecraft ports yeah which is there so you can just create that straight away okay so Chad hopefully that answers your question so to recap yes we do protect TCP applications cool so the next question we've got this is actually so when my Twitter handle no one follows me I don't know why I think I'm funny Matthew Prince and JGCR CTO retweeted and Stefan actually asked this on my on the Twitter handle so and it was actually a complex question so I'm glad I get to pass it over to you and so the question is let's talk about the physical network infrastructure where do your partner providers link plug into and the services how is the info configured to help outbound path e.g.

for Argo upstream path selection so a lot to digest so let me try I'm gonna try answer this in different parts because it is a pretty pretty big question so first first and foremost we let's talk about the network a moment so zoom down Cloudflare we operate you know hardware all of the world remember we have presence in more than 200 cities now we don't actually own the data centers themselves we essentially co -locate our own hardware in those data centers so we fully own the hardware we're not co-locating servers from hosting companies or anything like that but we we don't own the actual data center itself now and the the network of course we're always trying to add more capacity to it so it's it's an always evolving and increasing network in terms of size and performance etc we also try to increase the capacity as we as we add more customers to the platform and in terms of where do we connect with customers or with other networks and Cloudflare actually operates an open peering policy so we do have our own AS our own autonomous system of course actually now that I think about it I can share quickly the screen my screen for a moment and we have actually over the years built a peering portal so for anyone who's curious actually you can go on the Cloudflare peering portal and request access to it and essentially we you know as of today we're probably one of the most connected very well connected network I believe we connect several that to other several thousand networks worldwide and if you use a peering DB you can actually you know if you put our autonomous system number you can observe all the other networks we connect to and at which points you or another network can basically connect to us the the other thing to note is we try to deploy our hardware and Internet exchanges that essentially gives us the best ability to connect with as many other networks as possible in one go so if I think about the Cloudflare points of presence I can probably classify them into two categories mostly one is the actual Internet exchanges and that's where you know everyone tries to connect together but we also have the concept of ISP pops so sometimes an Internet service provider will want to basically allow us to host our own hardware in their data centers and that is because they actually get benefit from Cloudflare caching and basically serving the traffic from their network directly because it reduces their egress and ingress traffic as we're serving you know their customers from from one of their data centers so depending on where you are in the world and what ISP you you're using for your residential connection or your business you will actually be routed based on either directly to an Internet exchange or based on your ISP depending on where they on where they connect back to us and if we want to talk a little bit about the hardware itself so first of all inside a Cloudflare point of presence we have many different generations of servers of course over the years we're always trying to improve and make them better and increase their performance as well as sometimes we just need to upgrade the point of presence to add capacity so visit visit mix of hardware in there and actually our head of infrastructure recently published a really interesting blog post and I would advise anyone to go have a read if you're interested in this sort of things that basically specifies the sort of thinking that went into our generation X server so our 10th generation of our hardware and normally I mean we do use off-of-the-shelf components when we build our servers but of course we optimize and fine-tune them and they can choose exactly what components we need so that we're optimized for the type of workloads we observe on the Cloudflare network and historically for example with the Gen X servers we used to use into the Intel based CPUs and but more recently we actually made a switch to AMD EPYC processors and several reasons we made that decision you know all the details are in the blog post but essentially under the workloads we observe they just perform better and also had better power management in terms of WAP consumption per number of requests so hopefully that gives you a little bit idea of course there's a lot more hardware than just the servers themselves going into a pop we also manage the routers etc but hopefully that gives you a little bit of an idea of the sort of hardware we do deploy in our points of presence and the other thing I think which was mentioned in the question was you know how do we decide on the routes and more specifically I think we're talking about Argo smart routing so just for context the Argo Argo is a technology that allows us to accelerate dynamic traffic what I mean by that is that normally you know static traffic is very easy for us to accelerate we just keep a cache of it at the edge and with Argo smart routing we are able to route traffic through different paths in the Internet to potentially avoid congestion and avoid other problems that may appear in the Internet the Internet is not stable as we would all like it to be of course and by doing that you know if you're using Argo smart routing you normally get better performance even for some of your dynamic traffic if we have to connect back to your origin server now the way Argos would choose the routes for Argo smart routing is essentially from every data center we operate from we essentially ping customer subnets and we also ping every pop from each other so we have basically matrix of latency times between the different pops we have and between the pops on specific customer subnets we use customer subnets because we don't want to be pinging every single origin over and over again so you know if we have several customers hosting in the same data center likely the path between our pops and the data center is not going to change from one customer to another so that's why we ping specific subnets and it's basically mass calculation if we know that connecting from one pop to the origin is slower compared to hopping by another pop on the cloud for network we will essentially make that decision the way that pops connect to themselves is simple we just have a unicast map for every single point of presence so the routers themselves know which interface to put the traffic on to connect the next following pop and we sort of keep this live map of latencies and we update the routes we use when you're using Argo smart routing based on the results we see real-time from those latency metrics so and the other thing to note of course is between so that's how we route traffic between the pops and how we make those decisions and the actual customer facing part of the network so the IP addresses which all of our customers are using when their users connect to Cloudflare or potentially proxy to Cloudflare those IPs are unicasted so we're advertising those IPs exactly the same ranges across all of our points of presence so it doesn't matter where you are again it will depend on which ISP you know you're using for your home connection you will be routed to the closest point of presence which where we likely have a peering connection with the relevant ISP and I think that answers most of Stefan's question yeah I think most of it was covered obviously if if not please get me in touch and if you allow us to do in all of these yeah and Matthew told Stefan if you're listening to us Matthew told me you were the first person to send us the question over so I think he wanted to make sure we we got you an answer and if you want to if you want us to dig more you know in in more details send us a follow -up question and we'll make it more technical and I'm sure it's a later time there's going to be the network team or someone actually talking about hardware and Cloudflare on Cloudflare TV so people to take the schedule yes so okay so let's go to the next question so we didn't get a name from this person the question is really really good nonetheless so the question Matt for you is do you do you Cloudflare patch DDoS bypasses for websites okay so just to touch yet so this came in I think from Instagram so ovh.xyz is yeah either a handle or even Musk's kid and he's already sending Instagram things but so this is sort of interesting and I think it's two two ways to look at it so the first part is that the ever-changing landscape so people are obviously trying to find new vulnerabilities new ways of sending amplification attacks so that's like if I send a really small query how can I get it to then reflect to my target with a larger amount of data so there was one I think was memcache which was a big sort of vulnerability that was patched I think last year there was obviously IOT devices that got compromised Mirai botnet that launched sort of layer 7 attacks so these attacks are ever-changing and they are being launched in different ways from different geos from different locations so part of our DDoS mitigation product is because it's ever-changing and our DDoS team is actually quite relatively small and they need to sleep and they can't be working 24-7 and they don't want to be sort of pulled in on call all the time so a lot of this has been automated so we actually look at and if you it's talking about the hardware that Michael alluded to earlier a lot of what we are doing is if we're looking at the network devices or CPU where we can actually drop traffic we actually want to drop it as early on as in a request as possible because if we're spinning CPU cycles or moving it down the stack then it doesn't become performance and our data centers become overloaded so what we are doing is a few things so one we have a couple of products or internal names one is dusty and one is gate bots and what they are doing is they are sampling traffic across all of our network to understand the fingerprints or weird anomalies that are happening throughout our edge throughout all of those cities and if DDoS D or gate bot detects something that's weird and a thing and it will generate a fingerprint and these will then be based a block will be pushed to our edge to start dropping traffic and that could be at layer 4, layer 3, layer 7 to mitigate it as close to the edge as possible.

With what I said about CPU cycles we also created something called IP jails where if we see a massive layer 7 sort of attack from multiple IPs globally a big botnet we have a way now of moving our sort of layer 7 up to layer 4 and dropping them at the IP barring IPs from hitting us so sort of answer the question is do we patch yes we're always on looking for new vulnerabilities or seeing new things and deploying fingerprints but a lot of it is automated and this allows us to protect all the properties that are on Cloudflare to drop traffic to drop DDoS without needing to patch or to update anything so it's a self-learning self -healing system where if someone was to launch an attack against my domain and we sort of see it and mitigate it we then have that fingerprint we then know exactly what that attack looks like we roll it across a global network and protect all of our sites so if any of you are Star Trek fans and like if you know the Borg where they sort of get killed and but they understand how to then not be killed because they've adapted to it it's the same sort of weird technology and that's how I look at it I've never heard of that analogy yeah but no I think that that's spot on Matt and so I mean in summary yes one of the mission you know the mission of Cloudflare as well was help the smaller players on the Internet and we definitely still have that community component to us where as Matt said any attacks we see across our customers we can definitely you know we definitely use that intelligence and help others and I also say like we with the reason for is building all of this cool tech is we released unmetered DDoS mitigations for everyone if you're on a free plan and you're all of a sudden you get one terabits per second five times like we will stand there and drop that traffic and we won't kick you off the network and obviously we have to get smarter and build out attack it's a war of attrition where attackers are trying to find you exploits and vulnerabilities and then we're playing you know it's a cat-and -mouse game yeah there are I know there are some you know depending on the attack some of the mitigations are part of our core you know DDoS product mitigation product I for example though some mitigations for layer 7 DDoS attacks we actually have signatures in the WAP as well right so if there is a botnet that is making layer 7 requests sometimes we do deploy some of those mitigations of layer 7 rules as the botnet might be really easy to spot yeah one of those I'm thinking key ones which we've seen is probably the WordPress ping back attack where just open WordPress servers send a info even though you're not running WordPress on your site it's just making requests go in and it was designed for hey I'm looking for an update please send it but you can actually say hey ping me from all of these services and just take you down later somebody's again we've got a WAF fingerprint that just looks for that and and we've also we've actually got a DDoS fingerprint as well now well we have to sort of drop it even earlier yeah so I guess one last thing I could add to this is as it sees of course we that's what we do on a basis is we help customers mitigate their DDoS attacks and a lot of them do get mitigated by the platform out of the box which is pretty pretty awesome and pretty impressive and we do have a bunch of tools of course that we use to essentially you know help customers it's other things like rate limiting there's bot mitigation and sometimes we do have to leverage some of those tools but all in all yes we're you know we're R&D heavy company we are always patching DDoS bypasses for websites but not only for websites right as you said earlier we protect TCP and Minecraft servers and all sorts of other things yeah and with Magic Transit our sort of network layer protection as well so yeah with any sort of DDoS yeah we have a way of trying to stand in front of it cool so Jamal has asked I want to set security level for visitors from a country to medium and low for visitor sorry and low for visitors from other countries what is the best way to do this now thank you for your question I think this is one where it'd be better for us to actually jump into the dashboard so I did read this question before and to make sure I knew what I was doing and so yes we can definitely we can definitely help with that so let me just share my screen so before we actually jump into the dashboard I think it's worth explaining quickly what the security level does so the security level is a setting that all customers have access to so if you scroll to your zone under the firewall tab there's a security level toggle and you can essentially select between a number of options which are available now what those options are are essentially a mapping to an actual threat score which is filtered on the incoming request so for example if you have your domain on Cloudflare and you go in there and you set your security level to high for every request that comes in if the threat score is greater than zero in this case I'm looking at the last row here which pretty much is through for all requests we will essentially issue a challenge request to a capture challenge request to the user so I'm not sure if everyone maybe you haven't seen this but a you know sometimes when you're browsing Cloudflare sites depending on the configuration that the webmaster has implemented you will be served a capture challenge and if you solve the capture challenge you then allowed through to the site and there's a timeout after which you may have to resolve the capture challenge again and that's also of course configurable from the Cloudflare dashboard and the reason we have this feature of course is to help some of our customers block out automated crawlers or bots as they would normally not be able to solve the capture challenge so as you can see high equates to challenging any request with a threat score greater than zero medium 14 low etc and then essentially off anything that's greater than 49 and now the threat score itself we use you know a third-party feed from project honeypot to basically give us some threat score information as things have evolved of course we're also augmenting a threat score with a lot of other intelligence that we have available within Cloudflare within the Cloudflare platform so we're essentially providing the threat score is essentially a proxy for us to providing our customers with with our intelligence now so the question is we want to challenge anything from a specific customer when the threat score is higher compared to the rest of the world as far as I understand it so let's jump in I do have a demo account here so I'm just gonna quickly move the zoom okay perfect so I'm inside of the demo account the way you would normally do this is of course by navigating to the firewall tab and then and we would actually use our newer firewall rules engine so this is something we've been building over the past a couple of years which is essentially allows our customers to be a lot more flexible in defining their firewall rules and it's very powerful you can basically write expressions and match against any part of the incoming requests and trigger an action if the filter matches your definition so in this specific case let's go ahead and try to implement what the what the question now is asking for so let's click on create a new firewall rule give it a name so what we want to do is if I'm not mistaken we want to challenge from a specific country if the threat level is medium so we actually have access to the to the threat score inside of the firewall rule engine so in this case we could do something like if the threat score is greater than and I'm gonna switch back to the medium so medium is essentially 14 and then it's coming from a specific country I'm originally from Italy so I'm gonna pick Italy as the country to challenge lots of scrupulous people from Italy yes then of course remember the threat the security level will essentially issue a capture charge okay so then we can I'm gonna save this rule as draft and then what we would do once you've once you've deployed the first rule is you'd create another rule and again you'd select the threat score sorry let's go so low is 24 and then depending on what logic you want to implement you could essentially say and country of course not equals to Italy in this in this case you could maybe decide to bypass or you could decide to do you know if you do want to challenge them then you can you can define your filters in the way you better suits your needs and you can decide maybe to create different buckets of threat scores so I could actually add another another level here which is a threat score less than or equal than 24 and then threat score greater than 10 for example then we could essentially challenge everything that is not that is not coming from Italy and now let's go back to the so that way we can basically build rules that match match the specific request in this question and once the yeah I'm not going to deploy this rule the other thing to note is that the rules itself so let me actually I'm just going to deploy it actually to show you so let's say we're gonna allow these and the rules in the dashboard actually get executed in the in the order they present themselves and you can actually toggle the order or change the order by drag-and -dropping and you have a few ordering options here in the dashboard itself so of course you'd want to put I have several rules in here and I'm going to spend too much I'm not going to spend too much time reordering the rules appropriately but you would want to put your higher security rules at the top of course so that you make sure that the higher security settings get kicking kick in before maybe some of your lower security settings as you can see here in this demo account we have about 16 rules and depending on what you're trying to achieve you would reorder these as necessary and so yes it can be done if you are not of course looking at customizing the security level on a per country basis then you can just go to the settings which I am assuming is what Jamel was using initially and you can simply choose a toggle from from the drop-down box here but that will simply equate to a rule that gets applied across your entire zone so we're just matching on every single request and then challenging anything that has a threat level higher the medium or high etc I think that answers the question yeah totally I think the other thing I would say is I security level in page rules can also be updated if you wanted to do it for a specific path if you want it to be more aggressive or less aggressive but yeah rather than doing across the whole zone yeah so next question we have is from Beth and she's basically asking us what integrations do we have with Qradar and Matt I don't maybe we can explain what Qradar is so Qradar and this is being put on the spot I is a scene platform I have a logging platform and I think is it by IBM or have I yes IBM IBM so IBM have a way of having logs which could be from any of your applications like the nginx could be your Apache logs could be feeds from other providers could be anything that goes in and then you create analytics from that so the first part is do we have a direct integration with them currently not yet so again I'm going to share my screen so as we keep probably so the developers docs are one of the new places where we have a lot of information about all of our products and one of those is actually locks so with the enterprise plan we have way we have a way of pushing logs from our the arrive at our edge to an endpoint so this could be a Amazon s3 bucket could be a GCP book it or is your blog really simple way to configure and to deploy locks now with that we actually worked with a number of scene providers and so you can see here we've got data dog we've got elastic that allowed us to create ways of if you are using these products to add logging for Cloudflare so simple ways you know you can with pre-configured dashboards and you can get all the nice look and feel of what that is simply without and you don't have to create this now this is great if as we said if you are running sort of an elk stack if you're running gray log look at all these now another thing that you can do is if you don't want us to push logs we can you can also use our API so we have a local API that allows you to download the specific logs for the zone and that will then you can save that to somewhere that you're running say Linux or it can be a hard drive on a machine could be something in the cloud could be something on-premise and once you have these logs and these logs are pulled in JSON format and you can see here what sort of that looks like once we get that information you can then pipe it directly to your scene so if I just quickly log in I wasn't expecting a live scene scene demo you're asking permission from a demo goats the demo goes yeah live demo is always the best and this is just running on a slow GCP instance that hasn't scaled so my boss and budget cuts me on my GCP usage so as you can see it's spinning but I wanted to hopefully show and this is my live demos always go horribly wrong is all the different fields that you can have access to so once you get this JSON blob and Q radar I believe can pull in that JSON blob you then get access to the fields where you can customize your own dashboards so with the enterprise logging and I know this self serve customers are probably really wanting to get their hands on this you can get really details on every request so path how many bytes it was served user agents what colo and classified colors served it and from this you can then build those dashboards so even though we don't have a direct integration with Q radar and I want to say I think there is a lot of requests for this for Q radar so it may come soon but don't hold me to that there is ways of pulling logs down and then pushing it to it direct use and get it to read the JSON and then that means you can create the nice dashboards within that so that allows you to yeah just drill down into what's really important and see what's happening from a security viewpoint is the WAF being triggered is it something else so not yet but hopefully coming soon but there is workarounds to that and with all of my customers like developers.Cloudflare.com is a great resource I would definitely look to bookmark that and there is some interesting things in there about all of our products and not just the logging yeah I mean from my experience just like Matt said pretty much all seems will be able to import JSON formatted logs so regardless you know the integrations we do provide are essentially templated dashboards but actually a lot of our customers do not use those they build their own dashboards and importing literally all you need to do is point the seam to the place where you're storing location the path where you're storing the logs and they should be able to import them out of the box and then you can build your dashboard filter which builds you want etc etc etc and it's it's the same with pretty much all teams yeah and the other thing I will add is obviously that's sort of enterprising we are and I think there's a lot of releases from the firewall analytics down to the caching analytics and sort of improvements are coming soon for the people that don't have access to this that our actual platform within the UI will be able to give you a lot more verbose information for you to understand and assimilate without needing the likes of Splunk, Qradar or Kibana but yeah it's obviously a good thing to have integrations with those if you are sort of using them awesome cool so one this is actually a really common question and this comes in from like we have to an inbound on sort of the enterprise form so this is from Wesley and it's in my nginx logs I only see Cloudflare IPs how do I see the actual client IP address good question Wesley I would say actually this is common to everyone I'm a Cloudflare user myself not on the enterprise I abuse the free plan as much as I can because it's just too good so I actually had to do this recently myself and so Cloudflare is a proxy of course in the in the normal use case where you're putting a website behind Cloudflare we're proxying HTTP and HTTPS traffic so in your if you're using Apache or using nginx or any other web server indeed by default if you were using your logs to spot malicious IPs you will no longer see those IPs in the logs themselves you will start seeing Cloudflare IP addresses instead because we we reinitiate a new HTTP connection from our point of presence back to your origin when we're proxying traffic to you and so this can cause some initial confusion especially when you're looking at the logs and you'll see that the request will be coming from all sorts of different IPs depending on your traffic on your customers patterns or your users patterns it will be a wider set or a smaller set you know if you have very localized traffic you may see a smaller set what what everyone can do though and this is something that we've implemented it's really easy to do of course is replace the Cloudflare IPs in your web server logs with the original end user client IP address now there's two ways that we provide this information first and foremost Cloudflare will add the X forwarded for HTTP request header that header may also we may have also received that header when we saw the request if the end user is proxying via some other service in front of Cloudflare and but in that header you will see essentially the client the actual client IP address and we will append you know our IP address to that as well as as as the request goes through but a different number of proxies however that is not the recommended field to use the recommended field to use is an additional header which we add specifically and it's a Cloudflare base header which is called CF connecting IP and you can essentially rely on that header and it also will never be spoofed because we are setting that header at our edge and it will only include one IP address it could be bear in mind it could be an IPv4 or an IPv6 address depending on you know how the customer is connecting to us and but the IP address will be the customers IP address so second question is okay so you have this information the web server on your side and your origin can essentially read this header how do you make it how you should make it show up in the actual web server logs now I'm gonna quickly we're not going to be configuring this but I'll just show the support page because our support team has actually done a pretty good job at documenting how to restore the IP address based on what web server you're using so I would advise simply just Google for restoring original visitor IPs and every different web server has a different way of doing this it's very simple so we do have all the most popular web servers if the web server you're not you're using is not in this list I'm sure eventually we'll add it as long as it's popular and it's not something you've built for for yourself of course and it normally equates to installing a module or something similar which allows you to basically specify which HTTP header the server can find the client IP address and then simply replacing that in the logs themselves so if we look at Apache and there's only a couple of commands you need to run and we've actually built a an Apache module which helps you to do this very easily it's called modclover for those of you who are familiar with Apache you sort of in the config file you load different module depending on what modules depending on what you're trying to achieve once you've installed modclover it comes pre -configured with all the definitions that look at the IP address from the relevant request header and then once you've got it running you restart your Apache server and then you can sort of tell your logs and your logs will now have the original client IP address rather than the Cloudflare IP address.

The one thing to note which I mentioned earlier for those of you who are you know we're originally serving your origins over IPv4 only bear in mind that by default Cloudflare is also advertising IPv6 at the edge so if you are parsing your logs and you're expecting IPv4 only once you install this bear in mind you may see IPv6 popping up in there so sometimes we've had some cases where all of a sudden customers customer log parsing scripts were not working as expected because we were you know essentially the client IP address for us was an IPv6 not an IPv4 you can turn that off at the Cloudflare edge you can you know force us to only be serving your traffic over IPv4 I highly advise you against that that's not towards building a better Internet helping build a better Internet but it's it is an option that is available if you so if you need it and I think that answers the question so in summary it's easy to do it's a couple of commands depending on the web server and just Google restoring original visitor IP and you should be able to find the web server you're using and if not it should be pretty straightforward to restore that IP address.

Perfect so that was Wesley hopefully that answers your question next question is from Colin so Matthew over to you and the question states I think this is an interesting one hi Cloudflare I just signed up to Cloudflare and wanted to know what are the best practices for protecting my origin I'm going to assume this is going to be generally speaking from DDoS attacks and you know hackers or any other web vulnerability so yeah are there any best practices?

So from the origin side so the first thing that you want to do now let you stop sharing your screen is you want to make sure that only Cloudflare can really phone home so when we onboard customers we have we always give them and it's Cloudflare.com and it's slash IPs so these are the IP ranges of Cloudflare and these are the only ones that if you are proxying all the traffic through your domain and it's all going it all should be orange clouded these are the only the IP addresses that should be hitting your origin anything else that's trying to circumnavigate and trying to directly connect you should try to drop so customers would update their IP tables to drop this or only to allow traffic from these IPs so that's definitely point number one if you have been suffering a DDoS attack or you suffered one previously we also have we do recommend trying to rotate your IP address at the origin once you have this set up reason being for that is there is websites or there is crawlers out there that look for history of IP addresses to host names and it can easily be used to find what the IP address was before Cloudflare and try and attack you around the side so definitely number one update your ACLs your IP tables to drop traffic unless it's from these IP addresses if there is anything that's trying to let does connect directly to the IP of your origin could be monitoring could be something different could be a pen tester obviously you would have to allow them free but good practice is to only allow these now we have one of this and one product that may be of interest and it's something called Argo tunnel so again on the developers documents that are here Argo tunnel allows you to install a lightweight daemon on the origin so this can be installed on Windows could be installed on Linux distributions on Mac and what this and this lightweight daemon does it allows you to authenticate with your account and then spin up a secure tunnel that is from your origin directly back to Cloudflare so it will auto provision a DNS record so if you have my account test our org and you say that I want that to then connect to my local host on port 80 or whatever you're running on the application this can easily dial out and create that secure connection and that will then punch through firewalls and you can drop all traffic and almost make your host undiscoverable because you are having a direct communication with Cloudflare and only traffic can come through Cloudflare's edge so this is one that if you want to get really hardcore on the protection is something that we would recommend and this can also integrate with the Cloudflare load balancer so you can add within setting that up with load balancer and if you are if you are doing things with kubernetes which is probably a bit out of my I would probably hand this over to another essay as soon as someone mentions kubernetes there is ways of create using our ingress controller or even doing a sidecar model with this to auto every time you provision a new kubernetes cluster pod that and I've probably messed up the terminology again it will be able to auto dial out and once it you it deploys and connect to our load balancer and start accepting traffic so this is definitely one if you are doing something kubernetes DevOps environment but also as a general just securing from Cloudflare's edge to your origin definitely something to look into the other thing that I would recommend and if I go back to my domain and this is just sort of making sure traffic is always provisioned over it's always sent over HTTPS so within the SSL TLS tab you can select an encryption mode mine's currently on full but we'd look to do full strict so this would mean that a connection coming in over HTTPS is then over HTTPS Cloudflare but the connection back to your origin is also over another TLS connection so full strict means you and full means you have to have a certificate on your origin and you can provision these with let's encrypt if you use a self-signed or entrusted CA you select full or you can actually use a Cloudflare provision certificate so on the origin server tab you can actually create a certificate and deploy it to your origin and then we'll use that to negotiate TLS and you can then have end-to-end encryption and again another best practice to sort securing origin and to securing traffic there is a load of things we could do about setting up but again we may be another session but for yeah for just basics IPs blacklist whitelist only allow Cloudflare IPs and look to set HTTPS end -to-end yeah so I guess to the three steps I for from my perspective just like Matt said with the goal when you're putting your origin behind Cloudflare is to hide your origin so nothing gets to you directly and so set up Cloudflare make sure all of your traffic is proxying behind Cloudflare and then one thing as Matt said is change your origin IP address once you know that everything's coming from Cloudflare because attackers will often try to scan your IP ranges and look for you as long as you change your IP address after you're behind Cloudflare and you implement firewall rules that only allow traffic coming from Cloudflare or if you're using Argo tunnel you can just shut off the firewall completely you should be good to go and that allows you to then deploy all your configurations awesome yeah so just cool so the next one from Flora and it's about 1.1.1.1 app or quad one as I abbreviate to is there a limit to how many DNS queries per second now I'm guessing that's that you can make to the resolver or just to our DNS infrastructure so thank you for your question this is a good one so for those of you who I guess a bit of context Cloudflare the way we provision we allow Cloudflare to be provisioned historically is by a DNS so that means if you're wanting to use Cloudflare you add your domain to Cloudflare you configure your domain to be the authoritative name server for your domain and then depending on if you're wanting us to proxy your traffic or not we will respond with Cloudflare IP addresses to DNS queries versus your origin IP addresses so what that meant is over the years we've actually built an extremely strong DNS server and we're really good at running DNS services and so you know as of today we're running more than you know 20 million applications on our platform and we tend to be the authority DNS server for most of those a couple of years ago we said you know why don't we leverage our our DNS infrastructure to also offer DNS resolver so DNS resolver is what you'd configure within your home network potentially or you know within your phone or your laptop to basically tell your laptop to use the specific DNS resolver to then do it a DNS resolution when you're connecting into various websites around the web and we've launched quad one so 1.1.1.1 it's a free public service anyone can use it for who is familiar with Google's 8080808 is that is an equivalent very similar service of course but it's powered by cloud front and gets all the you know fancy advantages you know of leveraging large network etc etc etc now because our infrastructure is so big and we do have so much capacity essentially to answer your question is no there is no limit to the number of DNS queries you can send towards 1.1.1.1 now having said that of course if you start using the service in a way which does not you know does not make sense or slightly out of the norm sometimes you may risk triggering some of our mitigation mitigation features which are applied to our own services as well but you you know I wouldn't I would say most people will never actually have to worry about hitting any of those mitigation if you have a really specific use case and you're sort of deploying our public resolver to millions of users then maybe you know reach out send us an email we can double -check everything is in line and I do actually when I just before when Matthew was replying to the question the prior question I looked up what our what our official stats is in terms of how many DNS queries per month we we we respond and essentially we approximately do 4.6 million DNS queries per second or better that is 400 billion queries per day so the number is huge unless you think you are going to make a meaningful impact to that number and again we have a lot of spare capacity to go beyond that then I wouldn't necessarily worry about limits on our public DNS resolver so yeah so go ahead deploy it use it it's it's really fast as well and hopefully you you enjoy it and if you notice anything that's not working correctly then for sure just ping us send us an email and we'll get back to you so thank you Flora for the question the next question we have five minutes left I think we can maybe fit in one last question before we close this off so Matthew keep it don't don't let's keep it somewhat short so we didn't get a name from this person and the the question another question we receive pretty often actually is when I visit Cloudflare I assume a website behind Cloudflare why does my request not hit the closest data center in in Dubai for the specific user so yeah what does that happen yeah and so let's start off by dispelling a myth and that is that we don't like plans that are announced like the free plan pro plan biz plan like and and we announced from all the pubs we only reroute data centers and you can see that on the status page if there is upgrades if there is there's an issue with connectivity and then we will sort of drop traffic out and sort of leave the paid plans in there if this is sort of a constant thing that you're not hitting the data center it's really about the architecture that Michael alluded to so we run an anycast network that we announced the same IPs across all of our network infrastructure and when you make a request to connect to an IP we are at the mercy of the routes that the ISP will take so an ISP can do a number of things it could either directly connect if we're pairing with them it makes sense for them to send it directly to the data center in Dubai if we're not pairing with them what they would tend to do is a couple of things one could be a some ISPs look for the cheapest option for them or others look for the best connectivity that they have so that could be where all these Internet exchanges are so within Europe you will see a lot of traffic hit the London data center you will see it hit the German data center and that is because there is good Internet exchange links there and even though that the the actual geo of how long it takes to do that it's probably actually quicker than going to a pop and being handed off to another one even as a crow flies so the thing that I would say is that if you're ISP and you wanted the ISP to pair with Cloudflare and they're not pairing email them try and create some traction maybe Twitter to say hey could you play with Cloudflare and try and drive that traffic directly to the closest point of presence but as I said there may be sort of issues where ISPs are looking for where they are best connected with and where they need to hand traffic over to get back to I know an origin that may be in Dublin or an origin that may be in Frankfurt and it makes sense for them to send it specifically back to data centers within Europe so we try our best but really we are at the mercy of an ISP on that first hop where they decide to transfer the traffic but it isn't because we are not announcing zones or from like limiting free plans from being announced from them it is down to sort of peering and things like that if something has changed where you were always hitting Dubai and it's gone to somewhere else you can raise a support ticket support at Cloudflare.com and sort of send a traceroute and the information like the more information and then that can be passed to the network team and they can check and debug if there is any issues but yeah that's sort of the end of the thing yeah go ahead I was gonna say I think we've got 50 seconds left so I'll probably just yeah round it up now I think thank you for everyone joining even though probably it was just my parents but yeah maybe maybe we'll do this again if you have more questions please reach out and use the forms tweet us and like yeah would hopefully do this again so thank you yeah we have received a few more questions during the stream so we'll try to answer them at our next our next session so keep an eye out so now yeah I'm we're gonna hand it over so to Haiti Andrew and Anthony and I'm gonna do everyone at the table and so again lots of cool viewing lots of different things hopefully see you again hopefully we don't get kicked off I think this is any worthy yeah thanks bye bye thank you very much