Originally aired on January 28 @ 9:00 AM - 9:30 AM EDT
Watch Cloudflare’s Product Manager Christian Draper and Director Product Marketing Ameet Naik talk about how to monitor your network with network flow analytics.
Read the blog post:
Visit the Birthday Week Hub for every announcement and CFTV episode — check back all week for more!
Hi everyone. Good morning, Good afternoon. Good evening, wherever you are. Thanks for joining us today on this session of Cloudflare TV, where we're going to be talking about Magic Network monitoring. My name is Ameet Naik. I'm a director of product marketing here at Cloudflare. And I'm very excited to have Chris Draper join me. Chris, why don't you give us a quick intro? Yeah, so my name is Chris Draper. I'm a new PM at Cloudflare, joined Cloudflare about six months ago and I'm the product manager for the Magic Visibility team. The goal of that team is to give customers more insight into their network or Cloudflare network if they're using Cloudflare as their own network. And I'm super excited about our product announcement today. So Chris, I know you've been you've been here for a little while and you've been cooking up some stuff, and I just sort of sort of saw the demo doing a dry run earlier. I'm sure everyone's excited to see that. But before we jump in, I just want to kind of preface this topic a little bit. Right? So we're going to be talking about network monitoring. So fundamentally, if you're a network engineer or network security engineer, you want to know what's going on in a network, right? You want to know what's congested with you want to do some capacity planning. You want to know how much traffic in networks gathering. Where are the chokepoints, where are the bottlenecks? And you want to kind of understand what good looks like, right? What is a typical baseline look like? Because, you know, on a good day when everything's working, it's fine, right? But the minute you have an issue with an application, the number one team that gets blamed is the network team. It's always the network's fault. Right. And we want to be able to sort of get into those situations. We want to be able to look at the network and understand what's going on, what's changed, and whether it's really the network's fault. So before we dive into an announcement today, so tell us a little bit about the field of network monitoring, why it's important and what are some of the techniques that are used in today? Yeah, absolutely. Happy to kind of dive into it. So obviously, network monitoring is important overall because networks are very dynamic systems. You know, they're changing every day and the general weather of the Internet can change on a dime. So the importance of network monitoring is making sure that all of your systems are running as expected, whether you're an e-commerce company or you're a SaaS company, or you need to run some internal services that rely on the Internet, it's always really important to make sure they're operating as expected. In terms of network monitoring, there's really two categories. The first category is active monitoring. So that's when you're creating synthetic traffic, synthetic probes and sending them through your system to actively monitor whether your systems are performing as expected. If you're an e-commerce site, maybe you have a sort of login. You'll set up a synthetic probe that logs into your e commerce website every 5 to 10 minutes to make sure it's working. There's also passive monitoring. Passive monitoring is about collecting a lot of statistics, storing them in some sort of database and having them available for network engineers to review or build alerts on. Some classic examples of passive monitoring are like S&P pulling, pinging all your devices, making sure that everything is working as expected. Everything is up and running and can reach to your central network. Another example of passive monitoring is flow based monitoring. Flow based monitoring is when you basically summarize your traffic at your router and then you can actually send that traffic summary to an external IP address, for example, Cloudflare. And that's what Magic Network monitoring is actually built on top of. So you give us sort of a quick sort of intro into what Magic Network monitoring is, but tell us a little bit more about what is the announcement today? What are we announcing? What's what's on offer and what can our users do with it? Yeah, So today I'm super excited to announce that we are offering a free network flow analytics tool to everyone. All you have to do is create a Cloudflare account and we are putting this product into early access. So right now, in order to be able to use a product, you can actually just apply for early access via a Google form and I'll reach out to you and get you all set up. This is super exciting. It's really important to offer amazing network and security tools to customers. Cloudflare's main mission is building a better Internet, and a great way to do that is making sure that everyone has the network visibility that they need to be more secure. And we're hoping that the free version of Magic Network monitoring will make sure that everyone can do that. So if I'm a current user of Cloudflare, I can just go into the dash and then request early access and I'll be able to use this right away. Yeah. Point My router is to send NetFlow traffic do Cloudflare and then what sort of insights will I be able to see on the dashboard? Yeah. So in the dashboard we're collecting a lot of different data. The top metrics that we're collecting are going to be things like network traffic volume over time, the different protocols that are going through your network. Source IP destination IP Source port. Destination port as well as which routers are sending your data to your various destinations within your network. And this is really exciting because one of the one of the things I like about flow monitoring is it doesn't just tell you that this traffic on your network. It tells you what that traffic actually is. Right. And the only other way to get this information is to do like do tap ports or or install traffic analyzers or go actually get into the plumbing of your network and actually do that, which is painful enough to do with a network that you own because it may be in a different part of part of the world from where you need what you need to monitor. But it's almost it's next to impossible for if you're using the public cloud, for example. Right. So I think flow monitoring is a really good way to get insights and visibility from many different types of networks, whether it's your networks, whether it's a third party network. And I think most of the public cloud services now let you export flow records on what's going on, on their end and get some insights that way. So I think this is being able to just point the net flow to Cloudflare and be able to see all the insights. And the dashboard is really powerful. And you said it's just a just an early access signup form on the blog. So if you go to the blog post at the very end, the very first paragraph, you'll see that there's a Google form or you can sign up for early access. You can also request early access by emailing MNM, which is Magic Network Monitoring. So it's M as in Mike and N as in November and M as in Mike at Cloudflare dot com. And we're happy to set you up with early access. That's awesome. And I'm sure that mailbox is filling up already since we announced the blog post. Yeah, super excited. We've had a ton of requests so far and we can't wait to give this product away for free to everyone. Yeah. Yeah. This is exciting and I would love for everyone to get in and try it and give us feedback. I use the product. Tell us what you can do with it. But I wanted to ask you, what are some of the questions and use cases and problems users can solve with something like Magic Network Monitoring? Yeah, absolutely. So in terms of Magic Network mMonitoring, I feel like there are like three categories of example questions that I have like off the top of my head that network engineers might run into on a day to day basis. The three general categories I have are like network capacity planning, network and security forensics, and then like basic network security hygiene. And I want to take a little bit to talk about each of those and talk about how Magic Network Monitoring can kind of give you data to solve problems in those categories. So one of the great things about Magic Network Monitoring is it will match, it will measure your traffic volume over time. So let's say you're a Cloudflare customer and you're running an e commerce website and you notice that your traffic to that e commerce website is increasing really significantly. You can open the Magic Network Monitoring dashboard, look at your traffic volume metrics and see, hey, my peak traffic typically occurs on Monday, Tuesday, Wednesday, in the evenings when people are off work and you can kind of plan your network capacity and make sure you have enough servers and everything that you need to be able to serve those customers efficiently. So Magic Network Monitoring can kind of help you do some capacity planning and some general traffic volume trends over time, which is super useful. And capacity planning is an interesting one, right? Like the nature of traffic, IP traffic is it tends to be bursty. It's very rarely even unless it's streaming video or something like that. And then you want to you want to make sure that you have enough headroom on your on your pipes to accommodate those bursts. Otherwise it results in sort of bad user experience and buffering and application performance takes a hit. So I think it's important to stay ahead of that and know where the chokepoints are, know what your traffic patterns are, how much traffic, what the distribution is through the course of the day. Right. And just to help you make better planning decisions and investment decisions and what you need to build out and where. Yeah, absolutely. The next category is network and security forensics. So let's say you're in a bad situation where a malicious actor has access to your network. You're going to have a lot of questions like what was the source IP of that malicious actor? What destinations did they access, what resources on my network that they break into and how much data did they take from those destinations? Magic Network monitoring is really well equipped to be able to answer questions like that. So if you know that someone malicious got into your network from 2:00 am to 4 am in the morning UTC, you can go and check their source IP, see all the destination IP that they accessed, and then measure the exact amount of data that they took out of your network. I guess it's not necessarily the exact amount of data because we're doing some sampling to do some estimations, but I think it can be a really helpful tool in that regards and it's really easy to use. I think it's great for that use case. Yeah, And what's nice about this is you can actually get flow data from almost every network device out there, right? Yeah. And from the public cloud. So. Even if you're not there, you don't have to put tap words and network. You can sort of go and log in remotely, configure a few lines of code, and then Cloudflare can start receiving all that data and help with the analytics. Yeah. Magic Network Monitoring has support for both NetFlow and S4 customers, which is super exciting. We have great coverage. We cover NetFlow V5, NetFlow V9, IP Fix as well as sFlow. We're getting support for Velocity and so keep posted on that. But we're really excited. I think the days of realizing that you need more network visibility and calling up a network engineer to go install a network tap at some data center, or having to call up a data center and pay a ridiculous fee for smart hands to install some device are long gone. I think what's really cool about matching network monitoring as well as any other like flow based monitoring tool is that you can set it up very, very quickly, nearly instantly, and start receiving a ton of network visibility all at once, which is very exciting. All you have to do is remotely log into your router and configure your NetFlow or sFlow. Cool. And then one more use case you mentioned was around basic security hygiene, right. Like identifying boards that might be left open that you did not expect. And that's the bane of most security admins our lives, right? Yeah. So I think that's like a classic example. Of. Some developer was working on your systems. They opened a port for testing, they went home or they forgot to close it. Lots of unexpected things can happen if you leave a port open to the internet. Magic Network. Monitoring is really cool because it will show the amount of traffic that is going to specific destination ports on your network and it will give you a port by port breakdown. So if you're seeing a port that you didn't really expect to be receiving traffic, you can then further investigate, hey, like why is this happening and why is this port all of a sudden exposed to the Internet? You know, it's I think it's just creates a lot of peace of mind to be able to have good visibility into all of the different ports that are open to the Internet and making sure that that's locked down as expected. And then the one last but not the least is another really important use case for flow monitoring that a lot of folks use it for, and that's to help identify DDoS attacks. Tell us a little bit about what is DDoS and why it's every network engineer's nightmare. Yeah, so a DDoS attack is a distributed denial of service attack and they have been rising in popularity quite rapidly over the last couple of years. Cloudflare does like a quarterly report on trends and one of the things we noticed is that quarter over quarter layer three and layer four DDoS attacks increased by 100% and 80% respectively, which is just off the charts. So DDoS attacks cause a ton of problems. They can prevent you from offering the services that your company needs to operate, whether that's to the public because you're an e-commerce website or whether you have some internal tools that are built on top of the Internet. They're just extremely disruptive to regular business operations, let alone the cost to actually have to pull your network and security team away from what they would typically be working on to kind of manage a fire and figure out how to protect themselves. So I think it's a rising problem. Fortunately, Cloudflare has some amazing DDoS protection tools and we can make sure that everyone saves time and money. It doesn't have to worry about details in the future. Yeah, I mean, one of the cool things about Cloudflare is if you have your website on Cloudflare or web application or API automatically getting DDoS protection, right? And you don't have to do anything. We are constantly defending the network and everything. We run against DDoS attacks, but one of the cool things is we also have a product where customers can actually put the Cloudflare network in front of their network and the between their network in the internet as a bouncer and actually get the benefit of cloud protection for themselves. Right. It's also called Magic Transit. Tell us a little bit about that. Yeah. So Magic Transit is an amazing product and it's the industry leading DDoS mitigation solution, which is awesome. So I think part of the power of Magic Transit is, like you said, putting Cloudflare network in front of your own network. Cloudflare runs a massive globally distributed network. We're located in 275 cities and we have over 155 terabits per second of capacity that our network can handle. When you have a network of that size, obviously you will attract some relatively large attackers and Cloudflare is mitigated to date some of the largest DDoS attacks that the world has ever seen, which is definitely not a small claim. I think if you become a metro transit customer and you put magic transit in front of your origins to make sure that your networks can be protected, probably the best protection that you can buy today, which is pretty cool. That's awesome. You know, we always recommend running Magic Transit and Always-on mode, right? Having it there all the time so that Cloudflare can detect and start mitigation and under as soon as under 3 seconds. And that's why Always-on is really important. But a lot of customers historically have been using DDoS protection using Scrubbing center based model, right where you have to in order to activate protection. You have to send all your traffic to a scrubbing center and that means more latency, worse application performance, more complaints. And as a but I mean that's the model that some people want to continue to use and we offer both. We always recommend always on where we offer both. But one of the things that we always need and on demand mitigation is you need a way to signal the system, to signal Cloudflare that, hey, there is something potentially bad going on kick in mitigation, right? And one of the best ways to do that is using data from flow monitoring. Is this something that Magic Network Monitoring can help our customers with? Yeah, absolutely. So I'm glad you brought up Magic Transit on demand. I think oftentimes Magic Transit on-demand customers have one primary question, which is when should I enable Magic Transit on demand? It's great that they have a button, but they need to know when to use it. Magic Network Monitoring is great whether you're an enterprise customer or you're a free customer. You can set up network traffic volume alerts for specific destination IP prefixes or just like destination IP in and of itself. What that means is if you have a publicly facing web server and it typically experience 500 megabytes per second of traffic, you can set a threshold at 1.5 gigabytes per second and say, Hey, if I start receiving traffic volume over this threshold, something that's 3x the average that I typically see, it's pretty likely that you're being dosed. And it's not just like a sudden jump in customer traffic. That's a really good signal that it's probably time to activate magic transit on demand. The other great thing about Magic Network monitoring is that it'll provide some data that is commonly used to identify DDoS attacks. So for example, we have a breakdown of TCP flags in the dashboard, and I'll show you that really soon in the demo. But that's going to be great for identifying really common attacks like SYNFlood attack floods. And that's that's awesome. I know we've been talking for a while, but I'm sure our viewers are actually excited to see what this looks like. So why don't we jump in to the demo? Yeah, I'm happy to do that. The other thing I wanted to quickly mention is that Magic Network Monitoring and Magic Transit on demand are a great combo. You can actually configure your Magic Network Monitoring alerts to automatically enable magic transit on demand, which is super exciting. It's a feature called Auto Advertisement, and I'll point it out in the dashboard when we do the demo. The idea of auto advertisement is that there are small security teams and they don't want to get woken up at two or 3 a.m. to be able to tap to mitigate yet another DOS attack. The beauty of Magic Transit on demand being automatically activated is that if Magic Network Monitoring detects a DDoS attack, it will automatically activate Cloudflare mitigation system, which is magic transit. You can sleep at night and then wake up in the morning, see that Magic Transit is activated. Review the data in Magic Network Monitoring dashboard and then take action from there. So I love nothing more than seeing an incident that was already taken care of. Yep. It's as nice as it can get. All right. So let's jump into the demo and let's take a look at Magic Network Monitoring. Yeah, let me share my screen here. Awesome. So this is the Magic Network Monitoring dashboard. And can you see this? Okay. I can see this. Okay. Yes. Awesome. So the way that you get to this dashboard is you're going to go to the account level in the MetroTransit dashboard. Then you're going to go down into the analytics menu and click Magic Monitoring, which is right here at the bottom, and it's marked as beta product. So you'll be taken to this dashboard where you can view all of your wonderful network data. So up at the top we have traffic volume over time here we can see that on average our network is experiencing about 1.2 megabits per second of traffic. You can even zoom in to view a more detailed traffic profile if you have a specific incident. And you can see here how the time frame will adjust when you zoom in after that, we have traffic by protocol here. You can see that on our test account we're receiving a variety of traffic, whether it be ESP, GRE, TCP, ICMP, or UDP. Additionally, we have a breakdown of traffic by source and source port as well. So for example, if you wanted to say, hey, I think that there's a particular person that may be malicious and you want to be able to filter by all the traffic that they're sending to your network, you can click this filter button. The filter will appear at the top of the dashboard and then you can see that this source IP is sending exclusively GRB traffic. And then you could see the amount of traffic that they're sending to you over time. I love this ability to kind of do quick drill downs and I automatically have filters just by clicking on an IP address. Yeah, it's really nice. I think one of the advantages of just like the Cloudflare UI in general and something that sets Cloudflare apart is the ability to drill down into very specific sets of data, really understand a scenario at a very low detail level, and then click a couple of buttons and zoom back up and get an overall perspective. I think our front end engineers are amazing. They did a great job of building the dashboard. Awesome. So moving on. I think one. Of the things that you'll see here, too, is that there's a surf of zero that's for traffic that typically doesn't have a source defined. An example of that can be like any ICMP traffic or any traffic that's been encapsulated. So if you wanted to filter by, let's say we're looking at a specific destination and you want to see for a specific source for that filter, let's say you want to filter by a specific source port, you can kind of get a breakdown and see. Okay, so the majority of my traffic that's coming to me that doesn't have any source port specified is ISP traffic. And gray traffic goes a little bit of ICMP, which is always good to know. Additionally, you can filter traffic by destination. So if you have a particular web server and you want to say, Hey, you know, how much traffic volume is that web server receiving, it's super easy to be able to filter by that and get a breakdown of your traffic data. Finally, we have traffic by TCP flags. I think one of the really valuable things in terms of TCP flags is there are lots of common DOS attacks, like we mentioned earlier, in particular syn floods and floods. And I think those types of attacks will be super easy to identify because you can very simply filter by specific TCP traffic to be able to get a breakdown of it. That's a really interesting one because, I mean, you mentioned earlier, we do these quarterly trends reports and almost every quarter, the vast majority of volume traffic that we get is stills in floods. So that still remains a dominant vector. And I think part of the reason for that is, is a lot of toolkits available out there that anybody can use to launch SYN flood attack. So so people are using them and testing them and playing with them and sometimes with malicious intent and sometimes it's like, do see what they can do or very often also do probe network and probe infrastructure to figure out what's there or what's vulnerable so that they can take advantage of it later on. Yeah, absolutely. And so let's say in this scenario, you get a Magic Network Monitoring alert. You're not exactly sure if it's a DDoS attack you're going to start investigating. You know, you see a huge increase in the number of packets that you're getting. For example, you can filter by act packets. And then, hey, if you see that there's a really small number of sources that are sending a relatively large number of active packets to your network, it's pretty likely that you're being dosed. And that's a great signal that you should activate magic transit on demand or figure out how to update your firewall policies so that you can block those malicious users. Awesome. And then the final thing in the dashboard is you can also sort traffic by or source. This is going to be particularly useful as you're trying to set up your configuration. It's just going to be important to know whether or not your routers are successfully sending NetFlow or SFO data to Cloudflare. Awesome. So one of the other really important things is configuring Magic Network, Monitoring Traffic volume alerts. I think traffic volume alerts in particular are interesting. I'm sure every network and security engineer has had the experience where you set up a series of alerts. They're a little bit noisy at first. You have to fine tune the thresholds or maybe you're not getting enough alerts and you have to adjust the thresholds over time. I think this dashboard in particular is really built towards that use case. They're kind of adjusting thresholds and finding the happy medium so that you receive alerts when it matters most. So one of the easy things that you can do is kind of like look at your traffic in this dashboard and then switch over to the configure Magic network monitoring screen. Here you can configure specific rules. So for example, this new rule is for this specific IP prefix that's a slash 32. And then you can go into edit and you can update the rule name. Let's say this destination is in a particular data center. It's really easy to update the rule names that they're easy to sort through and easy to identify. You can set a threshold, so let's say you set your threshold a little bit high. You're not. You've got a DDoS attack, but you didn't really get an alert. It's easy for you to lower that threshold so that you can make sure that you're getting the right alerts. The rule duration is also easy to set. So that's going to be I want to see a specific volume of traffic over a specific period of time. In this case, let's update the rule duration to 60 seconds and then obviously the IP prefixes that fall underneath that rule. You can put a slash 32 up to a slash 24 or even a slash 16 into this as well. And you can combine multiple IP prefixes across your network and do a single rule. So when you configure a Rule four alert rule, where do the alerts typically go? Yeah, so alerts will typically be sent to your email address, but we also offer webhook alerting as well as pagerduty alerting. I think webhook alerting and pagerduty alerting in specific are really popular. That's going to make it easy to get alerts in a Slack channel or to whatever pagerduty configuration and whatever rotation system your team is running. And WebEx is really flexible, right? And it really allows you to build some automated workflows, trigger a ticket, trigger an incident trigger page, and maybe even sort of automate some steps in the process, right? Yeah. Absolutely. And then the final thing that's worth really quickly mentioning as well is there's also a configure routers screen. This is going to make it super easy to add or subtract routers from your network. So you register a specific router with Cloudflare and that router is public IP address is matched with your Cloudflare account. So when we receive sFlow or NetFlow packets from that router, we're going to match them to your account and know that that data should be displayed in your Magic Network Monitoring Analytics dashboard. I think a lot of times customers are going to run proof of concepts with only a handful of routers, one or two. When they do that, they're going to kind of want to test it out, see how things go, and then they're going to want to add additional routers over time. If you're a network hobbyist, maybe you only have one router, and that's kind of as far as you go in terms of configuration for your larger enterprise customer. Maybe you have 30 or 40 routers across your global network and this is going to make it super easy to onboard all of those routers. And Chris, this UI is easy enough, but the one question I had is I see the link for the API documentation in there. Can users configure provisioned all of this to an API? Yeah. So I think Cloudflare in general is an API first company and we want to make it super easy for people to write automations, to update rules and configurations as needed, especially if you have a lot of rules or a lot of routers. So yes, you can change all of these settings through the Cloudflare API. So you can in theory do all of the setup rules, set up, configure out or set up alert rules. Consumer alert rules do it all through the API and automate the workflow without ever logging into the dashboard. Yeah, exactly. Now. That's really great. Now, this is a really great demo. I'm sure a lot of our users and viewers are itching to try this out. But before we kind of get go there, one question I have is flow monitoring is not new. What's new about Magic Network Monitoring? Right. And why should customers choose this or this? Yeah. So what I've seen in a lot of existing flow monitoring dashboards is that you kind of have two issues. One issue is it can be very difficult to drill down into very specific sets of data. I think the way that these easy access filters are built into the Cloudflare dashboard makes it very, very easy to drill down into data and see exactly what happened in that specific scenario. The other thing that I've noticed is that on the flip side, if you want very custom views and the ability to drill down into very specific data, you need to learn a separate query language to be able to build all of those dashboard components and to make them interactive. Cloudflare dashboard comes right out of the gate with a lot of really interactive dashboard components and dashboard views that'll let you drill down into the data. And I think that's really what's going to set this flow based monitoring product apart from others. But this is really exciting. I am excited to try this. I haven't gotten my hands on this yet, so I'm going to play with this a little bit later on. I know we're almost out of time, but can you give us a reminder, viewers again on how they can get access to it? Yeah, absolutely. So right now, Magic Network Monitoring is free and it's an early access. If you want to sign up for early access, you can go into the blog post announcement and click the Google form and put down your Cloudflare email address and I'll make sure to give you access as soon as possible. Additionally, you can email MNM. That's Mike November Mike at Cloudflare dot com and request access and I'll make sure that I reply to your email and collect some feedback from you and we'll get you set up as soon as possible. Thank you, Chris. This was a really great session I think we had. I enjoyed this discussion. Enjoyed seeing the demo once again. Can't wait to get my hands on the product and try it out myself. Again, I want to thank all of our viewers for tuning in, listening into our Birthday Week session. Stay tuned. We've got an exciting agenda to the rest of the week. Check out Cloudflare TV for the rest of the schedule and we hope to join you to join you again and we hope to see you using our product. Give us feedback, try it out and talk to you soon. All righty. Hi, everybody.
