Hi everyone. Welcome to a special security focused segment of Cloudflare TV. So in Celebration of Security Week, which is Cloudflare's Innovation Campaign, highlighting exciting security related developments in our products and our services, Derek and I are going to be speaking about some work that we've done coming out of the security team. So we'll be talking about a partnership that we made with the product teams to take and adapt our existing products to actually isolate IoT devices and in turn make sure that we're preventing lateral movement if those IoT devices were to be compromised. So we're going to get into all the details. But first, we can actually take a moment to introduce ourselves. So I'll go first. My name is Molly. I'm an engineering manager on the Security Strategic Projects team, and a big part of my role is kind of closing the gap between our security team, our product teams and our go to market teams. And really kind of playing off the fact that we're in a unique spot. Being a security team at a company that builds security products, how do we help better and inform our products? Kind of like the way that we did here and we'll discuss today to better enforce our security posture and also enhance our products themselves. So that's my background and I'll turn it over to Derek. Hi, I'm Derek. I am a staff engineer on the security side and I kind of do a little bit of everything, primarily focused on infrastructure, but it's kind of a default bucket for everything that we do from the hardware that we build to how our services are protected on our platform. I deal with a lot of innovation too, so I've worked on a few different projects and this being one of them, we were really excited to kind of collaborate with and a lot of talk about it. I feel like Derek's intro was very humble. I feel like he's like kind of the mastermind in the background, like really pushing the team forward and always coming up with new ideas that might be like blind spots for some other people on the team. So a humble intro. But we'll take it for Cloudflare TV. I'm going to back up and kind of start by orienting us and everyone who's watching with where the idea actually came from and kind of where the motivation for this project we're going to be talking about today came from. So I think anyone who deals with network security or anyone who has an Internet of Things device in their home or in their offices might be used to being aware of news articles and different kind of announcements of various vulnerabilities that are exposed in these Internet of Things devices. And part of the reality of that, which Derek can also go more into, but is the fact that these devices are quite small. So that means that there's limited computational capacity and that kind of goes in an exchange for building in necessarily the right access controls or really making sure that there's only so much you can do right in a very limited space in size. And so even though we always vet and trust any of the Iot devices that we bring into our environments, there's sort of inherent risk just with the scale limitations of an IoT device. And so there's been a lot of very high profile incidents in the news with regard to IoT devices being exploited and basically utilized to kind of pivot laterally and move laterally into otherwise protected networks. So one of the most famous examples is the 2016 Mirai Botnet attack, where basically all these IoT devices were essentially used as a network of kind of almost just servers to carry out essentially all of the pivoting that these hackers wanted to do. And it made all these successful exploitations come to life. And I think at Cloudflare in general like we are. So we're in a very strong spot because we do often use our own security products to help secure our environments. But it's something that we're always thinking about, which is what are we bringing into our sensitive spaces, including our data centers and our offices as well? And a year or so ago, we experienced something along the lines of this, which was written about in our blogs about one of our camera suppliers had sort of a similar incident related to exploitation and attempt to move laterally, which we were unaffected by because all of our networks are behind zero trust, our own products. But, still we flagged something that I think had already been on our minds because of Mirai Botnet and just the news and things like that, which is how do we actually make sure our environments are doubly secure by not only protecting the rest of our network, but also isolating the IoT device itself. And so this was sort of flagged on our team by one of our colleagues. And Derek and I got to talking and started to. Think about, well, we have all these security products that Cloudflare has already built. How could we potentially repurpose those to create a virtual separation and isolation of IoT devices that we're deploying? So giving us that like double guarantee, right, that we're protected both from any lateral movement from that device and of course, maintaining our zero trust properties on the rest of our network, too. So I think it might be helpful to back up and maybe even think about like what are our solutions outside of what we've created that people might turn to if they have these same set of concerns around isolating IoT devices? Like what is sort of available today? I think Derek would be helpful if you could go into that a little bit. Yeah. And adding to some of the context of where we're at with IoT devices, a lot of people kind of forget that we have these little embedded devices in everyday appliances that we use. We have Internet connected refrigerators, a coffee machine that now uses an app, and these are all connected to usually the same network. And so when we think about larger environments or businesses, we have cameras, we have sensors, we have interconnected controllers to control temperature. So all these things generally have are either on the same network or might be segmented in some way. So traditionally what we've done is we've usually put them on a separate VLAN and a VLAN is a virtual local area network that creates a logical layer of isolation within a shared environment. So when you think about if we're connecting devices to a switch, a switch might have the ability to be able to be managed in a way that we can create separate VLANs to create these logical isolations of traffic. But a logical isolation does not mean that it can't talk to each other. When we break down network layers while they're on the same what we call Layer two network, and they can communicate ways they can communicate via this layer two network. Great, great example. What happens is, is generally from a routing perspective, they're allowed to still talk to each other so they can enforce different types of access controls to prevent that lateral type of movement. But what we do is we end up using access lists, and access lists on a network on a network switch, require you to be able to define the source and destination. What direction is it really going in? Are we managing the state of the connection? Meaning that are we tracking that connection to allow that return traffic to happen automatically? Or are we keeping the stateless where we require two different entries, one for outbound and one for inbound? And then furthermore, like, how are we actually detecting this traffic? Are we logging that? Are we logging successful attempts that could create a lot of logs? Or are we logging only the denies? And how what are we doing with that? So those are kind of like the traditional things that people are accustomed to. But as we go a little bit further, we can get into things like private VLANs, which creates this level of isolation that we really want, but it's really only dedicated to that individual switch. So if we wanted to automate this or if we had the same type of devices that we were trying to protect and we're trying to do it, say, in one location as well as another location or potentially all over the world. That creates a separate layer of complexity of how do you automate that and how do you enforce that same level of protection within each individual device, within each switch, within each data center, within each office building. So it just makes things exponentially harder. Can you do it? Absolutely. You know, being in this industry for a long time, I've seen large teams, you know, create, you know, automation that can do that. But it's not something that's easy for the everyday user or, you know, small to medium sized business to be able to enforce that level of control. And at the same time, too, there's not a lot of awareness as far as these type of impacts. But what we've seen is as these type of attacks start becoming more mainstream now, the concern is, well, how do we protect that and how do we protect all of our locations if we have these type of devices in play? Yeah, I think one of the scariest things too is sort of a sense of false assurance, right? You can go to all this effort to set up this environment with VLANs and feel like you're getting those guarantees. But it sounds like there's so many pitfalls where like one misconfiguration or lack of configuration could actually potentially expose you to something like lateral movement. Yeah. And it's also not really understanding like what these devices are connecting to at the same time to these devices are usually not updated that frequently firmware updates are not that are not that common. And if so, you might have to be a manual process. If they're an automated process, where are they talking to? Are they talking to a remote cloud server or are they talking to something local? And how much control do you have over it? Traditionally, security for IoT devices have usually been on how do you access the device and not necessarily what the device is communicating to. And so while we can address the administration of the device, our focus kind of pivoted to, well, what exactly is this talking to and how can we control that? Yeah. And I feel like, like every instance of kind of like IoT exploitation that has made the news in that way is exactly about that issue. Like where is your IoT device connecting? And I think one of kind of like the strongest powers of Cloudflare is to extrapolate a lot of this kind of like confusing access list policy related type information into something that is very point in click, like being able to actually write policies and access and gateway and being able to instantly have them apply. And I think that I would imagine as a network administrator to have to configure that and have that knowledge sort of live in your head and make sure that it's well documented. That only adds to the complexity, right? So yeah, you want to you want to make these things repeatable and at the same time to it's if you had a policy for, hey, these are how all my cameras are being protected or these are how all of my sensors are being protected, then it's that level of consistency that you kind of you kind of want to achieve. And again, it's that layer of visibility. You know, if you had a single pane of glass where you can actually control all that and see what's being connected to it kind of reduces the outage that you might take by inherently blocking something that you don't know because now you're getting that kind of like rich telemetry as far as what these devices are talking to and you're able to build that profile and make it consistent across the board. Yeah, I like that word profile. I feel like that's kind of exactly what the goal is to build there. Should we get into like what we actually did here in the solution, I can show a diagram based off of the kind of proof of concept we built out. If Derek, you want to start taking it away there. Of course. Of course. So this is about as simple as it gets. And where we had we simulated this camera that was in a data center that was connected to a switch that potentially had other things that were connected to it. So the first thing we were looking at is like, how do we control the outbound, what we call egress traffic? We realized that this camera needed to make a update to a system, a cloud based system, and that's where it pulled for more updates. Anything that was relevant to the camera that would require it to operate normally or whatnot. This is also potentially where it would send logs and whatnot. So we wanted to ensure that we not only were managing the inbound connect connectivity, but also seeing where it went outbound. At the same time to this concept of lateral movement. So if we're actually able to control the outbound connection, could we control it from, you know, connecting to other devices that we're connected to the same switch? Again, getting back to that original diagram, that original concept, you can build some of those logical controls, but it becomes exponentially harder to be able to kind of manage that within each individual switch that might be involved in connecting these devices. So we connect to this camera, we connect it up another server, you know, to simulate this concept of having like a neighbor next to it on the switch. It's connected via power over Ethernet, which is what powered the camera. And so we had this physical connection cable that was connected to an edge router. And this edge router is typical of being of of having a routed device within your network. From there, we created an encapsulated tunnel and this encapsulated tunnel allowed us to be able to. To connect to what's considered our edge. And our edge is what kind of powers Cloudflare network has, kind of like a lot of our secret sauce in it. And from there it was that this is where policy decisions are actually made, where we use a product called Gateway, as well as the internal product called Magic, that allowed us to be able to create a secure tunnel to this to this to this edge router at the same time, to also be able to do policy enforcement as far as where it needed to go to. So this policy decision affects this because as every route, as every rooted packet on that same switch, whether it be from a server or from a camera, it was going up to its gateway, which is used in order to be able to determine what the next hop or what the next route would be. It was enforcing these policy decisions because of these, because of our integrated products. So as we created a policy to say, Hey, we want this camera to be able to talk to this server, to be able to get its updates. Well, let's deny it from being able to talk to anything else that is connected to that same switch, which we were actually really successful in being able to see as it looked, as it did a next hop lookup and pointed that traffic to its gateway, as that gateway was integrated to our products, it made we had a policy effectively saying deny anything outbound except for the connections that it needs to be able to make to be able to work properly. So at the end of the day, two different products that we were using. One was this magic connectivity that allows us to be able to create secure tunnels to wherever we want to or wherever a customer wants to. And also our gateway products that allowed us to be able to filter out what we didn't want to see or, you know, effectively bad traffic or in this case, good traffic. Yeah, that was a really helpful explanation. Thank you. And it's kind of crazy to see this diagram and hear you explain it because it's so simple. Right. But Derek and I know how much kind of thought and effort and partnership with our product teams went into this to kind of take products that already existed, but just think about their application in a slightly different way here. So it's kind of cool that it's now at a place where we're like, Oh, it's all it takes is a diagram like this to explain it. Yeah. And a lot of our focus primarily was just on inbound traffic because that's the biggest concern and that's how a lot of our products were have been built. You know, whether it's stopping denial of service attacks or filtering out Web traffic, a lot of it was just thought about how do we handle the traffic inside. Now we're starting to pivot to like, well, how do we handle traffic that leaves our network? And that's what's been able to kind of enhance our products by being able to have that level of granularity and control. So if you're having traffic go through Cloudflare, we're not just protecting you inbound, but now we're able to kind of control what we what, what, what your devices are talking to. Yeah, and I think that's a really good point of clarification because a lot of people might not be aware kind of this pivot, which I think we've talked about in other contexts or rather this expansion. And what was sort of cool for us is as a security team and we're thinking about how to better the security posture of us as a company. And so we're thinking about IoT devices as a potential risk and a potential path for exploitation. And Derek and I are working together to think about this kind of new, innovative way of protecting ourselves. And it was totally sort of hand in hand and well aligned with the direction the product teams were going, with the direction the Magic and Gateway teams were going in terms of thinking about grass protection and applying rules in that domain. And so it was kind of a nice example of the ideal kind of partnership between security and product and really being able to amplify the work that one another is doing. So it's pretty cool. Yeah. It allowed us also to be able to kind of take a different perspective, even from a networking perspective, because we always thought that, oh, well, we do this in different ways or we kind of self protect. And this whole focus was primarily on things that really didn't do a great job of protecting themselves and or protecting or creating that noisy neighbor situation. You know, if you have a noisy neighbor, you know, you want to find a way of being able to silence that. And if you can't do that within the device itself, then how can we leverage our products to be to be more effective in that scenario? And it also gives you that layer of visibility where you traditionally haven't had that in these kind of type of devices. If you want to be able to get logs, you can collect network logs from different levels. But again, having that single pane of glass and allowing us to be able to see that, have that layer of visibility and then from there be able to make those policy based decisions made building these kind of profiles a lot easier. There's always the question like, well, what exactly do they talk to? And that's kind of like the running problem or the running joke with a lot of development teams. Well, what are your services talk to? In this case? We were actually able to see that, see how it changed, see if it was something that if it was communicating out with the server once a day at different intervals or whatnot, deviations for it. So we were able. To kind of profile what was legitimate traffic to ensure that we still maintain that level of connectivity and continuity when the device without creating an outage or without creating other problems. Yeah, that's such a good point of like kind of almost like learning from the logs. Like, I mean, depending on the vendor that you're working with. Like I think historically all the vendors that we work with for IoT devices have that clear picture of like, okay, we know what our expected connections, but depending on kind of what you're putting in your environment, right, you might need to kind of learn from the standard pattern and use that to determine the policies that are put in place. I'll go into a little bit sort of how did we actually put this solution in the field now that we've talked a bit about the architecture? So I've alluded to it a little bit, but when we first came up with this idea or recognize this need and this potential for application of our own products, we partnered quite closely with our product teams on the Magic and Gateway side who, like I said, were already thinking about control of egress traffic. And so it was exciting to be able to kind of align our needs there and multiply one of those efforts. So we worked directly with some partners on the product teams who definitely you've heard talk on Cloudflare TV probably about things related to this. And then from there our goal was to develop a proof of concept that we could deploy. So our step, our first step was to deploy a proof of concept in one of our offices that uses a vendor for cameras. Right. Of course we have cameras in our offices and basically to just make sure that performance was unchanged. I mean, the ideal, right, if you're putting something behind Cloudflare's network is actually performance gets better, which is the case, of course, but wanted to make sure that this architecture was making sure that things still worked as needed, but just adding this extra layer on top. And so once we'd gotten kind of that check mark that this had worked in our offices, we decided to deploy and one of our data centers. And so I actually spoke about this yesterday on Cloudflare TV with Evan Johnson, our product security and Application Security Lead. But we have a data center that's kind of like fully devoted to dogfooding, which is called Dog, which runs traffic through it, but is used really to kind of test the newest, latest and greatest products. And so we decided that we were going to deploy this architecture within Dog and determine whether or not in that higher stakes atmosphere, all of those kind of truths held up. And it was kind of funny. I still have the memory of like watching Derek scramble over the cages to like literally connect the CNI cables. And, you know, we really did the thing and then we brought in a pen tester. So one of our trusted pen test partners to confirm what we this was designed to do, which is prevent lateral movement. And of course, the results of the pen test proved that. So it was kind of just that extra validation that a third party who plugged in what they sort of emulated a rogue device as into our network was not able to move laterally from this from a technically exploited IoT device. So the next step here, now that we validated that this concept thoroughly works, is talking about it, which we are doing on Cloudflare TV and our blog post that we just posted about, because I think that as a security team, right, we kind of get the pulse off of what a lot of other security teams experience and wants is something tells me that this idea will be interesting to a lot of existing enterprise customers. And like what you're discussing, this is really just a repurposing of products that Cloudflare already has. And so it's sort of an easy win for a lot of customers who are concerned about these kinds of things. And continuing to deploy internally is like one of the biggest things. So Richard Sumner, who's our head of physical security, has a lot of thoughts and exciting endeavors ahead when it comes to securing our data centers. And this is definitely part and parcel of the work that's going on there, making sure that no stone is left unturned and that the work that we've done here, we kind of you know, we say it and we also do it right, like making sure that this is part of our security posture and our architecture going forward. So it's it's really exciting that this is something that just truly began as an idea. And now it's something that not only are we using in a more widespread fashion internally, but it's something that our customers can actually start to use, too. Yeah, you made it, actually. Really, really good point. We had our pen tester install a rogue device and ideally we all classify these as rogue devices. It's easy to be able to put them on some guest VLAN or whatnot, but it really doesn't create that layer of isolation when it comes to how things are. Crowded. And so as we get more accustomed to kind of classifying these as kind of rogue devices or potentially non trusted devices, that allows us to be able to say, hey, we will build these kind of profiles and determine everything that's within our network that we can kind of like throw within these profiles and create additional layers of protection and give greater levels of visibility and anything kind of that deviates from that. We'll kind of have that, that, that layer of visibility to be able see that. So yeah, I'm excited to see that, especially partnering up with physical security because of all their new devices and gizmos and gadgets, they'll kind of follow the same path where we know that they're being protected. We're being protected from those devices as well. And then we kind of have this layer of visibility that is kind of unified when it comes to when it comes to these profiles that we build. Yeah. I'm curious, Derek, for your thoughts on kind of like where this technology can go both from like the Cloudflare side and then also as IoT devices are developed and or developing. And I think one of the things that brings this top of mind is I feel like the US government is talking a lot about cybersecurity in general, but also with an eye to IoT devices, recognizing that they're notoriously insecure. And so I'm just curious, I feel like you have a pulse on kind of the direction that these things are moving. And so what are you concerned about or. I think it's hard because as a device is built, you know, it's usually meant to be as simple as possible or be as be a single purpose as possible. So it's not like a typical server that might have a full operating system that has complete control that you can do, even you can tinker with in any way. These have kind of embedded Linux systems or real time operating systems that have small functionality. So the question is like, do you really enforce a lot of controls on those devices and expect them to really make a lot of that decision making process? Or do you push it further up? And that's what we've done with our services. We are an edge computing company, we're a security company and we like to be able to push as much stuff to the edge as possible because that decision making process, let us expand the computing power to be able to make that determination. So I think, you know, partnering up with a lot of government entities are trying to figure out new and innovative ways to be able to protect these devices, some of which might be device based, but some of which might also be kind of like our bread and butter and how do we control the network and how do we make those kind of polices decisions as well? New protocol comes in. This is something that we can easily kind of adapt to as well. So whether it's doing updates, whether it's doing file updates, whether it's doing firmware updates, you know, if we're able to be in path and make those policy decisions, I think that kind of empowers a lot more trust in a company to be able to incorporate these devices fully knowing that they might not be as patches as they like, but allowing us to be able to make that level of control and decisions for them or allow them to be able to build that level of control as far as what access they want to have internally. Yeah, and I think it's kind of cool just extrapolating a little bit that, you know, some for a lot of enterprises, right, these IoT devices that they wouldn't normally think about necessarily in the same category of other devices that they have, like the fact that through this setup, they're getting those same kind of benefits of Cloudflare Edge and Cloudflare's network. Right. It's not just security, it's the performance, it's the speed as well. So it's kind of cool to all push that sort of under that same umbrella and really create that more robust architecture. Yeah, that's the added benefit is like just allowing us to be able to make kind of like the intelligent routing decisions. This is like one of the benefits knowing that you're going to experience the same performance issues that you would if you were controlling it through different types of networks and then being able to filter out the noise or being able to filter out what's bad, I think that's great because if they're doing it internally already with other services, then this is extending it to again what we'd like to define as rogue devices. These are, you know, if you if you truly don't trust them, then allow us to be able to help you control that level of trust and be able to police that traffic Totally. And I know we're almost out of time here, but kind of one of the last points that I wanted to make is that this is such an exciting example of the work that our team is pushing towards on something that we call security innovation. And I alluded to it at the beginning of this conversation, but what I think it is, is we're just in such a unique position being a security team at a company that builds security products. And that means that not only should we be the number one consumers of those products, which is something that Evan and I talked about yesterday during our session about dogfooding, we can also help inform the direction that those products go and kind of spot new uses for them, like in this instance of deploying magic and gateway to isolate IoT devices and prevent lateral movement. And so I'm just excited because I feel like this is sort of just one example of more work that is going to be coming out of the security team in partnership with product and engineers. To really help solve four needs that we are experiencing as a security team. And we know that our colleagues in the same industry are experiencing as well. So it's pretty cool to be able to build that muscle and definitely build that strong relationship with product that I think everyone knows that they want to have. But actually to do that in practice is something to definitely be proud of. Yeah. It allows us to also bring that level of awareness where people might not know that or not having to do these traditional mechanisms in order to be able to protect your network, it allows us to be creative and kind of spread that awareness amongst our customers as well as through our blog posts, allows them to see that how we're being innovative in that sense. Yeah, exactly. All right. Well, thank you, Derek. I look forward to more Cloudflare TV sessions from us on all things related to security, innovation. And yes, if anyone watching is interested in this technology, feel free to check out our blog post for more information or reach out to your Cloudflare representative to discuss further. Awesome. Thanks.