Fireside Chat with Jerry Perullo (CISO - Intercontinental Exchange)

All right, great. So, we have the honor, Jerry and I have the honor of closing today and we stand between you and drinks on the trading floor of the New York Stock Exchange. So that's why you needed water, Jerry, because you're in the hot spot. Plus it's a fireside chat. Right, right, right. There we go. So welcome. Thanks so much for being here. Thank you to everyone for being here today. So Jerry, he is the CISO at Intercontinental Exchange, which runs many trading exchanges around the world, including the New York Stock Exchange. And you've been their Chief Information Security Officer for almost 20 years. Yeah, 18. 18, for 18 years. And so welcome. Very happy to have you here today. And so I have some questions I want to go through, and I'm also going to open it up and give you, the audience, an opportunity to ask questions halfway through. And then I have a long list of things to go through. And Jerry's infinitely interesting, so the next 30 minutes is going to go by like that, like a snap. So why don't you start by telling us, tell us about your job, Jerry. What does the CISO of the New York Stock Exchange and all these other exchanges around the world, what do you actually do all day? That's a really interesting question. So one thing I've noticed since being a Chief Information Security Officer is that every time I meet another one, we usually have very different lives. You know, we do different things. Some people are basically cloud engineers, and some people are basically public policy experts, and then everything in between. So that can be troublesome, especially in a lean organization, because we're a very profitable organization, and we do that on the top and the bottom line. So we control costs really carefully. So I can't just have 800 people worrying about every headline and everything that affects every company. So I really do need to focus and think about what we care about. So we have this notion of threat objectives, where we really try to capture everything that you read about in the news, or that you read about in your email box, probably. The rest of us have to read the news. And so, for example, PII theft, that's a big portion of the media stories nowadays about cybersecurity, and it's really important for some orgs, but not for all. And so what we wanted to do is to say, how does that affect us? Do we care about that? Do we need to stop everything when the latest news story comes out? Do we need to worry about PII controls that are really for that? Things like data leakage protection and that sort of thing. And so we bucket in these 10 things and then rank them by what we thought was germane to our company and our footprint and what we're doing. And I'm oversimplifying it, of course, but to cut to the chase, we find that sabotage and large-scale asset theft, I always joke about this urn over here, actually, as an example of asset theft in the physical realm. And usually, I'm based in Atlanta, so it's nice having a prop so close by. But so sabotage, you introed exchange, and that's what we run, critical infrastructure. And so data theft, which it's really easy to fixate on in the cybersecurity world, is not the top billing for us. It's sabotage, data manipulation. And when I say asset theft, if you look at our balance sheet, it's clearinghouse collateral, so cash theft. And now we're even in the cryptocurrency space, so we do custody for digital assets as well. So that's really, so my day job, I have a second line function that's really the strategizing and figuring out exactly what I just described and setting the mission. And then I have the first line half, which is executing on that. All right, let's go out to the techniques, tactics, and procedures and the threat intelligence, figure out what it would look like, replay with our red team and make sure we stand up well, and if not, close the gaps. That's great. Thank you. Super helpful. And so are you worried about that boss being sabotaged over there? Well, no, it actually comes up because my group has physical security as well under it, which you see an outstanding presence here. So obviously, it's disproportionately geared towards 11 Wall Street. And this is where our main security, physical security office is. So we did the same thing when we absorb physical security. We said, let's start with the threat objectives, you know, because when all you have is a hammer, everything looks like a nail. So instead, we said, what are the big physical threat objectives with this theory that we were forced to learn things in cyber that would actually benefit a lot of traditional security orgs. And so it's kind of a bit of a pilot, but it's playing out. So in that case, one of the top ones that we've identified is notoriety versus active shooter, right? I mean, that's a real thing. It's a real threat, but it's kind of equal at any organization of a certain population density, whereas we have a symbolic presence here. And then we have petty theft in the physical realm and the urn and high value asset theft. And each of those is different. You know, smash and grab is petty theft. That's a different set of controls and a different set of actions for the policing staff than the Sean Connery threat, which is, I always think he's going to steal that urn one day. There is a Sean Connery potentially somewhere here. No, I'm just joking. Okay. All right. Great. Thank you. That was super helpful. So as you are planning for you and your team, as you're planning for 2020, what's on your mind? What are some of the objectives that you're thinking about for next year that you can maybe share with the audience here? Sure. So one really unique thing about being here a long time, and so I've been here 18 years and literally here in the New York Stock Exchange, we, ICE, Intercontinental Exchange, headquartered in Atlanta, purchased the New York Stock Exchange in late 2013. So that's been going on six years now here. But one really good thing about having a long tenure, which is extremely unusual in chief information security officers, is that we, I've had enough continuity to actually deliver long term projects. I really feel for a lot of the CISOs that rotate out, you know, in two or three years, just because it's hard to start and stop anything, you probably want these little quick wins and these little sprints and only that because it's hard to have a real continuous mission. So we have a lot of stuff that delivers that have been two, three, four year projects, and it's really exciting. So things, and we were just talking with your team as well, because Cloudflare has a good role in some of those. But things like on our network perimeter, adjudicating IP addresses in real time and figuring out if we all get a lot of threat Intel lists. And it's really, I always say, whack-a -mole. And our project lead on this is sitting right here. So I'm going to embarrass her a little bit here, too. But I always said, it's not worth blocking IP address because you're playing whack-a -mole. You know, we get them off these Intel lists. So we've had a long running project to automatically look at every IP adjudicated against over 600 lists and then kill it in real time. And a project like that, it even involves politics, right? You're threatening potential customers and cutting them off. And so it takes years to do something like that. But we're delivering on that. So extending that out to the edge, and that's where Cloudflare comes in, is a big 2020 initiative. We're always doing a lot of M &A. So there's always this constant cycle of finishing the integration of whatever we bought last year. And then next year, it'll be, you know, whatever we buy next. So that's certainly a constant cycle. And then when it comes to cloud, we enjoy the, I used to say we enjoyed the luxury of being a little late to the game, because the economics weren't really there for us. And we weren't, we didn't have a lot of money to save on infrastructure. We had a lot of sunk investment in data centers. But nowadays, just to buy the best product, you have to meet a lot of vendors in the cloud. So they kind of drag you in there and force you. And it's what, SaaS is one thing, but now you have things like Snowflake, where you have to sync up data into S3 repositories, and you have to do a good job of it, you're gonna have a lot of trouble, regardless of how secure Snowflake is. So getting, shifting mindset from IP addresses and ACLs into workloads and compute and applying that mindset there is a big shift. So we have a big cultural transformation going on there as well. Great, that's great. Thank you so much. So, you know, one of your areas is cybersecurity, these threat objectives. What's, what are one or two areas that caused you, kept you up at night three years ago, that you worry a lot less about now? Like where you feel like you've made a lot of traction against? And maybe, maybe what are one or two areas where you're like, this really is a burning issue for our team? Well, you guys would love to talk about DDoS, because you're good at it. So I don't blame you. But that's not why I'm bringing this up. We've gotten, knock on wood, right, we've gotten pretty good on DDoS. And I think we're really in a sweet spot in that we do get attacked. And if you don't get attacked, it's really hard to prepare, especially with DDoS, it's really hard to red team that, you know, there's a few services, but wants to try them out. So we get attacked enough to be trying it out in live fire, but not so much that it's all consuming and all we have to care about. And over the years, it's allowed us to really gravitate towards a really strong architecture. And you know, not just with CDN, and with the likes of Cloudflare, with Cloudflare, literally, actually, but also even at the carrier level. So we had shifted a lot of our DDoS protection into the carrier level so that we could scrub individual addresses and targets and that sort of thing. So now, DDoS events, and we've also done a lot of data visualization. So when we have one, instead of being phones are ringing, and then someone's calling someone else, and you have that panic of what is it and you're trying to figure out, what is it? Is it a DDoS? Which data center? Which IP address? What kind of attack? Instead of that, it's really now almost academic and fun, you know, to say, oh, wow, look at the graph, we can tell immediately and we're kind of standing there without anything on fire and analyzing visuals and saying, oh, wow, this looks like, you know, an old school NTP reflection attack, haha, that's so silly, or whatever it may be. So I think that's a good example of not keeping up at night so much. Yes, that's a great example. Okay. So I want to shift gears to your team. So you've been there for 18 years, you've had to build a team. So can you tell us a little bit about what your leadership team looks like? Maybe we start there. Sure. So I mentioned earlier, I kind of broke it up into this first line, second line idea. And at most companies where they use the lines of defense model, which is what I'm referring to, and I think that Ernst & Young might have coined that 20 years ago before they applied it to cybersecurity. But generally, the idea is the first line is doing stuff, and the second line is analyzing risk. So I sometimes say that the second line looks for problems, and the first line fixes them. So at most companies, security is one or the other. And I like to think that a lot of that if you had a CISO come in, because you had a breach, they're usually a first line CISO, and they report to the CIO. And they're really dealing with with firewalls and technical controls a lot. Whereas if you have a CISO, because of reg or governance or some kind of pressure, or your directors on the board, just read the latest NACD consortium, and they told them that they should have a CISO. They're often more of a second line person who comes from a risk background will report to a CRO or general counsel. And usually it's one or the other. And so when I and we get examined a lot, right, so as you might imagine, we have a lot of heavy regulators, and we're global. So we have them all over the world. And a lot of what regulation looks at is governance and more structural things. They're not really good at analyzing firewall rule sets for better or worse. So they focus on what they can, which is often governance and structure. So from that, I know that they have just visited companies that had just a second line security department or just a first line. And I was afraid of being pigeonholed into that and either not getting credit for the other half that we're doing or having expectations set unrealistically. So I structured the whole department to have separation. So the first line and second line are about the same size. They each have an independent manager that's on my management team. But they're both within information security. So they get along just great. Our red team is in the second line. A good example would be we have these sessions after every red team event that we call Ralph and Sam sessions. So the old sheepdog and wolf cartoon, they used to meet and be best friends, and then the whistle would blow and they'd go in and fight all day and then come back out. So I think we're at a pretty good balance, though, where they're independent, but they're friends and they respect each other. And I'm talking about individuals, but it's the whole departments. And then within those areas, we were able to build out somewhat traditional teams. So the second line, I mentioned the red team in there, governance, risk and compliance lives there, too, and application security. And to me, you know, at a lot of shops, GRC is kind of a paper job. You know, it's all about this, just writing things up. And it's an unfulfilling job for some people, and it's a high turnover, and it can be hard. And I really want my GRC professionals analyzing risk and really understanding, seeing a vulnerability and being able to contextualize it and say, this is hot, or this isn't that big a deal. Because otherwise, the people that they go when they go talk to developers and tell them to patch something, if it's, if the developer knows it's not really exploitable, you lose credibility right away. So GRC has to know their stuff. And having the red team right next to him is perfect. And on the first line, you know, incident response, of course, and reacting. And then we have an architecture team in there that really just goes through red team results and says, Oh, we missed a detection on this, what can we do differently, package it up, deploy it, put it in whatever monitoring infrastructure is relevant. Next. So for example, they'll instrument Cloudflare logs, which we suck down and put into our data lake. And so we have use cases against that, where if this, that and the other, then, and it's all bespoke, I find the most value comes out of the bespoke rule sets. Instead of a 1500 rule IDS out of the box, six rules that were actually written by people looking at your data are often far more valuable. Great, awesome. Okay, I have one more kind of group of questions, then I'm going to open it up to the audience. So if there's things I'm going to give you an opportunity in like three minutes. So staying on people and team, how many how many open roles do you have in your organization? We actually have just one right now, which is fascinating. Yeah, how and how is that possible? Because I actually I don't know, we did a prep call. And that was something that came up. So I guess I kind of was leading the witness. But it's the idea of like, I've never met any organization where they've only had one open role in your sort of organization. And so how have you been able to attract talent and train your team, your team and keep them at your organization? I think sharing some of those insights here would be really helpful to us. Yes. Well, what's good is that there's three of them sitting right there. So now they know they're not allowed to quit anytime soon, or I'll be really embarrassed. I mean, come on, right here. But well, first of all, it's another knock on wood, right? Anything can change. So I understand that. And I'm really thankful for what we have. And we've had problems, you know, it is challenging in the industry as a whole. And I think that, you know, compensation is obviously a thing, and that's pretty black and white. And so you got to be in the market and figure that out. And one of the problems in in really every role in this room, but especially security, is this notion of, you know, people coming out of school and spending three years, and all of a sudden, they're a senior, and it's really hard for a company internally to match what they could do going out on the market. So, you know, we've really had to make efforts to be able to structure that and band and say, Okay, this person's having a, what used to be a internal small promotion, but then we need to be able to reflect that in compensation, key people. And so that's something that, you know, let's say any company can do. And you'll see what I mean in a minute. Another huge piece that is not easy to fake is mission. And that's something, you know, that Alyssa touched on earlier, you guys have a really fun mission, people probably work there and continue to work there because of that. And, and we do too, I feel. Now, that said, we could hide that and insulate it from people. So the fact that we have this great mission, it's important for us to expose that and get people involved and, and not say, Well, you're only working on Project X, and you don't get to see the New York Stock Exchange, you know, we need to get people out here and, and to see this great asset that we have to get excited about the mission. So otherwise, that's, that's pretty much impossible. And then the third thing is really the culture within the team. You know, one of the things that we all wrestle with is buy versus build. And the constant argument is, well, if you build everything, then when that one person who built it leaves, you know, you're screwed, for lack of a better term. So you should just buy a commercial off the shelf application instead. But I find that there's a, you know, an intangible value in in letting people build things, because it's rewarding, and it's fulfilling, and that's what they really need. And my philosophy on build versus buy has turned into I call buyers or builders buying. So a lot of times, it's worth prototyping internally, even if it doesn't scale, even if you know, it's not going to continue forever, but at least you learn the true pitfalls. And then when you go to the market, you're much smarter. And you can say, Oh, what you're describing is pretty trivial. We built that last week, versus Oh, you solve that problem that we worked on for six months straight. This is a real vendor that I want to work with. That's great mission comp. And then that culture and that, you know, that that 80s, you know, real genius type, fun environment in the sock. That's great. Actually, I've heard that from other never put that way. So articulately, but this idea of when you're working on kind of the innovation within the company, then then people want to stick around and see it through. And so I think that's really smart. All right, when I open it up, I have other questions, but I want to see if anyone in the audience has questions for Jerry. I'll give you an opportunity to ask a couple questions, then we'll wrap up. Maybe can you introduce yourself, say your company, and then you can ask the question. Oh, one second. It's Omar from DigitalOcean. Great company. Yes, you're blocked if that's what you're asking. I'm scared. That's too easy. funds, high frequency trading, and just software eating finance, right? And exchanges have been, you know, at the center of the increase in information appetite. And so how has security and the various security threats changed over the past few decades? Yeah, so if I can repeat it, and then you can correct me if I got it wrong, too, is just with the amount of financial data in particular coming out of the exchange space has just increased and increased in the appetite for it. So how has security affected that? Yeah, so you know, it goes back to the threat objectives again, because when, when I walked into this space 18 years ago, I didn't know anything about trading and markets. And over time, I learned and we were traditionally more commodities geared. So energy was our first play, and then agricultural commodities, and now equities over the last six years with the stock exchange. And I don't like living in a world where I just don't understand that stuff. And I just say, Oh, it's the stock market, you know, money and stuff, therefore, VLP controls, where can I throw money? Not only is it a waste of money to just buy everything, but if you don't understand the problem, but it's also a big blind spot. You know, if you don't understand it, you are going to have a hack from somebody who took the time to actually figure it out. So back to the threat objectives. You something that people will always say from from, you know, 20,000 feet is, well, can't hackers interrupt all the order flow. And there was even a news story that broke on that BAE broke, sorry, BAE, I don't mean to dump on you there, just kind of blurted out. But they had broken news stories saying that hackers had front run orders out of co located servers at a major exchange or exchanges for hedge funds. And we're routing the orders to Eastern Europe, so that they could get ahead of them and insert orders and beat the market. That sounds plausible at 20,000 feet, and it sounds really dangerous and scary. We're dealing in microseconds in the exchange, there's no way you could beat anyone if you even moved it to Eastern Europe and pinged it right on back. You know, people are we actually have to measure the fiber in the data centers to make sure everybody's equitable. Because if somebody has an extra 10 meters, they can notice that latency. So that didn't hold any water at all. So we rang them up. And I said, Hey, we know which one was it? And it was Oh, well, you know, I worked with this lawyer at an old firm, I can't tell you who it was. And then we got the Bank of England to chase him for an answer. And then a redaction came out two weeks later, and the whole thing was a lie. And it was guys as old as this was a red team exercise that accidentally got portrayed as a real incident. Oops. But what I'm getting at is, so with market data, for example, you know, we have to study well, what could go wrong with that. And some things are unauthenticated. And that's okay. And we know that. And so we don't have to worry about things like scrapers. And sometimes, when we look at visualization for things like DDoS, we see scrapers all the time. And I think it'd be wrong to just jump to the conclusion that automated scraping of our websites is evil is terrible. You know what, maybe it's actually of commercial interest to us to have our data become the data of record. It's free data anyway. So I'm sorry, it's a long winded answer to it. But we haven't seen a ton of issues related to the data output. And then most of the big consumers on the financial data are latency sensitive. So they are co located are very close. And the amount of throughput that they're running, and these low latency environments is so high, that it almost makes a really comfortable ground from a denial of service perspective, because you can't even, you know, generate, it looks like a denial of service all day, every day, as I'm getting at. Great. That was amazing context. Thank you. One back here. Go ahead. Yes, Brian Gorgas from Blackboard Insurance. I got to imagine that one of your top risks, you know, are people, people make mistakes, people do things you don't want to do. What are some of the top things you do to mitigate that risk? Thanks. No, no, that's a good one. I do think about that a lot. So security awareness training, and on. We've, so we're a big automation shop. I mentioned builders buying and it doesn't just stop at firewall rules and ACLs. It even goes into security awareness training. So, you know, the easy kind of dumb answer is, oh, well, we do phish testing. And then you're thinking, yeah, of course, everybody does phish testing, and you know, all the pitfalls that come with that. We do phish testing of 6000 employees every quarter, we vary the campaigns, we grade the results by the whole population for that campaign to wait how hard it was, we automatically email the user, we use, we'd run a screenshot of the email in of the phish, and copy their boss and boss's boss, so that they can all self, you know, adjudicate how silly it really was. And then we combine that, their performance on a gradiated scale of, oops, I clicked the link and entered data to on the other end of the spectrum, you, they click the report button, thank you very much. And in the middle, there's things like ignored it, or just click the link, but didn't enter data. We have a scoring system for all of that, you put it all together, and it gives you a percentage score. And then that and if you fail three times, then, well, the disposition is towards dismissal, it's not a policy or anything like that, but in practice. And then we take that data, along with mandatory awareness training that everyone has to take, and then that is scored by how on time you are. And then we have every incident has a negative or positive contribution if it if there's a human involved. And then we run crack your password every single day or try to we spend eight hours on a routine cracking the entire database of passwords. And if your password is cracked, you get an email telling you it was cracked and asking you to reset it. And so if you ignore that email for many days, that's also, you know, a metric. And you take those four things and each have a weight. And that's a composite score that's graded against the whole population. So at any given point, we can say you're a C, C plus student. And even worse, your organization is a D minus and your peers is a B plus what's going on. And HR and those types of groups love that kind of data. Because you know, like it or not, in human resources, it's really hard to have quantifiable data for how people are doing. And so it's really nice to have something that hard science to accompany the hey, you were pretty marginal, and we didn't like your decision making. Oh, and by the way, you have a D plus on security vigilance. And you can imagine when you run that kind of program, you're going to get a lot of legal potential legal challenges. There's a lot of questions there. So you have to have good data, or at least has to be defensible, it has to be repeatable, has to be consistent, has to be auditable. So, you know, that's where the automation comes in and putting all that work that I mentioned into it. I'd say it works fairly well, because of, for one, we've seen over time that all those metrics I mentioned improve, you know, from the fish click failure rate to the passwords and that kind of thing. And then on the other side of it, we did some what I called fish marking. So we got a big for consulting, we have PwC, I don't care, to and they have a good red team. I know it sounds crazy to do. But we got them to run a test with us and 12 other big financials to do a blind fish test, the same test across all the companies without any advanced notice, no idea the infrastructure to formatting anything like that, for the sake of benchmarking. And that gave me comfort to that we stacked up. Well, I think it's been effective. Wow, that's, that's incredible. I'm curious, what, what grade does your department have? It has a very good grade. We didn't have any fish failures. Yeah, we didn't have any fish failures at all. That's good. No, that's amazing. I mean, that's, that's incredible. I you could, the entrepreneur in me thinks, wow, that's a company in there. That's, that's incredible. I'm over here. Oh, hi, my name is Trey. I'm with Cloudflare. I'm curious, as a CISO with 18 years of experience, what are some of the patterns or practices in the security industry that you think will prove to be a failure in three to five years? What are the things we're doing now that are a mistake? Oh, that's great. So, um, I think you, somebody teed you up on this one. Trey, I've had the pleasure of working with Trey for many years, and he has the best questions. Him and Usman, who runs our engineering team, they ask really, really good questions. So, great question, Trey. Well, it's funny, because I have this little slide deck I've used a few times. It's called the contrarian CISO, where I just go through this bullet list of things that I think are silly. So, um, and, you know, each of them is offensive to a different group of people. So, can't really win. So, the funny one I would always throw out, because, you know, most of my peers that I work with, other practitioners, and we compare notes and joke around all the time, are bigger proxy shops. I always say proxy server is a quick answer to that, but if nothing else, because it's such a location-specific control, and if there's anything about, you know, Cloudflare, and the cloud in general, and decentralization, it's that you're not going to have control over the endpoint and be able to route them through things. So, that's a control that, you know, I think if people are only relying on that, and having full visibility in the traffic, and allowing everything, but just counting on inspecting inside actual, you know, HTTP headers, and making decisions of saving the day there, that that's going to run out of steam at some point. I like to say, I guess this gets a good, you know, soundbite to, patching is overrated. And just all of these breaches that are then attributed to a lack of patching, and all the pundits roll their eyes, I can't believe they didn't patch Windows, they had a whole 38 days, and people don't patch in 38 days, they don't patch in 38 years, most of the time. And when something like Nietzsche came out, the companies that were affected weren't affected because they hadn't patched, or among other things they were, but to put it the other way, the 99.9% of companies that were not affected weren't saved because they were good patchers, not at all, they had the same patch failure rate as anybody else, they just didn't have the fundamentals right, or they did have the fundamentals right, the ones who didn't. So, for example, simple, like, block port 445 from the Internet to your workstations. I could have told you that 20 years ago, and it was really important 20 years ago, people have gotten owned every five years on that same problem, it happened again, but we missed the opportunity to fix it again because we went to patching, oh, they didn't apply this Windows patch. I think encryption is way overrated as well. Now, you guys see a lot of places where it really matters. I always use dissidence as an example of kind of like one end of the spectrum of people that you're trying to protect, but in a corporate or an enterprise environment, how many of you have gotten those emails from a financial services company that says, you've received a secure email, whatever that means, to read this, save the attachment, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going to use it, and you're going So, you can open the attachment, double click it, and do everything it says. That just blows through all the security training any of us have ever issued, and it's all in the name of encryption. I don't remember a single instance in threat intelligence of email being intercepted for lack of encryption. It's hard to get an email to anywhere without it accidentally getting encrypted these days, and there they are, going through those hoops for end-to-end encryption. So, I think it's, yeah, that's definitely one. Great, perfect. Maybe one more question from the audience then, if there are any more. Hi, my name is Mike. I work at LuckyVitamin.com. We're sort of a mid-sized e-commerce company. So, from your perspective, if you were to step into a company of that size, how would you recommend performing an initial security audit? It's something that we do periodically. We think deeply about it, but we're not security experts. So, where would you sort of start or guide a team in that endeavor? I know I'm gonna have a really good answer to that, like, on the plane home and just think, man, I missed that opportunity. I mean, you can look at a lot of different ways, like literally the 30, 60, 90-day plan or what are the best initial tools or practices to put in the place. Having run a lot of them at different cost points, one of them, the best bang for the bucks, bangs for the buck is bounty, bug bounty programs. I guess putting another way would be if we bought your company, right? If there was an acquisition, and that's something we live with a lot. And we've tried different approaches. You know, we've tried, so compromise assessment is a nice sounding thing, which means you're gonna check all the end, you're gonna check the network to see if there's any existing compromise. The way most big firms are gonna do that, it takes six months. They're sending out network hardware. They want you to configure span ports. Imagine the change control nightmare. It's not gonna happen. If you're in a jurisdiction with a customs group that's gonna hold the hardware for six months, so you pay some VAT, and it's just a mess. And they want to deploy their endpoint agent on every endpoint. And you know how that's gonna go. First, it's not gonna happen. And if it does, it's gonna be on all the ones except the important ones, because you can't risk putting something on those. But bug bounty, on the other hand, you know, give them a couple of URLs and let them do subdomain iteration on it, and then just go nuts. And they're competing with each other. And we run a private bounty program, so it's not even public. But they're competing with each other to not have duplicate findings. So they all know they have to be creative. And you'll get ten things that could actually get owned tomorrow. So that's the quickest fix to know stuff that, you know, you want to shut off immediately. And then the next step is to make that thematic. So then take those ten highs and those hundred mediums and whatever it's gonna be, and go over them to say, wow, we're deploying bad code. So, okay, maybe we need to emphasize on application security. Or, wow, they're coming in through infrastructure vulnerabilities. Wow, we need to do like some IP scans and see what's going on. Or, wow, they found some denial of service and simple old-school, I mean, we need a WAF. Maybe we need to turn on the Cloudflare WAF. And yeah, I don't know if this will come out as denigrating at all. It's not meant to be. But you know, when you use a service like Cloudflare, they have an amazing WAF. But it really takes, you know, a few minutes to turn it on. So I bet a lot of you probably have that. And when you go into the control panel, you see all these beautiful tick boxes that are off. And it's kind of scary to turn them on. But if you had a bug bounty and had those findings, you'd probably get a lot more air cover to start trying them out. Click the boxes. Great, great question. Okay, I have one more question and then I'm gonna wrap up. And this has been incredible, Jerry. So we talked a lot about your job internally. We talked about hiring people. How about externally? Like how do you keep up-to-date? Like what do you read? How do you keep up-to-date with the latest and the greatest? Who are your peers? How do you connect? Can you give us some insight? The trust-based communities are really valuable nowadays. So in financial services, we have the FSISAC. And that is Information Sharing Analysis Center. There's generally one for every sector. They have different levels of interaction, of action within them. Participation, I should say. They also have different levels of funding. But I don't think that's really the issue. It's really how much people actually participate. And the best way, if you're in a group and no one's participating, the best way to fix that is to participate yourself and really lead by example. So that one in particular, and really the biggest vehicle is mailing list. It's a somewhat closed group. You know, you get to know a lot of people. And it works from the tactical level of, hey, this business email compromise happened. Here's the indicators on it. You know, really busy mailing list. Those are going on all day. To the strategic level, as of this morning, I was kind of, that's probably what influenced my earlier answer, but about red teams and living in the second line, we were debating that. You know, where do you like to have it in your program? Other CISO, and CISOs are all talking. On this like closed email list. Super helpful. And then outside of our sector, though, there's a lot of trust -based groups that are kind of vouch based. You know, people know each other and they vouch each other in. And I know Justin Payne with Cloudflare through one of those, for example. So it tells you vendor, consumer, technology consumer, different verticals, all talking. Those are the most valuable to me. And then as far as the raw news, we didn't have a threat intel department for a long time because it just seemed like a nice to have. And maybe they're basically just reading the news. So when we finally got one, we made sure it was really tactical and had hard deliverables. And I found that they are really curators and it's amazing. We just got a, you know, a write -up this morning that was such high fidelity, so tailored, and it condensed, you know, hours of random web surfing for news down to, hey, we did that for you. Here's what you need to know at ICE. That's great. That's amazing. Jerry, this was wonderful. Thank you so much. I will say that I feel that the ICE is very lucky to have you because you're so articulate and clearly on top of all these things. So thank you so much for your time this afternoon. Thank you. Hi, everyone. I take it the people in this room don't really care about performance and security. No, today we'll be talking about how you can actually build your application so they're performant and secure from the get -go and you don't have to deal with all the other troubles that they're talking about over there. So my name is Rita. I'm the product manager for workers. Hopefully, if you attended Usman's talk earlier today, he got you really, really excited and thinking about, okay, serverless is the future, workers is the future, and maybe you were sitting there and thinking, okay, this is really cool. How do I get started? And that's a question that we get a lot and so hopefully between me and my wonderful colleagues today, we can help answer that question for you a little bit better. So there's no single way to adopt workers. Everyone kind of has their own unique Cloudflare Workers journey and that includes Cloudflare itself. So we'll have Matt Weinberg here today from HappyCog talking about their workers journey so you can hear it directly from someone who just went through it. But we here at Cloudflare also just had our own workers journey. We originally launched workers two years ago and it started out as a function service that runs in response to requests on the Cloudflare edge. So our customers came to us for their security and performance needs and they were like, hey, we need this to be way more customizable and what better way to allow someone to customize something than ultimately to let them run code. So we used our network of 194 data centers and decided that that was a suitable place for our customers to be able to write code there from now on. I feel like we talk about workers a lot and this idea of writing code on the edge, but everyone has this very mythical, mystical image in their head of what the code on the edge actually looks like. So let's walk through the little bits that you see here of what that code would actually be. So we have a very simple worker up here. We have the add event listener, which listens on fetch events. So that means that any request that comes in, we will take it and respond with the handle request function, which is defined right below, right? So now we have this handle request function that takes in a request as a parameter. And then this next bit is we make a fetch to this request. So this is what's called a no-op worker or a proxy worker, where if let's say a request comes in on www.Cloudflare.com, everyone's favorite website in here, it will basically continue through to the origin. And with that, we can add any modifications either on the way to the origin or on the responses way back. So as Usman mentioned earlier today, we quickly realized that stateless code is great, but that barely scratches the tip of the iceberg of what you can actually do with workers. And we expect that what we'll see more and more is people writing full -fledged applications with workers. But in order to be able to do that, you don't need only code, you need storage, which we ventured into with KV. And Steve will be talking a little bit more about how we think about storage in a bit. Monitoring and observability. So I know my worker is doing something, but I don't know exactly what and what it looks like in production. And actually, for those of you who are very interested in that stuff, we have a lot of exciting things coming down the pipes in the next few months in that category. So stay tuned. Developer experience, right? So when you're writing code, you want that quick time to dopamine, right? You want to feel like you've accomplished something. And so we have Ashley and her great team over here to talk about how we think about getting you to be able to experience that satisfaction as quickly as possible. And then coordination, data processing. So being able to determine what runs when and why. And getting us closer to that dream of the drone walking the dog, right? So with all of that, workers, we realize, doesn't have to be this thing that sits between your eyeball, the customer's eyeball, and your origin. It can in and of itself act as an origin except from 194 data centers around the world, which means that it's fast, it's secure, and it's basically infinitely scalable. So here we have a perfect example of a very simple worker that has no origin behind it at all, right? We have the whole thing here. We respond with a new response that just sends back a hello world with a status 200. So workers is not just a function as a service platform, but it's a full serverless platform where we allow you to not just run code, but bring over more and more of your workloads. Which brings us to our vision of allowing you to write code and only worry about that. So we really believe that the code, the actual function that you wrote, should work the exact same way whether it's running for 10 users or whether one day you have tens of millions of customers. And you should work on making their functionality much better, not on scaling your infrastructure and all of that. Ultimately, everyone's favorite question in meetings, right? But how is it going to scale? That should be our problem and we'll worry about that. And you can run the application itself. So let's talk about how you actually get started in terms of what are some of the use cases, right? If you haven't used other serverless frameworks before, you haven't written workers, it's kind of hard to imagine what you can actually do. So generally we see people go on this journey where they'll start with something pretty small or adding some modifications to their existing sites and applications. And over time they bring more and more of the logic away from their origin or as they're thinking about bringing new services online, workers is the tool that they reach for. So if you're trying to modify things on your application, custom caching is a great thing to get started with. So take more load off of your origin by being able to cache things like post requests or being able to cache authenticated requests, header modification. You have this perfect layer sitting in front of your infrastructure where you can address all of your course headers, needs, CSP headers, all of that stuff without ever having to touch the configuration at your origin or worrying about messing it up. Rewriting URLs with workers is really great and really easy. Next is responsive APIs and services. And the reason that you might want to think about using workers for that is, again, that user experience for your end user where if you can bring your APIs to the Cloudflare edge, that experience becomes much, much faster for them and much faster for you to actually bring online. So webhook endpoints are a really great thing to get started with. Easy REST APIs. We have several tutorials on how you can get started writing an API for workers. We realized that images and PDFs and assets are things that people are generating a lot. In fact, one of my colleagues, Adam, wrote this really great invoice generator. This is built entirely in workers and you guys can actually go to this website, even from your phone, and as you type these things in, they will update in real time because the worker is able to really quickly and seamlessly generate this PDF of an invoice that you can then send to your customers. We'll have the team talk a bit more about how you can use workers to deploy static sites, especially with our new worker site solution and all of the great things that your website can benefit from by using workers for it. So here we have a great kind of very quick example of how you can use worker sites. And you can see that in just the time that I'm talking about this, which is not that long, you can fully get your application from having nothing at all to being fully live and published on the Internet. And Gabby and Christian here will walk you through how you can do that for your website as well. So how do you actually, actually get started? You have an idea, you have a use case. Find the interface that works for you. So a lot of the time we see customers start out with a UI, you're just tinkering. It's very easy. You can open up the online editor that we have and it'll have a small example, a little bit of boilerplate for you to get started. Generally, we see as you advance towards more complicated use cases, you'll start using the CLI, which enables you to have a bit more control and a bit more consistent configuration. And for everything else as well, we have the API that you can integrate with everything that you're using. I highly recommend anyone starting out with workers to check out workers.claffleur.com. We have a lot of great resources there. We recently revamped our docs, so we have end-to-end tutorials that take you all the way from installing our CLI to getting your first worker deployed. We have a template gallery, again, to help you think about the different components of your application that you can move to workers, and you don't even have to write that much code for them yourself. A lot of them have really great, pretty fully-fledged examples. So I would like to invite Matt Weinberg here from HappyCog to come talk about his worker's journey. Thanks, Rita. So I am Matt Weinberg. As you can see, it's like my huge face and name on there. I'm the president of development and a co-founder of HappyCog. We are an interactive agency. We're based here in New York on 29th Street in Madison. We're about 70 people, and we do just a variety of work for clients. Most of our work is development-based. So it's like the majority of our employees, the majority of our revenue, and we do a lot of different kinds of projects, publishing, e-commerce, CMSs, anything you can imagine. And we love Cloudflare. Matt, how did you originally get started with workers? How did you hear about workers? Yeah, so a lot of our clients use Cloudflare. We work with a lot of large customers that were already using Cloudflare. In some cases, we've recommended Cloudflare. And so we keep track of the product announcements really closely. And so we saw workers was announced a couple years ago, I guess 2017 or whenever it was, and we knew that it was interesting, but we weren't exactly sure what we should do with it at first. So how did you go about picking your first project? And the first project is really cool, so I'm excited for you to tell us about it. So, as I said, we work with a lot of big customers, big clients, big websites, a lot of mission-critical stuff. And so workers was announced, and we knew it was cool, and we knew it was interesting, but we couldn't exactly figure out how. We couldn't exactly figure out where we could use it. And so we have these monthly lunch and learns with our developers where people can just talk about anything. And I think I brought up workers. I said, this is cool. Cloudflare just announced this. It seems popular. What can we do with it? And a lot of times what we do internally is we try out new technologies using just like internal play projects, like unimportant stuff that we can do internally. We can just throw some extra time at it and just kind of figure out how it works and some best practices. So we keep this list of like problems, not super serious problems, but just like problems that maybe technology can solve internally. And whenever we have this, like, oh, maybe we can apply one of these new technologies to it. And so we were thinking about workers, and we said, all right, well, this workers is JavaScript -based, so it's something that, which one of these problems can we solve with JavaScript? It seems like a good middleware type application because it's kind of sitting in between requests and responses. You know, maybe something that would require a REST API, something that could be managed by almost any developer. So we're looking through our problems, and we realized that one of them would be a perfect fit, and it's something we called DoorBot. So I made a diagram. This is me making this diagram, not our UX team. Although I am proud of aspects of it. So in our old office, people had to like buzz up our front door to kind of come up to our office. And we have a lot of lunch deliveries every day, and so what happens is our office manager was like constantly answering her phone, hitting 9, hanging up. Like answering her phone, hitting 9, hanging up. And it happened like, you know, 40 times in between 11 and 2 every day. And I know you're all like, oh, this is a stupid problem. Who cares about this? But this is exactly the kind of problem that we try to solve with like play-around projects, right? So we decided to make a bot that would fix this for us using workers. So do you want me to kind of go through it? Yeah, that'd be awesome. Okay. Again, like I made this diagram, so we're going to do my best here. So the way it works is this. We made a...we started with a Slack custom slash command. And a custom Slack slash command, when it's entered, it can kind of make a GET or POST request to any URL you choose. So we created a Slack custom command that, whenever it was entered, would do a POST request to a URL we specified. That URL we specified was a Cloudflare Workers URL. And basically, the Cloudflare Workers...the Cloudflare worker we implemented was just some JavaScript to check that the request is coming from our office manager or from me, check that it was like a Monday through a Friday, check that it was like during work hours. And basically, if our office manager entered in Slack slash front door and then unlock, what the worker would do is the worker would get that and then store it. It would store in workers.kv, which is like a database that you get access to with workers. It would store the fact that our office manager has chosen to unlock the door, okay? Then what happens is we had a delicious lunch delivery here. You can see that we don't do it with the bad lunches. It's getting cold, right? Yeah, it's getting cold. Let us in. Our devs are hungry. Their code has bugs. You know, the doorbell rang. What we decided to do was hook up the number that the door buzzer rang to Twilio, okay? So when they... Twilio is like an API for messaging, like text messaging and voice messaging. What happens is they ring our bell. Twilio gets an incoming request that somebody's ringing the bell, which is really just like an incoming phone call, and it does an API request to our worker. Our worker checks with KV, checks if the door is supposed to be unlocked, and if so, Twilio dials nine and lets the person in. So what happens is in this like one and a half second span, and I know it sounds like a lot, but it's really quick. The person rings, request goes to Twilio, workers, hits KV, person is let in. And now we can let in like 40 lunch delivery people a day without our office manager's phone ever ringing. So that's DoorBot, which is the very first thing we did. I should open source it. I think this is a very common problem for a lot of folks. If you want to submit a Wrangler template, we would love that. The really cool thing about this too is the Cloudflare worker actually runs from the nearest edge location, right? So it's only waiting for the request to go to like Newark and back, not even to like DC or anything. Exactly. I mean it's crazy because it actually, it all happens really fast. It probably happens faster than honestly our office manager was able to pick up the phone and dial nine. It happened in less than a second. And I know this sounds silly, but first of all, it actually solved a real problem for us. A silly problem, but a problem nonetheless. But this is important for us because it let us play with workers without putting a client site at risk. That's really important for us. We need to be able to, that's how we first learned about Redis years ago, that's how we learned about new content management systems. So I definitely encourage finding like these little toy problems that you can all play with. You might be able to throw workers at, and if it breaks, like who cares, right? Who cares? Our office manager cares, but realistically like our clients don't care. So most of your clients, you help them get websites up and running, right? So what are some ways in which you've been able to use workers for that? And I know a lot of them also have spiky traffic, so how do you think about scaling those websites and how does workers help you achieve that? Yeah, so after DoraBot, we actually put it towards real problems, which is interesting. I would take a couple of good examples of real problems. First of all, a couple of our clients are very spiky in traffic, like they typically have constant traffic, but they can get big spikes of interest. Maybe they're a publisher that gets like a big news story, maybe they're an e-commerce client that does a big sale, and we don't want to have to spin up a bunch of resources. I'm sure many of you have dealt with this issue and you can kind of solve it with Fargate or Docker or some other things, but we decided to use, for a couple clients, workers to do caching. It's something we played around with a little bit internally. We talked to one client about it and then we've implemented it for a couple other clients. Basically, the way it works is that, you know, workers can get a request and give a response without ever touching your origin server, without ever touching your front-end server, but it can do that conditionally. So you can do any JavaScript logic you want to decide should workers just give some response to this or should the worker actually forward the request on. That's a caching system, right? That's Varnish. That's the same as kind of any other caching system. So for a couple clients, what we've done is taken advantage of workers.kv, which is again that key value store, that database that you get access to with workers, and use that to store compressed HTML of cached pages and then use the worker to serve those pages, like literally the entire HTML, the headers, the HTML, everything directly from workers, directly from the KV cache, and that removes the need for like a varnish layer or something else. So we saw a few customers doing effectively that and so to us the natural next step was to allow you to circumvent the whole putting things in an origin first and let you deploy it just fully to the edge, which is why we launched worker sites. So is that something that you see customers asking for? Yeah, I mean, so that's another good point, right? The more that you move on to workers, the less you probably need on your origin. And so I think Cloudflare is solving this in two ways. First of all, just kind of going back to workers.kv for a second, workers .kv has a REST API. So you can actually store stuff into workers.kv from outside of the worker. So if you have like another system or content management system or anything else, you can save content, render a page, use the REST API to send it directly to workers.kv, then have your worker just read it. At that point there's no origin pulls involved. You don't have to worry about your front-end ever getting hit with traffic. But as far as worker sites, so a lot of our clients use static site generators. It's very popular these days. I'm sure many of you are hearing about this, like Gatsby.js is a big one. Traditionally when we've done work with something like Gatsby.js and static sites, we've hosted it on S3. Maybe we fronted it with Cloudflare. You know, you used to get the nice CDN, you get their points of presence. But Cloudflare, you could never really store that file. But that's changed because now they've announced, you know, worker sites, which lets you store your artifacts, your static site files, directly inside of workers. So almost all of our clients that we have built Gatsby .js projects on, we're talking to them about worker sites right now. Because it's just so easy. If you're already using Cloudflare, and again a lot of them were for firewall reasons and speed reasons, security reasons, being able to host your files right there is a no-brainer. I mean, it saves money because you don't have to pay for S3, but it also saves latency because there's not any of that origin pull. And it just removes one extra, like, layer from that stack and a little bit of complexity. So you just went from being a Cloudflare or workers novice, to deploying a doorbot, to now running code that runs in production for customers. What are some things that were surprising to you as you were becoming familiar with workers? Or what are things that you wish others that are getting started would know? So one big thing is Wrangler. So Wrangler is the workers command line interface that actually did not exist when we started using workers. And that was, I would say, a little bit of friction, just because it took a little more effort to kind of upload things. And we were using the serverless framework, and there were other ways. But the Wrangler CLI actually changes things significantly. So if you're playing with workers, I would definitely recommend checking out the Wrangler CLI, because it makes things a lot easier. The other thing I'd mention is, it used to be you had to attach workers to a domain. Like, you'd have to put a domain into Cloudflare, turn workers on it on certain routes. And then, like, we had basically some test domains we were using. But now they have this workers.dev, which you can register for. You can get your own subdomains. Like, we have happycog.workers .dev. And that lets you run workers without tying it to any of your other domains that are in Cloudflare. So that's really easy, too. Cool. Any other wisdom you'd like to impart? Yeah. I would just say, I think workers, for our team, has been a lot conceptually simpler to use than something like Lambda on AWS or something like Google Cloud Functions. And I say that because, like, workers, it just makes more sense to developers. And to me, I think, like, it's sitting between the request and the response. You can make modifications, or you cannot make modifications and just kind of pass them through. You can really do whatever you want. And so even just putting up a simple worker that doesn't do anything except adds a header, it's, like, a really nice, easy way to get started. And I just think conceptually a lot easier for people to work on. All right. Well, thank you so much for joining us here today. We're really excited to have you and really excited for more people to be able to get their lunch deliveries much more easily. Thanks, Rita.