This Week in Net: 2022 US midterm elections (and Developer Week intro)

Hello, and welcome to This Week in Net, our weekly review of stories we've been writing in our Cloudflare blog, but also things affecting the Internet and, of course, also Cloudflare. With me, I have, as usual, our CTO, John Graham-Cumming. Hello, John. Hello, good morning. For those who don't know, we are both based in Lisbon, Portugal, so that's our time zone, in a sense. Let's start the conversation in terms of elections, because there was a big election in the United States this week. We spoke last week about the Brazilian elections, also a major election in Brazil, and now we have a blog post related to the midterms in the U.S., right? Yes. Now, I'm no U.S. election expert, but the idea of these midterm elections is they happen essentially out of sync with the presidential election. They're sort of in the middle, I think, of the presidential cycle, and what happens is there's all of the seats in the House of Representatives, which is the one that has hundreds of seats, comes up for grabs, so they get contested, and a chunk of the Senate, which has 100 seats, by the way, and a bunch of governors' races as well, and then a load of other stuff gets thrown in as well, because in U.S. elections, there's lots of things get voted on at the same time. And so those all happened this week, on Tuesday, and depending on where you were in the U.S., you were voting for maybe a governor, and maybe you're a representative, and maybe something local. And why does that affect Cloudflare? Because we have a thing called the Athenian Project, which is a project dedicated to protecting state and local governments when they're running elections. So protecting the election infrastructure, and what we mean by that is it's not the voting machines we're talking about. We're talking about the website you go to to get information about where you should vote. Where's my local voting place? Who are the candidates? What are the rules? Do I need to bring an ID with me? All that kind of information. And anything that a government authority needs to put online around the voting procedures. And then we have another thing called Cloudflare for Campaigns, where we protect political campaigns for party websites and the internal applications that campaigns use, so we keep the campaigns protected as well. And then we have Project Galileo, and Project Galileo is quite well known for protecting journalists and things like that, but it also protects voting rights organizations, sites that publish election results. And so all of that kind of stuff comes together when there's an election. We saw an election in Brazil, obviously, last week, and then now in the U.S., and in the U.S., we have those projects that are helping people. And we have really expanded our protection for parties, for the authorities that are dealing with stuff. And so to give you a scale of it, we protect infrastructure for elections in 31 states, so a huge percentage of the U .S. And what's interesting, I think, about this particular election is that there really wasn't a lot of attack traffic against the states. It was a little bit higher. I think we saw it be like 3% to 4% higher in November on the election day than it was the previous month. There's still a lot of threats, right? We saw something like 16 million threats per day against websites because this sort of stuff goes on all the time. But it was not dramatic, and I think what's good about that is that I'm glad it wasn't dramatic. And I think it also indicates that people realize that a lot of the websites and the infrastructure being used for elections is getting good protection, and so there's no point attacking it. Exactly, and these are threats that are mitigated, so they weren't successful threats. We block them, right? We block them, yeah. And then on the other side, we protect campaigns, so 34 Senate campaigns, House campaigns. They saw a little bit higher, actually. The campaigns saw higher attack traffic than the actual infrastructure for elections, which is kind of interesting. Although it's higher in terms of percentage, it's about three times higher in November than in October. On the other hand, the actual number of threats per day was a lot smaller. It was about 150,000 per day as opposed to millions per day. So there was attack traffic, and it was similar kinds of things. So I'm glad we protected it. We'll protect more, and hopefully we'll get to expand these protections outside of the US in the coming months. Exactly. One of the things for me that's really important and relevant here is, first, these are – and I'm studying a bit of the elections in the United States because I'm writing a blog post about the midterms also – and first, there's so many different sides between states and different types of sides. Some sides are a little bit more modern, others less modern. So there's different even ways to presenting things, and having them protected, have this layer of security there, it's quite important. And one of the things I'm surprised is how much people use those sides for where should I vote, how can I vote, which is important. And of course, the US also has mail-in votes, types of voting, which is really relevant too. So where to cast my vote in terms of mail-in voting is also important. I think in general, I mean, obviously, website information is important globally for getting information about elections. But my sense is, having voted in Europe and in the UK, that the US voting system is much more complicated because there are variations from state to state, because there are so many things to vote on, and just because of the size of the country. So whereas, when I was voting in the UK, I always voted in the same place, and it was written on my ballot, the thing that called me to ballot. I think in the US, there's more of a need to get that information, to figure out how you're going to get somewhere. And so it becomes more important to keep those websites online, keep the information online, and keep election result information online right as well. Because there's, you know, obviously, over the last few years, people have been calling into question the integrity of elections. And so the information being made available is really important, both by the official websites, which are going to end up certifying results, and also by observers who are going to talk about what's happening in the election. So this all comes together in Athenian, in Cloudflip campaigns, and Project Galileo. For sure. And let's hope that I can try to write the blog post in time of today. A few trends I can highlight are things that I've been seeing there. First, and we spoke about this earlier, the US is a really big country, and there's a lot of people in different time zones. It's like three hours from East Coast to West Coast. And that's really relevant when you look at data. And for example, with the Brazilian election, we could see clearly a drop in traffic, like 20% drop in traffic during election. They compared a Sunday with a previous Sunday. And in the United States, elections aren't like in a Sunday or a holiday, a federal holiday. And because of that, and because of mail-in ballot voting, there's not a big difference. There's a slight difference in terms of Internet traffic during the day, because of all of those patterns. The funny thing is, last week, Halloween, on Monday, had a bigger impact. We could see like 12% drop in traffic after 7 p.m. on Eastern time, actually, until like midnight Eastern time again. So a big drop in traffic in this amazing country because of Halloween. And we didn't see as much drop in traffic related to the elections, which is interesting. I'm assuming the reason for that is partly that if people are going to go out and celebrate Halloween, it's more restricted in time, whereas voting is over a long period, so people will be moving through the voting process during the day, whereas Halloween is at a very different point. But it's interesting that Halloween is such an important event in terms of people, and it really shows up like that in Internet traffic. It's also interesting the contrast with Brazil, because Brazil also has four time zones, right? It's split. It has, but I think it's like 90% of the country lives in the same time zone. Right. It's the difference, the main difference in terms of the U.S. And it's not 90% of the country that lives in the same time zone in the U.S., for sure. It's more broad, in a sense. And there's also major importance of mail-in ballots in terms of having that possibility. That's also important. A lot of people vote that way, which is interesting also. Yeah. But it plays a role, for sure, which is something I'll try to put out today. Yeah, I look forward to it. I mean, I look forward to seeing if there are trends in there. And of course, we had daylight saving time change as well, right? So that's the other thing that happened. That was a major problem. A very technical problem, to be honest, that we had. And I was working with our radar data scientist, Carlos Azevedo, with this. And it's very difficult because usually we use UTC time, and UTC doesn't change. But in the United States, this past weekend, on Sunday, the clock went back one hour. And because of that, it's really difficult to compare a Monday or a Tuesday with the previous week, in terms of code and making Python work, to see at the same time. Yet another reason to stop doing this insane changing of the clocks every year, which is just nutty at this point. So please, can we stop doing that? That's my appeal for this weekend. That is, can we have this weekend no more changing the clocks around for random reasons? Because believe me that solar noon doesn't change, right? It's still like solar noon. That's true. And this is really interesting, not only in terms of data and checking data, but also difficult for companies to work. First, because, for example, in Europe, the hour change in October 30th, the previous Sunday, and the hour change in the US a week after. So in a company like ours, when you have a schedule like a meeting, the meeting time changes because the hour changes in Europe, but not in the US. It's really crazy, to be honest. And I was reading about this because of all this data thing. I was reading about how that impacts the human brain, in terms of there was a shift. One hour, you go to the bed. One hour, a difference. In terms of Internet traffic, there's not a big change. You can see a slight change, but not a big change. But it's really a weird thing, even in terms of data, but also human experience, for sure. So I second that appeal, for sure. And we should also discuss about Matthew Prince. He's in Europe this week. He was in London, in our London office. And now he's in Lisbon. We had a fireside chat yesterday with Celso Martinho, from our Kloffler Lisbon office, with Matthew, which was great. And next he will go to Munich. That's right. We have an event in Munich, where people can get together with him. It's great to have him visiting Europe. So yeah, he's here. In terms of Lisbon, he stated something that we stated before, that Lisbon is our fast -growing office. And it's our European headquarters at this time, in a sense, in terms of having this broad scale of continental Europe and Brexit and all that. Yeah, that's right. Just to talk about Lisbon shortly. I think what's happened here, just the quick history, is we came here in 2019, with initially six people. Now we're over 200 in Lisbon. And actually a few folks in Porto as well. So we have some folks around. But the main office is here in Lisbon, where we get together. And it's been a big success for us. We have grown very rapidly. And there's a lot of projects being done here, both from an engineering perspective, but also an HR perspective, finance perspective. It's a full-stack office, if you like. So yeah, Lisbon's been great. And Matthew's obviously in town, visiting for a few days, and then continuing his tour around Europe. Sure. And next week we'll have a big Innovation Week, Developers Week, where we'll have a lot of announcements related to workers, to developers, what they can do with our platform, in a sense. Right? What's coming? What can we say about Developers Week at this time? So, I mean, I think the big thing to think about is, to talk about, is obviously next week there'll be a bunch of announcements, as with all of our Innovation Weeks, where there'll be multiple blog posts a day and announcements about things. Some of those announcements will be partnerships. Some of those announcements will be products. Some of those announcements will be detailed explanations of how things work. And I think, you know, if you're a developer and you're thinking about building an application, this is a good moment to stop and say, you know, what is Cloudflare doing? Should I be building on Cloudflare's platform? And the reason I say that is, first of all, there are a huge number of developers building on Cloudflare. Just, you know, it is really, really massive. And it's become a very important part of our business. That's one thing. And the reason why that's happening is, I think, fundamentally is developer productivity. And so, if you go back five years, when we launched Cloudflare Workers, you know, you could push code out to Cloudflare's network. We would deploy it everywhere. We would execute it for you. It was super fast. And it was up to you to do most of the work to build the application. And over time, we've added other functionality. So, the key value store, the database, D1, the R2 service for objects handling, right? And we've added queues for queuing. We've added the ability to process email and so on and so on. And I think we've reached a critical mass where, if you're building an application, you can build it on our service. If you tied in with Cloudflare Pages, which is fantastic for doing static websites, if you add to that Pages Functions, which allows you to take static websites and make them dynamic, so you can do anything you want to with functions added in there, I think Pages is the best JAMstack platform there is out there. I think it's hooked in really nicely with workers through the functions functionality. And then you think about Durable Objects, which allows you to really fundamentally build, you know, interesting computer science type stuff, right? So you can build a database with it. You can build on top of it in a really fundamental way. And if you think about the way in which we optimize for you, so we figure out the scaling, we figure out where your data should be, we figure out where your code should be. I think it makes a platform where you can say, okay, look, I'm just going to build something and Cloudflare will scale it for me. I will just not have to worry about it. And it's sort of the dream of any developer to be able to go from, well, it works on my machine, to it works at Internet scale. And that stepped Internet scale is pretty hard to do actually, if you have to do it yourself, right? And figure out how to scale it. So we deal with that. The other thing is we are super efficient at how we do this. We're super efficient because that's how we run our business and we want to be super efficient, but also we're super efficient because. If you're a developer, what you don't want to do is think about how, which what virtual machines am I maintaining? Do I have to keep my code hot? Otherwise it won't work properly. And people do crazy stuff, keeping, preventing cold starts and stuff like that. And those things can take you a lot of time and time. Money, engineers. Yeah. Completely. And if you think about the history of, of, of SAS and the cloud, a lot of it was, a lot of it is really eliminating lots of work. So if you go back and if you go, go back long enough, it was like, I'm going to buy some software and I'm going to install it on my machine. I'm going to buy a CD and install it. Right. Or disks even, or, you know, or I'm going to install it on servers. And it's like, but nobody wants to do that. And then, you know, what really happened was you've got SAS applications, right. Which we're all using. So that was one thing. And then you, then you're like, well, I need to buy servers to install some of this stuff on the, even the stuff I'm doing. So, you know, Amazon really, you know, pushed for this with EC2 and with their storage, which is like, don't buy them, you know, use virtual machines. So you're constantly trying to remove roadblocks to the fundamental thing, which is, I just want to write code and get it running. And I think Cloudflare because of a combination of the scale, the services and the efficiency, it means you just stop thinking about any of that stuff. You're not thinking about the container. You're not thinking about provisioning things and you're also not doing stupid stuff like, you know, oh, what if I get a burst of traffic and I need to scale up? Well, I better keep a bunch of virtual machines running, sort of on hot standby, which I'm going to pay the provider for, even though I'm not using them, you know, so we are very much a pay for what, you know, pay for what you use and also in a very, very reasonable way. And so I think it fundamentally, if you're a developer, you just want to be very sort of selfish and be like, I want to make my thing work and I want to make it stay. And that's what you do. And so next week is going to be all about that realistically. And I really think it's amazing. First, it's like an ecosystem. You have more tools to use and to build your stuff. But even if you're not a very experienced developer and most of them are just figuring out things, discovering new problems, you will ship like more easily with less problems and less need to learn by doing mistakes, mistakes, type of situation using like our global network, our platform and our new tools in a sense. Right. Well, I think the thing is you can really easily iterate what you want to do. Right. So you can sit in there and that's, you know, you can write some code, you push it out and you can debug it and you can, you know, you can really make it make it happen. And I think that, you know, if you're a developer, that's the other thing you want, right. Which is you want to be, you know, you just want to be able to iterate really, really fast and having a platform where deploying is fast and rolling back is fast and getting, getting, you know, tailing the logs is fast that allows you to be super productive. And so that's what the platform offers. And you're not sitting there going, well, I need to go through a build step and then package up a container and then deploy the container to some service and load onto this thing. And Oh, once I've actually got it deployed, I need to worry about what happens if there's, you know, lots of traffic and how can I afford this? Because maybe I've got to keep this stuff on standby and I've got these functions, but they go cold and I get bad performance. And Oh, what if I suddenly get a bunch of people signing up in Indonesia? I chose the wrong region. I chose, you know, US East or something. And now the people are over there. You shouldn't have to think about any of that stuff. It's just like, I'm running some code. I want it to work at scale. Here's how I've decided to solve my problems. Here's the APIs I'm using. Here's the thing. And by the way, you know, because we open source, the workers run time running on your local machine, make sure it works for you. You know, it, you know, and with winter CG, we're trying to standardize all of the APIs that are being or as many as we can. So there's cross cross companies, there's similar APIs. So, you know, I, what I would say is, you know, I would take a really serious look at what we've built. I know things. The other week I put out on Twitter and ask people to tell me, you know, what are you building on, on Cloudflare? And, you know, I got to just a ton of emails from somebody. So, okay. There's somebody who's building a no code tool for people who need to build marketing websites entirely on Cloudflare Workers doing it like that. Right. There's a whole bunch of people who've done work, which is integrates with Zen desk. So if you're using Zen desk, you can, you're building like templates for emails and integration with different different bits of functionality. There's somebody who's building a fun, a gamified fitness app, totally built on Cloudflare workers. You know, it's just on and on and on. And so I think, you know, at the beginning maybe workers was like, you know, you can do this little thing here and that's great. Now I think it's a completely different matter. I mean, people are building entire applications and, you know, we just expect that the ease of use, the scalability, the pricing, you know, just this is why people want to build on Cloudflare. And it makes total sense to me, me too. I've started switching stuff to Cloudflare, which, you know, years ago I had other places and it's like, it's like a breath of fresh air. Exactly. We showed two weeks ago, anyone that wants to see your shift, your YouTube channel, amazing YouTube channel to a proper page, a proper blog. Yeah. I mean, Cloudflare stream. I mean, obviously I knew, I knew about Cloudflare stream, but I'd never, like I like to use Cloudflare products myself. And I found this excuse, which was to take my YouTube channel down and put it, you know, put it on my own website. I did the whole thing Cloudflare, I bought domain with Cloudflare. I set up a Cloudflare pages website using GitHub to keep the, the actual, you know, HTML, CSS and everything. And then I use Cloudflare stream to host the videos and I have Cloudflare analytics, right. Which are privacy preserving. It was just like, well, the full bundle there. I didn't, I just was like, there it was. I don't have to worry about it. And, you know, whereas previously, I probably would have been like, well, I'll probably get a VPS. I'll probably install Nginx or Apache or something. And then I'll set that up and I could have done all that. And then there's a video part. And how do I do that? And honestly the video part in Cloudflare stream was so easy that I've started using it for other stuff now. So on my personal blog, I had some of the little videos, like a little snapshot of something I was doing and I was like, how do I embed it? Oh, oh, Cloudflare stream. I'll just drag it into there and put it in there. It's just so easy. It is even as a user, it's good to easy and visually nice to use. So it's really well done props to the team there. It really, really is smooth. I mean, it uploads and it serves up exactly the right format whenever you are. Exactly. So that's the wrap, John. Thank you. And let's see what the developer week next week will bring us. Brilliant. Yes. Stay tuned for developer week. Lots of blogs coming. And because Matthew Prince, our CEO, was in Lisbon this week, here's a clip of the fireside chat hosted by Celso Martinho. Going, going back to our culture, two questions. Why do we blog so much and why do you think John is obsessed about it? And, and, and why do we have the Cloudflare weeks? Why, why are they so important to us? You know, if you, the, we used to make when people started at Cloudflare, we used to make them read all of the Cloudflare blogs in order, starting with number one and working their way up. Now that today is impossible. We didn't have a quiz. There was no quiz. We're not, we're not, we're not sadistic. We're just a little bit masochistic, but it's, but we would have people read it. And it's interesting. You can actually go back. You can, I think, go to the bottom, scroll the bottom of the page. There's a, there's a little thing. And if you jump to the very last page, you can read the first Cloudflare blogs. And what you'll find is they were awful. Here are the top 10 reasons you need a fast website. I mean, it was just, it was so lame and it was like everybody else's corporate blog and it just sucked. And I, when, when I, and I, John might point to a different point in time. And keep me honest, but I, but when I remember that changing, Was there was one day when Google went offline for a huge part of the world and there was a network engineer on our team still, still on our team today. And this was, we were probably had, I don't know, 30 employees. And it's a guy named Tom Paseka. Some of you might've worked with, he's now, now on the infrastructure team. And and we're eating lunch and Tom goes, well, here's what happened. Here's why Google went offline. And instantly a group of really smart engineers and me were like in absolute rapt attention because we all depended on and use and made our, made our livelihood from making the Internet. But what we realized didn't really know how the Internet worked. And here was this guy who was an expert. He was explaining BGP and route leaks and all, all this stuff. And it's like, why did it go offline here, but not here. And, and we get to the end of it and we're like, you got to write that up. That's amazing. And, and Tom's like, I'm, I don't know. I'm not that comfortable. Does anyone really care? And I'm like, yeah. And we were still small enough that we could just do crazy things. And so we said, you know, write it up. And Tom wasn't, you know, it was like, I'm supposed to be doing other things. I'm like, I will, if you just tell me the story again, I will ghostwrite it. You correct the mistakes. We'll put your name on the top of it. And we'll, and we'll go from there. And the following things then happened in order. The first was the blog went absolutely viral and had like a hundred thousand people read it over the next little bit. Cause who doesn't want to know, like there's this big company called Google and, and all of a sudden you now you understand why they went offline and they're not telling you, but we did. And we were right. That was, that was good thing. Number one, bad thing. Number two was Google immediately tried to hire Tom. But that's fine. And, and to Tom's credit, he's like, I'm not going to go work for Google. Cause they're never going to let me write a blog post like that. Just true. Right. And then I think the third thing that happened was overnight, we got a huge number of network engineers who said, I want to come work with Tom and we got an influx of new applicants. And at that particular moment in time, we realized that the purpose of the blog, most companies think the purpose of the blog is to get customers. We realized the purpose of our blog was to get employees. And once you think about the audience, not as being someone who's going to buy from Cloudflare, but think of it as someone who you want to have as your colleague in the future. And you want to show them one, we work on interesting problems too. We're pretty smart. And three, if you come here, we'll let you tell stories too, that that becomes catnip for, especially really, really, really smart engineers. What I wish we would do. And I begged Chris for years and years and years, the rest of sales is transitioning from running our sales team, but did for a long time. I wish we would tell the story of here are the interesting things we do in sales. Here are the interesting things we do in customer support. Here are the interesting, like, if we share those stories and we think of this as one of the best ways to recruit people, then it'd be good. So one of the things that I hear from the Lisbon team sometimes like, Oh, it's so hard to hire people in Lisbon, then blog more, right? Talk about the interesting work that you're doing here. Smart people want to work with other smart people. And that's one of the real opportunities to do it. And frankly, this is John's pushing back on this, but we have kind of migrated to using the blog as here's where announcing products, which is just one barely hair step above. Here's the top three reasons you need a fast website, right? Instead, like talk about the really hard problems. And you're like, well, customers don't care. I don't care if customers care. The purpose of the blog isn't to attract customers. The purpose of the blog is to attract employees. And, and it's been, it's been, you know, one of, one of the real secrets to, to our success, which is strange to say about a corporate blog. In terms of the innovation weeks, the history of it was, was so we launched at TechCrunch Disrupt in September, on September 27th of 2010. And to be, and we thought we didn't, we didn't know you could just sort of watch vaporware at these conferences. We thought you had to have a working product and and the whole team was sitting in the audience. And when Michelle and I got called to go backstage with a 30 minute warning before we were going to get on pitching, there were still 27 critical bugs that the service wouldn't work if they didn't get fixed. And we're sitting backstage, literally watching us. They're like, okay, I got number 13. I got number 26. I got number four. And they're ticking them off, like fixing those problems along the way. And, you know, it's really easy if you don't have a deadline to just let projects take forever. And what was amazing. And we learned just incredibly viscerally from it was, we couldn't move then we had, we had a time on stage on September 27th of 2010. And the service had to work by then. And, and when I stepped on stage, I wasn't sure if it was going to, and yet the next minute, several thousand people, you know, signed up and it mostly worked. Stuff broke, but it mostly worked. And so a year later, TechCrunch invited us back and they invited us back about a month before, just to give kind of a one-year update. And they said, it'd be really great if you could announce something because we, it's great to have news that's around whatever it was. And so we were like, what can we do? And we're like, well, and we finally came up with this idea that we would do an IPV4 to IPV6, IPV6 to IPV4 translation service. And, and I remember, first of all, you know, we had a deadline getting right around the end of September. And so we had to hit that and we were going to, it's going to be on stage. We're going to sort of announce it. And, and when we were talking about it, they're like, do you think that's going to get you any more customers? And I was like, no. And do you, do you do what, but then why did you build it? And I said something like, well, cause it's the right, that's the right thing for the Internet to do is taking us too long to migrate from IPV4 to IPV6. And, and we could make that easier. And in the process, that's the right thing to do for the Internet. And our mission, we hadn't come up with a mission at the time. It was just, that was, that was sort of what was out there. And TechCrunch, the tech publication wrote a story that was like Cloudflare, the company that's doing the right thing for the Internet, even if it doesn't get them any new customers. Cause that was like the most interesting thing that came out of the talk. And then the next thing, you know, we have a whole bunch of people who apply to us who say, I want to do things that, that work for that. And so that then started a tradition where every week, or it was originally every day around our birthday. And then when it turned out, we would try and think of what are some things that we can do that are the right things for the Internet that will be, you know, super successful, but that we want to, you know, that, that we won't make any, any real money off of. And that's how birthday weeks started. And what we found in the process of doing that was people really loved having a deadline and they loved having something that just what you couldn't move. And by putting a bunch of product announcements together, it became a bigger story than anyone on their own. And effectively this is the same thing that most companies do with their user conferences, like a ton of products get launched at AWS reinvent. And if you think it's any different, like their teams are working until the last minute to get it ready to go on stage. And, and then, you know, they're either thumbs up, thumbs down on whether you're going to be on stage. And so, and it's, and it sucks if the answer is thumbs down, Dreamforce, same thing, Oracle world, same thing. Right. And, and that they, they do that. And then we were like, well, that's fine. But now everything is around this one week of the year. And so the question was, how could we get that same benefit of a user conference, but spread it out over the course of the year. And so we distilled it down and said, let's basically pick eight weeks a year. We'll give each of them a theme, but the sort of secret of all the innovation weeks is you can kind of squeeze any product in any week. And, you know, it will create some deadline for people to get to, but it's, but it's not the end of the world. Like you miss it. There's at most a month later, we've got another one coming along. And it gave us this sort of cadence to get things out. And so I think that that, that combined with a really efficient developer platform taking on a big challenge, that's, that's caused us to, again, just really prioritize innovation. And I think that prioritizes, prioritization of innovation is, is still what causes, you know, most of the people who come and want to work for us say, I want to go there because I want to ship products. And, and we do that faster than anyone else. So, okay. Yeah.