2023 phishing trends, useful intern projects, and Cloudflare's lava lamp history
Presented by: John Graham-Cumming, João Tomé
Originally aired on November 19 @ 6:30 PM - 7:00 PM EST
Welcome to our weekly review of stories from our blog and other sources, covering a range of topics from product announcements, tools and features to disruptions on the Internet. João Tomé is joined by our CTO, John Graham-Cumming.
In this week's program, we discuss Cloudflare’s inaugural 2023 phishing threat report , which delves into millions of malicious emails, brand impersonations, identity deceptions, and other predominant attack trends based on a year's worth of email security data.
Next, we venture into a more technical realm with a blog post that offers developers an improved method to debug Rust and Wasm using our developer platform, Cloudflare Workers. It is followed by two projects, conceived by Cloudflare’s interns, that evolved into features:
We also shine a spotlight on our new AI hub: ai.cloudflare.com . Here, we explore how both individual developers and expansive enterprises are using Cloudflare to build AI tools. This includes the creation of ChatGPT plugins and a tour of some of our AI tools, like Constellation (allowing everyone to run machine learning models and perform inference on top of Cloudflare Workers).
Finally, we explain the history behind our use of lava lamps — and a lava lamp wall in our San Francisco office — for Internet security. These have come to symbolize Cloudflare’s commitment to security, rooted in the scientific principles of randomness and entropy. We'll retrace the journey from the idea in 2013 to its implementation in 2017. We'll also discuss how our lava lamps story became part of the culture, the subject of news articles , inspired a season of the TV show NCIS , and was popularized by a segment by science YouTuber Tom Scott .
Hello everyone, and welcome to This Week in Net. It's the August 17th, 2023 edition, and we're going from phishing threats to some interesting interns' projects.
But also we're going for a bit of lava lamps and Cloudflare's history. I'm João Tomé, based in Lisbon, and with me I have, as usual, our CTO, John Graham -Cumming.
Hello, John, how are you? Hello, I'm very well, thank you. It's summer here in the Northern Hemisphere, so I'm feeling a bit like T-shirt weather.
I almost put sunglasses on for this event, although I am actually indoors.
There will be a day that we'll do this indoors, although we did it like a bit during WebSummit.
We did, yeah, during WebSummit.
Maybe we'll do that again in November. But you're looking very Steve Jobs today.
I am, although it's a T-shirt. You've got the stubble going on, the little beard thing going on there.
So, all right, so you're going to channel Steve Jobs.
I'm just wearing a really worn-out Cloudflare Lisbon.
I don't know if this is the original Cloudflare Lisbon T-shirt, which was designed by Silvia, I think.
And it even says my name on the back of it, which I'm always a bit embarrassed to walk around with it.
Everyone will know your name that way.
Exactly, at least it's my initials. Those are known initials. They're known initials inside the company, certainly, yes.
Everybody refers to me as JGC and not John.
True. And even externally, at least in some technical or technological...
I actually don't use JGC as my handle. If you go to Hacker News, I'm not JGC on Hacker News, and I am not in places.
So, anyway, there you go. But your blog is JGC.
It's true. I have owned that very rare three-letter domain name for a very, very long time.
So, yes, JGC .org. And your blog... Yeah, go ahead. Your blog has been on Hacker News for many times.
Many times. Yeah, true. I actually originally set up JGC.org for email, because what happened was, a really long time ago now, I suddenly realized that everyone at the time was using their ISP's email address.
This was before Hotmail came along and people started getting permanent addresses.
And I thought, the problem is everyone's going to get tied into this thing.
You see this sometimes, right? I mean, you see it here in Portugal.
Sometimes you see a business has a sapo.pt address, right? True. In the UK, sometimes you have like btInternet.com or something, because they've got stuck with their ISP.
In the US, it's AOL, right? To a certain extent, AOL, yeah. Earthlink .net.
And I was like, I don't want to get stuck. So, I want to get my own domain and then I can move my email from provider to provider, and my email address doesn't have to change.
So, that was my thinking, way back when I registered that, which I think...
I'm trying to remember now. It was probably 1997, I think. So, it's probably 26 years ago.
Yeah, 26 years ago that I registered that domain. That's interesting.
To be honest, I was the Hotmail guy in the 90s, 96, I think. And I was so happy I had my own name with Hotmail.
But the interesting thing, and to your point, Hotmail became like, you have Hotmail?
That's so old-fashioned. It became like that, right?
At the time, it was like, wow, Hotmail. The whole idea that you could do email on the web was wild.
I mean, Microsoft bought Hotmail for some crazy amount of money.
I think it was like $400 million. Of course, to me, you got Gmail and there are many other services, but that idea that you could sign up.
And also remember, Hotmail was the original viral phenomenon on the Internet because you sent a mail with the Hotmail, it said, sent from my Hotmail.com address.
That was cool at that time. It was a very good idea, yeah. And 10 years later, it was the uncoolest thing in the world.
Yeah, I was ashamed. I still have a Yahoo.com address, but I probably could log into it and find out.
I'm sure it's full of spam.
I hope there's no one trying to contact me there. It's been a while. True.
I've created an alias, Outlook.com for my Hotmail account just because I knew that people will judge me for it.
Sounds a bit more professional. Yeah, there you go. There you go.
Oh, well, we have a few blog posts this week. And there's a new kid in town in terms of reports.
It's our 2023 fishing threats report. It's the first time we do one of these.
It comes mostly in terms of data from our Area 1 part of the company in terms of email.
So I'll share my screen and we'll take it from there.
Yeah, I think perhaps people don't realize that Cloudflare has quite a big email protection business called Area 1 where email goes through Area 1 and we filter out the bad stuff, the phishing, the malware, business email compromise, all this kind of stuff.
And because of the number, we've handled 13 billion emails over a year's period.
And we've been able to look into that, blocked about a quarter of a billion malicious messages.
And so this report is really looking at what are the trends here.
And I think, I mean, the headline, of course, is that something like 90% of cyberattacks start with an email.
Kind of steal a password, trying to deliver malware, trying to trick somebody into doing something.
So email is really, really, really important thing to protect against the threats that come in email.
So this is all about that.
So we're looking at that one year period from May 2022 until May of this year.
What are the sorts of things we're seeing? What are the businesses that are being impersonated?
And I thought that was quite interesting. There was a section about who's being impersonated.
Microsoft being up there because Microsoft files email and also identity.
So no doubt. Even the World Health Organization.
Well, yes, because of course, you have the pandemic, right? And so one of the things that cyber criminals do is they will use any emotional event to get emails to you.
So sending emails, pretending to be the World Health Organization, warning about something, telling you about some new thing.
A little while back, there was actually a bunch of emails from Boeing, and it was supposedly from Boeing.
And if you remember when the 737 MAX planes crashed and they had to redraw it, and there was a lot of controversy, again, cyber criminals jumped on that and started sending emails as Boeing.
And maybe you're allowed to get compensation because your flight was canceled or some nonsense like that.
So we see this kind of stuff.
But we also see just like the big names, Microsoft, Google, Salesforce, Apple, Amazon, all these folks getting spoofed in emails.
I'm curious about Louis Vuitton.
I don't actually know the story behind that. Maybe that's like, you've won a free Louis Vuitton bag.
Just give us all of your life details and we'll steal everything from you.
It's possible. That's the too good to be true type of thing.
Too good to be true. Elon Musk wants to give you a thousand Bitcoin.
Click here. Bill Gates. You can tell which billionaire is the most well-known at any one time by what's in spam.
At one point, it was Bill Gates, if you remember.
Bill Gates was going to give you money. If you scroll down, there's a great example of Microsoft credential harvesting.
This is just so classic. Your password has expired.
Do you want to keep the same password? There's a grammatical error in here.
That's always a good sign of phishing. This verification is for its intended receiver.
That's a mistake. People fall for this sort of stuff all the time.
The number of times I get asked by family members, I've got this email. Is it genuine?
It's definitely not. It's been here. My mother has always sent me. Back to the original thing we talked about with my email, my favorite ones are I get phishing emails from the administrator of jgc.org, which that's me, saying that my password needs to be reset or something like that.
I haven't fallen for any of those yet.
Good to know. Although there's also this little thing that sometimes you just click on links by mistake.
You're having a bad day and sometimes you just do what in another day you possibly won't do.
That's true. Also, I think one of the other areas is shipping companies.
The number of SMS messages I get, this is slightly different from email, smishing where they're saying your package is held in customs or there's a fee you need to pay or we couldn't deliver it with some really weird link.
Sometimes those folks actually send me an SMS and the domain they're using uses Cloudflare.
That's a really bad idea on their part because I then report that domain to our trust and safety department and they usually take it down about two minutes after I've reported it.
Top tip if you're a phisher, don't send me email containing a domain that's on Cloudflare.
Actually, there's also a mention the SFB brand.
Silicon Valley Bank. Exactly. Earlier this year, they declared bankruptcy.
Well, they sort of got rescued. There was an orderly transition of Silicon Valley Bank to other things.
But yeah, you're right. Very quickly afterwards, people started sending out these various real-looking messages from Silicon Valley Bank saying we need to review these documents, we need to sign stuff.
Using DocuSign it looked like, so the sort of things that people might expect to be signing all the time.
Cybercriminals will jump on events and use them to try to fool people.
The pandemic was full of this kind of stuff, especially when people were working from home suddenly, unfamiliar environment, unfamiliar with what it meant to work at home and people were in there preying on them.
There's also some mention here in terms some of those categories can be and we have actually a chart in terms of detections by threat category.
Deceptive links are definitely at the top.
There's more, right? Absolutely. That means if you want to get the full report, there's a downloadable report.
If you want to understand all the things that are going on with email and how to protect yourself, take a look at it.
I think it's based on our data. It's based on a survey of people across the world.
You can get it from here. Exactly. And it's quite extensive, so you can get a sense of things that you should be worried about.
There's actually some advices in terms of how you should proceed in some cases.
It's great to explore.
Absolutely. We also have a very specific blog post about Quorum dumps. It's nerdy times.
It's nerdy times. Well, what's interesting about this is like, okay, so Cloudflare Workers based on V8 uses JavaScript, TypeScript, and of course WASM.
WASM being a format in which you can compile other languages like Rust, C, C++, et cetera, into WASM and then you can run it on our system.
And that's all wonderful.
You can use the language of your choice. But most of programming is debugging.
This is a very, very famous quotation by one of the very early pioneers of computing in the 1950s saying that they just got the first computer.
And he remembers the very moment he realized that the majority of the rest of his career was going to be spent finding faults in his own programs, which is a depressing thing that programmers often have to get over.
One of the ways in which you get fault information is called a core dump, which is basically, if you imagine I took your computer right now, Joao, which is in front of you, and I froze the memory exactly as it is.
I just took a copy of it. I could have a sort of like an, if you like, an x-ray of what was happening on your machine at that moment.
It doesn't change.
It's like, this is what's happening. You can use debuggers then to look at those core dumps.
And they're usually generated when a program crashes. You can get one at any time if you want, but when a program crashes, you want to know what was the state of that program at that point.
Very popular. This is how we debug many things.
Cloudflare famously looks at core dumps very carefully because we're looking for memory unsafe behavior, which could result in privacy leaks or other behavior.
And there's this, there are a bunch of proposals for implementing core dumps within WebAssembly.
So if your program goes wrong, you can get a snapshot of where it was and use debuggers.
And this kind of long blog post by Sven tells you about how to do that on Cloudflare workers, giving an example of a Rust program that crashes.
And then how do you get a core dump and debug it? So hopefully, you know, if you're into writing code and debugging it, which, you know, hopefully you are, if you're using our platform, then if you want to use Rust, you want to use WebAssembly, you want to debug it.
Well, there's a debugger for that.
Wasm GDP. In a sense, it's all about making a task that you're probably not very excited about more easy or simple, right?
You save time. I don't know.
It depends if you like debugging or not. I mean, sometimes debugging is a great, like, you know, mystery.
It's like a side quest in programming, where you go off and try and solve some problem.
Sometimes it's just really, really frustrating.
Explains how it works in a lot of detail, how to use the debugger, how to understand things.
And so, yeah, absolutely. If you, you know, I think one of the key elements, I remember I went to a talk by Donald Knuth at Kepler's bookstore in Menlo Park a while ago, 15, 20 years ago, it was a while ago.
Anyway, at the time, Java was a really big deal.
And somebody puts their hand up and asks him a question.
And then they say, which do you prefer, Java or C++? Classic baiting somebody into giving it, you know, and then you're going to argue with them about why they made the wrong choice.
And Knuth's response was one line. He said, which has the better debugger?
So debugging matters. Debugging matters. And also a reference to our developers, Discord community.
Absolutely. Drop in on us.
We're happy to, you know, get your experiences using the debugging features. What do we need?
What more do we want? And it's also a way of showing how Cloudflare workers can be at service in different ways, right?
Yep, absolutely. We then have a couple of blog posts that are essentially with projects from interns, right?
Yes. I mean, Emily's project here on debugging queues. Queues is a cleverly named queuing product from Cloudflare that you just send messages between workers in our platform.
And, you know, great enhancement. If you want to create things that are kind of loosely coupled with each other, you get guaranteed delivery, you get batches of messages, you get concurrency and all sorts of stuff.
But again, kind of on the sort of debugging front, it was a little bit difficult to tell, like, okay, this message was generated.
Where did it go? How do I understand what's happening?
And so there's a nice new implementation where in the UI, you can actually send messages, look at what message is there, acknowledge them and see how messages going through the queue actually get handled.
So this was what I love about.
This is a great debugging feature. Very, very helpful for somebody.
They can get, you know, great visibility into what's going on. And it's also written by an intern.
And I think one of the things that's key about Cloudflare's internships is we give our interns real jobs.
I mean, our interns ship stuff.
And I think that's really, really exciting. And hopefully, you know, Emily thinks about us when she looks for a full time job.
And the other interns, I think we had about 40.
But you know, if you want to come to Cloudflare and be an intern, you're going to get to work on something real.
And if you're not on the tech side, you might be in product management, or you might be in finance, you're really going to get exposed to how the company really operates.
So yeah, a nice little feature, very helpful for debugging queues.
Because you know, debugging distributor systems is another whole ballgame.
Exactly. And in this case, you get support from the team, right?
The team helps you with an idea with something.
So you definitely learn, which is the most interesting thing. Yeah, you definitely will learn stuff.
Yes. And there's also this introducing scheduled deletion for Cloudflare Stream.
That's right. We mentioned briefly last week. Yeah.
I mean, if you're using Cloudflare Stream to stream video, it may be you want that video to be available for an hour or 30 days or something like that.
Well, you can.
You can specify that now in Stream, and we will take care of deleting it. So you know, if you want to make something available for a limited time, maybe after that time, it becomes an archive and is only available to subscribers or something.
All can be done very simply through the UI and through the API for either existing things, also for new streams that you're doing through our product.
Cloudflare Stream is pretty cool product for streaming live video. I mean, and also for storing video.
I use Stream for one of my websites, and I love it. I think it's a really great product.
And whether you use the UI or the API, now you can do this.
You can decide when things disappear. And it's also good to save costs, right?
Because if you have some of these videos there, it could mean a cost for you.
It could, right. Yeah, storage costs. And so if you don't want them anymore, then maybe this is something that stays around temporarily.
Anyway, you can set it up now.
That's available. Exactly. And again, also an intern project. In this case, it was Austin.
Great. Another great one. We still have time. I think we covered the blog post for this week.
By the way, before this new section of our site, Cloudflare AI, where all things AI related to Cloudflare are here.
It's like an AI hub.
What is Cloudflare AI all about? It's all over the place, right? It's on the Zero Trust tools for protecting our customers and helping them deal with AI tools their employees are using.
It's Constellation, which is the ability for us to run your models on our network globally.
It's ChatGPT. You can use ChatGPT to write code with Cloudflare.
We have a dedicated plugin. We also have a plugin right for ChatGPT with Radar.
So if you want to query the data that's in Cloudflare Radar, you can.
It's how we use AI internally for finding zero days before they've been released to the public for dealing with bots.
There's just a ton of stuff with AI at Cloudflare at all levels that's been happening.
I'm not giving too much away, but next month there'll be more.
Stay tuned. Stay tuned. Next month will be our birthday week.
It will. One of our innovation weeks. The main one, the first one in a sense.
Innovation week of Cloudflare. It was the one where we started releasing things like Universal SSL, where we gave away SSL free.
Workers was announced during a birthday week and a major change to how people program on the Internet.
Lots of interesting, fun announcements coming up towards the end of September.
Exactly. There's also our cursor, Experimental AI Assistant to answer questions, Cloudflare's developer platform.
Exactly. If you don't want to read the docs, you can ask Cursor to read the docs for you and Cursor will help you.
I passed through quickly here, but there's a few examples of AI companies because there are a lot of AI companies, big and small, that use Cloudflare.
There are a few examples of those here also.
Absolutely. If you're doing AI stuff, you can build on Cloudflare.
You can store your data in inexpensively. If you've got big models, come, we can help you out.
Exactly. We still have time and I want to go to some of our history.
Lava Lamps has become a part of Cloudflare's history. Not in the beginning right away, it was like a construction, but how did Cloudflare become the news in terms of Lava Lamps?
Because we have a wall, it's all about science, randomness in terms of security.
How did that come about? I think the original way it came about was there was a discussion between myself, I'm trying to remember now, Matthew Prince, I think Nick Sullivan, who was head of research at Cloudflare, about entropy.
A long time ago, Silicon Graphics had done this Lava Lamp random number generator.
I believe the original suggestion actually came from Matthew because I was chatting with him over one weekend and we wanted to think about random number generation.
The thing is the motion of Lava Lamps is unpredictable.
If you put a camera in front of it, you can basically hash that and get random numbers.
What we decided to do was use the output of that to generate random numbers that really, really are predictable.
They're not pseudorandom, they're genuinely random. We did it. This is 10 years ago now.
Nick said it's a great idea. He started producing this signed random numbers.
He proved these random numbers came from Cloudflare. It just grew from there and there.
What happened was it took off in a way I think that none of us really expected.
We'd seen the thing that Silicon Graphics did a long time ago.
What we did in San Francisco was we had this wall of Lava Lamps, which actually causes a lot of power.
The reason the Lava moves is because they're hot. You have to use an old style incandescent light bulb which heats up.
Each of those is let's say 100 watts and you have a wall of these things.
You're starting to talk about kilowatts of power.
We actually started having them turn on and off in batches so the power was not too much.
There it is, yes. There's a Wikipedia article about it.
The original Lava Rand is Silicon Graphics. You can see that there's a row of Lava Lamps at the bottom that are off to save power and heat.
Now we have other random number generators.
In London, we have a wall of double pendulums.
If you attach a pendulum to a pendulum, the motion is chaotic and we have a whole wall of them.
We have something in the Austin office which is a whole moving thing which is moving randomly with air currents.
Lots of fun ways of getting random numbers.
We need them to feed into our system so we get good random number generation.
I can see in your tabs NCIS. Let's just talk about NCIS there because this thing really crazily got away from ourselves.
For those who don't know NCIS, it's a very popular, at least a few years ago, TV show all about investigation detectives in the Navy.
Look at this. There was an episode in which there was a plot that some company was using Lava Lamps for encryption.
They got the idea from now. They were using in a slightly different way than Cloudflare was.
At some point, a number of nuclear reactors started to melt down and they lost control of everything because of the poor implementation of encryption with these Lava Lamps.
Very different to what Cloudflare does. Luckily, the day was saved by this gentleman who throws a chair at the Lava Lamps, thus cutting off the encryption.
Honestly, I don't know how this works. Valderrama, the actor Valderrama.
There you go. You see, you know more about this now.
This is the only episode I've ever watched of NCIS. The reason he was so happy to destroy the Lava Lamps, here's the backstory, was that as a child, he had a Lava Lamp which smashed and burnt him, which stopped him being a baseball player because of the burns on his arm.
This is his revenge on Lava Lamps. Anyway, this was funny because this is obviously based on Cloudflare.
And so much so that we ended up making fun of it on our blog at some point too.
We did. Globeam Technologies.
Globeam Technologies. That's actually a picture of me and Matthew looking at the real Lava Lamp wall in San Francisco.
But I wrote this rather serious sounding blog post about how we had acquired Globeam Technologies.
But we wanted to assure people that the way in which we were doing encryption was nothing like the poor quality that they'd been doing that resulted in, I think, a number of nuclear reactors melting down or something like that.
It's like an open letter of sorts.
It was. I believe we still own globeamtechnologies.com as a domain name.
We own a lot of interesting domain names. Interesting. I found this 10 years blog post.
It was from Nick Sullivan in 2013. And it was all about randomness and entropy.
In a sense, it was, like you were saying, what led to the Lava Lamp wall.
It was 2017, I think, the Lava Lamp wall was created, right? Well, I think 2013 was when we had the original idea.
And, you know, over time, you know, we had to kind of like, I think, at the beginning, we had like one Lava Lamp.
And then over time, it got bigger and bigger and bigger.
And it became this crazy thing.
And then, I mean, Tom Scott came and did a YouTube video about it, right? With Nick, I think.
He did. Here it is. Helps keep the Internet secure. That's a great, his description is a very good one.
If anyone wants to understand about it, there's Nick describing it.
And, you know, Tom will take you through why we have a wall of Lava Lamps.
To be honest, it's scientific, but it's also pretty much security-based.
It has security in its essence, randomness. But also, it's, generally speaking, for the general audience, pretty much interesting.
It's a story that never stops being told.
It's unbelievable to me. We, you know, we did this thing, and it just, the number of times there's a press inquiry about it, or I see people taking a selfie outside the office in front of the Lava Lamp wall.
Who knew?
Well, I guess Matthew did. He suggested it in the beginning. True. And I think it's kind of cycles.
I was a journalist. And sometimes I've seen the story back in the day, and sometimes it's a new news article.
Someone remembers it and just tells the story again, because it's a pretty much good story.
People see it, and they're like, wait, what?
It's just such a funny idea, right? It's like, wait, there are Lava Lamps that protect the Internet.
And over time, and this is 2017, there was a bunch of blog posts in terms of deep dives, explaining the science that is there.
So it has been around for a number of times. And you'll be happy to know, I have a Lava Lamp at home.
So in case any disgruntled ex -impossible baseball player destroys the Cloudflare Lava Lamps, I've got the backup Lava Lamp.
Have the backup. I have a very small one. There you go. Thank you. You're all part of the solution.
Not really. This is too small. I bought this for my kids, to be honest.
But here it is. We had to end the show with a real Lava Lamp appearing.
There you go. I should have brought mine. It's in the other room over there.
Well, that's a wrap. Thank you, John. It was great. And we're going to be out for a couple of weeks, I think.
I think that's right. Something like that. Maybe it's this week, next week.
So we'll see you in September. Have a good summer slash winter, depending on whether you're on the top or the bottom, whichever way around the world it's meant to be.
Exactly. That's a wrap. See you.