Deep Dives Live - Episode 0

Hello, everyone, and welcome to This Week in Net. It's the March 22nd, 2024 edition. And this week, it's an episode where we're going to get technical.

So we start a new segment called Let's Get Technical.

I'm João Tomei, based in Lisbon, Portugal. And with me, I have, returning to the show, podcast, RCDO, John Graham-Cumming.

Hello, John.

How are you? Hi, João. It's good to see you. Good to hear you. It's strange to be talking to you when we're actually in the same office, but we're in different boxes.

A few meters away from each other. It's like he's over there. Exactly. But we want to share the screen.

So there's advantages in doing it in this way. So let's start.

Let's get technical. Our new segment with you. This week, what do you want to highlight in terms of technical little things that we want to share about Cloudflare?

Well, I think we should talk about random numbers. Random numbers are a good subject.

I do like randomness and entropy of switch. And I think it's perhaps not appreciated by many people how important random numbers are to the functioning of the Internet.

In particular, all the secure connections we make on the Internet.

So for example, like the secure connection I'm making with you over Zoom right now, right?

Or a secure connection you make to a website. All of those rely on good sources of random numbers.

And we've talked about random numbers at Cloudflare quite a lot in the past in the context of lava lamps.

And so I have a silly website.

I'm going to show you the silly website and then we'll talk about that website.

It's called Lava Millions. Let me share my screen. OK, so this is Lava Millions.

And this is a website for picking lottery numbers, in particular, for picking the Mega Millions lottery in the US, which has five balls and a golden ball at the end, a pick like that.

And this website, which you can go to, is all on Cloudflare, uses Cloudflare Pages for the website itself.

Cloudflare Pages functions for the dynamic part of it, which is picking the balls, and it gets random numbers from Cloudflare's entropy system, in particular, from lava lamps.

Now, we've actually extended beyond lava lamps now, but I think we're best known for our lava lamp random number generator.

And so this website will get you a new set of numbers.

Every time you hit refresh, you get a new set of lava lamp chosen Mega Millions numbers.

This won't make you win any more random numbers, but hey, they're picked by lava lamps.

So there we go. They're pretty random.

So we had, during Security Week, a blog post that updated a bit some information about the randomness that we use in different offices.

You mentioned the lava lamps wall that we have in San Francisco, but there's also, in Austin, we have suspended rainbows.

In London, we have double pendulums, a wall full of double pendulums.

And in Lisbon, we also have something potentially in the future. And we have a blog post about that, so you're absolutely right.

So the original lava lamp wall, which people know fairly well, is still operating in San Francisco.

In London, we have these double pendulums.

And the reason we use double pendulums is their behavior is chaotic.

So unlike a pendulum, which swings back and forth very regularly, which is what we wanted to do, if you attach a pendulum to a pendulum, the motion made is chaotic.

I cannot easily predict. I mean, the starting conditions cause wild changes.

And we have a whole wall of them. And we'll get a picture up here and show it in a bit.

And then in Austin, we have a thing that's a bit like a mobile.

So it's multicolored things hanging from the ceiling, which is affected by air currents and constantly in motion.

So giving you something else random.

And in Lisbon, it's a bit of a sneak preview. In the new office, we're going to have a wall of waves.

So it's actually water, cold water, orange, in plastic containers that rocks back and forth, creating a wave-like motion.

And if you're my age, you may remember these were a popular desk toy at some point in the 80s.

And so now they're back in the cloud for office here. The idea of all of this is all of these things that are moving in this very, very hard to predict way are in front of a camera.

And the camera takes a picture periodically. It then takes that picture and what's called hashes it.

So it makes a random, that makes a number from the image.

And then those numbers are combined together and sent out to our network and they are used to kick off the process of random number generation on all of our machines in a way that is, well, we hope, impossible to predict.

That's the idea.

When we mix in lots of these sources of randomness from different offices and also from the machines themselves, so that when we need to pick a random number, we're able to pick that random number and, you know, use it for communication or a website or whatever is going through Cloudflare's network.

You mentioned there something really interesting.

First, that's like a news that the Lisbon office will have that specific wall.

And it relates to the history of Portugal, like the relation to the sea.

So there's a relation also there. And I remember I'm, I was born in the 80s and I remember those little things, those rocking things.

Those were amazing.

Even like as a child, you will play with those for a long time. Exactly.

The motion of the water is, I mean, just things like that. So we're, we're having some custom made for us and there's going to be a wall of them.

We can actually bring up that blog post.

I have it right here. If you want to take a quick look at it, let me share my screen again.

The blog post is called Harnessing Chaos in Cloudflare Offices, which is a fun title.

It's not about the chaos of the people, but the chaos of things moving.

And so if I scroll down, this is the lava lamp wall in San Francisco, but the motion of the lava there.

And then in London, we have these double pendulums that look like this.

You see they, and they move in this wild fashion.

If this at all looks familiar to you, then you have to go look at, watch Iron Man 2 again.

Because Tony Stark has one of these on his desk and it's from the same company.

And so you can actually buy these, they're, they're fairly, they're just powered by batteries, but their motion is, is wonderful.

And then you can see a whole line of them.

And then in Austin, we have these mobiles, which are, you know, they're taking the light and also the, the, the air currents in the room and they move around again.

And we take a photograph of it and we feed it into the system.

So if you're interested, you can see what's there and you can also see how this stuff actually gets mixed together and turned into a random numbers that gets sent out to our entire network.

And that's the, that's our system that we're using.

And you can then use that yourself. So there is a thing called the DRAN, which is a distributed source of randomness.

There's a whole bunch of companies come together and create a random source, which is something that can be verified that it was who it came from.

I think called the League of Entropy, which has been around now for about five years.

If you need random numbers, you can use that.

It's verifiable and public randomness. It's a cool name. Yeah, it is.

The League of Entropy is really good. And there's some really good graphics done by the graphics team here around the, around the League of Entropy.

I'm sure we should be able to see that blog post.

And this is, this is a collaborative effort around, around randomness.

We're also going to introduce our own endpoint.

This has not been released yet for Cloudflare customers. So if Cloudflare customers want to get random numbers from Cloudflare directly, they can through, through, through the web.

And so that will be, that will be coming out soon because in my Lava Millions website, if we, if we dig into the code, let's just get some random numbers again, there you go.

This is the actual code in Lava Millions, it's on my GitHub, it's, it's publicly available.

And I actually call out to a, for a website, csprng.xyz, which is run by a previous, was created by a former Cloudflare employee.

And it actually uses the Lava Lamps to get the random numbers.

That's going to get replaced with a Cloudflare owned randomness source.

So you'll be able to do this kind of thing yourself. But this is a lovely example of using Cloudflare pages.

So the Lava Millions project, which is here, is, is the website you look at, and it is entirely created through Cloudflare pages.

And the main page is through a pages function, index.js, which is a little bit like a Cloudflare worker, one of the, you know, of code that you can write, but it is attached to a static website, which is through Cloudflare pages.

And so this actually gets executed, and that's actually what's getting, this code is what's getting executed when I do this and refresh.

So there's the code and you can have a look at it and you could make your own version.

You will also see that all the images are here.

These are, these are all the images of the balls and everything.

And those images are on Cloudflare pages too. So the whole website is in this one repository.

And if I were to make a change here in GitHub, I, I could do it through the user interface or through command line and push that change to the master branch.

Then the, that would then get automatically built by Cloudflare pages and pushed out.

And it just takes a few seconds. It's incredibly powerful. It's one of my favorite Cloudflare products and they're all my favorites.

It's not like children, right?

You can't have a favorite, but Cloudflare pages is such a slick thing.

If you're creating a website that consists of static assets and dynamic things and definitely with pages functions, it's really, really powerful.

So there you go.

That's, that's Lava Millions. Now come on, let's get some more numbers.

These are the good ones. Maybe, maybe we'll add Era Millions for Europe in one of these days.

Europe. Yeah. We should have a Euro Lava or something. Yeah. We can, I mean, easy to create.

Maybe someone can fork my code and create it. Lava Era Millions or something.

That would be great. More, more things you want to share.

We also had a few blog posts recently with the more technical side. That's actually something very frequent in our blog, technical blogs.

Anything you want to mention?

There was one that really fascinated me, which is around an attack on AI chatbots.

And the reason this really fascinated me is what they do in this AI chatbot thing is that they, you know, if you use a chatbot, if you ever gone to one, you, when you type stuff in, you get the reply back in a progressive fashion.

Like words appear on screen.

And that's how, you know, these chatbots are working. They're predicting the next word it should say.

And you get that sort of, it's almost like it's typing on the other end and words are appearing.

Right. Well, there's some researchers at Ben-Gurion in Israel who worked out that even though they couldn't see what words were being created, if they could sniff on the network, they could look at the network communication.

So between you and that, so suppose you're using chat GPT, right?

Or our case, a chatbot on Cloudflare's workers AI.

Suppose you're communicating with that as well. And all I can do is I can snoop on your traffic or you're in a coffee shop and I'm in the same coffee shop and I look at the traffic, but I can't decrypt it because it's obviously an encrypted connection because each word is appearing one by one, it's actually possible to see the individual lengths of the packets being sent to your machine.

And then knowing how a chatbot tends to produce its responses, what the researchers did was they built their own model of how the chatbot works without ever looking at the decrypted traffic and were able to predict what you were reading on your screen.

And it's a fascinating piece of research because you're actually using like as an AI, you're sitting there using an AI and they're essentially using an AI to predict what the other AI is saying.

And one of the reasons I found this super interesting is if you go way, way back to 2004, so 20 years ago, I was working in email spam using what we called machine learning.

If I wanted to, I could call it AI now.

And I built this thing which did email classification, did spam filtering by looking at the words in a message and predicting whether it was spam or not.

It was quite successful, an open source project. You can still find it. It's called Popfile.

And there was a conference at MIT called the MIT Spam Conference.

And one of the things that I had done was I'd presented Popfile the years before and I was like, because it got quite good, because the machine learning spam filters got quite good, I was like, how is someone going to beat them?

And I said, well, maybe they're going to use machine learning, we'll call it AI now, to beat another one.

So what I did was I pitted my own program against itself.

So I had a, I had a, what I did was I made a program send spam emails and then on the other side, they got classified by the spam filter, right?

Machine learning thing.

And if one got through, then the end user would read that email, right?

Cause you'd see it, cause it got through. And you would embed in the email, a little signal that goes back to the spammer.

So often people do this for marketing purposes, a little hidden image, right?

So the spammer could know, ah, this message got through.

And what they would do then is say, ah, well, that was this particular message.

I will, and it would train its own AI to say, ah, this is the kind of message that gets through.

And by the way, I never heard back about this other message.

So that didn't get through. So I, I trained, basically trained a mirror of itself.

And so we could learn for an individual user, what words and what to say in an email, they'll get it through their spam filter.

So it was pretty good AI against AI.

You were, you were acting your own system with your system in a sense.

Two, two AI machine learning things talking about. So this thing from Ben-Gurion about chatbots, a similar kind of thing, right?

Which is like, they built an AI that could recognize what another AI was saying and then said, well, we can do that.

Now, um, they reported to us and OpenAI and a few other people.

Um, and the solution it turns out is not very complicated. You just don't leak the information about the lengths of the words, right?

So when you're sending back, you throw in some extra random data and then the other end can't tell how long the word was.

Um, because it turns out that if you can see the sequence of words, you can kind of guess what the sentence was and just by knowing the lengths, so we throw in some randomness and it's fixed.

So we, we patch that, we fix it in all the stuff that uses us.

We fixed it on our API gateway, sorry, our AI gateway as well.

So if you're using our AI gateway in front of your AI, we will also add randomness so this can't be a problem.

And other providers fixed it too.

But it's kind of interesting, my perspective, because I was doing kind of this stuff 20 years ago.

But also that AI's pitted against AI's is, is a thing. And so, you know, I thought that was a fascinating piece of research and we have a nice blog that goes into a lot of detail about this.

Here's the blog, right? It's this one, right?

That's the one. That's the one. And it was written by Celso that is here also in the Lisbon office.

Yeah, I don't know if he's here in the Lisbon office right now, is he?

He is, he is. He is. Okay. Okay. Well, so yeah, there was the, you know, there's a, this was a side channel where you could, without being able to do any decryption, you could predict what someone was doing.

Side channel.

Yeah. But it's interesting, you mentioned something there that I think is really fascinating, which is even hackers are using old methods, right?

Even if it's in a more sophisticated way, there's other things, but there's old methods to mitigate, but also to hack in a sense, a situation, right?

Well, this was very clever, right?

This was a very clever use of an AI to predict what another AI was doing.

And so I think we'll see this kind of like adversarial AIs going up against each other, which will be super interesting.

True. Yeah. There you go. That was, that was one of those fascinating reports came out.

Over the last few weeks.

Any other blog you want to emphasize? We had a, an undersea cable failures that brought 13 countries.

You should talk about this because you wrote this.

This was really, this is, this is, was it off the coast of Cote d'Ivoire? Is that where it was?

It was off the coast. It was. Yeah. It started actually in the Gambia country specifically.

So it was, it started South of Senegal and I can actually show on radar in the outset center, the perspective that shows in Africa, the specific countries where this happened.

So Senegal is here. So just North of Gambia.

So it was after Senegal that this happened. So apparently undersea cables were, had more than one actually had a few failures.

And then it was like 5am where it started in Gambia and then it went to Guinea and went to Liberia, Cote d 'Ivoire, Ghana, Burkina Faso.

So it was in total 13 countries. It went all the way to South Africa also.

In South Africa, it was just one network, Vodacom, that was impacted, not the full length of networks in the country, Nabibi also.

In some situations like South Africa, it was just for a few hours, the outage.

In others, which is the case of Cameroon and Ghana, after more than one week, it's still impacted in some of those countries.

It's Cameroon, the case, and also Ghana. So it's a good reminder, I think, of how undersea cable failures could have a real impact in country Internet, in a sense.

And was this made worse because of the cuts around the Red Sea area?

So there's not like an official, someone putting in an official way that that happened, but Microsoft, for example, did a report where they considered that could be of influence here.

So because there's less redundancy in terms of where could the Internet flow or reroute, if there's a problem, in this case, where three undersea cables in the west of Africa, because on the east part, on the Red Sea part, there was already problems from a few weeks ago.

There was less redundancy, so less ways of traffic to reroute, in a sense.

So that could be also at play in terms of the impact that this had.

Yeah. Looking in your outage center, it looked like we've got some outages in Ukraine today.

We did, actually.

It was by military action. So it affected power stations in the region of Kharkiv.

And it was since 3 a.m. this morning that that happened specifically. And we had a tweet about it to explain the impact.

Apparently, it was like three or four regions that were impacted.

One of those, it's where there's the nuclear power plant, forget the name of the nuclear power plant.

But so this was still ongoing at this moment.

And there was also a cyber attack last week. But this was a Friday, March 22nd incident that happened specifically.

And I think I saw Myanmar as well on that list there somewhere.

Myanmar, I think, yeah, it started last week.

Last week, yeah. Regarding maintenance. Very cool. Very cool. And Philippines also recently, already on Monday.

And that was also a cable cut. Cable cut for an hour or so.

Yeah. So a good reminder of the Internet, it needs resilience.

Also a good reminder of all the stuff there is on Cloudflare radar. True, true.

We have a lot. We have a lot of stuff in there. Yeah. Even recently, we had email security.

We added email security. Yeah. Yeah. Yeah. That's a really good, really good thing as well.

Yeah. Well, good. It was lovely talking to you.

Thanks for having me back on this week in there. Thanks for doing this and potentially let's do it again.

Yeah. In the future. One thing before you go, it was last week that was Pi Day.

Do you have any story related to Pi Day or to Pi specifically?

Well, so you've touched a difficult subject, right? Which is it's only Pi Day if your dates are American style, right?

Oh, that's right. Because it started in America.

Yeah. So the idea of that though, which a friend I used to work with described the US date system as middle Indian, right?

We have, you know, big Indian and little Indian numbers and the US is middle Indian, you have the middle beginning.

So it is, I mean, I was fascinated by Pi when I was at school and I, me and another boy decided to memorize it and I memorized it to about 50 decimal places.

And I still, I still use it, or at least I still know it as I just, one of those random things.

And occasionally I will, you know, it's a bit like a Rubik's cube.

I can do Rubik's cube and occasionally I'll get out a Rubik's cube and I'll just do it like, oh yes, I can still do it and put it away.

Memorizing Pi is a bit like that.

I'm like 3 .14159265358979323846264383279502084197169399. And then I think it's 37510 and I'm like, great.

I can still remember that. And there's a really lovely website, which is how many digits of Pi does NASA use for navigation in space?

And I know it's way more than necessary to do some navigation. In fact, 50 decimal places, if I remember this correctly, is enough to do like measurements and calculations at the size of the observable universe.

Down to, yes, I think that's the one, down to, um, with the accuracy of the size of a hydrogen atom.

So it's not that useful. In fact, yeah, here it is. You see, they only use 3 .141592653589793.

That's enough for some, you know, interplanetary navigation.

So when I learned it to 50 decimal places, it was overkill.

But it's quite fascinating as a concept, a mathematical concept.

And of course it's useful not only in space, but only also to the Internet, to more small settings in a sense.

It's always important in a sense. Well, I'll tell you a funny story about it from a security perspective is that, um, a long time ago I had a credit card whose pin number I kept forgetting.

And I was like, well, I want to write this pin number down.

And, but I don't want anyone to know, no, I don't want us to do. So what I did was I had a, I had a notebook in which I used to keep notes.

And so on one page I wrote out, um, pi to 50 decimal places.

I also wrote out a few other things like E, the square root of two, stuff like that.

So it was like a crazy nerdy page, right?

But because I knew the first 50 decimal places of pi, I made a mistake some way into the expansion where I put the four number, um, pin number of this, this, this credit card.

And so to me, I could look at it and go, those aren't the four digits of pi.

That must be my pin number. And so that was a way of, uh, security sort of security a bit, but you would have had to be particularly nerdy to discover the pin number of that credit card by spotting errors in the expansion of pi.

That's a John Graham coming for sure. Then it makes sense. Makes sense.

It adds up. And that, that's, we, we ended with pi, so that's good. So we ended with mathematics and that's wrap.

Thank you, John. Cheers. See you out. Bye. Cheers.

Bye. Hi, I'm Ryan and I'm a community manager here at Cloudflare based in San Jose in the United States.

And here are some of the recent social highlights that we wanted to share with you.

Up first, there was a site operator not using Cloudflare who incurred a whopping $100,000 bill for their static site.

Their Reddit posts generated more than a thousand comments and 8,000 upvotes with many of them suggesting Cloudflare as an alternative.

On the bright side, they've already updated that they've learned their lesson and have migrated to Cloudflare.

News of that horrible billing experience spread to X where NixCraft commented that if you want to avoid billing surprises, always hide your site behind Cloudflare.

We agree.

Good tip. Still in the realm of avoiding billing surprises, a developer's post on X highlighted the importance of easy to understand pricing, stating, quote, it's impressive how easy and straightforward it is to understand the pricing for Cloudflare Workers compared to AWS Lambda.

I think that's correct. We received some great responses from Security Week announcements, one of them being the new firewall for AI.

Developer and influencer LevelsIO amplified the news to his 400,000 plus followers, including another comment about billing.

Seems that there's a bit of a theme here.

That being said, not everyone is always focused on billing.

One of the most common things we see on social is excitement about how easy Cloudflare products are to deploy.

Under an hour is a pretty common proclamation with a lot of our different products.

Here we've got someone who's clearly excited that we are open sourcing our Rust framework, Pingura, which if you didn't know, is used for building programmable network services.

One of the other areas that we see a lot of activity around is observations from Cloudflare Radar.

A recent blog post about cable failures in Africa even caught the attention of BBC Africa, and they went on to write about it.

And last but not least, some chatter around our Xeroz product and how it brings one solution for all types of third party tools.

Remkus DeVry, a WordPress veteran and performance specialist, made a tutorial about it.

And that's it for this round of social love.

See you next time. So before we go, let's share some recent Cloudflare news from the week.

Let's start with our blog.

On Wednesday, we announced that we introduced Warp Connector. So in a sense, it's for Zero Trust administrators to deploy our new Warp Connector for simplified any to any connectivity.

We also discussed hardware, in this case, redefining Fleet Connector for management at Cloudflare, all about processes around service provisioning, maintenance windows, repairs and diagnostics reporting were reaching their limits.

So we had some solutions there. We already mentioned the undersea cable failures cause Internet disruption for multiple.

We also explained the impact for Cloudflare users regarding the upcoming Let's Encrypt certificate chain change.

So stay tuned there. And also we have the Security Week 2024 wrap up.

For those who want to see all of the blog posts that were published during our Security Week on the different topics at hand.

On other news, Cloudflare was named a strong performer in the forest wave, the Security Service Edge solution specifically regarding Q1 2024.

There's a download that could be made of that report if you want to see details there.

And also this announcement, the Kendrill and Cloudflare announced global strategic alliance to drive enterprise network transformation, multi-cloud innovation and Zero Trust security.

So this is an alliance to focus on modernizing IT infrastructures by streamlining connectivity to multiple clouds and data centers.

Before we go, there's still a small teaser with a conversation I had with our very own engineer of the research team, Thibaut Meunier.

So here's Thibaut. Hey everyone, I'm Thibaut, working at Cloudflare as a research engineer based out of France.

And in the next episode of Cloudflare Research Corner, we're discussing how Cloudflare is working towards making an end to CAPTCHAs.

If you don't know, if you've ever clicked buses, if you've clicked fire hydrant, well, CAPTCHAs is probably what you did.

And Cloudflare is definitely working to end, to make an end to this madness.

And one of the ways to do that is with Privacy Pass.

It's a privacy preserving protocol that will allow you to go through CAPTCHAs without revealing much information about yourself.

To understand more deeply how it works, you can listen to the research corner that will happen.