Zero trust in railway OT: What worked, what broke

I thought about what to share with you, and what I'm not going to talk about is AI. I realize that some 60% of the content for this event is focusing on AI, but there is one thing that we don't want to do. We don't want to put train safety or the safety of the power grid in the hands of AI, because we definitely want to reach home safely, not probably reach home safely. And AI, being a probabilistic engine, a bloody good one, is still a probabilistic engine. So I'm going to talk about something rather basic, at least I hope it's rather basic, but nevertheless important, because our adversaries are using those nice tools against us. And if we don't get the basics right, we're going to be on the headline saying that Chachipiti hacked them. No, Chachipiti didn't hack them. It discovered the control that we failed to implement, and it used it, like any other good hacker, with skills. So that's what I'm going to talk about in the context of Critical Infra and OT and how to protect it with Zero Trust. I am Tõnu Tammer, I'm Director of Technology for Estonian Railways, and previously as a Director of National CERT. So I have seen some incidents in my life, quite a lot, actually. Let's see if this is working. It doesn't want to change. Ah, nope. Ah, we have another one. Yeah, so... Cheers. When I put the abstract together, I was thinking back in July what to talk about. And when making slides, you know, life does go forward. So some things are not quite as I thought about that. Yes, we are using tunnels for also OT. We are using Zero Trust Client or Warper, however you want to call it. That all works quite nicely. But that's more or less where the divergence between IT and OT become two different worlds. We have zones and conduits go per vendor. And we hope that logs outlive the device, because not all devices in that area are even capable of basic logging. But I will cover all of those aspects a little bit deeper later. A brief background. So over a thousand kilometers or a little less than a thousand miles of tracks, a number of stations, typical OT lifespan is 25 years. So Windows XP, folks, is a feature, not a bug. You are laughing, but that's what it is. And on the OT side, the technology in terms of maturity is where everything you put on a wire by default is trusted. There is nothing that explicitly forces authentication. Some of the newer things bring encryption, but even that to a degree is frowned upon because speed and deterministic approach is first and foremost that matter. You don't want an emergency shutdown valve to fail if temperature in the nuclear reactor goes a little bit too high. Perhaps authentication is not the most critical question at the time. But again, this is just to highlight that OT and IT are a little bit of a different species. So for us, a typical approach when it comes to OT is that it's very custom built, usually by a vendor or an integrator, typically operating abroad. And of course, in such a condition, you have need for vendors to occasionally come and check and do maintenance. And they do need access around the clock. And typically every supplier used to be a firewall change. If you work in an industry that requires this approach, raise your hands. Is this the case for you? Voila, we live in the same world still. So what worked? Putting stuff in a traditional way means exposing your services to the public world. It means that every port that you put out there can be hacked. Every door that you expose to the Internet can be opened. Lock, we say, is only to keep animals in bay, not humans. And one week tunnel means trusted insider is in and they are able to roam around wherever possible. Now with Cloudflare, we have taken out the doors. No door, no lock, nothing to pick. Easiest way to defend is by denying the opportunity to attack. You cannot brute force a door that is not there. This is what we gain by having tunnels. And easiest way to put different services through wire or through Starlink or whatever Internet you want to use is through tunnels. That doesn't matter anymore because it's an inside out connection, not outside in. Put CDN in front of that and you have a pretty amazing tech stack to even cater over a satellite because CDN actually consumes a lot of volume. We have quite a number of different partners that we need to work with. Trust me, not all of them are as mindful of security as we are. We even have a vendor who has introduced ransomware to a number of their customers. Why? Because certain solutions you don't have many options to choose from. If you know who your vendors are, you also need to adjust how you work with them and how you defend given the mindset that they are. On the OT, there is no identity. A SCADA controller or an ICS controller does not have an identity. It only has a function. It does what it's programmed and that's it. However, people that work with those systems, they have an identity. So identity in OT needs to move with a person, interlocking, nothing. It does what it's told and programmed. An engineer who works or maintains the system has a name, has a device, has a multi-factor authentication and has an identity. As long as he or she works for our contractor, he or she also works for us. So we can control how they have an access to our system. And that is the best part of the technology that Cloudflare has to offer in this context. So we rebuilt a perimeter from the classical approach of having a VPN. And we call it the Zero Trust. And, you know, in some sense, you can argue, is it? But it definitely is not a wrong answer. So we have and we are working on that to have a number of routes, one route leading to one destination. It's easier when we have virtualized services. But as I said on the OT side, if you put everything on the wire and how you normally do, you divide systems into zones which have the same security requirements. And inside you can have multiple different VLANs. But because those systems don't have an identity, you can't put a tunnel on a SCADA controller. The only thing you can do is you can create tunnels to specific zones. And you can control access through those tunnels to these specific zones. And you can regulate how much of the surface you are going to expose. I hope it makes sense. And what we do is basically we have a number of jump hosts. And everyone who has in principle access, they can connect through that jump host into a specific OT network segment. The beauty is that now we have logs. Before, when people used to go to a data center physically, connect to a serial port, guess how many kilobytes or megabytes of logs were left behind? None. Because logging is for weak. I promised that I wasn't going to talk about AI, but I maybe promised too much. A practical use case, when presenting or preparing for another speech about a month ago, I asked one AI agent to help me with slides. And the best way to help with slides is if you give them access to a real configuration, because I'm not in the habit of lying. So what would be the best to actually show what is there? And it made some nice slides and perfect. And the day before I went through, I was like, one point I did not quite catch. I did not remember a case going through that was on the slide. And so I kept poking. Okay, explain to a dumb person like me, what is on that picture? It took some time, but I realized it found a misconfiguration in the system. A tunnel that was half decommissioned, but not fully decommissioned. It took a further half an hour to back and forth. How do I explain that issue into a Jira ticket? So that a network admin can then completely decommission that leftover weakness. None of this triggered an alert, because at one point this was part of the configuration. Public DNS still pointed to a tunnel, but it did not resolve anywhere. And nothing alerted, because everything was by the book. Typically, these things are one of the biggest cause of incidents that happen. People making mistakes. If you think that hacking is about trying to dig through a vulnerability or a CVE, yes, this can happen. But the easiest is you find misconfiguration, because we're human. We make mistakes. AI is great in helping you understand what you have put in place and what you wanted. So hands up if you have read your public soul and verified that everything there is correct. There's a couple of hands raising. That's good. I am in awe. But this, I think, is worth checking every now and then, just to make sure that the house is still clean and intact. Another case to mention. One of the very reputable vendors in the field also wanted to do remote access and maintenance. If you know anything about OT, the typical approach is you never connect it online. It's very isolated. Put that into the context that they want to work remotely, and we can debate over a beer if this is considered putting it on the Internet or not. However, this is not putting it on the Internet. But nevertheless, they took an old 4G router, plugged it directly into an OT network because they wanted to do some maintenance, and, oh, it got compromised. Because no one ever does it like that. It means that no one ever updated the firmware. People probably did not even recognize that it was there. It took some sharp questions for them to say, yes, this probably was not the best idea. But this wasn't about punishment. It was just for them to recognize that, guys, let's not do this again. So how did it come out? When it started sending smishing text messages. And who noticed? The mobile operator. Yay for the good guys. It managed to send a few thousand of text messages. And that's that. How did it get compromised? Who knows? Because a firmware a few years old connected to the Internet over static IP. I mean, I have seen devices if you take out of the box with a default configuration, you plug it in the Internet and in 20 minutes, someone else also lives there. So the world is what it is. That's why denying an access is, I think, the best way how you defend your crown jewels. So the best Zero Trust control is actually a power button. When I say Zero Trust, it's named account, MFA and reason. If you let your contractors into your data center, raise your hand how often they need to be accompanied by one of your own engineers. Some hands raised. Good. On Cloudflare, it's also possible to have four eyes principle in the sense that if they request an access, you get a notification whether you open the door or not. Just because they have access doesn't mean that they can access. You have to physically open the door. If the jump host you operate, does it need to be up and running all the time? Or do you only bring it up when there is the maintenance time? Again, the way how you mitigate risk, multiple ways to do a little bit at the time. And the beauty is, all of this is logged and forwarded automatically. And when you ask, is it safe? Well, you could argue, but it definitely is secure. And in my view, you cannot build security better than this. So, where Zero Trust bends in terms of OT, we have a lot of legacy protocols. And Cloudflare is built for modern worldwide web. So, one size doesn't fit all. And we have to find ways to make it work. Zero Trust and safety, some things still need to go on the web. Not because it's necessarily a better way. But this is the mindset of people who still write the standards. Two years ago in a conference, one guy writing a standard was proud, like being on the stand where I am, and say, now the standard supports TLS. I nudged my colleague, I was like, I wonder which version? The first one, probably, he goes. So, at the time when we talk about post -quantum cryptography, this is the world we live in. Safety certified systems, like no agent, every change needs a vendor assessment. As I said, probably safe is not safe. It needs a very deterministic approach. And jump posts from vendor, replace them with yours. Because you are ultimately responsible. So, if you're here, you're a customer, use them. Three things to check on Monday. Check your links. Make sure nothing is visible that is not necessary. Check your DNS zones, especially for those guys who didn't raise their hands. And widest route. Find it and ask where it really goes. And I finish my presentation with a humble question. If your vendor's box starts sending weird text messages, who would tell you? Your CM, your mobile operator, your vendor, or your ransomware operator? Ladies and gentlemen, thank you.