Immerse Sessions: Performance and Security with Printify and Hostmasters

Thanks once again everybody. Yeah this session is going to be similar to the one we had earlier on but primarily around performance and security and of course I've got Milan and Dimitro here with me from Printmasters and from Hostmasters and I'd just like to start with the first thing is um if you could introduce yourself i'll introduce myself i'm tim down i'm the lead of the solution engineers and um it's great to see you and we did speak briefly uh last week about uh what we'd be discussing today but if you could just introduce yourselves and uh and the businesses you represent so milan first hello hello uh thank you for having me very excited to speak here today my first time so bear with me uh yeah well i work as a SRE product manager in Printify. What Printify does, who we are. The company was started in 2015, Riga, Latvia, with one bold idea to bring financial independence to anyone. So we are print on demand platform. What that means? That means that we connect main pillars together first one is merchants how we call them it could be you me company youtuber singer whoever basically or with an idea or brand he or she or whoever wants to put on some white labeled product what the white label product is for us it's trousers hoodies t-shirts caps hopefully those Cloudflare bags are printed by Printify. Maybe not anyway. So we currently have more than a thousand products in the database. The second pillar is our print providers who are basically the facilities who can print your design or your idea on top of the widely product and send to the end customer. We have roughly 90 plus print providers in more than 110 countries around the world so pretty big market and the last pillar are end customers so you might ask how we connect those together we connect those through the e-commerce platforms marketplaces and so on so we support the the biggest one you might Etsy, Shopify, by the way, your customer, then Amazon, eBay, Walmart. We support API. So if you have any other eShop platform or eCommerce platform, which supports API, you can sync products from our platform to eShop of your choice. Or just with a few clicks, you can open the eShop on our platform. on our infrastructure and start selling and monetizing your ideas basically instantly. Great. Thanks very much. So, Dimitro, your own organization, Hostmasters, can you just give a little overview of what it is that you guys do? Sure. Okay. Thanks for inviting me here. I'm thrilled to speak in front of closer audience and, well, your company. Uh, we are... I should call ourselves the railroad of internet right it's an old technology domain name system quite boring rusty it's great when it works when it breaks you're getting angry so Cloudflare helps us to deliver data faster so every time somebody types the domain name which ends in .ua or maybe it's used as part of the email address so maybe there's some kind of back-end processing it's one of these hundreds of servers run by us and by our partners not just call fair others and we are doing this as a private company from 2001 uh yeah i was one of the original founders i'm still at stuff sometimes i feel it's like a lifetime job and but i do have other interests so that including my kids so um yeah we can talk about this in more detail but let's say it's not a typical company which would buy a services like so from printify it's more like a what they call back-end infrastructure and we do have a few other things we do but these are mostly concerned with domain sales our registry partners we have about 100 customers about 10 of them are outside Ukraine we have about 20 staff we have not had serious incidents since beginning of full-scale Russian invasion to Ukraine from 22 but as everyone in the we are affected by the war and we're continuing to be affected I'm currently residing in Lithuania I'm actually the only staff that is located abroad we have more people looking up located abroad but with the time they move back to Ukraine thanks so I hope that's was enough and we can follow up later yeah that's a good that's a good overview absolutely well I mean let's let's just cover off filming one of the first things you enabled was DNS firewall I believe right when you first started using cloudflare um tell me a little bit about your experience when you first enabled that that led you to to go on further yeah uh yeah dns firewall yeah we use three products dns firewall is one of them huh actually my first encounter with dns firewall was back when my good friend and former employee of cloudflare martin levy was still working there he was one of people who i guess made this product popular it was Basically, a thing that takes your DNS traffic, resolves it, and then caches the data results that provides by your back-end. It's similar to WebFirewall, in a way, because the same thing happens. Bad agents hit the Cloudflare infrastructure, it responds, filters out bad requests, then the good ones are optimized and ultimately handled by your back-end. The difference is that while DNS is different from the web, we use... it uh we actually started using cloudfire products before this new war started it was if i recall correctly end of 2021 and by early 22 we were in extensive talks with cloudfire stuff uh to choose from the big set of products which one would be best for us it was one of the three we also use web firewall which is your typical website organization that everybody knows and loves and we use magic transit, which you can talk about later. So we still use Sardinas firewall. We use it with the ComUA domain, the biggest Ukrainian domain. I'm happy. It's one of the providers we use. It's not the only one. So we still handle the traffic using our own infrastructure. It basically acts as one of these resiliency parts. Like, you know, we have multiply ways to ensure things don't break. So it's one of them. Yeah. So we'll come back and talk about Magic Transit in a little bit. wanted to come back to you Milan you talked about how you have a lot of print providers out there and then you've got a lot of people that are printing their own merchandise for either their websites or maybe instagram or whatever it might be so there's a lot of uh different types of people that will be using those products does and does that almost to an extent that sometimes people can be polarized by either content creators or types of people do you often see attacks on particular brands or merchant shops that have been singled out, maybe through their actions or words or any other reason that somebody felt was important? Yeah, absolutely. Well, you know, if you are a small company, no one is attacking you. Why? There is no reason for that. But as soon as you grow up and you basically extend your. surface of attack, which we definitely did when we provided our eShop platform where users can, on our infrastructure, basically start selling whatever they want. Then obviously there are some topics or themes which are polarizing, let's say. So yeah, we immediately saw DDoS attacks on these websites, which effectively, you know, scaled all our infrastructure you know I just don't want to repeat what was said already you know of course you have a scaling so you scale but you cannot scale indefinitely you hit some limits and then you have a problem right luckily we were with the Cloudflare just before even that though on the free tier so that was you know maybe hit for us or right time to go to business or even enterprise level and utilize uh you know security features you you you have in your in your suite uh so one of those uh ddos protection uh very very cool feature and together with bot management actually which is also a part of the attacks we we see that because of course if you have a stolen credit card you try to buy whatever you want whenever you want and get get the profit out of it right so we need to we have to basically distinguish whether it's a human or a bad bot or a good bot and so on. Everything was said before. So both these technologies, both together, were able to mitigate the really vast majority of the of the threats we saw. One number every month, Cloudflare blocks more than 100 million security events. our on our platform 100 million security events every month so it's a quite substantial number and with these technologies we typically after we implemented everything and of course we were hit by DDoS first right everyone everyone did so after we implemented hopefully correctly everything nowadays we realize that there is a DDoS attack just because of your analytics engine and emails with reports we receive so hopefully fingers crossed we are covered but not only these technologies we use also also other from your security suite i'll come along to that sure absolutely but one interesting thing i think is um yeah the 100 million uh security events but also i would like to talk about some of the credential theft checking you're doing as well so we'll come back to that because that's pretty interesting um so dmitro uh we did talk about um how you're protecting the dot com ua uh domain one of the things you did to protect that was put on magic transit so you want to tell me a little bit about what led you to put magic transit on and how that helped you protect that domain oh sure uh well yeah magic transit protects more than just the domain and by the way i was really thrilled with the operational support we got because it's not just about this product generally huh i mean we had open tickets with cloudfare during our initial deployment and i would say it wasn't always smooth sailing and it wasn't always our fault okay sometimes it's hard to say no there are too many pieces this lego puzzle is you know it's harder than some of my kids are doing so the thing that magic transit is not a solution by itself it's more like an instrument which can use to decide which part of infrastructure are kind of um how to say try to explain it like a magically mirror it to the cloud fair network what does it mean exactly imagine and really trying to build an analogy here imagine you know you have a house right you can build a physical security you can make the wall secure right you know how the platform works you have another house right the one that everybody comes to and then they send your packets over through well magic transit is a bit different it works with every kind of traffic and it's yes it's sent to the network the cloud that provides then these packets are then tunneled so sent over some special protocol to your network so basically you're still kind of dealing with the traffic and again cloudfair network protects you from that where we use this well we have multi-played data centers well usually these are computers located in europe or united states or ukraine and we have used our own ip address space with them so that's a prerequisite and what you do is that you describe the connection between your network with cloudfair by submitting the special letterization so you let cloudfair be your isp in a way that they're kind of your magic well virtual ISP that ultimately sends traffic to you through usual connection that can do stuff like swart any kind of attack or probing towards you you can also control that because that control panel you have allows you to set up some rules it's not just yes or no you can turn these things on and off and I was asking for IPv6 support I was told during our latest call with our assigned engineer from UK office that it's going to be delivered real soon. I know we are not the one customer waiting. There is a challenge because the PV6 is still relatively 20 years in the making, yet new technology. But yet we have never had this thing that I mentioned failing, and it allows us to sleep peacefully and not think just about the DNS specifically or web sites. Let's say about our email server or about our custom so-called EPP, domain registration protocol, front-end. So that's more than that. It's something that, in my opinion, every company with its own hosted infrastructure should use. Yeah, there are not so many of these, but we are one of them. And so I think this is really geeky, well-designed, and I would say probably unique product. I was really looking for competitors. because it's kind of a due diligence we have to do. Like, hey, what if Lawfare goes bankrupt? Okay, what if we cannot afford that? Okay, wait a second. Okay, what if something else happened that we have to change the provider? Well, I really can't find a company which is offering comparable service at comparing feature parity. There are some options there. I don't want to name them because that's not that kind of event. Still, I wouldn't be changing. So, yeah, I guess I'm admitting this is like the best partner in the world, almost like Well, yeah, so I hope that answers the question. We're still using this daily. We don't turn it on and off. I know, for example, a company which you use, which is one of DNS providers, they use it in an on-demand way. They can also do that, meaning that you turn it on only when your staff or maybe your automation decides it's worth doing. Like you detect an attack, you turn it on, you turn it off. We chose not to do that because the traffic amount we have is usually low. They only bill you for the actual useful traffic. So unless you have like high traffic website, that's really not worth it. So quick follow-up. Yes, a quick follow-up question there, Dmitry, then. So you're hosting. You said you've got some hosting locations, some in Ukraine, some in US, some in other parts of Europe. Is that hosted in individual? data centers or do you have some of those in hyperscalers like aws or gcp no we don't use uh amazon or google we call magic promise it actually we can't no it's it's some hardware that we either lease or in rare cases own well typically nowadays it's like a provider which gives us a rack with equipment in it and we use that uh providers facility yeah so no no we don't use the google cloud i don't know about but the amazon would allow you to bring your own ip you can in theory bring your own mp to amazon and yet have these ip addresses server called fair magic transit i would say don't try this at home yeah uh i would be thinking of this slide as so before of this like new cloud fair idea of like being the whole loop of networking infrastructure i guess that would be a good product to bring into that mix and that probably has to be done with engineer we're not planning on the using amazon for the high traffic applications yet we use amazon partially but it's for some specific use cases yeah so brilliant really speak for that thank you very much um coming back to you uh we did talk briefly about um some of the credential theft prevention and i believe that you use workers to identify that based on some databases that you have acquired or that you inquire of could you tell us a little bit about that yeah well uh we have uh i would say very good security team. So we are using WAV, of course, from the from the Coopler, where there is a security or leaked security managed rule which tries to figure out whether those were leaked somewhere. However, what our security team did was that they went to dark web, downloaded several free databases of stolen credentials they calculated hashes out of these and store those in the in the Cloudflare key value storage and utilize the workers so right now when the customer logs in we calculate the hash we compare with the database and if there is a match we basically tell the customer you should change your credentials very quickly so that's one of the examples where if there is no feature in the Cloudflare for that, you can build it in the workers. So I hope you are familiar with the workers, how it works, basically a piece of code which you run the traffic through and you can do whatever you want to do with that. So we use it for that particular use case. That's a particularly interesting one, I think, especially when you're using a fairly interesting data source. that um do you do sandbox those uh communications in any way when you are acquiring the database shall we say not my business security team yeah just let them deal with it right they just give you the database and then you take it from there exactly they do it completely on their own we just you know look at it don't ask don't tell exactly all right good stuff okay um so you also uh do a lot image processing at Printify as well. I guess because people are uploading images to then apply onto t-shirts, mugs, stickers, whatever, ties, pies, you know, all sorts of things. So you process something like 50 million images, is it? We process actually even more. That's just very part of whole thing. So yeah, you can imagine we deal with a lot of images, you know, we have images in... catalog, we need to build mockups out of your designs, we even generate for users. That's specifically why I said that anyone with an idea can create a product on our infrastructure because we have an AI tool where you can say generate me an astronaut on the moon and put it on the t-shirt and it will do it for you. So you just need to have an idea and through AI you have a product and within a few clicks it's on the market. place. But anyway for all these things we need to generate mock-ups with you know different angles, people, how it will look like and so on. So yeah I would say 70% of our traffic is somewhat related with the images. And what you are asking specifically was that one particular team who is responsible for a catalog. They said, OK, we have plenty of images. We need to serve them in different resolutions, different sizes, different settings, different lightning, different compression algorithms, and so on. And we don't want to develop these things. We don't want to store these images on our site. We just want to have a source image and someone else will do the job. So that's why we specifically looked what are options. surprise, surprise, there is an option in the Cloudflare called Cloudflare image or image resize. So we use that feature where you basically put the source image, you define your format, again compression ratios, whatever, and Cloudflare does that. And it just ends there. There are other benefits, caching, that all these versions are cached immediately in edge locations. So when our customer, and again, we have more than 110 locations of print providers. However, we send millions of orders to almost all countries around the world. So it really saves a lot of traffic. Yeah, I mean, I guess also then when you're sending everything around the world, you also use Argo, don't you, as one of your products? We do, we do. yeah well basically there were two big reasons why to partner with the Cloudflare security of course a lot of a lot of stuff discussed already and the second one was performance right because when we started you know young startup you want to save every dollar you want to be cost effective so to serve basically to whole world all continents and everything you have a couple choices one of obvious is to deploy your stuff everywhere but that's very expensive you need to sync everything you have maintenance headache blah blah blah or you can do something else and something else is is a cloudflare of course because with all those data centers which were mentioned already the caching we cache more than one with more than one-third of our traffic is cached in all those CloudFare locations. And there is yet another piece in the puzzle and that is if you have your database or your system or platform in one region how can you serve the same experience to everyone around the world. And there is Argo in the CloudFare so that's the tool which routes your traffic dynamic one stop. want the best possible way the easiest way the fastest way and it's just a matter of click that's the one of the two of the tools or or you know systems where you just enable and forget uh so we enabled and we haven't forgot we looked on the results and the results were that after a few weeks we saw 20 decrease in the in the in the latency and more than 40% decrease in the response times. So it's a huge impact on our ecosystem, on our platform, and that allows us, apart from other things, to be very cost efficient and be, let's say, in one region and serve the planet. Yeah, so that, you know, in essence, rather than you having to scale to match that demand, you're able to balance it and use things like Argo and caching to make sure you've got... that scalability and performance without having to physically scale or as much equipment or images yeah interesting okay great um you mentioned um you you know you've got a lot of different apis for different marketplaces such as etsy and many others as well um do you use anything like api protection for that or is that something you look into in the future uh well yes We consume a lot of API requests, we also have public API, which you can use to publish our stuff on the platform of your choice. So yes, API, you know, it's different than a real human, right? You can write a script and call API a million times a second and basically do an attack or an unintentional attack, if you will, to that... to the system, right? So, of course, we have our API secured, you need to do authentication and so on and so on. However, we also had to figure out how many times you can call it. Like, you know, calling something a million times a second, it's definitely, you know, something is wrong, right? Or in our case, it could be something wrong. So, we used, again, software for that to help us distinguish basically scrape the API endpoints we use and suggest the correct limits for or global correct limits for those API endpoints. So rate limiting is part of WAF again it's security feature it's I would say in the middle of the performance because it helps with the performance but also it's a part of security so we are rate limiting but it's not enough because those are just the global limits or limits per IP per cookie per whatever but typically you need to know who is behind those IP addresses session cookies and so on so that's actually something we built right now maybe we'll use a call for that because we need to you know, somehow connect the information who is behind that because we have free customers, we have paid customers, we have customers with zero orders, we have customers with million orders. So we would like to distinguish how many calls they can have. So that's something we are working on right now. Okay. Dimitro, just coming back to you again on, you said you leave Magic Transit sends it on all the time. Are there any situations where you see a large... spikes in traffic whether it be DDoS or similar types of attacks where you need to react quickly can you tell me a little bit about what happens when when you get those kind of attacks and how Cloudflare helps you to respond approached us, we started creating zero trust micro-access to them, and not only it was more secure for them, we also automated most of the things with using API in their forum because we are API-first platform, but it also gave them the capability of scaling up, which was really important because they're always growing and they have cloud. So, I want to thank you very much for your time listening to me, and if you have any questions, answer them always. can you go click meet after I can show you the demo. Thanks. Thank you Alex. Yeah, it's like Zero Trust is a very interesting topic. And as Maxi mentioned, that we entered this field or this part of our industry several years ago. Of course, we used our Zero Trust for years. We tested it for ourselves. And I see that the customer, they're like, I should post it. the audience of these two enterprise customers that are using our zero trust platform here in Czech Republic breakdown so if you will be interested we can like I can introduce you later thinking like strategically I guess is never to rely on any single vendor for everything well okay I mentioned that there is unique position but let's say for for the website we do use Cloudflare and that's certainly us but uh for the DNS we have multiple providers even if one of the providers we have is hit than the others who take its place so there is no single point of failure so to speak in our network that I can think about. So when it comes to you know so what you're saying to me then is that when there are large-scale attacks or DDoS attempts you pretty much don't see it you just get a report that says it happened. yeah well yeah there were big attacks actually the traffic volume of attacks went down i guess the actors who tried to do this discovered that it doesn't really work and but there was a big attack in ukraine on all of the government related infrastructure and such in february about a week and a half before the war but i wanted to add that i see many ukrainian companies and government enterprises use Cloudflare web firewall we just ran a quick check today during lunchtime I was speaking to one of your staff and I can't say the exact number but let's say of all of the gov UA government related domains we use about slightly under 10% use Cloudflare web firewall right now I know there's significant amount I was just asking my staff member to do the quick calculation so I don't want the actual number But, well, 10% may not sound like a lot, but think about this 80-20 rule, right? Or 90-10 rule, you know. There are only few of these sites that are really important, right? There are many, like, let's say, local government sites, you know, they're not so popular. There are some of these domains that are not even in use. So I think it's a big number. And it doesn't relate to us, really, because we are not a web hosting company. But it just shows that while people use the services... because they're valuable so if it works it works and you don't notice right you know it's like it's kind of what you want it's like being healthy you know you don't notice you're sick right till you are so i guess yeah but it's but it's also great that it's if you if you have something that's managing those threats um if you never see it then it's working right it's one of those things where actually you don't want to know you don't need to know that it's working oh we do notice things like for example If somebody does what's called an enumeration attack, by the way, that's a new feature, Dines Farwell, we haven't enabled it yet. Enumeration attacks mean that somebody tries different names. So imagine you have, let's say, certain names in your domain database, and they try the same name repeatedly. Well, they can try it a million times per second. It would be resolved through Cloudflare network. Well, if they try a name like AA001, AA002, and such, every time the request is different, well, caching doesn't work because every name is different. And you have to check. that it doesn't exist they would check that it doesn't exist in the cache that will hit you so that kind of thing does happen to us and that's not something that call fair can defend on except now it can in a way there is a feature that dns firewall has that kind of detect these things i guess it's by combination of i don't know if i don't want to call it artificial intelligence let's call it machine learning for now maybe it's also a combination of source address space we do see that i mean okay but our own dns infrastructure is resilient enough to handle that. And I guess people just don't attack DNS that much because it's kind of boring. There is no website you can say, ha, it's down, right? It's kind of, it still works because there is so much resiliency built into protocol because every customer of DNS uses their own, either provider cache or maybe company like Google or even QuadOne itself. Yeah, so that's sure it's like a double defense. So it's also- So we're using QuadOne DNS. Quad1DNS would first use that, then they use maybe Cloudflare powered DNS or not. So it's a multi-layered defense, let's say, and some things may just be masked by others. And of course, there's the best way to have more than one layer of defense. You mentioned something there about AI versus machine learning. I wouldn't want to derail this conversation because we could go into that ad infinitum because that is quite an interesting subject. I think you've got a point there. Some of it isn't... just AI it is machine learning but that's for a different session I just wanted to come back to you Milan on we talked a lot about what you do today but I'd like to wrap up these sessions with what you want to do tomorrow what are the things that you want to then implement in Cloudflare or areas that you think you can improve your usage of your of the current Cloudflare platform yeah well the one of the plans is as I mentioned the rate limiting pair merge basically tailored rate limiting so that's something we work on uh it's in early stages so maybe we'll be in the cloud flare uh uh the second feature and and uh it's a coincidence right uh it's about ai so one of the one of the problems we are trying to solve right now is to figure out what the people want to print on on their on their white label products, right? Because you don't want to have Nazi symbols or something really polarizing or something which is trademarked, copyrighted, and then you can be sued by, you know, those trademark holders. So this is something we want to figure out. We are for now using third-party services. where we send those images and they are you know trying to figure out through the image recognition and an AI what's on the picture what's on the what we what do we need to print and tell us okay you should you should tell the customer this is enough this really does sound like you should be using workers and workers AI because yes as I'm sure Mark would like to avail you of we do have an image recognition so models which are available in workers Yeah, so for now, we are using third party, but that's something we want to develop in-house and look at that. So workers and specifically AI workers, maybe, yes. Yeah, it's interesting. So you do get a lot of either people just putting copyrighted imagery or offensive imagery onto products as well, I guess. Yeah, and there is a pattern, you know. It's not just about the trademark. and copyrighted stuff. But the more and more polarizing the image is, then it's more likely that that specific issue will be hit by DDoS and so on. So there is a correlation between those. So we are also trying to protect us not only on the end, you know, of that like DDoS on the issue, but proactively like, okay, this could lead to some attacks. So even before we print or publish that, we are trying to solve that. Interesting. Yeah. Well, I look forward to seeing your results when you do that in Worker's AI, because that'd be fantastic. Or maybe not, but I hope you do. And Dmitry, finally to you, what sort of things are you looking to implement in the future, in the near term, if anything, at Hostmasters? I say, I mean, what do I? Anything. in the future that you want to... Oh, generally not related. Well, not related to workers AI, no. What would you, what would you, what are you looking at currently in the Cloudflare portfolio that you think would be advantageous to the next stage of how host masters? Yeah, sure. I'll, well, we'll continue monitoring the product advancement. I was impressed by this graphic. What is this, you know, this line, the timeline, right? And it's like more and more things happening like, hey, I don't even know how to do it. of the things and by the way i'm still well the magical incv6 i mentioned i guess uh maybe really again i don't want to use word artificial intelligence maybe more intelligence more automation in threat detection maybe these reports that we see weekly can be no no i don't want them hourly but maybe they can be a bit more detailed that can be useful maybe they're in ipi that we can utilize and get the custom them will look what i want to mention i mean heard you saying about this you know protection of well intellectual property and such i'm curious you know with jurisdictional issues and the european union trying to you know go more into regulating digital space you know i'm not even mentioning as to all these other things you know how do you decide what's legal or not you know the trademarks are relative right no trademarks have always jurisdiction it's for this country for this particular space you know trademark can be known for their for the music you know for the major the apple computer versus apple music like you know they always have the area of applicability right the hate symbols well okay the nazi symbols are explicitly prohibited in many countries of europe and then there are some of them which are more or less you know you can have a racist jokes print on the t-shirt or language that your staff doesn't know do you think that artificial intelligence helps you to read between the lines well we'll get this this is an interesting area where we go i think that is an interesting humans are smart but i mean computers are smart humans are smarter that's true but it's just important you mentioned about this too we do have a session coming up after this around this too um in particular around compliance uh so yeah maybe we could actually use uh workers ai to help us identify where we might have a compliance issue with certain items or materials or maybe even logos uh phrases or whatever we have as well so who knows i think the uh with any of these things it's how good the training data is right um and then how good the model is using that training data to identify these things so we'll see um the the page has not been turned finally on ai we're certainly all still feeling this out um new models come along every day so we'll see but i think right now it'd be really interesting to see some of this uh image identification uh in operation and see what we can actually do with it that really is interesting to me okay um i'd like to wrap it up there um thank my uh panel guests here dimitro and milan um it's been great having them here i hope everybody's enjoyed uh what they've got to share about their businesses um and what they're going to be doing in the future with cloudflare and uh thank you very much