Episode 396 – Lessons from a real-life ransomware attack

February 27, 2025

Episode Transcript

Welcome to Episode 396 of the Microsoft Cloud IT Pro Podcast. In this episode, Ben walks Scott through a ransomware attack on an on-premises VMware environment. As they discuss the attack and the results of it, they also talk through some lessons and considerations that could have helped mitigate such an attack.

Your support makes this show possible! Please consider becoming a premium member for access to live shows and more. Check out our membership options.

Show Notes

About the sponsors

Would you like to become the irreplaceable Microsoft 365 resource for your organization? Let us know!

See full episode transcriptTranscript is autogenerated by AI

1 00:00:03,359 --> 00:00:05,919 Welcome to episode 396 2 00:00:05,919 --> 00:00:09,039 of the Microsoft Cloud IT Pro podcast recorded 3 00:00:09,039 --> 00:00:12,019 live on 02/21/2025. 4 00:00:12,240 --> 00:00:14,480 This is a show about Microsoft three sixty 5 00:00:14,480 --> 00:00:16,344 five and Azure Azure from the perspective of 6 00:00:16,344 --> 00:00:18,824 IT pros and end users, where we discuss 7 00:00:18,824 --> 00:00:20,824 a topic or recent news and how it 8 00:00:20,824 --> 00:00:24,265 relates to you. For today's episode, Ben walks 9 00:00:24,265 --> 00:00:26,904 Scott through a ransomware attack on an on 10 00:00:26,904 --> 00:00:28,530 premises VMware environment. 11 00:00:29,089 --> 00:00:31,089 As we walk through the attack, we also 12 00:00:31,089 --> 00:00:33,729 talk through lessons learned, things not to do, 13 00:00:33,729 --> 00:00:36,469 and ways to help mitigate such an attack. 14 00:00:38,370 --> 00:00:40,210 So here we are. We did no planning, 15 00:00:40,210 --> 00:00:42,604 Scott, So we're telling stories. Should we tell 16 00:00:42,604 --> 00:00:44,125 the story today? I think you have a 17 00:00:44,125 --> 00:00:46,844 fun story to tell this week. I I 18 00:00:46,844 --> 00:00:48,704 always think it's interesting to 19 00:00:49,164 --> 00:00:49,984 eye the 20 00:00:50,524 --> 00:00:53,189 more material and conceptual things back 21 00:00:53,590 --> 00:00:56,070 to the real world. So today, you've got 22 00:00:56,070 --> 00:00:58,149 a nice fun story. You know, I was 23 00:00:58,149 --> 00:01:00,329 trying to think about, like, what could we 24 00:01:00,390 --> 00:01:02,309 title this one? Because, you know, it's fun 25 00:01:02,309 --> 00:01:04,390 to kinda game the titles in the beginning 26 00:01:04,390 --> 00:01:05,989 and and see where they would go. And 27 00:01:05,989 --> 00:01:08,174 I kinda landed on just lessons from a 28 00:01:08,174 --> 00:01:11,334 real life ransomware attack. And we'll have we'll 29 00:01:11,454 --> 00:01:13,295 and take us through a real life ransomware 30 00:01:13,295 --> 00:01:16,034 attack and some things that went wrong at 31 00:01:16,334 --> 00:01:18,414 client that you were working with. And we 32 00:01:18,414 --> 00:01:20,575 won't name names or anything like that, but 33 00:01:20,575 --> 00:01:21,954 I think it's a good opportunity 34 00:01:22,334 --> 00:01:23,269 to talk through 35 00:01:23,729 --> 00:01:24,229 potentially 36 00:01:24,769 --> 00:01:26,549 what that client of yours 37 00:01:26,929 --> 00:01:28,069 could have done better, 38 00:01:28,450 --> 00:01:28,950 especially 39 00:01:29,250 --> 00:01:31,090 looking back in time. And, you know, I 40 00:01:31,090 --> 00:01:32,849 think there's some good lessons in there around, 41 00:01:32,849 --> 00:01:33,909 like, perceived 42 00:01:34,289 --> 00:01:35,984 PCO of things, like 43 00:01:36,465 --> 00:01:38,864 having multiple copies and all that good stuff 44 00:01:38,864 --> 00:01:39,924 that we can get. 45 00:01:40,305 --> 00:01:41,825 So why don't you start us out at 46 00:01:41,825 --> 00:01:44,224 the beginning? Yeah. So this is absolutely one 47 00:01:44,224 --> 00:01:46,064 of those boat one of those scenarios. And 48 00:01:46,064 --> 00:01:48,484 like you said, names, people, places, 49 00:01:49,025 --> 00:01:51,525 things will be left out to protect 50 00:01:52,120 --> 00:01:53,959 said people. I'm also on the fence guide. 51 00:01:53,959 --> 00:01:55,319 I'm always like, do I wanna call them 52 00:01:55,319 --> 00:01:56,920 a client? Like, I've been working for them 53 00:01:56,920 --> 00:01:58,939 for two weeks. They were not a client 54 00:01:59,000 --> 00:02:01,799 before two weeks because I don't wanna be 55 00:02:01,799 --> 00:02:03,799 like, hey, Ben, why didn't you protect these 56 00:02:03,799 --> 00:02:06,040 people sooner? That type of scenario. But they 57 00:02:06,040 --> 00:02:08,055 are a client. It is a something I've 58 00:02:08,055 --> 00:02:10,134 been working on for two weeks. So this 59 00:02:10,134 --> 00:02:10,634 was, 60 00:02:11,334 --> 00:02:12,314 I think it was 61 00:02:12,694 --> 00:02:13,194 maybe 62 00:02:13,495 --> 00:02:15,254 a week and a half ago, something I 63 00:02:15,254 --> 00:02:15,754 was, 64 00:02:16,134 --> 00:02:17,735 I can turn this into a whole narrative 65 00:02:17,735 --> 00:02:19,814 too. So, yeah, I was actually talking to 66 00:02:19,814 --> 00:02:21,414 a partner I have and we were talking 67 00:02:21,414 --> 00:02:23,590 about another project. He was like, Ben gotta 68 00:02:23,590 --> 00:02:24,090 go. 69 00:02:24,789 --> 00:02:27,590 Cyber incident just came in. So it was 70 00:02:27,590 --> 00:02:29,110 like, okay, go let me know if you 71 00:02:29,110 --> 00:02:29,769 need anything. 72 00:02:30,069 --> 00:02:31,669 Didn't think much of it. I don't always 73 00:02:31,669 --> 00:02:33,989 get pulled into these because again, I deal 74 00:02:33,989 --> 00:02:36,169 primarily Microsoft three sixty five Azure, 75 00:02:36,525 --> 00:02:39,245 but got pulled in a little later, talked 76 00:02:39,245 --> 00:02:39,905 to them. 77 00:02:40,205 --> 00:02:41,665 They thought everybody 78 00:02:42,284 --> 00:02:45,504 was out. They definitely had been compromised, 79 00:02:45,965 --> 00:02:48,759 but thought they caught it early enough, pulled 80 00:02:48,759 --> 00:02:51,960 network access to on prem hardware soon enough, 81 00:02:51,960 --> 00:02:54,199 all of that. So chatted with him for 82 00:02:54,199 --> 00:02:55,180 a couple more hours. 83 00:02:55,560 --> 00:02:58,039 Great. Was sitting in my office. I should 84 00:02:58,039 --> 00:03:00,360 have gone to bed. One lesson learned, Scott. 85 00:03:00,360 --> 00:03:02,120 After that, just go to bed and turn 86 00:03:02,120 --> 00:03:04,525 off the cell phone. Always good. Either recommended. 87 00:03:05,145 --> 00:03:06,905 Yeah. So I was sitting in the office. 88 00:03:06,905 --> 00:03:08,985 11:30 at night, I get a text message. 89 00:03:08,985 --> 00:03:10,365 Hey, Ben. Are you still awake? 90 00:03:10,824 --> 00:03:13,145 Dilemma. Do I answer do I respond or 91 00:03:13,145 --> 00:03:15,564 do I pretend I'm asleep? Yeah. I'm awake. 92 00:03:15,784 --> 00:03:18,550 All of the VMs in this environment just 93 00:03:18,550 --> 00:03:20,229 shut down. We lost all of them. And 94 00:03:20,229 --> 00:03:21,430 it's not a call you wanna wake up 95 00:03:21,430 --> 00:03:23,129 to. No. This is not 96 00:03:23,430 --> 00:03:26,229 what you wanna hear. So okay. Let me 97 00:03:26,229 --> 00:03:28,250 jump on the call. Lo and behold, 98 00:03:28,870 --> 00:03:32,185 these attackers had gotten into the network, And 99 00:03:32,485 --> 00:03:34,504 this is where the first lesson is, Scott. 100 00:03:34,564 --> 00:03:37,384 This was not an attack I had necessarily 101 00:03:37,525 --> 00:03:40,504 heard of before. In this particular scenario, 102 00:03:41,364 --> 00:03:42,985 there were some unpatched 103 00:03:43,924 --> 00:03:44,424 networks, 104 00:03:44,805 --> 00:03:45,305 firewalls 105 00:03:46,069 --> 00:03:46,969 in this environment, 106 00:03:47,349 --> 00:03:48,889 and there was a 107 00:03:49,189 --> 00:03:50,090 certain vulnerability. 108 00:03:50,550 --> 00:03:52,409 And, again, I'm not sure of the vulnerability. 109 00:03:52,550 --> 00:03:54,310 I'm not gonna name brands or any of 110 00:03:54,310 --> 00:03:56,790 that, but it allowed them to actually create 111 00:03:56,790 --> 00:03:56,950 a 112 00:03:57,590 --> 00:03:58,730 based on my understanding 113 00:03:59,110 --> 00:04:00,550 of this, it allowed them to create a 114 00:04:00,550 --> 00:04:03,194 VPN connection to the firewall to get inside 115 00:04:03,194 --> 00:04:05,915 the network. So this was not a user 116 00:04:05,915 --> 00:04:06,415 compromise. 117 00:04:07,115 --> 00:04:09,275 This was not somebody clicking on a phishing 118 00:04:09,275 --> 00:04:10,974 link and credentials getting accessed. 119 00:04:11,355 --> 00:04:14,235 It was, oh, here's a vulnerability. Let's establish 120 00:04:14,235 --> 00:04:16,395 a direct connection into the internal network through 121 00:04:16,395 --> 00:04:17,055 the VPN 122 00:04:18,079 --> 00:04:20,160 and see what we can find. So first 123 00:04:20,160 --> 00:04:22,480 rule, servers are not the only things you 124 00:04:22,480 --> 00:04:24,639 should be patching and updating, and the only 125 00:04:24,639 --> 00:04:25,620 things with vulnerabilities 126 00:04:26,079 --> 00:04:26,899 are workstations. 127 00:04:27,360 --> 00:04:28,579 Probably not. 128 00:04:29,120 --> 00:04:31,439 You know, gen generally, firewalls are an important 129 00:04:31,439 --> 00:04:33,654 thing to patch. Your firewalls, your switches, 130 00:04:34,455 --> 00:04:37,495 let's just say your entire infrastructure And end 131 00:04:37,495 --> 00:04:40,375 to end. Keep these updated. So once they're 132 00:04:40,375 --> 00:04:42,134 on the network, now it's like, let's go 133 00:04:42,134 --> 00:04:44,694 see what we can find. They managed to 134 00:04:44,694 --> 00:04:45,754 get their way into 135 00:04:46,295 --> 00:04:46,795 the 136 00:04:47,199 --> 00:04:49,439 ESX servers that were hosting all of their 137 00:04:49,439 --> 00:04:51,220 on prem VMs and 138 00:04:51,759 --> 00:04:52,259 encrypt 139 00:04:52,720 --> 00:04:53,939 everything. Encrypt 140 00:04:54,639 --> 00:04:55,139 drives, 141 00:04:55,680 --> 00:04:56,180 encrypt 142 00:04:57,279 --> 00:04:57,779 configuration 143 00:04:58,160 --> 00:04:58,660 files, 144 00:04:59,595 --> 00:05:00,095 Everything 145 00:05:00,395 --> 00:05:03,115 was encrypted, and this is why said servers 146 00:05:03,115 --> 00:05:05,134 went down. They were just no longer 147 00:05:05,514 --> 00:05:06,254 in existence, 148 00:05:06,555 --> 00:05:08,235 or they weren't in existence. They were just 149 00:05:08,235 --> 00:05:11,615 no longer accessible anymore. Oops. They went away. 150 00:05:11,675 --> 00:05:14,810 Oops. Right? So first thing, right? When everything's 151 00:05:14,870 --> 00:05:15,370 encrypted 152 00:05:16,149 --> 00:05:17,449 was, okay, where's your backups? 153 00:05:17,829 --> 00:05:20,550 Like, do we have backups of these? Do 154 00:05:20,550 --> 00:05:24,229 we have anything like that? Well, our server 155 00:05:24,229 --> 00:05:25,910 that runs all of our backups was also 156 00:05:25,910 --> 00:05:26,810 on these hosts. 157 00:05:27,144 --> 00:05:29,464 Okay. So your backup server is encrypted, so 158 00:05:29,464 --> 00:05:30,664 we're not gonna be able to get into 159 00:05:30,664 --> 00:05:33,004 that. Where was this backup server 160 00:05:33,464 --> 00:05:35,784 saving these backups to? Well, that was to 161 00:05:35,784 --> 00:05:38,344 another storage device that was also attached to 162 00:05:38,344 --> 00:05:39,084 these hosts. 163 00:05:40,149 --> 00:05:42,329 So backup drive was a massive 164 00:05:42,949 --> 00:05:45,529 VMware file or VMware virtual drive, a VMDK 165 00:05:45,669 --> 00:05:48,709 file that this was interesting, Scott. It was 166 00:05:48,709 --> 00:05:51,509 not actually encrypted. It was large enough and 167 00:05:51,509 --> 00:05:53,349 the storage was of a size that I 168 00:05:53,349 --> 00:05:54,975 don't think there was size to encrypt it, 169 00:05:55,134 --> 00:05:57,694 but an attempt was made and it corrupted 170 00:05:57,694 --> 00:05:59,535 the entire file. Like, we were able to 171 00:05:59,535 --> 00:06:01,214 get it mounted and get back into it, 172 00:06:01,214 --> 00:06:03,395 but there was nothing salvageable 173 00:06:03,855 --> 00:06:06,975 on the VMDK. Like, folders would disappear as 174 00:06:06,975 --> 00:06:08,574 you were clicking around it, and you ran 175 00:06:08,574 --> 00:06:10,970 disc checks and there were flags flying everywhere. 176 00:06:11,350 --> 00:06:13,430 So, okay. So we don't have backups. What, 177 00:06:13,589 --> 00:06:15,930 what about a Doctor plan? Well, 178 00:06:16,389 --> 00:06:18,710 our Doctor plan is when we think there's 179 00:06:18,710 --> 00:06:20,710 going to be certain events that could cause 180 00:06:20,710 --> 00:06:23,285 a Doctor, we back up said servers to 181 00:06:23,285 --> 00:06:24,264 a cloud location. 182 00:06:24,645 --> 00:06:25,305 And then 183 00:06:25,685 --> 00:06:26,904 after said 184 00:06:27,285 --> 00:06:29,685 events, we just delete them because we don't 185 00:06:29,685 --> 00:06:32,485 wanna pay to host them up in the 186 00:06:32,485 --> 00:06:34,084 cloud full time. So we don't have any 187 00:06:34,084 --> 00:06:35,444 Doctor. So what's the plan to get back? 188 00:06:35,444 --> 00:06:37,139 And frankly, at this point in time, we 189 00:06:37,139 --> 00:06:40,180 had nothing. There's no backups. Everything's encrypted or 190 00:06:40,180 --> 00:06:42,199 corrupted. There's no off-site backups. 191 00:06:42,819 --> 00:06:43,800 It's all 192 00:06:44,580 --> 00:06:46,819 like, it was essentially we were, do we 193 00:06:46,819 --> 00:06:49,860 just rebuild everything from scratch? Like, do we 194 00:06:49,860 --> 00:06:50,360 have 195 00:06:50,725 --> 00:06:52,485 files? Do we have data? Do we have 196 00:06:52,485 --> 00:06:54,404 applications? Do we have all of these where 197 00:06:54,404 --> 00:06:54,904 we're 198 00:06:55,205 --> 00:06:59,205 rebuilding everything from ground zero? Because there was 199 00:06:59,205 --> 00:07:02,745 nothing to restore from. It's kind of brutal 200 00:07:03,125 --> 00:07:05,740 all around. Even once you mitigate it, it's, 201 00:07:05,740 --> 00:07:08,879 it's not like you can go in and 202 00:07:09,019 --> 00:07:10,079 trust your 203 00:07:10,779 --> 00:07:13,339 ESX host at that point. You're kind of 204 00:07:13,339 --> 00:07:16,319 paving everything. You're potentially paving your firewalls, 205 00:07:17,324 --> 00:07:18,305 It could be paving 206 00:07:18,605 --> 00:07:19,584 inline infrastructure, 207 00:07:20,285 --> 00:07:21,504 like, managed switches. 208 00:07:22,204 --> 00:07:25,665 You you're paving your virtual machine hosts, which 209 00:07:25,884 --> 00:07:28,845 have all that other stuff on it, you 210 00:07:28,845 --> 00:07:31,569 know, not just the the probably the the 211 00:07:31,569 --> 00:07:32,629 the business applications 212 00:07:33,009 --> 00:07:35,170 and the surrounding infrastructure, but things, you know, 213 00:07:35,170 --> 00:07:36,770 that might be important to a business, say, 214 00:07:36,770 --> 00:07:39,029 like your, I don't know, your active directory 215 00:07:39,170 --> 00:07:39,670 and 216 00:07:40,050 --> 00:07:41,889 things like that. That was that, like, we 217 00:07:41,889 --> 00:07:43,910 have two domain controllers. There was a failover, 218 00:07:44,134 --> 00:07:46,055 a primary and a secondary domain controller, but 219 00:07:46,055 --> 00:07:47,735 they were both sitting in the same VMware 220 00:07:47,735 --> 00:07:49,495 hose. Well, they were sitting in the same 221 00:07:49,495 --> 00:07:51,274 VMware environment, same storage, 222 00:07:51,894 --> 00:07:54,294 both DCs were encrypted. And like you said, 223 00:07:54,294 --> 00:07:55,194 we were finding 224 00:07:55,814 --> 00:07:58,649 accounts popping up on firewalls. Like, the breed 225 00:07:58,730 --> 00:08:01,209 the attackers had gone in and created, I 226 00:08:01,209 --> 00:08:04,490 mean, accounts everywhere, accounts in AD, accounts in 227 00:08:04,490 --> 00:08:04,990 switches, 228 00:08:06,089 --> 00:08:06,589 firewalls. 229 00:08:07,930 --> 00:08:08,430 And 230 00:08:08,730 --> 00:08:09,389 it was 231 00:08:09,845 --> 00:08:12,085 like, we found some of these that accounts 232 00:08:12,085 --> 00:08:14,324 that actually had like first name, last name, 233 00:08:14,324 --> 00:08:15,845 but then usernames are, 234 00:08:16,564 --> 00:08:19,545 like looked like named accounts, but we're definitely 235 00:08:19,685 --> 00:08:22,585 not employees at the company. So 236 00:08:22,965 --> 00:08:24,645 it's a lot of work going in and 237 00:08:24,645 --> 00:08:27,519 cleaning it up. And we were able to 238 00:08:27,579 --> 00:08:29,740 end up decrypting some of the data, get 239 00:08:29,740 --> 00:08:31,740 some of this back, go through and clean 240 00:08:31,740 --> 00:08:33,019 it all. But there were a lot of 241 00:08:33,019 --> 00:08:34,700 lessons learned, like, even, 242 00:08:35,179 --> 00:08:38,059 we were talking about the ESXi hosts. The 243 00:08:38,059 --> 00:08:38,559 password 244 00:08:39,215 --> 00:08:40,355 for said ESXi 245 00:08:40,815 --> 00:08:42,995 ESXi host was seven characters, 246 00:08:43,295 --> 00:08:46,415 and, well, it had a mix of letters 247 00:08:46,415 --> 00:08:47,634 and special characters, 248 00:08:48,254 --> 00:08:50,815 was absolutely one that if I was going 249 00:08:50,815 --> 00:08:53,475 to go try to brute force a ESXi 250 00:08:53,535 --> 00:08:55,830 host hosting VMware, would have been on the 251 00:08:55,830 --> 00:08:57,289 short list of 252 00:08:57,750 --> 00:08:59,190 attempts that would have been made when I 253 00:08:59,190 --> 00:09:01,269 was going into brute force it. Yeah. The 254 00:09:01,269 --> 00:09:03,669 passwords seven characters. Have you ever looked at 255 00:09:03,669 --> 00:09:05,509 some of the password things, like, starting with 256 00:09:05,509 --> 00:09:07,750 this one? Because I think it's interesting to 257 00:09:07,750 --> 00:09:08,924 talk through this and some of the best 258 00:09:09,084 --> 00:09:11,024 practices and things we talked about as well. 259 00:09:11,084 --> 00:09:12,445 Have you looked at some of these for 260 00:09:12,445 --> 00:09:14,225 how long it takes to brute force certain 261 00:09:14,284 --> 00:09:17,004 passwords? It's very quick, depending on the amount 262 00:09:17,004 --> 00:09:19,084 of confusion that's sitting behind them. So you're 263 00:09:19,084 --> 00:09:21,644 often dependent not only on the password and 264 00:09:21,644 --> 00:09:24,160 the complexity of said password and all that 265 00:09:24,160 --> 00:09:25,299 goodness, but also 266 00:09:25,919 --> 00:09:26,419 the, 267 00:09:26,960 --> 00:09:29,040 you know, the compute on the other side 268 00:09:29,040 --> 00:09:32,000 and the capabilities of a system to mitigate 269 00:09:32,000 --> 00:09:34,179 things like maybe, like, password spray attacks 270 00:09:34,639 --> 00:09:36,720 and and bringing that down. And as I 271 00:09:36,720 --> 00:09:39,815 recall, like, ESX, one of those things where 272 00:09:39,815 --> 00:09:41,735 it's like, oh, I'm gonna have a super 273 00:09:41,735 --> 00:09:45,654 sophisticated system for preventing password sprays, you know, 274 00:09:45,654 --> 00:09:48,215 compared to maybe, like, the things that you're 275 00:09:48,215 --> 00:09:51,095 used to in the world of SaaS products 276 00:09:51,095 --> 00:09:53,070 in the cloud. Like, you know, I think, 277 00:09:53,070 --> 00:09:55,429 like, Entride, like, m three sixty five Azure 278 00:09:55,429 --> 00:09:58,190 are good examples. Like, they have built in 279 00:09:58,190 --> 00:10:01,230 passwords, right, protections. Heck, your Google account. Your 280 00:10:01,230 --> 00:10:03,149 your Gmail account has it. I would bet 281 00:10:03,149 --> 00:10:05,294 AOL accounts have it at this point. But, 282 00:10:05,294 --> 00:10:06,995 you know, ESX on prem, 283 00:10:07,615 --> 00:10:09,294 not so much. This is one. So I 284 00:10:09,294 --> 00:10:11,315 just pulled it up. Like, number of characters. 285 00:10:11,375 --> 00:10:12,975 There's a chart out here, and we can 286 00:10:12,975 --> 00:10:14,754 put it in the show notes because this 287 00:10:14,815 --> 00:10:17,134 is a chart. This is back from 2021 288 00:10:17,134 --> 00:10:18,835 too. So four years ago, 289 00:10:19,294 --> 00:10:20,274 this is from 290 00:10:21,100 --> 00:10:22,720 Hive Hive Systems 291 00:10:23,259 --> 00:10:23,759 commando 292 00:10:24,220 --> 00:10:26,540 dot com website. Again, we'll put that list 293 00:10:26,540 --> 00:10:28,480 out there. Seven character password, 294 00:10:28,860 --> 00:10:29,919 if it's numbers 295 00:10:30,460 --> 00:10:31,519 only or 296 00:10:31,899 --> 00:10:33,519 lowercase letters only, 297 00:10:33,914 --> 00:10:36,394 They say four years ago, those are pretty 298 00:10:36,394 --> 00:10:37,455 much you can instantly 299 00:10:37,834 --> 00:10:40,714 brute force a seven character password. If it's 300 00:10:40,714 --> 00:10:43,914 uppercase and lowercase letters, twenty five seconds. If 301 00:10:43,914 --> 00:10:47,534 it's numbers, upper and lowercase letters, it's about 302 00:10:47,779 --> 00:10:50,100 one minute. And if it is numbers, upper 303 00:10:50,100 --> 00:10:52,419 and lowercase letters and symbols, it's about six 304 00:10:52,419 --> 00:10:54,580 minutes for a seven character password. Not long 305 00:10:54,580 --> 00:10:56,759 at all. To your point, right, AOL, 306 00:10:57,379 --> 00:10:57,879 AOL. 307 00:10:58,259 --> 00:10:59,779 You got AOL in my head because my 308 00:10:59,779 --> 00:11:01,620 wife still has an AOL email account, and 309 00:11:01,620 --> 00:11:03,875 it drives me absolutely up a wall. So 310 00:11:03,875 --> 00:11:06,115 does mine. Our wives need we need to 311 00:11:06,115 --> 00:11:08,834 have interventions with our wives. But no, like 312 00:11:08,834 --> 00:11:10,674 Apple. I'm I know I've done this. My 313 00:11:10,674 --> 00:11:12,514 kids have done this, and it infuriates me 314 00:11:12,514 --> 00:11:14,195 when they're trying to get into my phone 315 00:11:14,195 --> 00:11:15,714 or the computer and they type in the 316 00:11:15,714 --> 00:11:17,975 wrong password. And you get, like, 317 00:11:18,299 --> 00:11:20,539 after five or six attempts, and I think 318 00:11:20,539 --> 00:11:21,899 they've gotten up to, like, 10 or 12 319 00:11:21,899 --> 00:11:23,019 where it's like you have to wait twenty 320 00:11:23,019 --> 00:11:25,019 four hours before you can try your password 321 00:11:25,019 --> 00:11:27,580 again. Yes. Every systems should absolutely have that. 322 00:11:27,580 --> 00:11:29,179 But to your point, I don't know 323 00:11:29,820 --> 00:11:31,519 I have not looked enough at ESXi 324 00:11:32,205 --> 00:11:34,205 host lately, but I don't know that they 325 00:11:34,205 --> 00:11:36,684 do. And when you look at this chart 326 00:11:36,684 --> 00:11:37,184 even, 327 00:11:37,565 --> 00:11:40,285 eight characters eight characters is kind of the 328 00:11:40,445 --> 00:11:42,605 I would consider eight characters the bare minimum 329 00:11:42,605 --> 00:11:45,485 if it's if it's complex. Eight characters, upper 330 00:11:45,485 --> 00:11:48,889 lowercase numbers, symbols, all of that is eight 331 00:11:49,190 --> 00:11:52,170 hours. Once you hit the nine, ten, 11 332 00:11:52,629 --> 00:11:53,129 characters, 333 00:11:53,670 --> 00:11:56,309 like, eight to nine characters if you're doing 334 00:11:56,309 --> 00:11:58,070 complex. So all of those goes from eight 335 00:11:58,070 --> 00:12:00,790 hours to three weeks. Nine to 10 goes 336 00:12:00,790 --> 00:12:02,965 from three weeks to five years. Ten to 337 00:12:02,965 --> 00:12:04,904 11 goes to four hundred years. 338 00:12:05,284 --> 00:12:07,365 Eleven to 12 goes to thirty four thousand 339 00:12:07,365 --> 00:12:09,764 years. Twelve characters has kind of turned into 340 00:12:09,764 --> 00:12:12,325 my new minimum, but it still has to 341 00:12:12,325 --> 00:12:14,504 be complex because even a 12 character 342 00:12:15,205 --> 00:12:18,184 letters only is like three weeks. 343 00:12:19,419 --> 00:12:21,500 So a first lesson here, nobody should be 344 00:12:21,500 --> 00:12:23,419 doing, in my opinion, less than a 12 345 00:12:23,419 --> 00:12:25,919 character password. And even then, that might be, 346 00:12:25,980 --> 00:12:30,379 like, not enough. Most password managers default to 347 00:12:30,379 --> 00:12:32,534 something a little bit longer than that. I 348 00:12:32,534 --> 00:12:34,634 think, like, if you did, like, a 349 00:12:35,014 --> 00:12:37,815 default one password dash lane kinda thing, isn't 350 00:12:37,815 --> 00:12:40,054 it, like, 16 to 20 characters? I know 351 00:12:40,054 --> 00:12:41,975 I probably messed with mine in, like, one 352 00:12:41,975 --> 00:12:45,334 password, but generally, like, 20 is my min 353 00:12:45,334 --> 00:12:47,550 bar. I'm up there too. Mine is sixteen, 354 00:12:47,610 --> 00:12:50,410 twenty, 20 four. Like, when I'm creating admin 355 00:12:50,410 --> 00:12:52,029 accounts, something like 356 00:12:52,410 --> 00:12:54,970 root access to an ESXi host, if I 357 00:12:54,970 --> 00:12:57,290 can, if the systems support it, that's the 358 00:12:57,290 --> 00:12:58,809 other thing. I've ran into a few times 359 00:12:58,809 --> 00:13:00,725 where it's not supported. I'm usually going, like, 360 00:13:00,725 --> 00:13:02,345 24 to 32 characters 361 00:13:02,804 --> 00:13:05,924 and completely random for those. You should never 362 00:13:05,924 --> 00:13:08,084 be typing them enough that they need to 363 00:13:08,084 --> 00:13:09,684 be short or they need to be easy 364 00:13:09,684 --> 00:13:11,524 to remember or they need to be a 365 00:13:11,524 --> 00:13:13,125 phrase. Like, you should be able to do 366 00:13:13,125 --> 00:13:13,919 a 32 367 00:13:14,159 --> 00:13:17,700 character, 24 character complex password for those systems. 368 00:13:17,759 --> 00:13:20,079 Even if the passwords are complex, though, for 369 00:13:20,079 --> 00:13:23,279 things like your root user, you still have 370 00:13:23,279 --> 00:13:25,600 to worry about your admins and everything else 371 00:13:25,600 --> 00:13:26,980 down the chain. So 372 00:13:27,434 --> 00:13:29,035 ESX is a little bit of a weird 373 00:13:29,035 --> 00:13:30,955 beast. It's been a while since I've touched 374 00:13:30,955 --> 00:13:32,315 it, but you used to be able to 375 00:13:32,315 --> 00:13:33,514 do it, like, a bunch of stuff in 376 00:13:33,514 --> 00:13:34,014 vCenter. 377 00:13:34,554 --> 00:13:36,634 And, you you know, your admins are connecting 378 00:13:36,634 --> 00:13:38,715 typically through your vCenter. They're not just, like, 379 00:13:38,715 --> 00:13:41,215 SSH ing into the host themselves. 380 00:13:41,600 --> 00:13:43,860 You know, you're over on these other privileged 381 00:13:44,000 --> 00:13:44,500 systems. 382 00:13:45,120 --> 00:13:46,980 Those users might have passwords 383 00:13:47,440 --> 00:13:48,660 in active directory. 384 00:13:49,120 --> 00:13:51,600 So if your active directory gets compromised because 385 00:13:51,600 --> 00:13:53,360 the the these attackers come in, and they're 386 00:13:53,360 --> 00:13:56,259 looking for lateral movement however they can. So 387 00:13:56,495 --> 00:13:58,274 they they might come in on the side 388 00:13:58,654 --> 00:14:01,054 and all of a sudden go, oh, I'm 389 00:14:01,054 --> 00:14:02,754 through your firewall and I'm in your environment. 390 00:14:02,894 --> 00:14:04,495 Well, great. If they can land on one 391 00:14:04,495 --> 00:14:06,495 of your active directory servers and find a 392 00:14:06,495 --> 00:14:09,450 vulnerability there and then they see admins and 393 00:14:09,450 --> 00:14:11,210 other things there, they'll just change the admin 394 00:14:11,210 --> 00:14:13,149 password for that admin, and 395 00:14:13,610 --> 00:14:17,290 they too could have access to vCenter or 396 00:14:17,290 --> 00:14:20,190 to SSH in and and come across. So 397 00:14:20,250 --> 00:14:22,730 it's bad once they're inside. Like, it's very 398 00:14:22,730 --> 00:14:24,524 hard to stop them once they're entrenched past 399 00:14:24,524 --> 00:14:27,004 the edge. Yeah. A %. And another thing 400 00:14:27,004 --> 00:14:29,504 that we encountered, I would say another 401 00:14:29,964 --> 00:14:32,605 lesson learned because we're on the passwords and 402 00:14:32,605 --> 00:14:35,164 credential managers and all of that. Another question 403 00:14:35,164 --> 00:14:36,684 that came out was like, where are your 404 00:14:36,684 --> 00:14:38,544 credentials stored for all these accounts 405 00:14:38,924 --> 00:14:42,519 for not just on prem accounts, but even, 406 00:14:42,820 --> 00:14:45,220 like, maybe cloud services or other applications you 407 00:14:45,220 --> 00:14:47,379 have. And they were like, well, we have 408 00:14:47,379 --> 00:14:49,220 a password manager installed on one of these 409 00:14:49,220 --> 00:14:51,779 VMs that they encrypted. It's doing you a 410 00:14:51,779 --> 00:14:53,540 lot of good now, isn't it? Right. One, 411 00:14:53,540 --> 00:14:55,575 it's, well, if we can't get back to 412 00:14:55,575 --> 00:14:57,174 it, where else can we get some of 413 00:14:57,174 --> 00:14:59,495 these credentials? But the other thing is, like, 414 00:14:59,495 --> 00:15:01,815 if they exfiltrated these, some of these are 415 00:15:01,815 --> 00:15:04,774 not big. You don't know necessarily right away 416 00:15:04,774 --> 00:15:07,029 how long they've been in there. If they're 417 00:15:07,029 --> 00:15:07,529 able 418 00:15:08,389 --> 00:15:09,910 to get credentials and get to some of 419 00:15:09,910 --> 00:15:11,669 these, and if they're able to take that 420 00:15:11,669 --> 00:15:15,350 VMDK file and upload it somewhere online and 421 00:15:15,350 --> 00:15:17,910 run off with a VMDK file and they 422 00:15:17,910 --> 00:15:20,955 have your password manager, they have an unlimited 423 00:15:21,174 --> 00:15:23,654 amount of time to spin that up, get 424 00:15:23,654 --> 00:15:25,754 into it, and compromise 425 00:15:26,375 --> 00:15:29,095 every single credential that was in there. And 426 00:15:29,095 --> 00:15:30,855 I think password managers are always one of 427 00:15:30,855 --> 00:15:32,615 those I think about. Like, I have my 428 00:15:32,615 --> 00:15:35,014 password manager. I'm pretty happy with it. I 429 00:15:35,014 --> 00:15:36,830 think it's everything I've read. It's one of 430 00:15:36,830 --> 00:15:38,589 the more secure ones out there. I think 431 00:15:38,589 --> 00:15:40,750 we've talked about them before. You've mentioned them 432 00:15:40,750 --> 00:15:42,589 already. You can guess it. I don't know 433 00:15:42,589 --> 00:15:44,289 that I would ever run an on premises 434 00:15:44,509 --> 00:15:46,750 password manager. I don't know. Like, there's risks 435 00:15:46,750 --> 00:15:48,975 too of the cloud versus on prem. Right? 436 00:15:48,975 --> 00:15:50,334 Like, everybody's always, well, if it's in the 437 00:15:50,334 --> 00:15:52,334 cloud, anybody can log into it and anybody 438 00:15:52,334 --> 00:15:54,575 can access it. Based on what I've read 439 00:15:54,575 --> 00:15:57,294 about some of the cloud ones, they're pretty 440 00:15:57,294 --> 00:15:58,735 secure. The way they do it, the way 441 00:15:58,735 --> 00:16:00,815 they encrypt things, where keys are stored, all 442 00:16:00,815 --> 00:16:03,289 of that. But your password manager better have 443 00:16:03,289 --> 00:16:04,909 complex long credentials 444 00:16:05,289 --> 00:16:06,649 for as well. Those better not be a 445 00:16:06,649 --> 00:16:08,570 seven character password. You know, the cloud ones 446 00:16:08,570 --> 00:16:11,209 are better than there's varying degrees of everything 447 00:16:11,209 --> 00:16:12,889 here. You know, I think it's funny going 448 00:16:12,889 --> 00:16:16,490 back to how it started. Like, the firewall 449 00:16:16,490 --> 00:16:19,154 attacks are a massive thing. They're always ongoing 450 00:16:19,294 --> 00:16:21,134 and going in the background, and it seems 451 00:16:21,134 --> 00:16:22,674 to cycle through the vendors 452 00:16:23,134 --> 00:16:23,634 as 453 00:16:24,014 --> 00:16:27,475 various zero days and things come up. So 454 00:16:27,534 --> 00:16:28,815 I thought it was funny when you 455 00:16:29,455 --> 00:16:31,159 funny, sad, not funny, 456 00:16:31,620 --> 00:16:32,899 You know, when you said, oh my gosh, 457 00:16:32,899 --> 00:16:35,299 this thing happened with a customer because I 458 00:16:35,299 --> 00:16:37,700 was reading an article back in a couple 459 00:16:37,700 --> 00:16:40,659 weeks ago about how Fortinet firewalls were going 460 00:16:40,659 --> 00:16:42,259 down all over the place because of zero 461 00:16:42,259 --> 00:16:45,454 days. And I used to work as in 462 00:16:45,454 --> 00:16:47,714 a shop where I was installing and configuring 463 00:16:47,774 --> 00:16:48,274 Fortinet, 464 00:16:48,575 --> 00:16:50,414 you know, and and FortiGates in a in 465 00:16:50,414 --> 00:16:52,414 a former life for customers. So I was 466 00:16:52,414 --> 00:16:54,334 like, oh. You know, that one was interesting 467 00:16:54,334 --> 00:16:55,949 to me because it was old gray hair 468 00:16:55,949 --> 00:16:57,309 and gray beard. That was a former life 469 00:16:57,309 --> 00:16:59,070 we used to live, but it's definitely a 470 00:16:59,070 --> 00:17:01,230 thing that happened back then and and still 471 00:17:01,230 --> 00:17:04,109 happens today too. Right? Because there was the 472 00:17:04,109 --> 00:17:06,029 one, two it was a couple years ago 473 00:17:06,029 --> 00:17:06,929 now, right? SonicWall 474 00:17:07,390 --> 00:17:09,329 had a big vulnerability? 475 00:17:10,669 --> 00:17:13,784 Yep. Sonic, Cisco. Every everybody's had their kinda 476 00:17:14,085 --> 00:17:16,244 day in the sun when it comes to 477 00:17:16,244 --> 00:17:18,325 these types of things. And I think going 478 00:17:18,325 --> 00:17:21,284 back to takeaways, the other thing from this 479 00:17:21,284 --> 00:17:24,005 was, and we're going this is conversations we're 480 00:17:24,005 --> 00:17:25,544 having now with them, is 481 00:17:25,990 --> 00:17:26,490 thinking 482 00:17:27,190 --> 00:17:29,529 through what are you doing for a backup? 483 00:17:29,669 --> 00:17:31,990 Like, you mentioned the whole the one, two, 484 00:17:31,990 --> 00:17:34,390 three approach for backups. Like, you need multiple 485 00:17:34,390 --> 00:17:35,849 backups in multiple places. 486 00:17:36,230 --> 00:17:36,970 On prem 487 00:17:37,349 --> 00:17:40,089 attached to your VMs, I would say absolutely 488 00:17:40,149 --> 00:17:42,265 has a place. Right? Like if a VM 489 00:17:42,265 --> 00:17:43,565 accidentally gets deleted 490 00:17:43,945 --> 00:17:45,884 or you have corruption 491 00:17:46,505 --> 00:17:47,005 or 492 00:17:47,465 --> 00:17:48,445 self imposed 493 00:17:49,384 --> 00:17:52,505 issues on a VM, having them directly attached 494 00:17:52,505 --> 00:17:53,565 where you can quickly 495 00:17:53,945 --> 00:17:56,845 restore from a snapshot, restore a backup that's 496 00:17:57,039 --> 00:18:00,799 local, that's locally attacked storage. Perfect. 100%. 497 00:18:00,799 --> 00:18:02,740 But that should not be your only backup 498 00:18:02,799 --> 00:18:05,440 as evidenced here. Like, where are those cloud 499 00:18:05,440 --> 00:18:08,480 backups? Where are your immutable backups? That's one 500 00:18:08,480 --> 00:18:10,394 thing we've been talking a lot about is 501 00:18:10,875 --> 00:18:13,355 why are you not backing stuff up or 502 00:18:13,355 --> 00:18:16,075 paying to have backup somewhere where they can't 503 00:18:16,075 --> 00:18:19,035 be encrypted because it's a read only copy 504 00:18:19,035 --> 00:18:20,955 of that database or it's an or that 505 00:18:20,955 --> 00:18:24,015 server. It's a mutable backups of your VMDK 506 00:18:24,234 --> 00:18:25,295 files where 507 00:18:25,690 --> 00:18:27,769 you can go pull them down in an 508 00:18:27,769 --> 00:18:30,490 emergency. Maybe it's not quick because it's not 509 00:18:30,490 --> 00:18:33,070 directly attached. Yeah. Maybe you're spending 510 00:18:33,609 --> 00:18:35,210 a few thousand dollars a month on it. 511 00:18:35,210 --> 00:18:38,250 Maybe this is costing you 10 or $12,000 512 00:18:38,250 --> 00:18:39,470 a year to 513 00:18:39,845 --> 00:18:41,365 keep these up there. And even that is 514 00:18:41,365 --> 00:18:43,765 probably, I would say, it's high for some 515 00:18:43,765 --> 00:18:46,345 SMBs. It's very low for 516 00:18:46,724 --> 00:18:47,224 enterprises. 517 00:18:47,845 --> 00:18:50,325 But I would say the cost probably varies 518 00:18:50,325 --> 00:18:52,244 in terms of the harm that it can 519 00:18:52,244 --> 00:18:53,730 do. When you look at the amount of 520 00:18:53,809 --> 00:18:55,410 time that has been spent on this, the 521 00:18:55,410 --> 00:18:56,869 time of man hours, 522 00:18:57,410 --> 00:18:59,809 cost of lost business, I would say it 523 00:18:59,809 --> 00:19:02,309 probably scales relative to the enterprise 524 00:19:02,690 --> 00:19:05,169 where I mean, I spend I probably spend 525 00:19:05,169 --> 00:19:07,329 a few hundred dollars a year on backups 526 00:19:07,329 --> 00:19:09,164 for myself because the amount of time and 527 00:19:09,164 --> 00:19:10,684 effort it would take to recover some of 528 00:19:10,684 --> 00:19:12,605 that. Same thing here. It would've cost them 529 00:19:12,605 --> 00:19:14,924 probably a few thousand dollars a month, way 530 00:19:14,924 --> 00:19:16,924 less than what it has cost in the 531 00:19:16,924 --> 00:19:18,704 last two weeks of 532 00:19:19,244 --> 00:19:22,589 man hours to have something like that. That's 533 00:19:22,589 --> 00:19:25,569 a good lesson. So so lots of customers 534 00:19:25,710 --> 00:19:26,450 sit down, 535 00:19:26,750 --> 00:19:30,429 and they look at the cost today. And 536 00:19:30,669 --> 00:19:33,230 Yep. I get it's the total, like, oh, 537 00:19:33,230 --> 00:19:34,450 you're a backup 538 00:19:34,785 --> 00:19:35,845 storage salesperson 539 00:19:36,225 --> 00:19:37,525 kind of thing to 540 00:19:37,904 --> 00:19:39,664 come in and and and figure those things 541 00:19:39,664 --> 00:19:41,205 out. But it's really 542 00:19:41,664 --> 00:19:42,164 just 543 00:19:42,785 --> 00:19:45,184 it's what's your total cost for that thing, 544 00:19:45,184 --> 00:19:46,565 and and what's your ROI? 545 00:19:46,940 --> 00:19:48,299 And like you said, over the course of 546 00:19:48,299 --> 00:19:50,059 four years, if you get hacked once and 547 00:19:50,059 --> 00:19:51,519 it costs you $50, 548 00:19:51,740 --> 00:19:53,660 well, if you only paid, you know, a 549 00:19:53,660 --> 00:19:55,819 thousand bucks a year for backups or that, 550 00:19:55,819 --> 00:19:58,559 depending on your size, it's well worth it 551 00:19:58,779 --> 00:20:00,535 to come through and and get to where 552 00:20:00,535 --> 00:20:02,634 they need to be. Yeah. A %. 553 00:20:06,615 --> 00:20:08,695 Do you feel overwhelmed by trying to manage 554 00:20:08,695 --> 00:20:11,095 your Office three sixty five environment? Are you 555 00:20:11,095 --> 00:20:14,315 facing unexpected issues that disrupt your company's productivity? 556 00:20:14,670 --> 00:20:16,509 Intelligink is here to help. Much like you 557 00:20:16,509 --> 00:20:18,509 take your car to the mechanic that has 558 00:20:18,509 --> 00:20:20,590 specialized knowledge on how to best keep your 559 00:20:20,590 --> 00:20:21,250 car running, 560 00:20:21,549 --> 00:20:24,430 Intelligink helps you with your Microsoft cloud environment 561 00:20:24,430 --> 00:20:25,890 because that's their expertise. 562 00:20:26,269 --> 00:20:28,590 Intelligink keeps up with the latest updates in 563 00:20:28,590 --> 00:20:30,724 the Microsoft cloud to help keep your business 564 00:20:30,724 --> 00:20:33,044 running smoothly and ahead of the curve. Whether 565 00:20:33,044 --> 00:20:34,964 you are a small organization with just a 566 00:20:34,964 --> 00:20:37,444 few users up to an organization of several 567 00:20:37,444 --> 00:20:40,244 thousand employees, they want to partner with you 568 00:20:40,244 --> 00:20:41,544 to implement and administer 569 00:20:41,845 --> 00:20:43,625 your Microsoft cloud technology. 570 00:20:44,349 --> 00:20:47,809 Visit them at inteliginc.com/podcast. 571 00:20:48,029 --> 00:20:54,769 That's intelligink.com/podcast 572 00:20:55,230 --> 00:20:57,309 for more information or to schedule a thirty 573 00:20:57,309 --> 00:20:59,355 minute call to get started with them today. 574 00:20:59,755 --> 00:21:03,035 Remember, Intelligink focuses on the Microsoft cloud so 575 00:21:03,035 --> 00:21:04,734 you can focus on your business. 576 00:21:06,875 --> 00:21:09,855 I think it covers that well. Like, 100% 577 00:21:09,914 --> 00:21:11,294 have some backup 578 00:21:11,595 --> 00:21:14,289 somewhere, and it is absolutely worth the cost. 579 00:21:14,289 --> 00:21:15,970 I was talking to another client after it, 580 00:21:15,970 --> 00:21:18,289 and it's knock on wood, I hope I 581 00:21:18,289 --> 00:21:19,970 don't ever run into it, like, with my 582 00:21:19,970 --> 00:21:21,650 home stuff, but more and more of the 583 00:21:21,650 --> 00:21:24,070 mindset does not if you have a ransomware 584 00:21:24,130 --> 00:21:26,049 incident or a security incident, but when you 585 00:21:26,049 --> 00:21:27,670 have it. I feel like they're becoming 586 00:21:28,125 --> 00:21:30,924 so common. And it's really one of those 587 00:21:30,924 --> 00:21:32,704 things you need to think about is 588 00:21:33,085 --> 00:21:35,644 take that mindset of not, oh, this isn't 589 00:21:35,644 --> 00:21:37,744 gonna happen to me. I'm a small business. 590 00:21:38,045 --> 00:21:40,445 Again, this was not like a fortune 500 591 00:21:40,445 --> 00:21:44,049 or fortune 1,000 company. It probably they probably 592 00:21:44,109 --> 00:21:45,890 fall into the SMB space 593 00:21:46,190 --> 00:21:48,670 in terms of size, but it's these attackers 594 00:21:48,670 --> 00:21:50,690 aren't out there, I would say, looking for 595 00:21:50,990 --> 00:21:53,390 who's gonna be our most profitable person to 596 00:21:53,390 --> 00:21:54,990 attack or where can we get the biggest 597 00:21:54,990 --> 00:21:57,865 ransom. Chances are they're out there running scripts 598 00:21:57,865 --> 00:21:59,644 looking for these zero day vulnerabilities. 599 00:22:00,105 --> 00:22:02,505 And once they get in, they're taking advantage 600 00:22:02,505 --> 00:22:04,825 of getting in, not really caring about who 601 00:22:04,825 --> 00:22:06,924 it is, what size company it is, 602 00:22:07,225 --> 00:22:09,305 any of that. They just wanna get in 603 00:22:09,305 --> 00:22:11,809 and get that ransom in. You need to 604 00:22:11,809 --> 00:22:14,529 have that mindset of, how am I going 605 00:22:14,529 --> 00:22:17,170 to protect myself when this happens? Or how 606 00:22:17,170 --> 00:22:19,410 can I best position myself for when it 607 00:22:19,410 --> 00:22:21,990 happens to me so that I can be 608 00:22:22,049 --> 00:22:24,690 up and recovered quickly so that damage is 609 00:22:24,690 --> 00:22:25,190 minimal? 610 00:22:25,650 --> 00:22:26,984 All of that. Yeah. It's 611 00:22:27,785 --> 00:22:30,424 a rough place to be. So so let's 612 00:22:30,424 --> 00:22:32,444 see. They broke through the firewall. They encrypted 613 00:22:32,825 --> 00:22:33,325 all 614 00:22:33,944 --> 00:22:36,204 the VMs. They encrypted the 615 00:22:36,585 --> 00:22:39,704 backup storage along the way, lost all your 616 00:22:39,704 --> 00:22:40,204 DCs 617 00:22:40,950 --> 00:22:42,169 and all your other 618 00:22:42,470 --> 00:22:42,970 supporting 619 00:22:43,750 --> 00:22:44,250 infrastructure. 620 00:22:44,950 --> 00:22:46,789 The other thing that stands out to me 621 00:22:46,789 --> 00:22:48,950 is the only way to recover from this 622 00:22:48,950 --> 00:22:52,149 stuff is to kind of save it when 623 00:22:52,149 --> 00:22:54,575 you're done. Because even if you, you know, 624 00:22:54,575 --> 00:22:57,054 go ahead and and pay the VIG and, 625 00:22:57,054 --> 00:22:58,494 hey, I'm I'm gonna pay and get the 626 00:22:58,494 --> 00:23:01,294 decryption key because that's the the thing to 627 00:23:01,294 --> 00:23:02,815 do. And and it seems like that's been 628 00:23:02,815 --> 00:23:05,240 happening more and more lately too with, you 629 00:23:05,240 --> 00:23:06,440 know, some of them. I've seen some of, 630 00:23:06,440 --> 00:23:08,440 like, the bigger hacks on, you know, like, 631 00:23:08,440 --> 00:23:10,840 medical facilities and things in The US where, 632 00:23:10,840 --> 00:23:12,680 you know, these entire hospital systems are going 633 00:23:12,680 --> 00:23:14,039 down and they're just paying it because they 634 00:23:14,039 --> 00:23:16,460 have to to get out. But you can't 635 00:23:16,680 --> 00:23:18,299 trust it at the end of the day. 636 00:23:18,474 --> 00:23:18,974 It's 637 00:23:19,514 --> 00:23:20,254 not viable 638 00:23:20,634 --> 00:23:22,554 to to, you know, to to go down 639 00:23:22,554 --> 00:23:24,474 that path. You're gonna have to come back 640 00:23:24,474 --> 00:23:27,274 in and say, hey. Guess what? It's time 641 00:23:27,274 --> 00:23:30,554 to pave that ESX O store set of 642 00:23:30,554 --> 00:23:32,494 hosts and and that entire environment. 643 00:23:32,890 --> 00:23:33,630 It's time 644 00:23:34,329 --> 00:23:35,390 to pave your 645 00:23:35,929 --> 00:23:36,429 firewall. 646 00:23:36,809 --> 00:23:38,970 It's probably time to pave all your managed 647 00:23:38,970 --> 00:23:41,210 switches and and everything else that sits in 648 00:23:41,210 --> 00:23:43,369 between. And it could be time to pave 649 00:23:43,369 --> 00:23:45,325 your clients as well that we're connecting, Like, 650 00:23:45,325 --> 00:23:47,244 even if they weren't encrypted because you never 651 00:23:47,244 --> 00:23:49,725 know what an attacker leaves behind. Yeah. A 652 00:23:49,725 --> 00:23:52,525 %. And that's always, like, I mean, I 653 00:23:52,525 --> 00:23:53,884 would say one of the first things is 654 00:23:53,884 --> 00:23:56,045 once you start getting it back, it's get 655 00:23:56,045 --> 00:23:59,105 all your antivirus malware, security detection, 656 00:23:59,725 --> 00:24:00,990 all of that up to date to start 657 00:24:00,990 --> 00:24:02,589 watching for it. But it is, it's always 658 00:24:02,589 --> 00:24:05,150 like, well, where is an account? Where is 659 00:24:05,150 --> 00:24:07,970 a random PowerShell script sitting? Where is 660 00:24:08,269 --> 00:24:10,190 all these things to your point that you 661 00:24:10,190 --> 00:24:12,589 can't really trust and you're always on the 662 00:24:12,589 --> 00:24:15,884 alert and there will be stuff continuing on 663 00:24:15,884 --> 00:24:17,484 to try to get it back. The other 664 00:24:17,484 --> 00:24:19,244 thing I did not mention this to you 665 00:24:19,244 --> 00:24:19,744 earlier, 666 00:24:20,204 --> 00:24:22,444 it affected them, but it also would affect 667 00:24:22,444 --> 00:24:25,085 somebody like you mentioned a hospital. And I 668 00:24:25,085 --> 00:24:27,660 had not thought of this before. 669 00:24:28,279 --> 00:24:29,799 I would say because in my scenario, I'm 670 00:24:29,799 --> 00:24:32,460 not in it. But as more and more 671 00:24:32,840 --> 00:24:33,740 physical devices, 672 00:24:34,200 --> 00:24:37,500 you think IoT type stuff, right, is controlled 673 00:24:37,720 --> 00:24:39,980 by applications or by servers. 674 00:24:40,345 --> 00:24:41,644 What is the default 675 00:24:41,945 --> 00:24:42,445 state 676 00:24:42,904 --> 00:24:43,644 of certain 677 00:24:44,025 --> 00:24:44,525 physical 678 00:24:44,904 --> 00:24:45,404 devices, 679 00:24:45,945 --> 00:24:47,325 or what is the 680 00:24:48,105 --> 00:24:50,684 situation that happens when physical devices 681 00:24:51,225 --> 00:24:53,545 lose connection to the server that controls them? 682 00:24:53,545 --> 00:24:55,450 And that was one of the first questions 683 00:24:55,450 --> 00:24:55,950 that 684 00:24:56,809 --> 00:24:59,150 got asked in this scenario was, 685 00:24:59,690 --> 00:25:02,330 this specific stuff, what is gonna happen because 686 00:25:02,330 --> 00:25:04,650 the application that it normally talks to is 687 00:25:04,650 --> 00:25:07,309 gone? It's, yeah, all these different scenarios 688 00:25:07,610 --> 00:25:09,845 that I would say not people don't think 689 00:25:09,845 --> 00:25:11,924 of until they're in them. Again, hopefully from 690 00:25:11,924 --> 00:25:13,764 this, people can think through some of this, 691 00:25:13,764 --> 00:25:17,224 some lessons learned of different aspects to consider 692 00:25:17,524 --> 00:25:19,065 to best protect 693 00:25:19,444 --> 00:25:20,184 your organization 694 00:25:20,619 --> 00:25:21,740 as you think through some of this? I 695 00:25:21,740 --> 00:25:22,940 mean, I think there's a couple ways to 696 00:25:23,019 --> 00:25:24,539 one is you can think about it from 697 00:25:24,539 --> 00:25:26,460 just the front end of, like, what are 698 00:25:26,460 --> 00:25:27,200 best practices 699 00:25:27,659 --> 00:25:29,659 and and and what happens along the way. 700 00:25:29,659 --> 00:25:31,359 I think one of the things that always 701 00:25:31,659 --> 00:25:34,079 strikes me, like, when I'm having conversations 702 00:25:34,380 --> 00:25:35,440 with folks about 703 00:25:35,894 --> 00:25:37,994 these types of things, ransomware attacks, 704 00:25:38,375 --> 00:25:41,734 you know, business continuity, disaster recovery, they're often 705 00:25:41,734 --> 00:25:44,775 thinking about kind of the best practice aspects 706 00:25:44,775 --> 00:25:46,375 of it from the front end. I I 707 00:25:46,375 --> 00:25:48,634 also like to come in from the 708 00:25:49,015 --> 00:25:51,990 back end of things. So if you're gonna 709 00:25:51,990 --> 00:25:53,289 go out and look at 710 00:25:53,590 --> 00:25:56,650 what are common attack factors for ransomware, 711 00:25:57,269 --> 00:25:59,289 yeah, absolutely. Go out and look at that. 712 00:25:59,430 --> 00:26:02,150 You should also go out and look to 713 00:26:02,150 --> 00:26:04,890 a certain degree around what does incident response 714 00:26:05,105 --> 00:26:08,224 look like? What happens when somebody who's, like, 715 00:26:08,224 --> 00:26:11,505 an IR professional comes in, and what do 716 00:26:11,505 --> 00:26:14,065 they look at, and what's their checklist, and 717 00:26:14,065 --> 00:26:15,664 what do they run through? And a lot 718 00:26:15,664 --> 00:26:18,384 of that material is out there on the 719 00:26:18,384 --> 00:26:20,880 interwebs as well to go ahead and read 720 00:26:20,880 --> 00:26:22,500 through based on your scenario 721 00:26:22,960 --> 00:26:25,440 and just how you land and how your 722 00:26:25,440 --> 00:26:28,420 environment composes as a customer. Because your considerations, 723 00:26:28,640 --> 00:26:30,240 I think, are very different if you're like 724 00:26:30,240 --> 00:26:33,220 that on prem customer, you're maintaining a firewall, 725 00:26:33,360 --> 00:26:35,224 or maybe you've got an MSP or somebody 726 00:26:35,224 --> 00:26:38,444 maintaining it for you. And then you've got 727 00:26:38,744 --> 00:26:39,244 the, 728 00:26:39,625 --> 00:26:41,144 you know, the customers who are all in 729 00:26:41,144 --> 00:26:42,664 on the cloud and then the hybrid ones 730 00:26:42,664 --> 00:26:45,224 in the middle and every permutation in between 731 00:26:45,224 --> 00:26:46,044 those. So 732 00:26:46,380 --> 00:26:49,099 go out based on your constraints and your 733 00:26:49,099 --> 00:26:51,680 environment and and do some of that research. 734 00:26:51,980 --> 00:26:54,299 And then hold up to examples like this 735 00:26:54,299 --> 00:26:56,539 as, like, you know, with your finance department 736 00:26:56,539 --> 00:26:58,779 and say, it might be worth us spending 737 00:26:58,779 --> 00:27:00,079 a little bit of money today 738 00:27:00,654 --> 00:27:03,315 to go ahead and not have these problems 739 00:27:03,375 --> 00:27:05,454 in the future. I will just say, like, 740 00:27:05,454 --> 00:27:07,615 there's a reason why some of, like, the 741 00:27:07,615 --> 00:27:10,734 most successful, like, SaaS companies out there tend 742 00:27:10,734 --> 00:27:12,494 to be in, like, the BCDR space. Like, 743 00:27:12,494 --> 00:27:14,255 if you look at them by revenue, by 744 00:27:14,255 --> 00:27:16,759 size, things like this, like, they're not inconsequential 745 00:27:17,059 --> 00:27:20,259 entities. It's because customers, like, they're customers of 746 00:27:20,259 --> 00:27:23,059 these Veeams and Commvaults and Rubrics and things 747 00:27:23,059 --> 00:27:25,160 like that of the world. The the barracudas, 748 00:27:25,460 --> 00:27:27,700 like, as a customer, you still extract a 749 00:27:27,700 --> 00:27:29,299 lot of value out of them as well. 750 00:27:29,299 --> 00:27:31,535 One other thing I didn't think about, primarily 751 00:27:31,535 --> 00:27:33,714 because it didn't apply in this situation, 752 00:27:34,015 --> 00:27:34,994 but the other 753 00:27:35,375 --> 00:27:38,174 aspect of all of this is to test 754 00:27:38,174 --> 00:27:38,914 those backups. 755 00:27:39,375 --> 00:27:41,535 Hey. You wanna test them? Come on. Right. 756 00:27:41,535 --> 00:27:43,875 We had no backups to restore from. So 757 00:27:44,089 --> 00:27:45,849 it was we didn't find ourselves in the 758 00:27:45,849 --> 00:27:48,750 situation where we had backups, but we couldn't 759 00:27:48,890 --> 00:27:50,970 get a good restore from them. But you 760 00:27:50,970 --> 00:27:52,570 do hear about those as well in these 761 00:27:52,570 --> 00:27:54,109 types of situations where 762 00:27:54,410 --> 00:27:56,250 they do need to go rely on backups. 763 00:27:56,250 --> 00:27:58,224 They have some some immutable backups somewhere. They 764 00:27:58,224 --> 00:28:00,305 have some offline backup somewhere. They go to 765 00:28:00,305 --> 00:28:03,265 restore them, and they can't restore. Or the 766 00:28:03,265 --> 00:28:05,984 backups are or they find that backups aren't 767 00:28:05,984 --> 00:28:08,545 being completed successfully, or there's corruption in the 768 00:28:08,545 --> 00:28:09,684 backups, or 769 00:28:10,065 --> 00:28:10,565 whatever 770 00:28:11,184 --> 00:28:12,869 that might be. I would say that's the 771 00:28:12,869 --> 00:28:14,869 other aspect to all of this is not 772 00:28:14,869 --> 00:28:16,950 just making sure you have all these different 773 00:28:16,950 --> 00:28:19,609 backups and you have plans in place for 774 00:28:19,750 --> 00:28:21,430 where it's gonna back up, but if it 775 00:28:21,430 --> 00:28:23,430 happens and you need to restore, have you 776 00:28:23,430 --> 00:28:25,484 tested it? Do you know it worked successfully? 777 00:28:25,484 --> 00:28:27,565 Do you know the policies that or not 778 00:28:27,565 --> 00:28:30,285 the policies, but the procedures to follow to 779 00:28:30,285 --> 00:28:33,585 properly restore from one of these types of 780 00:28:33,805 --> 00:28:36,525 situations? And, yeah, we talked about offline. We 781 00:28:36,525 --> 00:28:38,380 haven't talked about cloud at all. Like we 782 00:28:38,380 --> 00:28:39,359 were talking before, 783 00:28:39,740 --> 00:28:42,220 use Azure backup. Here's my cloud. We'll get 784 00:28:42,220 --> 00:28:44,539 cloud in a little last minute. Like Azure 785 00:28:44,539 --> 00:28:47,019 backup, ASR, I'm sure there's stuff in AWS 786 00:28:47,019 --> 00:28:49,019 as well. It can back up even some 787 00:28:49,019 --> 00:28:51,674 of these on prem VMware environments up to 788 00:28:51,674 --> 00:28:52,335 the cloud. 789 00:28:52,714 --> 00:28:54,714 It's not hard or complex to set up. 790 00:28:54,714 --> 00:28:57,194 It just takes some effort and some planning 791 00:28:57,194 --> 00:28:58,954 on your part. I put an article from 792 00:28:58,954 --> 00:28:59,454 Veeam 793 00:28:59,835 --> 00:29:01,035 in in the chat, and I'll put it 794 00:29:01,035 --> 00:29:02,555 in the show notes as well. But they 795 00:29:02,555 --> 00:29:04,980 talk about kinda the three, two, one backup 796 00:29:04,980 --> 00:29:06,980 rule and and having these multiple copies and 797 00:29:06,980 --> 00:29:08,980 things like that. But there's a section in 798 00:29:08,980 --> 00:29:11,059 their article that talks about how to think 799 00:29:11,059 --> 00:29:13,880 about this in terms of cloud as well, 800 00:29:14,259 --> 00:29:14,759 and 801 00:29:15,140 --> 00:29:17,460 how how some of that stuff composes and 802 00:29:17,460 --> 00:29:20,355 and what the benefits are to you potentially 803 00:29:20,355 --> 00:29:22,615 as a customer. And I say potentially because 804 00:29:22,674 --> 00:29:25,955 your situation's different than my situation, which is 805 00:29:25,955 --> 00:29:27,815 different from the next person's situation. 806 00:29:28,434 --> 00:29:30,595 So there might be things that, like, I'm 807 00:29:30,595 --> 00:29:33,789 interested in from a backup perspective. Let me 808 00:29:33,789 --> 00:29:36,850 say, like, maybe I'm interested in offline backups. 809 00:29:36,990 --> 00:29:39,009 I don't need instant availability 810 00:29:39,309 --> 00:29:40,990 and instant read, and I just want the 811 00:29:40,990 --> 00:29:43,470 cheapest option possible. So I'm gonna look for 812 00:29:43,470 --> 00:29:45,890 some, you know, kind of archival storage 813 00:29:46,585 --> 00:29:48,345 that I can go ahead and put that 814 00:29:48,345 --> 00:29:50,345 into. So that could be maybe like Glacier 815 00:29:50,345 --> 00:29:52,045 and s three or Deep Glacier. 816 00:29:52,505 --> 00:29:53,565 Could be archive 817 00:29:53,945 --> 00:29:55,164 in Azure, 818 00:29:55,465 --> 00:29:56,585 you know, and it or it could be 819 00:29:56,585 --> 00:29:57,485 one of the equivalent 820 00:29:58,025 --> 00:30:01,059 data classes over in, like, a Wasabi or 821 00:30:01,059 --> 00:30:01,640 a Backblaze, 822 00:30:02,420 --> 00:30:04,259 you know, r two, things like that. So 823 00:30:04,259 --> 00:30:05,700 that's one thing. Like, I might go with 824 00:30:05,700 --> 00:30:08,100 archive storage. You might go with, say, like, 825 00:30:08,100 --> 00:30:10,759 cool or cold because you need instant availability 826 00:30:10,980 --> 00:30:13,065 and some of those things along the way. 827 00:30:13,144 --> 00:30:15,544 Your redundancy needs are gonna be different than 828 00:30:15,544 --> 00:30:17,544 mine. I might be good with just backup 829 00:30:17,544 --> 00:30:19,865 to a single region or a single geo. 830 00:30:19,865 --> 00:30:21,544 You might wanna have that stuff spread all 831 00:30:21,544 --> 00:30:23,865 over the place because it's really important to 832 00:30:23,865 --> 00:30:25,004 you along the way. 833 00:30:25,319 --> 00:30:27,799 You might be concerned about immutability. Like, you 834 00:30:27,799 --> 00:30:30,839 mentioned immutability a bunch of times and worm 835 00:30:30,839 --> 00:30:33,960 storage. So just that write once, read many 836 00:30:33,960 --> 00:30:35,240 kind of things. So I wanna make sure 837 00:30:35,240 --> 00:30:36,759 that once I write it, that's it. No 838 00:30:36,759 --> 00:30:38,119 one else can write to it again, but 839 00:30:38,119 --> 00:30:39,559 I wanna be able to kinda read it 840 00:30:39,559 --> 00:30:41,174 back. That could be important to you. Couldn't 841 00:30:41,174 --> 00:30:43,335 be as important to me. Like, it all 842 00:30:43,335 --> 00:30:43,835 depends. 843 00:30:44,535 --> 00:30:47,494 But those options do exist, and they're out 844 00:30:47,494 --> 00:30:49,835 there. And I think the options for 845 00:30:50,375 --> 00:30:51,974 being able to back these things up and 846 00:30:51,974 --> 00:30:54,920 kinda having not only robust backup plans, but 847 00:30:54,920 --> 00:30:56,779 the ability to test, restore, 848 00:30:57,080 --> 00:30:59,880 test your continuity, things like that, those are 849 00:30:59,880 --> 00:31:01,500 just, like, more streamlined 850 00:31:01,960 --> 00:31:04,359 than they ever have been before. Like, I'm 851 00:31:04,359 --> 00:31:06,440 always super impressed, like, year over year when 852 00:31:06,440 --> 00:31:08,119 I sit down and I look at things 853 00:31:08,119 --> 00:31:08,619 like 854 00:31:09,025 --> 00:31:09,845 Veeam's integrations, 855 00:31:10,545 --> 00:31:12,785 convuls, all this stuff. Like, you know, it's 856 00:31:12,785 --> 00:31:14,944 always been kinda like next, next, next, but 857 00:31:14,944 --> 00:31:17,184 it gets smoothed out and smoothed out more 858 00:31:17,184 --> 00:31:17,845 and more. 859 00:31:18,224 --> 00:31:20,224 It's, oh, I could take that VMware VM 860 00:31:20,224 --> 00:31:21,744 and back it up over there, but I 861 00:31:21,744 --> 00:31:23,240 don't have to restore it back to VMware. 862 00:31:23,240 --> 00:31:25,259 I can restore it to a different target 863 00:31:25,720 --> 00:31:27,259 and have it go to 864 00:31:27,720 --> 00:31:29,720 go to a different place. Or I wanna 865 00:31:29,720 --> 00:31:32,440 back up to multiple targets and coordinate that 866 00:31:32,440 --> 00:31:34,680 all on different times. So, you know, I 867 00:31:34,680 --> 00:31:36,119 want this backup to be in the form 868 00:31:36,119 --> 00:31:38,955 of snapshots or versioning here locally, and I 869 00:31:38,955 --> 00:31:40,394 want it to be this other thing and 870 00:31:40,394 --> 00:31:41,295 this other construct 871 00:31:41,914 --> 00:31:44,634 over here. So, like, the tooling has gotten 872 00:31:44,634 --> 00:31:46,955 really good too. There's almost no excuse not 873 00:31:46,955 --> 00:31:48,575 to do it other than 874 00:31:48,955 --> 00:31:51,375 you're scared of paying the licensing. 875 00:31:51,755 --> 00:31:53,399 But I don't know. I think it's always 876 00:31:53,399 --> 00:31:55,480 a good lesson. Like, go out and look 877 00:31:55,480 --> 00:31:57,720 at what's happening with other folks when these 878 00:31:57,720 --> 00:31:58,779 things do happen. 879 00:31:59,079 --> 00:31:59,579 And 880 00:31:59,960 --> 00:32:01,960 it kinda sucks to say it, but it's 881 00:32:01,960 --> 00:32:03,179 it's really a question 882 00:32:03,480 --> 00:32:03,980 of 883 00:32:04,534 --> 00:32:06,934 if, not, you know, when it happens, if 884 00:32:06,934 --> 00:32:09,734 it's gonna happen. Yeah. If not, when yeah. 885 00:32:09,734 --> 00:32:11,515 Nothing. Sorry. It's been a week. Yep. 886 00:32:11,894 --> 00:32:13,974 I agree. Yeah. And I think the other 887 00:32:13,974 --> 00:32:15,494 way to think about that too, like, you 888 00:32:15,494 --> 00:32:17,095 talked about the paying for it. It's kinda 889 00:32:17,095 --> 00:32:19,220 like you pay for insurance. Right? Yes. You 890 00:32:19,220 --> 00:32:21,240 pay for insurance, not because 891 00:32:21,539 --> 00:32:24,180 you wanna use it or because you I 892 00:32:24,180 --> 00:32:26,019 would say you derive value from it, but 893 00:32:26,019 --> 00:32:28,099 it's when you get sick, when you need 894 00:32:28,099 --> 00:32:30,420 it, it's there. Your backups are the same 895 00:32:30,420 --> 00:32:32,394 type of thing. Like, look at it as 896 00:32:32,394 --> 00:32:34,075 a type of insurance cost. This is an 897 00:32:34,075 --> 00:32:36,474 insurance for when something goes wrong, when you 898 00:32:36,474 --> 00:32:39,355 get attacked, when something gets corrupted, etcetera. And 899 00:32:39,355 --> 00:32:42,075 I think too many companies, and maybe this 900 00:32:42,075 --> 00:32:43,934 is a message from to IT 901 00:32:44,369 --> 00:32:46,849 to whoever's writing the checks, is too many 902 00:32:46,849 --> 00:32:48,609 companies or if you're a CEO listening to 903 00:32:48,609 --> 00:32:49,990 this or an executive, 904 00:32:50,289 --> 00:32:52,710 you think of backups as an IT expenditure 905 00:32:53,329 --> 00:32:54,390 versus an insurance 906 00:32:55,009 --> 00:32:57,329 expenditure. Like, you don't think twice if you're 907 00:32:57,329 --> 00:32:59,670 a business about writing a check for 908 00:33:00,025 --> 00:33:02,845 your liability insurance or your business insurance or 909 00:33:02,984 --> 00:33:05,884 workman's comp insurance, all those different insurance policies. 910 00:33:06,105 --> 00:33:08,025 But then as soon as your insurance policy 911 00:33:08,025 --> 00:33:10,904 is my servers, my infrastructure, my data in 912 00:33:10,904 --> 00:33:12,849 lieu of paying for backup, you're like, no. 913 00:33:12,849 --> 00:33:14,529 IT can't spend that much. We need to 914 00:33:14,529 --> 00:33:16,630 cut the IT budget here because 915 00:33:17,250 --> 00:33:19,569 of IT can't spend that much. No. Your 916 00:33:19,569 --> 00:33:21,750 backups are it's just another type of insurance 917 00:33:21,809 --> 00:33:23,990 that you should have for your business. Absolutely. 918 00:33:24,625 --> 00:33:26,704 It absolutely is. And and depending on how 919 00:33:26,704 --> 00:33:28,065 you approach it, you might be able to 920 00:33:28,065 --> 00:33:30,784 find this in different ways as well. Like, 921 00:33:30,784 --> 00:33:33,025 there's maybe you as an individual or an 922 00:33:33,025 --> 00:33:35,984 SMB going out and owning this infrastructure and 923 00:33:35,984 --> 00:33:37,589 doing it yourself. Maybe you go to an, 924 00:33:37,669 --> 00:33:38,490 like, an MSP. 925 00:33:39,109 --> 00:33:42,069 And quite often, like, MSPs will have, you 926 00:33:42,069 --> 00:33:44,390 know, varying levels of service plans and things 927 00:33:44,390 --> 00:33:46,630 like that, and they might include Yep. Even 928 00:33:46,630 --> 00:33:48,650 additional insurance, like, protections 929 00:33:49,109 --> 00:33:51,744 in there or things that come back to 930 00:33:51,744 --> 00:33:53,984 you that way, or you can look to 931 00:33:53,984 --> 00:33:57,345 the SaaS providers. And I think kinda the 932 00:33:57,345 --> 00:34:00,625 the value that they bring to some of 933 00:34:00,625 --> 00:34:03,509 these environments. But it's incumbent on us as 934 00:34:03,509 --> 00:34:06,389 customers, like, understand, like, our use cases, our 935 00:34:06,389 --> 00:34:07,849 scenarios, our constraints, 936 00:34:08,389 --> 00:34:11,429 and then kinda line everything else up around 937 00:34:11,429 --> 00:34:14,309 that. But, you know, unfortunately, this stuff, like, 938 00:34:14,309 --> 00:34:16,635 it's just become a cost of doing business. 939 00:34:16,795 --> 00:34:19,514 Whether you you are actually a business, whether 940 00:34:19,514 --> 00:34:22,315 you're an individual, you know, you coming to 941 00:34:22,315 --> 00:34:24,554 me earlier and saying, hey. Like, let's talk 942 00:34:24,554 --> 00:34:27,054 about this ransomware attack. Like, I still cringe. 943 00:34:27,195 --> 00:34:29,034 Like, my so, you know, my on prem 944 00:34:29,034 --> 00:34:31,514 NAS here, I I have a QNAP NAS 945 00:34:31,514 --> 00:34:33,530 at home, and I had a backup service 946 00:34:33,530 --> 00:34:35,289 enabled on that NAS that had a zero 947 00:34:35,289 --> 00:34:37,469 day attack in it. My NAS got encrypted 948 00:34:37,530 --> 00:34:39,530 last year. It wasn't anything on my fault. 949 00:34:39,530 --> 00:34:41,849 It was a vulnerability in the backup app 950 00:34:41,849 --> 00:34:43,610 that came in and did it. Like, I 951 00:34:43,610 --> 00:34:45,769 wasn't even, like, exposed through my firewall or 952 00:34:45,769 --> 00:34:47,594 anything. Like, I didn't have a specific punch 953 00:34:47,594 --> 00:34:49,114 through the firewall to get this thing on 954 00:34:49,114 --> 00:34:51,355 the Internet, but it had that connectivity from 955 00:34:51,355 --> 00:34:54,234 the cloud service to broker connections back to 956 00:34:54,234 --> 00:34:56,394 my NAS, and I lost it and had 957 00:34:56,394 --> 00:34:58,875 to pave it. It was painful. Yeah. My 958 00:34:58,875 --> 00:35:00,394 wife was not happy the day the Plex 959 00:35:00,394 --> 00:35:01,295 server disappeared. 960 00:35:01,599 --> 00:35:04,019 Right? Again, another one of those. Not the 961 00:35:04,079 --> 00:35:06,159 if, but the when. Yeah. Did you have 962 00:35:06,159 --> 00:35:08,000 it all backed up somewhere? No. I did 963 00:35:08,000 --> 00:35:09,679 not. Oh, you did not? Rebuilt it all. 964 00:35:09,679 --> 00:35:12,159 Lost my entire iTunes library. That one really 965 00:35:12,159 --> 00:35:14,719 hurt. Oh, yeah. See, mine's backed up to 966 00:35:14,719 --> 00:35:16,639 Azure. I do have my NAS backing up 967 00:35:16,639 --> 00:35:18,375 to I don't remember if I'm going to 968 00:35:18,375 --> 00:35:20,295 cold or archive storage in Azure. So this 969 00:35:20,295 --> 00:35:21,815 is the thing. Right? I I was backing 970 00:35:21,815 --> 00:35:22,215 up to 971 00:35:22,855 --> 00:35:24,375 I was because I was using their hybrid 972 00:35:24,375 --> 00:35:26,135 backup. I had it enabled. I was backing 973 00:35:26,135 --> 00:35:28,135 up to s three. Yep. But the backups 974 00:35:28,135 --> 00:35:29,974 in s three weren't encrypted, and I was 975 00:35:29,974 --> 00:35:32,199 only keeping one copy. So by the time 976 00:35:32,199 --> 00:35:33,800 I caught it was encrypted, it had already 977 00:35:33,800 --> 00:35:35,320 run through, and it had just backed up 978 00:35:35,320 --> 00:35:37,000 an encrypted copy of it on the on 979 00:35:37,000 --> 00:35:38,680 the Delta. And, yeah, the the whole thing 980 00:35:38,680 --> 00:35:40,599 was a mess. Learn from my mistakes as 981 00:35:40,599 --> 00:35:41,960 well. Yeah. I was just actually gonna look 982 00:35:41,960 --> 00:35:43,640 at my Azure and see what my Azure 983 00:35:43,960 --> 00:35:45,559 I don't think that could happen. I'd have 984 00:35:45,559 --> 00:35:47,025 to look. I guess that's a downside too 985 00:35:47,025 --> 00:35:48,505 of the right ones. Right? Because if you're 986 00:35:48,505 --> 00:35:51,324 overwriting your backups and not keeping historical backups. 987 00:35:51,465 --> 00:35:53,144 Yeah. You gotta keep a couple out there. 988 00:35:53,144 --> 00:35:54,985 So but but, again, that was my constraint. 989 00:35:54,985 --> 00:35:56,425 Like, I didn't wanna pay all that money 990 00:35:56,425 --> 00:35:57,625 for that stuff because I was like, oh, 991 00:35:57,625 --> 00:35:59,589 it's just my Plex server. Like, I'm gonna 992 00:35:59,589 --> 00:36:00,210 lose, like, 993 00:36:01,070 --> 00:36:03,550 a couple of MP threes, not my entire 994 00:36:03,550 --> 00:36:05,469 library kind of thing. Or I was just 995 00:36:05,469 --> 00:36:07,150 thinking, hey, if a disk goes bad and 996 00:36:07,150 --> 00:36:08,590 I I don't wanna wait for the disk 997 00:36:08,590 --> 00:36:10,269 to rebuild and I gotta go pull something 998 00:36:10,269 --> 00:36:12,344 from s three, that'll be easy enough. But 999 00:36:12,744 --> 00:36:14,344 it didn't didn't turn out to be the 1000 00:36:14,344 --> 00:36:17,144 case. More lessons learned. Alright. Well, with that, 1001 00:36:17,144 --> 00:36:19,224 Scott, I have a meeting that starts in, 1002 00:36:19,224 --> 00:36:21,704 like, thirty seconds. Alright. Well, I'll let you 1003 00:36:21,704 --> 00:36:23,224 get to it. Thanks, Ben. And I will 1004 00:36:23,224 --> 00:36:25,920 let you go. Thank you. Enjoy your weekend, 1005 00:36:25,920 --> 00:36:27,380 and talk to you later. 1006 00:36:28,719 --> 00:36:30,960 If you enjoyed the podcast, go leave us 1007 00:36:30,960 --> 00:36:33,199 a five star rating in iTunes. It helps 1008 00:36:33,199 --> 00:36:34,880 to get the word out so more IT 1009 00:36:34,880 --> 00:36:36,960 pros can learn about Office three sixty five 1010 00:36:36,960 --> 00:36:37,784 in and Azure. 1011 00:36:38,244 --> 00:36:39,844 If you have any questions you want us 1012 00:36:39,844 --> 00:36:42,085 to address on the show, or feedback about 1013 00:36:42,085 --> 00:36:44,405 the show, feel free to reach out via 1014 00:36:44,405 --> 00:36:46,585 our website, Twitter, or Facebook. 1015 00:36:46,885 --> 00:36:48,804 Thanks again for listening, and have a great 1016 00:36:48,804 --> 00:36:49,304 day.

Digital Dispatch Podcast Podcast Artwork Image

Microsoft Cloud IT Pro Podcast

Ben Stegink, Scott Hoag

On the MS Cloud IT Pro Podcast, Scott and Ben discuss the Microsoft Cloud with a focus on IT Pros. They'll discuss the latest in Microsoft 365 and Office 365 News, Azure news and talk about their experiences with managing the Microsoft Cloud as well as interview industry experts on various cloud technology. They'll cover things such as SharePoint, Exchange, Microsoft Teams, PowerShell, Azure, Azure AD, Security, Networking, Storage, and the many other technologies and products that have made their way into the Microsoft 365 suite and Azure. To stay up-to-date on the latest in Microsoft Cloud news and gain some valuable knowledge as you deploy it within your own organization, make sure to tune in every week! Find out more at msclouditpropodcast.com.

Contact Show