Stedman SQL Server Podcast Episode 3

October, 2 2024

Recoding Date : 10-2-2024
Topic: How important is your SQL Server data? Exploring our IT Manager’s Whitepaper
Host: Steve Stedman

Steve Stedman emphasizes the critical importance of SQL Server data in business operations, highlighting the severe consequences of data loss. He recounts a ransomware attack on a regional hospital that resulted in the complete loss of patient data and backups, underscoring the need for robust backup and business continuity plans. Stedman stresses the importance of regular testing of backups and having multiple data centers to minimize downtime. He shares a case study of a client who, after implementing a business continuity plan, recovered SQL server operations within an hour following a primary site failure. Stedman concludes by urging businesses to proactively protect their data to avoid disaster.

Watch / Listen Now

Podcast Transcript- How important is your SQL Server data?

Steve Stedman 0:18

Hi and welcome to this week’s live stream, or actually podcast. We renamed it a few weeks ago. We used to just call it a live stream, but now it is our SQL Server podcast from Stedman solutions, and this is episode three. Excuse me, this week’s episode is, how important is your data? And my name is Steve Stedman, the host of this podcast. So one of the things we run into a lot of the times when we’re talking with people about backups is kind of varying opinions on how important their data is. We always want to make sure that we treat everything like it’s the absolute crown jewels or the most important thing in their business. But sometimes we don’t always find that the people who own the data feel the same way. Sometimes they do, sometimes they don’t understand. But today I want to kind of dive in a little bit to this. This is a topic that is also covered on our IT managers white paper, which we’ll have a link to download that at the end. But for now, let’s jump into how important is your data? Episode Three of the Stedman solutions, SQL Server podcast. And I should say it’s a Stedman solution SQL Server video podcast, because I got in trouble not with someone expecting that it was going to be just an audio and not being ready for video. So yes, it is the video podcast. So here we go into data. So in today’s digital age, data drives everything in business. And the question, if you really want to prove that, is, go turn off your SQL Server for a couple hours and see how many people scream, see how much stuff is impacted by that data that’s there, or maybe just do a thought experiment and ask people what would be impacted if you turn it off. You don’t actually have to turn it off, but the thing is, what would happen if something catastrophic happened to your data? Let’s say you’re working in an office space. You’ve got a great server room there, and you’ve got all your SQL servers, you’ve got all your other stuff, your web servers and other systems like that, and a fire breaks out in the data center room and in the rack where your SQL servers are, and that SQL Server is burnt to a crisp. Okay, well, what are you going to do about it? What are you going to do to get that back? And the question is, how long is that going to take and things like that. But really the thing to think about is, how is that going to impact business? Most businesses that we come across don’t necessarily have a robust backup and business continuity plan. Now, there’s some clients we work with that. If that happened? No problem. Flip on the second data center. Flip on the second site. Everything’s running at the second site. Well, yeah, it’s a problem. The first site’s got to be rebuilt. But as far as the systems that are running, it’s going to be a brief blip or an outage while we flip the second site on, not a big deal if they lose the primary data center. So think about things like that. How important are your backups, and what would happen if you were if you lost that SQL server and you had no way of getting that data back? So with this, this is vital information, whether it’s customer records, financial operations, things like that, got a call one time from a regional hospital, and they’d been hit with a ransomware attack. And first off, this was not a client of ours. If they had been, we would have had them in much better shape, but they got hit with a ransomware attack, and all of their patient data, the entire patient history, of everything they had had been encrypted, and as well as all the backups which happened to sit on that same SQL Server box, and as well as some of the backups that have been copied to the network, and they paid the ransom, they got the keys, or whatever the magic thing was that supposed to decrypt All the data. But surprise, the ransomware developers were not the best software developers out there, and their systems failed on really large data, really large files, so all their SQL Server files and all their backup files were unable to be decrypted. They were in a point where they had lost 100% of their data and 100% of their backups. They were coming to me and asking, Well, what do we do to get this recovered? And my answer was, gee, sorry, I can’t help you there, because you don’t have any good backups. And in my job, the thing I hate more than anything is having to tell people that they don’t have any way to get their data back. So. So the clients we work with, we make certain that we never have to tell them that. But these people were not prepared. They didn’t have good backups, and they lost everything. Now, could you imagine if you were a patient at that Regional Hospital and you had a surgery scheduled next week, and you went in and at your scheduled appointment time and said, Hi, I’m here for my surgery. And they said, Oh, who are you? Tell us about, tell us who you are. Tell us about why you’re here, because we have no record of anything. It’s all been lost that that would be damaging to that, I mean, and to that business, to the any trust that’s there, but I mean, it could completely cripple the business. And whether it’s a hospital or something dealing with finance, or you’re dealing with a factory of some kind, anything that happens with that server can can take out, take out the system. So the consequences of data loss. There’s a lot of things that can cause this. I mean, one, what are the potential threats? Okay, there’s the basic things like hardware failures, hard drive just crashes. Fails, has to be replaced, things like that. Hopefully you’re covered with redundant drive, so that doesn’t happen. But things could be worse than that. You can have cyber attacks or corruption, or even just human error. Sometimes can destroy data one client called us up and said, hey, somebody accidentally updated every single they didn’t update without a where clause, and it updated every single row in a table and set some column to the wrong value for all but one of them. And that was one of those human error had they not had a good backup plan, there wouldn’t have been a good way to recover that. Now the question is, what is the impact on that? Well, the impact can be a lot of things, from financial loss, from well, financial loss, meaning you’ve got to go and spend money to replace things, as well as you have operational downtime, which usually equates to loss of money. Whether you’re a factory that’s down and you can’t produce product, or whether it’s a website that’s down you can’t sell product, there is operational downtime there that costs things. But also with operational downtime, even if it’s just internal and the site’s down, how many employees are being impacted by that? Let’s say you’ve got 50 people internal that are all working on some system, and that system’s down, and all 50 of those people are just sitting there twiddling their thumbs, waiting for the system to come back up so they can do their jobs. Well, what’s the cost of each of those employees times the amount of downtime? It can be quite a bit, and also damage to your reputation or legal consequences. I mean, every time I get an email that says from a bank or something that says your your data has been compromised, well that’s a damage to their reputation. And I think there’s a lot of different industries where you have legal consequences that you have with that data loss as well. So can your business survive if your data was suddenly lost? And that’s that’s a question that you might have, and depending on who you are or what part of the business you work in have that with someone in management to say, Well, how would we recover if that data was suddenly lost? And there’s some people say, Yeah, we don’t care if that system was lost. No big deal. We can get by without it. But oftentimes, if that core system is lost, it may have significant impact overall in the business. So think about that and go check out in your business what, what would be the impact of that? Okay, and this slide looks horribly formatted, but we’ll take a shot at it anyway. Backup strategies and what do you need to know so when you’re dealing with backups, so many times I’ve talked to people that have had a system failure and they said, Oh yeah, we’ve got backups. And they go and look and find out, Oh yeah, we had backups four years ago, but somebody turned that off three years ago and we haven’t had a backup since then. We thought we had backups, or Yeah, we have backups, but they’re not consistent. What happens if you’re doing a virtual machine backup in your database files are in use and not getting a consistent copy, consistent copy of those database files when you do that machine backup or machine snapshot. Well, there’s a lot of things like that that can impact your backup, so the most important thing relating to those backups is testing them, testing them regularly to ensure that you can restore them when they’re needed. And you’re testing a couple things. When you do this one, you’re testing, do you know how to get them back? Do you know how to get those backups back when you need them? Then you’re also testing, do those. Backups actually work? Are those backups somewhere you can get them quickly and use them, or

are they even restoring the right thing? And what you need to do is you need to practice this once in a while, and I like to practice it one if we’re moving data from a production system to a dev system or a test system, things like that is a great way to test backups. Or you build out a temporary server and restore your backups there, or just restore them under a different name on the same server if you have space. But being able to test those backups and confirm that they’re coming back in a way that they’re usable is so important. So often we’ve seen that people think they have backups or they’re able to restore something, but what they get back is not what they’re expecting. So next we want to talk about business continuity plans in regard to losing your data. This is beyond just recovering your data, but what happens like in the example of a fire in your data center could be something simpler than that. I mean, I worked with a data center about 10 years ago, oh, gosh, maybe it’s more than that. About 15 years ago, where they had this switch that they that was a big switch that would switch from grid power to generator power, and they even had two different power companies. They were right in a place on the border between power companies. They had two different grid power companies providing data to them, and they had a disaster that took them out for almost a week. And what the disaster was, they were doing their regular, regular testing, and they flipped the automatic transfer switch for from grid power to generator. And when that happened, something arced. And it welded that switch in place in an open position where it could not get shore power or grid power or generator power at that point. Now that was something. The Switch was fried at this point. They could not there’s no way they could get it running. It was such a large amount of power they needed. They couldn’t just use jumper cables or something to get around it. And they were in a position where they had battery power in that data center. It was a pretty sizable data center. They had battery power to run for five or six hours, but at the end of that five or six hours, everything shut down. Problem was before that shut down, their AC system shut down because that was not running on battery power. And things started getting hot and hot and hot, and then everything shut down. That gosh, the company I was with when that happened were was down for more than a day while we got the data running somewhere else. After that, that was our wake up call, where we were able to get multiple data centers. We had three different data centers with all the data synced, and we got to a point in that case where we were in a position where we could whatever happened at any data center, from a flood or a fire to something crazy, like an asteroid falling out of the sky to completely destroy the data center, any of those things we had covered by having multiple data centers in multiple regions with different all of them that were accessible, and all of them with all of the data there, so that we could bring them up and running, and get them up and running quickly while I stayed in the time I was at that company, after getting those three in place, we never had another failure. But knock on wood, they got lucky on that one, and we learned from the mistakes, and we learned how to build it out. So my recommendation is business continuity plan. Look at what the worst case scenario is, what happens to your data center, and what are you going to do if that fails? Lot of our customers that we work with in our managed services capacity, one of the things that we’re working with them on is, how do you get that business continuity in place? How do we get things set up so that whatever happens at your primary data center asteroid falling out of the sky and destroying the whole building. That kind of thing happens. How are you covered? And how do we get you up and running at another location? And with that, you need to make sure it’s planned and documented and regularly tested. And one of the things that I like to do when we have multiple sites that we’re working with with business continuity plans is just schedule a regular, regular practice and say, well, once a quarter, we’re going to flip from our primary data center over to our secondary data center and run there for a week or two and confirm that all is good, confirm that it handles the load, confirm that everything works well. And then after we know that, we’ll flip back and run at the primary one ongoing for a while. So why do you need to get these business continuity plans in place? Well, the thing is, once you’ve had a catastrophic failure, it’s too late. There’s nothing you can do about it at that point in time, except for maybe go figure out what backups you might have access to. So planning ahead of time will minimize your. Downtime. And when we talk about minimizing downtime, it might be the difference between a day or two to a week of downtime versus five to 10 minutes of downtime while you turn on that second data center or turn on this other system in order to be able to accommodate the load when something fails. And really what it does is it puts you in a position where whatever happens at that primary system, you’re able to recover at that secondary site. And it doesn’t even have to be a second site. It could be a cloud based server, or even just, I mean, one of the things we do oftentimes is some cloud servers that are undersized, and in the event of a catastrophe, we size them up and get things running and get them running well. But really the key is, what are the odds of this happening? And in today’s world, the odds of something happening that would require your business continuity plan to be executed are probably pretty high. So some of the actionable steps that we need to take for data protection, we want to review the backup strategies regularly, ensure that they’re reliable and tested, develop that business continuity plan, have regular drills and planning and updates to ensure that you’re ready and stay educated. Keep up to date on what kind of threats are out there and best practices around them. So really, the thing we want to cover on this is, if data is your business, and this is core to your survival, you need to make sure that you take proactive protection. You need to make sure that all the servers that you have running SQL server are properly backed up if they’re needed. And sometimes people say, Yeah, we don’t need that server. Okay, no big deal. I mean, it’d be great to back it up so that you don’t find out after the fact that you don’t need it, but make sure that the servers that are important for your company’s future are properly backed up and properly taken care of, so that in the event of disaster, that you’re able to recover, get things running again. We had a client that we were working with where we had just helped them with their business continuity plan. We had just helped them get redundant servers running at another location in another city. They had a catastrophic failure at their primary site. And I don’t, I don’t even know what the failure was. All we were told was that the primary site’s gone for now, and we were able to, in that case, it took about an hour to get things running up at the second site, because it would, it had just been configured, and we had not yet practiced it. But after that hour, we then now have it practice, practiced, and we know we can get that up and running into the second site or flipping back back and forth, either way, now in less than five minutes time. So that’s one of those things that gives them a peace of mind. It lets them sleep better at night, knowing that if something happens, that their system will not be down for days if that primary site fails. And basically taking care of the data, there’s an expression, it’s like you only had one job to do, and that was to make sure that you will always have that SQL Server data available. We like to make sure that whatever happens, we’re covered. So call to action here is don’t wait for a disaster. Prepare now, because once the disaster hits, it’s too late. You can download our white paper. We have a white paper on managing SQL Server for IT managers, or maintaining SQL Server for IT managers. You can download it at stedman.us/maintaining, and what we’ve covered here is just chapter one in that white paper. There’s a lot of other things about what you need to do to take care of your SQL servers. Like I said, the thing that I really hate doing is having to tell people that your server is in a position that it can’t be recovered, and doing everything we possibly can to make sure that nobody is ever in that position. That’s my goal. And we do that through our managed services. We do that through our consulting. We even have a backup assessment we can do to go in and find out if people have the right backup for what they think they need and sometimes what an IT worker thinks they need may be different than what an IT manager thinks they need, which might be different than what the business owner or the president of the company thinks they need. And part of making sure that everyone is on the same page with that is really important in order to be able to do the best or in order to have the best backup and recovery plan so that you can recover from just about any disaster. At this point, I think that wraps up what we had scheduled for the podcast. Want to point out again, we took what we were doing as just a weekly live stream, and now we’ve turned it into a weekly podcast. And if you go to Stedman solutions.com and under the Home tab, there’s a podcast page now you can go and look and browse our previous episodes. This is episode three since we officially renamed it to podcasts. So thanks for watching and stick around. We’ll have another podcast next week.

Thanks for watching our video. I’m Steve, and I hope you’ve enjoyed this. Please click the thumbs up if you liked it. And if you want more information, more videos like this, click the subscribe button and hit the bell icon so that you can get notified of future videos that we create and.

Visit our main podcast page for more episodes.