Celerity – Security Panel – March 2020
ANDREW MCLEAN [00:00:17] Welcome to this very special episode of security panel brought to you by Celerity. This special episode is on data protection and we’re titling it, “Securing Your Backup Data”. There is my absolute pleasure in this panel to be joined by the maestros of data storage, Darren Sanders from Celerity and Lee McEvoy from IBM. Guys, welcome.
DARREN SANDERS [00:00:41] Thanks, Andrew.
LEE MCAVOY [00:00:42] Thanks for having us.
ANDREW MCLEAN [00:00:44] Well, for the benefit of the audience at home, would you like to begin by telling us who you are and what you do, Darren?
DARREN SANDERS [00:00:52] Yeah, no problems. Darren Sanders, a technical architect working for celebrity specialising in data protection storage. Been with for Celerity for almost 18 years now. And I cover the complete IBM software and hardware storage portfolio. We’ve worked with IBM products for over 25 years now. And so a few grey hairs showing. We are enjoying it.
ANDREW MCLEAN [00:01:16] And Lee?
LEE MCAVOY [00:01:16] I’m Lee McAvoy. And just to finish your Darren’s introduction, he’s also an IBM champion, which is a big thing. And congratulations again to him for that. So my name Lee McAvoy, I’m a storage specialist at IBM. I work with all sorts of different customers, but also have a focus on service providers who have some of the most stringent requirements based on what their customers need to have. And for that reason, I had to help my customers design solutions so they can bank host banking software solutions that were running payments and current accounts for the UK.
ANDREW MCLEAN [00:01:58] Okay, so let’s start with the very obvious question. You got backup. You got cyber security. How exactly does backup mitigate cyber security?
LEE MCAVOY [00:02:11] So I think Darren and I actually do a bit of a double act on this one. So if I start off by giving you the background as to where we are, how we got here. So obviously from the first user and computers for business, we recognise that things can and sometimes do go wrong. So typically we recognise that we need to have backups in case of system failures and obviously gets a lot more reliable than it used to be. But we still do backups in case of a system failure. But what we also recognised was that for more critical systems that we can’t be doing without. If there’s a failure of the local power grid or the network or whatever, you also need to be able to cope with a site failure. So that’s what we started doing, things like replication as well. But when you actually analyse the situations where customers had data issues or data loss, most of those circumstances have arisen due to what we euphemistically call finger trouble or human error. So we have to protect against those things as well. And we still need to continue to protect against all those things. But it’s on that basis that we then start to build up and recognise this new threat where it’s malicious actors are effectively doing what finger trouble used to do. And that’s cyber attacks.
DARREN SANDERS [00:03:33] Yeah, just to build on what Leigh said about how backups that has evolved over the years, that it’s no longer just simply a method for you to recover your data. You have to think about recovering the data to a point in time before potentials cyberattack, which isn’t as easy as it sounds because you got no idea how that threat has been resident in your infrastructure. How many backup restore points also contain that threat? There’s a lot more challenges now. The backup software itself can obviously help you mitigate the risk of cyber attacks. And that’s more around identifying the fact that a threat has happened by using trend analysis to monitor daily data ingest anomalies, highlighting debt reduction levels that are potentially out of the norm, to maybe pinpoint certain clients that may have been attacked by something like a malware where all the data is being encrypted. But these are backend security tools. So these are obviously highlighting something after the event, which isn’t always idea. it’s not the complete solution. So your backend security tools need to work hand-in-hand with your front end tools to help minimise the vulnerabilities and then mitigate the impact of those.
ANDREW MCLEAN [00:04:54] It’s it’s interesting. I’ve heard people talking about cybersecurity and backup. I’ve also heard people talking about cybersecurity and snapshots and know this new term that I had. I wasn’t too familiar with air gaps. What’s the difference between these three things?
[00:05:14] Obviously, we’re talking about backup, where backups are kept for a specified period of time. Short term, long time, long term retention. Snapshots typically are actually taken at point and time copies of a particular bit of data and stored on the same device where the source data resides. That’s the downside there is if you lose that device, you’ve lost your source data. And those snapshots of it. So snapshots is something which you need to use hand-in-hand with backup. And typically where you have low recovery objectives, where you need to sort of be taking a recovery point maybe every hour, every 4 hours, where it’s not feasible to run an entire backup job every hour because of the resources required. You may take you choose to take a an hourly snapshot at the storage level or at the hypervisor level. And again, where recovery time is critical as well, you need to be able to recover that system to that point in time as quickly as possible. And applying a snapshot is still one of the fastest way is available to do that. Air gap sort of takes the backup another stage further where backups tend to be online to enable faster recovery. And air gap is an offline isolated copy of your backup data or a subset of your backup data to protect from cyberattacks, to allow you to recover to a point in time before the attack took place. As we mentioned earlier, where possible that data that would be immutable as well. It’s an offline copy of the data. And with that data that can’t be changed, there’s a there’s no additional policies in place to protect the integrity of that data.
LEE MCAVOY [00:06:59] So to add to that, for many years I’ve been working with organisations where they’ve wanted to have snapshots. They wanted to have replication and they thought that that was enough. So they got well, I’ve got my LPO and I’ve got my site, my data replicated. And it’s one of those released statements. I can’t actually remember who it’s actually really attributed to. But if data is not in 3 places, it’s at risk. And if you’ve got data that’s still living on the primary storage system or it’s replicated pair, you’re still backing up onto yourself, which is something I’m never entirely comfortable with. So we’ve always strongly recommended you back up onto a different system. So if you’ve got a primary storage array, you back up onto a different storage array if it’s a true backup. With air gap. We’re taking that a step further and effectively saying that once you’ve written the data there, the primary system and the backup software that’s put it there. Shouldn’t be able to and shouldn’t be able to change that. Shouldn’t be able to take control of it, destroy that first, because under a number of cyber attacks, the most effective way of extorting money from people has been to get in, understand the lay of the land, destroy the backups, then go after the primary data. Because at that point, where are you going? And that’s what air gap is about. It’s giving me that safe place where they can’t have destroyed it.
ANDREW MCLEAN [00:08:34] Okay. So you’re both talking about, you both spoken about the theory of air gap. What is it? But let’s put it in a more applied sense there. So what would an air gap solution be?
DARREN SANDERS [00:08:47] I think if you look traditionally what the term air gap means, you know, it’s from it’s been around for a long time and it was used to come from a more from a network security perspective since to ensure that secure computer networks were physically isolated from unsecure networks. When we now talk about it from a data protection perspective, looking at backups, we mean taking a copy of backup data. There are a subset of that data and storing it securely off-line. The most common form, when you look at it from, you know, like how do you do it is physical tape. A physical tape cartridge isn’t plugged into the network. Duct tape cartridge can’t be modified unless it’s in a tape drive and is being accessed by an application. So physical tape is the oldest form of air gap that’s been around for a very long time, and it’s still one that’s used. It’s probably the most common form of air gap that’s still used now. Public cloud object storage of the storage technologies which are playing into this space as well, especially now the cloud storage providers are starting to offer more security capabilities, such as the ability to store immutable data on cloud storage. So expect more and more backup air gap solutions to start transitioning to cloud storage. We’ve always designed backup solutions. Isn’t Lee mentioned earlier on way three copies of your data? It’s this three to one approach that you store 3 copies of your data on 2 different types of media. And 1 copy is always off line. Air gap expands that now to what you often hear out I was the 3, 2, to 1 approach. And the extra 1 being the off line immutable copy. And that’s sort of where air gap has played into.
ANDREW MCLEAN [00:10:35] And going back to the terms that we used earlier. So RPO, RTO or recovery point objective, recovery time objective. We just were spelling it out. Recovery point is, if I lose. What’s the largest amount of data. I’m willing to lose is typically how we start talking about RPO in a traditional sense in cybersecurity. And how many points do I want to be able to go back to as well? And the recovery time objective, which is key as well, is how long should it take me to get my systems back up and running? Because it’s great that you’ve got your data. But if it takes you a week to get there, you probably don’t have a business anymore. So it’s just worth having gone through those in a little bit more detail and saying air gap. Along with a small RPO, a small RTO is actually quite hard to do. Because air gap is more of a backup, backup in the traditional sense rather than a snapshot and a recovery point in the more modern sense that we’ve been using for the past decade or so. So if you want to have an air gap solution, that sounds like a negative, but as Darren’s alluded to. The attackers can have been in your network for a long time, they can have been in your infrastructure. I think the typical figures are something like it’s 160 days on average before before the attack is initiated that they’ve been in your network. So at that point, they’ve probably had a good look around and decided what they’re going to do. And potentially, if you don’t have air gap the immutable copies of your backups, you’re potential in trouble already, even before they’ve actually initiated the attack. So having a small RPO, small RTO is what most customers want to have. Having that and an air gap solution, there’s a bit of tension there. And it’s probably also worth mentioning around immutability or the ability to have data that can’t be changed. As Darren’s alluded to. We had that we take for a while and we had it in tape, not just because you could write it to a tape cartridge and then leave it on the shelf. Traditionally, financial services customers had to be able to show a transaction was as it had been written, and it could not have been changed. And therefore we had right once, read many tapes or worm. So effectively that tape cartridge you’d written to it could never be changed. It was there. You could destroy it, but you couldn’t actually change the data. So that’s kind of what we talk about nowadays with the immutability. Certainly some parts of the IBM portfolio around object storage, we’ve got immutability there. But one thing that I’ve found has really been attractive to some customers is I give you a immutability, but I’ll give you a immutability for the time that you want that data to be immutable and then you can reclaim the space afterwards. So it’s a cannot be changed for 24 months if that’s what you deem, anything that goes into that bucket. Will leave 24 months, can’t be changed. But at the end of the 24 months, you get your space back because you can at that point delete it. So immutability is good. Some people hear that and go, oh, my goodness me. How much storage am I going to have to buy if I need to have 7 years of of backups and I’m putting everything into immutable, you then say, well, you’ve got to keep your monthlies for 7 years. Everything else you can put in immutable still to give you that level of protection, but potentially into a different bucket that’s got a different retention period.
ANDREW MCLEAN [00:14:13] So if I were an organisation, would you say that air gap is a cyber security solution?
LEE MCAVOY [00:14:18] Part of a cyber security solution and storage is only part of a cybersecurity solution as well? It’s the whole piece in the round multi aspect. And effectively, storage can help protect against impact. And it can also give you a warning when something’s happening. But things around network security are key as well. Things around patching things around or practises, making sure people are trained against phishing attacks or those other things, which according to scrape the surface there, they’re all key as well.
ANDREW MCLEAN [00:14:58] One of the one of the things that if we were talking about this movie, like 15 years ago, people would have their server sitting in the back of their office. And it was fairly straightforward. They need statistical backup tape in the way it goes. But when now in this kind of a lot of organisations now doing this all multi-cloud approach in different services and in different clouds and so on. What impact does this have?
DARREN SANDERS [00:15:22] I think multi-cloud is all about driving down cost and ramping up your service levels, increasing innovation, which is key these days. And there’s two sides to it. On a positive side, it can add the whole load of flexibility about what you can do with your data. You’ve got the ability to move data around. You can copy data that easily. It facilitates data reuse, which is more and more common now. It gives you access to a lot more features and functionality than if you’re running on just a single storage platform, which obviously means that also all your eggs in one basket from a security point of view, because you’ve got different tiers, are definitely multi-cloud pockets of storage. You’ve got the ability to have different data protection policies and recovery options for those different types of data. As always, there’s negatives to that as well. So there is the potential for increased operational complexity because you’re not managing the one bit of storage which, you know, inside out you’ve been working with for years. You now have a requirement for a much broader skill set in your operational teams. To help alleviate that you can look at things like the IBM software defined storage portfolio to have a software defined storage solution sitting on top the multi-cloud infrastructure. To again make it simpler, reduce the complexity, reduce the skills required, but still retain all the features and functionality that a multi-cloud infrastructure can provide.
LEE MCAVOY [00:17:05] I “decode” for darren said that. I think the key thing is multi-cloud. There are a number of good reasons to do these things. There’s a lot of flexibility to burst. Run the workloads in the right place, augment with functionality that’s potentially in some of those clouds. Great reasons to do it. But it does bring additional complexity. And realistically, customers are saying what we need to be able to do is irrespective where the workload is. I want to have one methodology to protect it and more importantly, recover it. I want to know that irrespective of where I was running it, I’d get it back within my RTO and that I can get it back simply and in a reproducible way. And potentially 2 a different cloud because it could have been on a cloud that’s having issues at that point. So being out to truly be multi-cloud and have a work that you’ve recovered from cloud 1 to cloud 2, watch your own private cloud and have the tooling and the automation and the orchestration to be able to do that. So that’s something that IBM is pushing really, really hard on. We have been IBM traditionally isn’t that fashionable and I’m not going to say look at my dress sense because I’ve put a shirt on, especially for this. But we’ve been talking about hybrid multi-cloud back when people were just talking about cloud. You’re recognising that actually the first 20% of workloads that can be run in cloud could easily be 1 in cloud. But when we look at our customer base. If I was going to count the number of cloud services that I’m interacting with on a daily basis, just as me but alone, some of the organisations that I work with, they’re probably touching 5, 6, 7 different clouds. Some of them are software as a service saying at that point you just have them, you buy that service and then there are hopefully running their solution properly. But if you are running your stuff across multiple clouds, it’s really, really key that. When you’re running it there, it’s running the same way. Not on the same stuff, but in the same way you can recover it. You can. Choose when to move it without having to then go and change your backup strategy. You have to have something that’s automated not just for the recovery, but also recognising when a work load is somewhere then isn’t and it’s somewhere else. The backup follows the workload. Having that kind of capability is really, really important in a hybrid multi-cloud world, which is where we’re moving in chapter two. 80% of customer workloads can’t or couldn’t move in chapter 1. They were too complex. They were monolithic applications. They were too sensitive for those reasons. Chapter 2, when we start moving these complex workloads, those things that have had crown jewel type security on them. In the on premises world, we need to make sure we can do that in a hybrid multi-cloud world. And that’s where IBM is pushing. That’s why we’re leading that our software defined approach.
ANDREW MCLEAN [00:20:24] I am… so I started this panel today calling you the both the storage maestros… Maestro, as I said. So if we were conducting our symphony on a cyber security, how would you recommend people actually protect their storage from cyber security attacks?
DARREN SANDERS [00:20:50] How far do you want to take it? There are so many different things you could look at. You know, to answer that, I mean one of the key things is when you’re looking at your backup data is trying to you need to limit the footprint that your data protection solution and your backup data has on your network. You know, to install your backup on network shares is obviously a good one. You know, for performance means a lot of people look at fibre channel storage hanging behind backup servers, which is great because as soon as your data hits your backup server, it’s off the network. It’s being transmitted over a fibre channel protocol, which isn’t as accessible as the the TTP IP stuck, a fibre channel desk, fibre channel tape, you talk about object storage and the whole “AGAP” thing. if possible, don’t have your backup servers in active directory. Someone hacks active direct… someone hacks you active directory. As Lee mentioned earlier, the first thing they’ll go after is your backup data. Once your backup data has gone, they go afre your production data. If they can’t get the credentials for your backup server, then your backup data is a lot safer. So stop the hackers getting those credentials and use the 3, 2, 1 or the 3, 2 to 1 approach. Store multiple copies, keep one offsite and if possible, have an egg after mutable copy as well.
ANDREW MCLEAN [00:22:22] Lee do you want to quick some for your supper?
LEE MCAVOY [00:22:25] Well, because we’ve primarily been talking about open systems stuff, but I know there are still customers out there that use either a mainframe customers. And typically, they’re the larger accounts that have the most demanding requirements. But we do have storage solutions that have a built in. air gap snapshots within their capabilities. So we call this safeguarded copy. And effectively, it’s so secure, even the storage admin, once they’ve set them up by default, can’t access them. So having safeguarded copy on our plan DSF thousand solutions has been a massive interest for those kind of bank financial services type customers who, let’s face it, are normally the biggest targets for people to go after. Having safeguarded copy where effectively you can have those snapshots but have the air gaps so that the storage admin, even if they weren’t having their credentials hacked, they wouldn’t necessarily be able to destroy their own air gap copies. That’s something I’d recommend people having a look at. Very much so. And one thing I’d say about IBM’s storage portfolio is being very, very, very good over the years of taking something that we developed at the high end. And then bringing it through the rest of our portfolio. So I aspire to be able to tell you at some point in the future what we’ll have done along the same lines of safeguarded copy on our DSF thousand solutions elsewhere in our portfolio. Because it’s a great idea. I would put a cheeky comment in that it helps you buy more storage, which is the storage salesperson I think is a great idea, but it’s buying more storage for a very, very good reason and providing you with an extra level of protection that, let’s face it, for a number of organisations, would have saved them tens, if not millions of pounds worth hundreds of millions of pounds of money from disruption. Recently, I was reading about “not patchier” and what that did to some very significant companies, especially around logistics and the like. And the impact of cyber can be had no hints. And that’s the best it can be. What happened in “mpt patchier” in terms of the impacts of those attacks, 40 foot containers. The docks are on ships. People having to figure out what was in them using paper manifests because of the circumstances that hit that particular company potentially could have been mitigated by some of the things Darren said. Some of the things I was saying around the safeguarded copy.
DARREN SANDERS [00:25:18] I think a safeguarded copy like a thousand storage. It’s a great new feature, which I know a lot of financial organisations are looking at. But to sure how the technology is being introduced, an expert in the mainframe world is to sort of extend out further is there’s also the ability we’ve talked about object storage, cloud storage that the IBM cloud object storage with the S3 connectivity that can also be plugged in at the back of DSF, thousands or “T-77,000” the mainframe virtual tape libraries to provide an object store which then can deliver worm capabilities without the need for physical tape. So there’s a multitude of options out there to help protect both open systems and mainframe data.
ANDREW MCLEAN [00:26:04] Brilliant answers, guys, but we’re going to bring it to a close now. And I’m going to ask you for some advice. Still, if I am a viewer who is thinking about implementing an air gap solution. What is some top level advice that you guys can give me?
DARREN SANDERS [00:26:22] I think to me, from my point of view, you have to think about the bigger picture. Air gap itself is only part of the puzzle. It’s all well and good storing an air gap copy a backup data. But you have to think, how do you recover it? Potentially a recovering infected data. You know, you got no idea if it’s infected or not. So the air gap needs to be part of something which you might call a cyber vault where you can recover your data, are into run forensic analysis on it and share the data as clean. Bring up an application to make sure everything’s accessible before you even consider putting that data back in the production infrastructure and the production storage, because you don’t get that right. You’re just going to reinfect everything, so there’s no there’s no point even doing it. I think a lot of as well. It’s all about breaking traditions, you know, cloud is insecure, tape is evil. It’s not it’s just not true. You know, we’ve talked about the immutability features of cloud storage. And to build on what Lee said before, even when you’ve got a immutable object stored within the cloud, even the cloud service provider cannot delete those objects that secure. So we’re breaking those myths around cloud storage. A few years ago, everyone was moving off tip tapes, making a huge comeback. Now, not only is it a low cost per terabyte, but it can be if a fundamental player in producing the air gap in the cyber vault because of its offline capabilities, worm storage capabilities. I’ll just say, look at the bigger picture, that there’s a lot of technologies out there. Look at the IBM portfolio, not from just a hardware perspective, but look at the software side and see how the software defined storage layer can help improve your data security and your data availability.
[00:28:17] So my comment would be. For company that’s operating at scale, the things that Darren and I’ve touched on should be easily achievable. As part of a wider ranging cyber security play. If you’re a smaller customer, some of those things are potentially a little bit more challenging to do. And that’s where I found that a number of organisations are going to specialist providers of data protection, either in a traditional retail way or increasingly as a backup as a service or data recovery, as a service offering. Because they’re able to put specialists who only do data protection it’s their full time, full time job. And they’ve got a breadth of skills and a breadth of team there. But they’re also able to leverage these leading edge technologies, especially around object and immutability and gap tape. Tape doesn’t have to be expensive. This morning, IBM tape “Bybee’s, 3 year Rackspace” and its modular so it can grow quite nicely. But realistically, if you are trying to operate using tape as an air gap, you’re probably trying to do that because you at scale and it can not. It’s not so easy, but if you’re working with a partner who is a backup specialist and they offer a backup as a service solution, they’ve got that scale. They’ve got that capability because they’re doing that for multiple organisations who typically are backing up petabytes and petabytes of data. And that doesn’t mean that it has to be one of the famous 4 of the cloud providers. There are UK based specialist providers. I think one of which is probably on this call it as of the moment, to offer that as a service and he should absolutely explore that.
DARREN SANDERS [00:30:19] Exactly. We know one of our core services that we offer to our customers is back up as a service. You know, this “DR” a service platform as a service. All of those can combine together to form your almost air gap as a service, because if you’re worried about where’s this clean room that I recover that data to, that can be delivered through platform as a service and for customers who don’t have the infrastructure at hand to do it themselves. That’s what we do and we provide that as a contracted service to our customer base.
ANDREW MCLEAN [00:30:50] So, Darren, Lee, thanks for joining us. Thank you very much.
LEE MCAVOY [00:30:56] Thanks for having us.
DARREN SANDERS [00:30:57] It’s my pleasure.
ANDREW MCLEAN [00:30:58] And that was Darren Sanders from Celerity and Lee McAvoy from IBM. Talking to us on this data protection episode. Securing your back up data. You’ll be watching a special episode of security panel brought to you by Celerity. Until next time. See you soon.