Interviewer
If you were given unlimited funding to design a system for storing and preserving digital information for at least a century, what would you do?
Mark Lantz
There’s two critical aspects to that question. The first part of it is just preserving bits. That’s the focus of the underlying storage technology itself. If you write some digital data to a medium, how long will that data be readable on the medium itself?
Stacked on top of that are a whole set of other problems. Everything in the IT world is constantly evolving. Even if I have a punch card reader from 1940 that can still read my punch cards, I can’t find a computer that supports the interface it has. Because everything is constantly evolving, the problem goes beyond just preserving the ability to read the bits.
There’s lots of great work in the academic community around how I can create a storage medium where I can write something, and in a hundred years or a thousand years, that physical effect that was used to encode the bits in the storage media hasn’t changed or faded much so the bits are still there. That’s the relatively easy problem. The much harder problem is preserving your ability to read that medium or that device and interface it to a computer that still supports that interface, an operating system that can still talk to that device with a piece of software that can still interpret whatever format that you wrote your data in.
There’s good work around things like archive PDF formats, where if you’re just storing text and image data, there are some solutions for this 50 to 100 year time frame. But if I need to store a 3D CAD diagram for an airplane design, there’s no guarantee that that format will be supported in the future, or that the company that created that software will still be around. So this preservation of digital information, especially anything beyond text data, is a really hard problem.
I would work on that neglected part of the problem. How do you preserve your ability to actually interpret that information at some future point, ideally forever? How can you somehow preserve the ability to interpret those bits?
Interviewer
I’d love for you to tell me a little bit about the current state of tape. I think it’s a really interesting medium because it has proven reliability, but in the wider world, I find it’s almost a secret or a revelation to a lot of people that it’s still being used the way it is.
Mark Lantz
The technology as a digital data storage medium has a long history. A lot of people that I meet have an image of tape that’s fixed in time. The last time they encountered it was video cassettes or audio cassettes, or depending on their age, loading reels onto an old reel-to-reel drive in a data center as their first job. But they don’t realize that the technology has really continued to evolve in parallel with computer technology. Most credit card transactions are processed by IBM Z machines and those machines are backed up by virtual tape, which has physical tape as a backend. So all your credit card transactions probably end up on a piece of tape somewhere. You’re just not aware of it.
The evolution is continuing today. Our latest enterprise class tape drive has a native cartridge capacity of 50 terabytes, compressed capacity of 150 terabytes, and a native data rate of 400 megabytes per second. Our research primarily focuses on scaling those two things. Of continuing to increase the capacity and increase the data rate of our tape drives. That helps to scale the cost in terms of dollars per terabyte. Today, I think tape is by far the lowest cost way of storing large volumes of data.Rebecca Frank had a more provocative take on green technology: should we just be throwing bags of hard drives into landfills?
Part of that comes from the very low power consumption. It’s a very green technology,Rebecca Frank had a more provocative take on green technology: should we just be throwing bags of harddrives into landfills? 90 to 96 percent less CO2E than something like storing data on hard disks. But of course, it’s not appropriate for all forms of data. The tape cartridges today have a length that’s more than a kilometer long. Depending on the drive technology you use, we’re seeking or fast forwarding at 8 to 12 meters per second. So on average, it takes maybe 30, 40 seconds to get to your data. Once we get there, we can serve it up really fast, 400 megabytes per second. But if you’re only looking for one megabyte, that’s not an efficient use of tape. If you can organize your archive so that you’re storing and retrieving much larger objects, then tape technology is really perfect for that kind of application.
Interviewer
One topic in your published academic work that comes up a lot is reliability. Reliability can mean a lot of different things depending on the humans involved. When designing hardware for the long term, what are some of the techniques and frameworks you use to define reliability?
Mark Lantz
That’s a great question. We tend to look at reliability through two lenses. One is not losing data. The other is making sure that the data is always available. The always available is a much harder problem. When you’re looking at that availability lens, then you need to consider the network and power failures in your data center or even in a local rack. So there’s more failure modes that can make your data temporarily unavailable, but there’s also solutions. The most relied on one at the moment is what I would call RAIL, which is like RAID technology, a redundant array of independent disks, but applied to tape. It’s a redundant array of independent libraries, where you take your file, break it up into chunks, add some parity, and you store those pieces on different tape libraries.
Depending on the kind of erasure coding scheme you use, if one or more of those libraries is unavailable, you can still reassemble the data from the pieces that are accessible. Then if you put those libraries in different data centers in different geographies, you can make extremely reliable and available systems. So we try to analyze reliability across these different dimensions. At the lowest level, it’s really the ability of the drive to read back and correctly return the raw bits that were written to it. In tape systems today, we write 32 channels in parallel, and we write quite large objects to tape. That means we can use very robust error correction codes. Reliability of the raw tape system is very, very good. Failure mechanisms tend to occur somewhere else in the chain. If the robotic system accidentally drops the cartridge, and it’s sitting in the bottom of the library. You didn’t lose any data, but you just can’t find it because you don’t know that the cartridge is down there. Or the barcode reader fails to read the barcode on a cartridge. So again, you don’t know that the tape is there. You think you’ve lost the data, even though it’s still there. It’s quite challenging to kind of get statistics on how often those kinds of things might happen. But those are very important for this availability lens, even if they are much less important for the real data loss or probability of data loss.
This is probably the most exciting time to be working on tape, even though it has this 75 year history of storage technology.
Interviewer
What are you excited about in the field right now?
Mark Lantz
The tape industry has this very long history of ups and downs. When I first started working on tape technology, it was the standard technology for backup and restore. But hard disk drive based backup with deduplication technology made HTD much more efficient. It’s fast compared to tape for recovering a small number of objects. That technology basically started to eat away at the tape backup market. So the industry was kind of in decline for, for quite a long time. But then, with the continued kind of exponential growth in data, which is an overused term, but it’s reality. Data is really growing at 40% compound annual growth rates, doubling, you know, every couple of years. That has driven a need for cost effective archive solutions, especially because the scaling of HTD technology has slowed down so much in the last few years. A lot of our customers are looking for technology to cost effectively archive data. And that has created a renaissance in interest in tape technology. So for me, this is probably the most exciting time to be working on tape, even though it has this 75 year history of storage technology. It’s crazy to me that it’s still actively used, let alone that I’m doing research on this topic. What is even more fascinating and what I am really excited about is how much potential tape technology still has to continue scaling its capacity and data rate. Eventually tape technology will face similar challenges to what the HDD industry is currently struggling with, at which point tape scaling will also have to slow down, but for the next decade or two we have a path to continue scaling tape at a similar rate to past scaling. It won’t be easy. There are lots of technical challenges to overcome and a need for lots of new innovation and inventions to address these challenges, which is what makes working on tape so interesting and exciting.