Storage strategies have evolved to accommodate a world with massive amounts of unstructured data generated all of the time. The growing demand for storage is driven by many industries: from scientific observations, to film and video storage, to data gathered for AI/ML training. And higher fidelity, higher resolution data means the need for storage will continue to increase.
“Everything is bigger, more immersive, more visual, and higher resolution,” says Brian Pawlowski, former Chief Development Officer of the IT storage company Quantum, which specializes in large, high-performance data stores, on the IT Visionaries podcast.
“It’s data growth, the massive explosion, the tidal wave of data coming at you,” Pawlowski says. “So there’s the hardware, the ability to store it that’s cost-effective, and the performance required to move the data seamlessly so that the data is where you need it, when you need it.”
From Structured to Unstructured
This flood of data is a new development. Decades ago, business data was mostly structured—think of things like order processing or payroll records. But today, there’s an explosion of unstructured data. Unstructured data is simply all the data that isn’t formatted as a database. The rise of AI has only accelerated this trend, making unstructured data more useful and more in demand.
Just to take one use case—consider the countless hours of security video footage recorded on a daily basis. Those video files are unstructured data, and they need to be stored somewhere.
Handling this amount of storage is one of the IT industry’s great challenges. IT organizations need to determine how to store their data, and in what formats, so that it is both cost-effective and accessible.
Organizations buy storage solutions on-premises and in the Cloud, and use software to control the movement of data back and forth for backups or to keep data where it’s easiest for collaborators to work with it.
“All storage is software,” Pawlowski says. “The hardware is the necessary evil. That is the physical place where you store the data, but… it’s our software that’s basically making sense of it.”
With software controlling the movement of data, whether to store data in the cloud or on-prem becomes a decision based on performance and collaboration needs. Most likely, this is a hybrid system of flash discs and tape, which remains a cheap and reliable way to store data for long periods. SSD flash storage is also critical for reliable application performance.
The Data Factory
Customers typically funnel their data through a transformation pipeline: Data comes in, then they process it. One way to think about this is like a data factory. The “data factory” model views data as moving through ingest, production and processing, and long-term storage stages.
The data funnel is where AI is making a huge impact, as data scientists apply AI and ML applications to look for patterns.
This is also where networking, like the low latency connections provided by Zayo, works hand in hand with storage. Data needs to be connected, and users need to be able to move it around.
Bandwidth, reliability and uptime, integration and interoperability, and network reach are all considerations when making your storage available over a network. Often the path to high-performance storage includes a fiber connection between on-premises and cloud storage.
Whatever the physical media format and location, the bottom line is data is mobile and reusable, going back and forth over the network. This helps teams optimize for reliability, security, and cost-effectiveness.
