tl;dr
Data virtualization is a technology that allows users to access data from disparate sources as if it were a single unified source, without the need for physical data movement or replication.
Background
The process of data centralization (migrating data into a single repository) was a big hot button topic over the past decade, and many businesses have spent a lot of time and energy trying to centralize their data into one region, warehouse, data lake, etc. The belief was that by centralizing their data, it would become more usable, easier to govern, and they could keep storage costs in check. The reality is that it created a lot of overhead on data engineers, costs ballooned because they needed to maintain two copies of data while they performed the migration (doubling storage), and governance got harder, not easier, since data was now mobile.
This trend is reversing. Centralization carries with it a lot of problems, as it often requires a complete overhaul of a company’s data ecosystem, much of which can’t necessarily be moved to a public cloud. The “lift and shift”, as it’s called, is essentially impossible for large enterprises, and isn’t always needed. Data that needs to be moved should be moved; wholesale centralization is rarely necessary. That said, most companies have some kind of a data migration strategy in place. Increasingly these are focused on moving some datasets from a legacy warehouse into a new cloud-based warehouse, or similar.
Companies are realizing that there are benefits to increasing, not decreasing, the mobility of their data. A one-time shift from a legacy warehouse to the cloud is less important than being offered the ability to dynamically move data from one warehouse to another, as needed. We support this using the platform ETL.
Although centralizing data assets into a single warehouse is falling out of vogue, centralizing data access is taking its place as a desired outcome. Data catalogs have historically tried to fill this gap by providing a solution that can be used to index and maintain all of a company’s data.
What is Data Virtualization?
“Data virtualization” is a way to allow people to access and query data inside a warehouse without accessing the warehouse directly. It is a form of zero-copy integration that lets a business index and access data without moving it.
Virtualization provides a “lens” to a dataset in a warehouse, so you can see it and query it, but it does not leave the warehouse. This is different from an ETL, for example, which extracts the data from where it resides and then loads it into a warehouse, from where it can be queried and used.
The ThinkData Works platform provides an ETL that is exceptionally good at centralizing data assets for a business (think flat files on someone’s computer or transferring data from a legacy warehouse to a public cloud instance). Many businesses need this kind of data migration. For other organizations it is better to create a window to data where it lives and catalog it there.
We have built data virtualization capabilities into the platform in order to help businesses create a centralized view of all of their data assets without needing to necessarily migrate them “into” the catalog.