ETL

Common Issues

Some issues with ETL processes include:

It's often expensive to keep them up to date with new versions (or even products) of source or target systems
Data is not actual. An IDC study (3rd Platform Information Management Requirements Survey, IDC, 2016) reports that 80% of data that is delivered by ETL processes is between 2 and 7 days old
Security: because the data is copied, additional effort needs to be taken to ensure that only the authorized people can access the data.
Compliance: because of the security concerns, compliance regulations such as the GDPR needs to be taken into account
Storage: it also needs to be taken into account that the copied data requires more storage to be present.

These issues are somewhat overlapping with the problem of redundant data in data warehouses.

Data virtualization

Because of these issues, it is proposed to use data virtualization rather than traditional ETL processes.

With data virtualization, data, on its different sources and with its different formats, is not replicated, rather it is accessed from a single point, usually referred to as data hub, using a standardized method.

The SQL Server feature that promises data virtualization is PolyBase.

ETL

Common Issues

Data virtualization

See also