Extract Transform and Load (ETL) was considered to be the most effective way to load information into a data warehouse. Early data warehouses were not viewed as being capable of handling the extensive processing required to perform the complex transformations involved in the warehouse load process. Instead, third-party tools like IBM's WebSphere DataStage and Informaticawere leveraged to orchestrate data movement between source systems and the data warehouse. With the advancement in both hardware and data warehouse software technology, warehouse designers can now consider Extract Load and Transform (ELT) as a viable option.
Loading a Data Warehouse
The process of loading a data warehouse can be extremely intensive from a system resource perspective. In companies with data sets greater than five terabytes, load time can take as much as eight hours depending on the complexity of the transformation rules. Most data warehousing teams schedule load jobs to start after working hours so as not to impact performance when a user query is being executed. However, as data volumes and warehouse subject areas increase, load times can increase even further and spill over into regular working hours.