Monday, 3rd August 2020

Top trends for modernising the data warehouse

Data automation technology holds the key to an increasingly data-centric market.

WhereScape has published the results of a global survey by the Business Application Research Center (BARC), commissioned by WhereScape. The survey revealed that nearly half of respondents found time-consuming development processes at the top of the list of primary drivers for data warehouse modernization as companies look to keep pace with increasing complexity in the data landscape. Lack of agility in infrastructure and internal processes, as well as a lack of business and technical know-how, however, are limiting business’ ability to act.

“This research gives statistical evidence and puts the BARC stamp on concepts that WhereScape has been evangelizing for some time. In order for companies to succeed today, we must automate repetitive manual tasks that force too many developers to solve puzzles and fight fires arising from outdated tools and methodologies. Only then can companies focus forward on innovation and agile collaboration to supply reliable data to the business at the right time,” said Rob Mellor, vice president and general manager EMEA, WhereScape.

Drivers for modernization

44% of survey participants found that time-consuming development processes were the most daunting tasks in the data warehouse landscape and the primary reason to modernize. This response was followed by restricted support of self-service BI (35%) and lack of ability to provide required data (30%). Of note, the inability to meet compliance and regulatory requirements was only seen as a major challenge by 14% of participants. Even less (11%) found insufficient technology or lack of functionality to be a limiting factor, suggesting technology has reached a sound level of maturity.


Methods of modernization

Data warehouse and ETL automation is the most important approach to modernizing the data warehouse (60%). Correspondingly, data integration and data warehouse automation are the technologies most frequently utilized. By trend, however, enterprises with more mature data warehouse landscapes are more geared towards moving into the cloud, and report less infrastructure and process challenges - clearly profiting from investments in a flexible data and process architecture and agile data services.


Challenges faced in modernizing the data warehouse

Participants cited lack of agility in infrastructure and internal processes (39%) as well as lack of business and technical know-how (38%), as the key factors limiting the ability to move forward with data warehouse modernization. Data and analytics literacy is not only an immensely important success factor for building an effective, cost-efficient and sustainable data warehouse environment. These skills are invaluable for the transformation to the digital enterprise, yet they are rare. Enterprises with less advanced data warehouse environments struggle more than those who are more advanced. However, it is vital to realize that this area requires investment in existing employees in any case.

Leaders, followers and laggards

In order to characterize the status quo of survey participants’ current data warehouse and analytics landscape, three groups of respondents were created: leaders, followers and laggards (see definitions below). Here were the standout statistics within the groups surveyed:

- The data warehouse/ETL automation approach to modernization was a popular approach with nearly half or more of respondents from all groups (leaders, followers, and laggards).

- For leaders, real-time processing of data is the most important (41%). This is due to real-time processing being one of the most important prerequisites for decision automation and being an important foundation for agile, interoperable data services.

- For leaders, analytical databases are utilized much more intensively than by followers and laggards (41% compared to 27% overall).

- For leaders, cloud services play an important role in approaches to modernization of the data warehouse with 46% citing cloud services as a promising alternative with regard to resource and cost efficiency of hardware.

- Laggards in particular suffer from a lack of support from executives for data warehouse modernization initiatives (44%).

Concluded Mellor: “In order for IT to advance and deliver timely business insights needed in today’s era of exploding data, an automation-first approach to data warehouse modernization is essential. Leading organizations are recognizing the need to focus skilled development resources away from repetitive coding and onto higher value projects by arming developers with the automated tools necessary to agilely deliver.”

Leading digital automotive marketplace embraces the power of data with Looker.
Qlik has launched two new resources that build on the recent global IDC study sponsored by Qlik, whi...
Innovate UK supported AIR Platform (Automated Intelligent Regulation) will be a ‘generational breakt...
Teradata will be a partner of the Volkswagen Industrial Cloud
New research from Qlik explores how advancements in use of analytics is delivering a data-informed h...
Google Cloud has kicked off, launching new solutions across its smart data analytics and security po...
Partnership focuses on meeting demand for trust, accuracy, and traceability of data to support data...
MariaDB Community Server 10.5 turns into analytics powerhouse with cloud-native storage.