New generation flexible data architectures have introduced new key concepts such as Data Fabric, Data Mesh or Data Lakehouse.
As the amount of data generated by companies increases, organisations require next-generation data architectures to provide them with the flexibility that the new business ecosystem demands. We discuss flexible data architectures and their key concepts.
According to Forbes, the total global volume of data is expected to grow from 64.2 zettabytes to 181 zettabytes between 2020 and 2025.
To provide some context, a zettabyte equals one trillion gigabytes, which means that in the next two years we will be storing the complete works of William Shakespeare 178 trillion times, or the 16,000 feature films of the Internet Archive 125 million times.
As the amount of data continues to grow, it is essential to develop new mindsets and approaches to ensure that data is being leveraged effectively and securely.
To make the most of data capture, data storage and data analysis, it is essential that companies adopt a holistic data management strategy. Moreover, for this strategy to become a sustainable source of long-term business value, it is crucial to incorporate an essential element: flexibility.
Flexibility allows companies to adapt to changing market and customer needs quickly and effectively. It also allows them to adopt new technologies and data management methodologies as they emerge, which is essential to keep pace in an increasingly competitive business environment.
For this reason, more and more companies are turning to flexible data architectures.
Don't have a defined data strategy yet? Download our e-book where you will find the keys to building a data strategy:
The importance of flexibility in a data architecture
Having an enterprise data strategy in place is critical to ensure the resilience of a business. To maintain this resilience and scale enterprise data operations to meet the competitive demands of the future, it is crucial to adopt an open and agile approach that allows for greater flexibility.
Today's companies must be able to leverage their data in innovative and rapid ways in order to adapt and change course as needed to stay competitive. Regardless of an institution's specific constraints and needs, a flexible data strategy can help extract insights from any type of data, whether structured or unstructured, in motion or at rest.
As data sources expand and the demand for data-driven insights increases, a strategy focused solely on current business objectives will soon outlive its usefulness and prevent progress. In this sense, introducing flexibility as a key element of the data strategy is imperative to ensure that an organisation will be prepared to meet new needs as they arise.
Ultimately, flexibility is essential to maximise the value of data and maintain long-term competitiveness. Adopting a flexible data strategy enables companies to adapt quickly to change and maximise the value of their data.
A flexible data architecture: Key concepts
The adoption of flexible data architectures by enterprises has introduced new concepts that are important to understand.
1. Data Fabric
Most organisations are faced with a complex and sometimes chaotic collection of data storage and data processing platforms. With acquisitions, new needs and organic growth, a typical enterprise may have multiple databases, data warehouses, analytic platforms with different user communities and data transformation routines dictated by short-term needs rather than a long-term strategy.
Data Fabric is a data architecture that unifies all these diverse data sources and applications in a secure and automated way, without changing where or how the data is stored. In other words, it provides access to data without the need to migrate it. This connected architecture facilitates, accelerates and secures the deployment of data-driven apps and automations, and makes information available to users on a self-service basis.
A Data Fabric architecture allows end users to see data in a unified way, although the data is still distributed across multiple on-premises and cloud resources. This architecture makes data management more efficient and effective, leading to better business decision making.
2. Data mesh
Data management is often complicated because of a long-established tradition of treating data and its architecture as short-term projects. Even if a particular project might be successful in the long term, the tools and techniques used to implement the particular solution are likely to have been established by a small team focused on specific objectives. Over time, this approach can complicate the design of the data architecture, create cumbersome organisation-wide rules for accessing and influencing data, and make data ownership and management difficult.
Data Mesh is an approach designed to solve this problem by focusing on structure rather than technology. In a Data Mesh, data is thought as a product rather than a project. A team of internal experts is responsible for one or more data domains and establishes rules for the workflow and delivery of data to end users. For example, the marketing department is in charge of marketing data and the finance department gathers financial data.
In contrast to the centralisation provided by a 'Data Fabric' architecture, in a 'Data Mesh' architecture, those in charge of each data domain act in a decentralised manner, but according to standardised interoperability and data governance policies.
A 'data mesh' is not a specific technology or something that can be purchased, but an approach that encompasses both the people and the processes that revolve around data. It is, more than anything else, a mindset that involves a change in the way you think about data and its management. While technology is important, it plays only a supporting role in the implementation of this approach
3. Data Lakehouse
The term "data lakehouse" refers to the recent evolution of a data warehouse, which combines the capabilities of a data warehouse and a data lake. Both concepts emerged to address the limitations of traditional databases in terms of storage capacity, scalability and flexibility.
In a data lakehouse, data is stored in raw, unstructured form, just as in a data lake. However, unlike in a data lake, the data is also transformed and structured in a data model optimised for analytical queries, similar to a data warehouse. In this way, data can be analysed efficiently and in real time.
In addition, the data lakehouse approach also provides for real-time data integration, allowing organisations to access more up-to-date data for decision-making. The use of cloud-based architectures also facilitates scalability and flexibility of the data infrastructure.
Data lakehouses emphasise access based not only on user roles, but also on data classification attributes, easy-to-scan and modify protocols around data governance and retention, and the ability to distribute both storage and computational analytics resources across a hybrid of on-premises and cloud systems.
In short, a data lakehouse combines the control, accuracy, completeness and strict data governance of a data warehouse with the freedom, flexibility and granularity of a data lake.
Next generation data architectures
Adopting next generation architectures implies an evolution, not a complete abandonment of existing data systems.
There is no single roadmap for adopting these approaches and the first steps will depend on business needs and technical legacies. The maturity of the organisation in terms of data and analytics is also an important factor in choosing the right modern architecture. For example, a company that handles large volumes of unstructured data, but struggles to extract value, may opt for a data lakehouse as a first step. To implement a data mesh, independent cross-functional teams with data engineers, data product owners and data scientists are required.
If you don't already have a consolidated data strategy or architecture, download our e-book to get the keys you need to do so.
