EA for Big Data and Analytics

Big Data and Analytics have moved from a specialist function to a core enterprise capability. Every decision — operational, tactical, strategic — is increasingly data-driven, and increasingly AI-augmented. The architecture that supports this has matured from bespoke pipelines into a reference architecture: a governed set of patterns for ingesting, storing, processing, serving, and governing data at scale.

The EA's role is to define and govern this analytics reference architecture so that analytics is trustworthy (governed), scalable (consistent platform), and accessible (self-service within guardrails). Without it, each team builds its own pipeline, producing analytics that cannot be compared, data that cannot be trusted, and costs that compound.

This page covers the analytics reference architecture, the lakehouse pattern, self-service BI within governance, the data product pattern, and the governance that makes big data trustworthy — and how AI now sits inside this architecture rather than beside it.

A modern analytics reference architecture is layered: ingestion (batch and stream sources), storage (the lakehouse), processing (batch and streaming compute), serving (warehouses, BI, ML feature stores, AI retrieval), and governance (catalog, lineage, quality, policy) as a cross-cutting layer. Each layer has standard patterns and approved tools.

The layers are loosely coupled so each can evolve independently — new sources can be added without re-platforming, new serving patterns (a vector store for AI) can be added without rebuilding storage. This loose coupling is what makes the reference architecture durable as both data volumes and analytic needs grow.

EA governs this by defining the standard patterns and the approved tools per layer, so teams assemble solutions from a known catalog rather than inventing. The reference architecture turns 'big data' from a series of bespoke projects into a platform with consistent, governed patterns — which is the only way it scales across a large enterprise.

The lakehouse has become the default analytics platform pattern because it resolves the long-standing warehouse-vs-lake tension. A warehouse is governed and fast for SQL but expensive and rigid; a lake is cheap and flexible but weak on governance and performance. The lakehouse — open table formats (Iceberg, Delta, Hudi) on cheap storage with ACID, schema enforcement, and warehouse-grade query — gives the strengths of both.

For EA, the lakehouse is significant because it removes a forced choice. Historically, the enterprise had to decide warehouse (governed, expensive) or lake (cheap, ungoverned) for each workload; the lakehouse lets the platform serve BI, data science, and now AI retrieval from one governed foundation. This is simplifying, not complicating — fewer platforms, governed consistently.

The architect's decision is which lakehouse engine(s) and table format(s) to standardise on, balancing portability, performance, and ecosystem. The trend is toward open table formats to avoid lock-in, with one or two query engines standardised — letting compute evolve while storage stays portable.

The promise of self-service BI — analytics for everyone — has historically collided with the reality of ungoverned spreadsheets and contradictory reports. The modern resolution is self-service within governance: analysts work with governed, certified datasets exposed through a semantic layer, producing reports that are trustworthy because they rest on governed data.

The semantic layer is the key artifact: a business-friendly abstraction over the physical data that defines metrics consistently (one definition of 'revenue', one of 'active customer') so that every report uses the same logic. Without a semantic layer, self-service BI produces a proliferation of metric definitions; with it, self-service produces consistent, comparable analytics.

EA governs this by owning the semantic layer and the certified-dataset catalogue. Analysts are free to build reports; they are not free to redefine core metrics. This balance — freedom within a governed frame — is what makes self-service scale without producing analytical anarchy.

The data product pattern treats shared datasets as products — with owners, contracts, quality SLAs, and discoverability — rather than as one-off extracts. A data product has a clear producer, a defined contract (schema, freshness, quality), and consumers who rely on it. This makes data reuse compound: a well-built customer data product serves many consumers rather than being rebuilt per team.

The data mesh extends this to an organisational pattern: domains own and publish their data as products, with a federated governance layer ensuring cross-domain consistency. Mesh is not a technology; it is an operating model for data ownership that complements the lakehouse platform.

EA's role with data products is to define what makes a dataset a product (contract, owner, quality, discoverability) and to curate the product catalogue. An enterprise with a healthy data product catalogue compounds reuse; one without it rebuilds the same datasets endlessly and never achieves a single version of the truth.

Big data without governance is a swamp — large, dark, and dangerous. Governance is what makes big data usable: a catalog that makes data discoverable, lineage that makes it traceable, quality rules that make it reliable, classification that makes it safe, and policy that makes it compliant. These are architectural concerns, bolted onto the platform, not manual processes.

Lineage has become especially important as regulation (e.g., BCBS 239 in banking) and AI both demand traceability — knowing what a model was trained on, or where a regulatory metric was derived from. Automated lineage capture, integrated into the platform, is no longer optional in regulated or AI-heavy enterprises.

Quality is the most undervalued governance discipline. Data that is discoverable and lineaged but wrong is worse than no data, because it produces confident wrong decisions. EA governs quality by defining quality rules per data domain, measuring them continuously, and treating quality as a product attribute of data products — not a one-off check.

AI has moved from a separate stack to a consumer and producer within the analytics architecture. Models are trained on governed data; their features come from feature stores that are part of the platform; their retrieval (RAG) draws on the same catalogued, governed knowledge. AI is now a serving pattern on top of the lakehouse, not a parallel world.

This integration is what makes AI trustworthy. A model trained on uncatalogued, unlineaged data is an unaccountable liability; one trained on governed, lineaged data is auditable. EA's role is to insist that AI data lives inside the governed analytics architecture — vector indexes catalogued, feature stores governed, training data lineaged — rather than in a shadow AI stack.

The same governance that makes BI trustworthy makes AI trustworthy: catalog, lineage, quality, classification, and policy, extended to vector data and features. The enterprise that governs its analytics well is most of the way to governing its AI well — another reason analytics architecture is a strategic EA concern.