The Embedded Data Architect

The Challenge of Data

It comes as no surprise that companies are sitting on a treasure trove of information, only a certain percentage of which is utilized to its potential. We face challenges with organizational awareness of the data that’s available to us, including data we actually have right under our noses. It leads to a few questions:

• How much of the data that is available TODAY to Shipping and Receiving – about the customer, product, distribution channels, heck (the weather!) – does Marketing really know about?
• What could Product Development do with those little nuggets that are picked up by Customer Service and Support?
• Is our data architecture as effective as it can be at cross-pollinating across organizational boundaries?
• Is it sufficiently institutionalized or woven into the relationship between IT and the business, or even into the partnerships between the various development groups within IT itself?

It’s not difficult to imagine the untapped potential buried inside any given organization’s existing data landscape. The challenge continues to lie in realizing that potential.

This is not a new problem. We’ve thrown technology, methodology, infrastructure, processes and procedures, let alone vast sums of time, energy, and money to come up with better ways to discover, socialize, conform, publish, promote, utilize, and repeat the cycle with valuable feedback for additional discovery, and ultimately monetize these hidden assets.

The Enterprise Landscape

The typical IT division has resources dedicated to building and maintaining operational transaction systems that handle the organization’s day-to-day work and coordinate its business processes. Quoting, Order Entry, Shipping, Receivables, Purchasing, Receiving, Payables, Ticketing, etc.

Separate teams also manage one or more analytical systems (data warehouses, marts, hubs, vaults, lakes, the occasional swamp, etc.) that bring all that information together for the benefit of analysts and decision makers who mine it to gain the valuable insights.

In my experience, an organizational, cultural, and technological chasm exists between the operational and analytical components of the typical enterprise data landscape:

• Object and 3NF data models vs dimensional models
• No SQL vs nothing-but-SQL
• Real-time vs batch
• Agile vs Waterfall
• Departmental vs enterprise
• Millennials vs baby boomers

Ok, maybe I’m stretching a bit with that last one, and certainly not all these distinctions apply within the same company, but you get the idea…a kind of wall exists.

Building Connection

What if we could break that wall down – or at least build a door through it – to more effectively wire the operational and analytical systems together organizationally in such a way that institutionalizes the kind of cross-functional, inter-departmental, business process-centric information awareness that we continue to strive for?

What if we had a small team of data architects reporting to a central or quasi-central enterprise analytical team but each with a dotted line to the manager(s) of one or more operational systems in which these data architects would each be embedded?

As I see it, this would open door to all kinds of value creation:

• Operational systems would get expert data modeling and the insights that come from that.
• The embedded architect would provide a valuable new source of ideas for data-related features from feedback from across the enterprise, up and downstream of the operational system’s place in a business process. Remember, the embedded architect has one foot in the departmental / operational world, and the other in the central / cross-functional enterprise analytical world.
• The analytical team would benefit from the deep domain knowledge that the embedded architect gains from her (or his) close collaboration with the operational systems she is assigned to, and the discovery opportunities that would inevitably arise over time.
• Periodic meetings among the data architects would provide a forum for model walk-throughs and other forms of knowledge sharing and brainstorming that could be taken back to the various operational systems and surfaced to the business.
• Similar, perhaps higher-level walk throughs could be given to the business users to promote awareness and discussion, providing an opportunity for the business to give feedback. This alone would go a long way towards promoting the kind of partnership we value between business and IT.

The Embedded Data Architect and Key Goals

Let’s go back and review the key goals we’ve been working towards for our data assets and how the embedded architect would help meet them.

Discover

The embedded architect would have specialized knowledge of the data domain of his (or her) assigned operational systems. The build-out of the logical and/or physical data models would provide a deep knowledge of the available data. Then there’s the top-down expertise the architect brings from his work with other architects and the analytical business partners. That plus the close collaboration with the developers and especially the business stakeholders responsible for the operational system would provide fertile ground for many “aha” moments.

Socialize

Discussions among the embedded architects in their periodic meetings and day-to-day collaboration would ferret out new opportunities for data insights that may not otherwise present themselves. An architect assigned to the shipping and receiving system may come up with an idea for a new dimensional attribute, or maybe even a whole new dimension or fact/measurement. He could pitch it to the rest of the architect team, and that idea might just resonate.

Conform

What better way to carry out the heavy lifting of dimension conformance than with a close-knit team of data architects with ties to the business, especially when properly supported with an effective governance and stewardship program.

Publish/Promote/Utilize

The same architects who handle the modeling tasks for the operational systems would naturally work on the data models of the analytical systems – if not directly, then certainly as valuable subject matter experts contributing requirements. The analytical IT team, with its relationship to its own user community, aided by data stewards and good data governance practices, would provide the vital distribution channels to the analysts and decision makers.

Feedback

The organizational opportunities for feedback abound – between embedded architects themselves, between the embedded architect and operational team, and especially between the business users and the architect team. Given a taste of what is possible, the business will naturally think of more opportunities that can be taken by the embedded architect back to the operational system for additional capture of new source data, restructuring, normalization and/or cleanup of existing data, and more.

Monetize

This, of course, is what it’s all about. Over time, richness of the shared data assets uncovered by the process that our team of embedded architects enables would provide insights that would translate into measurable opportunities for revenue enhancement, cost containment, and ultimately, profit.

Final Thoughts

We spend so much effort focusing on elegant technological solutions that it is easy to lose sight of opportunities that may be ripe for the picking if we approached them from a different direction. Hopefully, by presenting some of the ways the organizational idea of the “embedded architect” can help better leverage your existing data assets, we’ve not only provided a workable path towards improvement, but also stimulated further thought about how other non-conventional approaches can be applied to achieve feasible yet noteworthy results.