Ana içeriğe atla

Data Mesh Principles and Logical Architecture

 Data Mesh Principles and Logical Architecture The great divide of data What do we really mean by data? The answer depends on whom you ask. Today’s landscape is divided into  operational data  and  analytical data . Operational data sits in databases behind business capabilities served with microservices, has a transactional nature, keeps the current state and serves the needs of the applications running the business. Analytical data is a temporal and aggregated view of the facts of the business over time, often modeled to provide retrospective or future-perspective insights; it trains the ML models or feeds the analytical reports. The current state of technology, architecture and organization design is reflective of the divergence of these two data planes - two levels of existence, integrated yet separate. This divergence has led to a fragile architecture. Continuously failing ETL (Extract, Transform, Load) jobs and ever growing complexity of labyrinth of data pipel...

AlignmentMap

 Alignment maps are organizational information radiators that help visualize the alignment of ongoing work with business outcomes. The work may be regular functionality addition or technical work such as re-architecting or repaying technical debt or improving the build and deployment pipeline. Team members use alignment maps to understand what business outcomes their day-to-day work is meant to improve. Business and IT sponsors use them to understand how ongoing work relates to the business outcomes they care about.

Here’s an example scenario (inspired by real life) that illustrates how these maps may be useful. A team of developers had inefficiently implemented a catalog search function as N+1 calls. The first call to the catalog index returned a set of SKU IDs. For each ID returned, a query was then made to retrieve product detail. The implementation came to the attention of an architect when it failed performance tests. He advised the team to get rid of the N+1 implementation.

“Search-in-one” was the mantra he offered the team as a way to remember their objective. Given the organizational boundary between architects and developers and the low frequency of communication between them, the mantra was taken literally. The team moved heaven and earth to implement a combined index query and detail query in a single call. They lost sight of the real objective of improving search performance and slogged away in an attempt to achieve acceptable performance in exactly one call. Funding ran out in a few months and after some heated discussions, the project was cancelled and the team disbanded.

The above example may seem absurd but sadly, enterprise IT is no stranger to architecture and business projects that are cancelled after a while because they lost sight of why they were funded in the first place. In the terminology of organizational design, these are problems of alignment.

Visualizing Alignment

Broadly, IT strategy has to align with business strategy and IT outcomes with desired business outcomes. A business outcome may be supported (in part) by one or more IT outcomes. Each IT outcome may be realized by one or more initiatives (program of work—architectural or business). At this point, it may also be useful to identify an owner for each initiative who then sponsors work (action items) across multiple teams as part of executing the initiative. Depending on the initiative the owner may be a product owner, architect, tech lead or manager. Here's an alignment map for the “search-in-one” case. Had it been in public display in the team’s work area, it might have prompted someone to take a step back and ask what their work was really meant to achieve.

Global Map

A global alignment map for the IT (appdev+ops) organization may look more like this (although real maps tend to be much larger).

As with all information radiators, such a map is a snapshot in time and needs to be updated regularly (say once a month). Each team displays a big printout of the global map in its work area.

Big organizations are likely to realize value early in this exercise by collaborating to come up with a version 1.0 of such a map that everyone agrees to. The discussions around who owns what initiatives and what outcomes an initiative contributes to leads to a fair bit of organizational clarity of what everyone is up to. Usually, the absence of well-articulated and commonly understood business and IT strategies come in the way of converging on a set of business and IT outcomes. Well-facilitated workshops with deep and wide participation across the relevant parts of the organization can help address this.

Tracing alignment paths

Once a global alignment map is in place, it allows us to trace alignment from either end. IT and business sponsors can trace what action items are in play under a given initiative. Development team members can trace through the map to understand the real purpose of items they are working on. In addition to in-progress items, we could also include action items that are planned, done or blocked.

As illustrated in the map above, each team highlights their section of the map on their copy of the global map.

Qualitative benefits validation

Once a month (or quarter), IT and business people get together to validate if all the IT activity has made any difference to business outcomes. Business people come to the meeting with red-amber-green (RAG) statues for business outcomes and IT people may come with RAG statues for their side of the map. Both parties need to be able to back up their RAG assessments with data and/or real stories from the trenches (narrative evidence).

These maps can be combined

Yorumlar

Bu blogdaki popüler yayınlar

Data Mesh Principles and Logical Architecture

 Data Mesh Principles and Logical Architecture The great divide of data What do we really mean by data? The answer depends on whom you ask. Today’s landscape is divided into  operational data  and  analytical data . Operational data sits in databases behind business capabilities served with microservices, has a transactional nature, keeps the current state and serves the needs of the applications running the business. Analytical data is a temporal and aggregated view of the facts of the business over time, often modeled to provide retrospective or future-perspective insights; it trains the ML models or feeds the analytical reports. The current state of technology, architecture and organization design is reflective of the divergence of these two data planes - two levels of existence, integrated yet separate. This divergence has led to a fragile architecture. Continuously failing ETL (Extract, Transform, Load) jobs and ever growing complexity of labyrinth of data pipel...

Pair Programming Misconceptions

Pair Programming Misconceptions A bunch of common misconceptions about Pair Programming. You have to do pair programming if you're doing an agile process. This is utterly false. 'Agile' is a very broad term defined only in terms of values and principles, most notably in the Manifesto for Agile Software Development. The manifesto doesn't mention pair programming and most agile methods don't make it part of their approach. Since pair programming is a practice of XP it's had a lot of influence in the agile community. As a result it's often mentioned as an agile practice - meaning a practice that's commonly used by people on agile projects. But that's an observation not a prescription. Extreme Programming forces you to do Pair-Programming This is much more nuanced issue. Pair-Programming is one of the practices of XP and has been since its inception. The nuance here is whether XP practices are mandatory for a team that claims to be doing XP. This is actu...

ActivityOriented

  ActivityOriented Any significant software development effort requires several different activities to occur: analysis, user experience design, development, testing, etc. Activity-oriented teams organize around these activities, so that you have dedicated teams for user-experience design, development, testing etc. Activity-orientation promises many benefits, but software development is usually better done with   OutcomeOriented   teams. Traditionally, big businesses with large IT departments (Enterprise IT) have tended to execute IT development projects with a bunch of activity-oriented teams drawn from a matrix IT organization (functional organization). The solid-lined arms of the matrix (headed by a VP of development, testing and so on) are usually along activity boundaries and they loan out “resources” to dotted-lined project or program organizations. Common justifications for doing so include: It helps standardization of conventions and techniques in development if a...