Ana içeriğe atla

Data Mesh Principles and Logical Architecture

 Data Mesh Principles and Logical Architecture The great divide of data What do we really mean by data? The answer depends on whom you ask. Today’s landscape is divided into  operational data  and  analytical data . Operational data sits in databases behind business capabilities served with microservices, has a transactional nature, keeps the current state and serves the needs of the applications running the business. Analytical data is a temporal and aggregated view of the facts of the business over time, often modeled to provide retrospective or future-perspective insights; it trains the ML models or feeds the analytical reports. The current state of technology, architecture and organization design is reflective of the divergence of these two data planes - two levels of existence, integrated yet separate. This divergence has led to a fragile architecture. Continuously failing ETL (Extract, Transform, Load) jobs and ever growing complexity of labyrinth of data pipel...
Yorum Gönder

ActivityOriented

 

ActivityOriented

Any significant software development effort requires several different activities to occur: analysis, user experience design, development, testing, etc. Activity-oriented teams organize around these activities, so that you have dedicated teams for user-experience design, development, testing etc. Activity-orientation promises many benefits, but software development is usually better done with OutcomeOriented teams.

Traditionally, big businesses with large IT departments (Enterprise IT) have tended to execute IT development projects with a bunch of activity-oriented teams drawn from a matrix IT organization (functional organization). The solid-lined arms of the matrix (headed by a VP of development, testing and so on) are usually along activity boundaries and they loan out “resources” to dotted-lined project or program organizations. Common justifications for doing so include:

  • It helps standardization of conventions and techniques in development if all developers report into a single organization (arm of the matrix). Same for testing etc.
  • It helps the cause of mentoring, training and nurturing the competency in general if all developers have long-lived development/engineering managers. Same for testing etc.
  • It helps maximize utilization of talent (and thereby improve cost-efficiency) by staffing projects from pools of supposedly fungible developers, testers etc.

However, activity-oriented teams are prone to optimize for their own activity and not for the bigger picture of delivering useful software. This is a consequence of what they are held responsible for and how they are measured. It is common for a team of only developers to only be measured by their velocity. If they are only tasked with delivering scope, they will not think about whether it is going to solve the problems it was meant to. Even if they do so, they may be discouraged by the product management team - another activity-oriented team that is only responsible determining the spec.

Organizing by activity gets in the way of lowering batch size of work that is handed-off between teams. A separate team of testers won’t accept one story at a time from a team of developers. They’d rather test a release worth of stories or at least all stories in a feature at a time. This lengthens the feedback loop, increases end-to-end cycle time and hurts overall responsiveness.

High speed IT calls for motivated teams. Autonomy is a key motivator for teams. However, activity oriented organization can only be granted so much autonomy because they tend to use it for optimizing their sphere of activity.

A variation of activity-oriented organization are super-specialized teams that may result in the following ways:

  • Tool or skill centric teams: e.g. a WebSphere Portal Server team or a BizTalk team
  • Architectural Layer teams: e.g. a presentation layer team, middleware team, data layer team.

They are problematic because they have a narrow focus and they tend to optimize for team performance rather than the big picture. For sure, some tools may need specialists but that is no reason to isolate them in a separate team. Specialization isn’t the problem; organizing along lines of specialization is.

What works better

Software development is an iterative design process. In order to achieve true iteration and realize the value of fast feedback, its activities need to be performed with a single team (having a common reporting line) as far as possible. Many internet business and independent software vendors (ISVs) already operate in this way.

Yorumlar

Yorum Gönder

Bu blogdaki popüler yayınlar

Out come Oriented

 Out come Oriented  effort, better sales conversion, greater customer satisfaction, i.e business outcomes. Outcome-oriented teams are those that are mandated and equipped to deliver business outcomes, such teams have people with the capability to carry out all necessary activities to realize the outcome.. By contrast,  ActivityOriented  teams are neither equipped nor mandated to do so. They can only perform one of several activities required to realize an outcome. A mandate to deliver a business outcome is very different from a mandate to deliver a certain amount of scope. Scope delivery is easy, relatively speaking. Outcome realization requires real collaboration between those who understand the problem and those who can fashion various levels of solution for it. Initial attempts at solution lead to a better understanding of the problem which leads to further attempts at better solutions. This doesn’t work where the product management organization is separate from t...
Yorum Gönder
Devamı

AlignmentMap

  Alignment maps are organizational information radiators that help visualize the alignment of ongoing work with business outcomes. The work may be regular functionality addition or technical work such as re-architecting or repaying technical debt or improving the build and deployment pipeline. Team members use alignment maps to understand what business outcomes their day-to-day work is meant to improve. Business and IT sponsors use them to understand how ongoing work relates to the business outcomes they care about. Here’s an example scenario (inspired by real life) that illustrates how these maps may be useful. A team of developers had inefficiently implemented a catalog search function as N+1 calls. The first call to the catalog index returned a set of SKU IDs. For each ID returned, a query was then made to retrieve product detail. The implementation came to the attention of an architect when it failed performance tests. He advised the team to get rid of the N+1 implementation. ...
Yorum Gönder
Devamı