Ana içeriğe atla

Data Mesh Principles and Logical Architecture

 Data Mesh Principles and Logical Architecture The great divide of data What do we really mean by data? The answer depends on whom you ask. Today’s landscape is divided into  operational data  and  analytical data . Operational data sits in databases behind business capabilities served with microservices, has a transactional nature, keeps the current state and serves the needs of the applications running the business. Analytical data is a temporal and aggregated view of the facts of the business over time, often modeled to provide retrospective or future-perspective insights; it trains the ML models or feeds the analytical reports. The current state of technology, architecture and organization design is reflective of the divergence of these two data planes - two levels of existence, integrated yet separate. This divergence has led to a fragile architecture. Continuously failing ETL (Extract, Transform, Load) jobs and ever growing complexity of labyrinth of data pipel...
Yorum Gönder

ActivityOriented

 

ActivityOriented

Any significant software development effort requires several different activities to occur: analysis, user experience design, development, testing, etc. Activity-oriented teams organize around these activities, so that you have dedicated teams for user-experience design, development, testing etc. Activity-orientation promises many benefits, but software development is usually better done with OutcomeOriented teams.

Traditionally, big businesses with large IT departments (Enterprise IT) have tended to execute IT development projects with a bunch of activity-oriented teams drawn from a matrix IT organization (functional organization). The solid-lined arms of the matrix (headed by a VP of development, testing and so on) are usually along activity boundaries and they loan out “resources” to dotted-lined project or program organizations. Common justifications for doing so include:

  • It helps standardization of conventions and techniques in development if all developers report into a single organization (arm of the matrix). Same for testing etc.
  • It helps the cause of mentoring, training and nurturing the competency in general if all developers have long-lived development/engineering managers. Same for testing etc.
  • It helps maximize utilization of talent (and thereby improve cost-efficiency) by staffing projects from pools of supposedly fungible developers, testers etc.

However, activity-oriented teams are prone to optimize for their own activity and not for the bigger picture of delivering useful software. This is a consequence of what they are held responsible for and how they are measured. It is common for a team of only developers to only be measured by their velocity. If they are only tasked with delivering scope, they will not think about whether it is going to solve the problems it was meant to. Even if they do so, they may be discouraged by the product management team - another activity-oriented team that is only responsible determining the spec.

Organizing by activity gets in the way of lowering batch size of work that is handed-off between teams. A separate team of testers won’t accept one story at a time from a team of developers. They’d rather test a release worth of stories or at least all stories in a feature at a time. This lengthens the feedback loop, increases end-to-end cycle time and hurts overall responsiveness.

High speed IT calls for motivated teams. Autonomy is a key motivator for teams. However, activity oriented organization can only be granted so much autonomy because they tend to use it for optimizing their sphere of activity.

A variation of activity-oriented organization are super-specialized teams that may result in the following ways:

  • Tool or skill centric teams: e.g. a WebSphere Portal Server team or a BizTalk team
  • Architectural Layer teams: e.g. a presentation layer team, middleware team, data layer team.

They are problematic because they have a narrow focus and they tend to optimize for team performance rather than the big picture. For sure, some tools may need specialists but that is no reason to isolate them in a separate team. Specialization isn’t the problem; organizing along lines of specialization is.

What works better

Software development is an iterative design process. In order to achieve true iteration and realize the value of fast feedback, its activities need to be performed with a single team (having a common reporting line) as far as possible. Many internet business and independent software vendors (ISVs) already operate in this way.

Yorumlar

Yorum Gönder

Bu blogdaki popüler yayınlar

Continuous Integration with Visual C++ and COM

  William E. Caputo ThoughtWorks Oren Miller ThoughtWorks July 2002 The Continuous Integration principles are applicable in many different development environments. While the principles remain largely unchanged from project to project, the implementation of this practice can vary considerably. Variables such as language, platform, team size & team location provide unique implementation challenges. Here we will outline how we've implemented CI in a COM/Windows environment for a project developing primarily in Visual C++. The More Often the Better What Is a Successful Build? Single Source Point Building the Code Self-Testing Code Automated Build Dependency Management What We Could Have Done Better Summing up The More Often the Better As noted in the main article, one of the least intuitive notions about integration is that less often does not result in less difficulty, it results in more difficulty. This is especially true when developing with C++. The build time on a development...
Yorum Gönder
Devamı

Rotation

  Rotation I've spent a lot of time of the last year wandering around ThoughtWorks, talking to lots of people on lots of projects. One message that's come home really firmly to me is the value of rotation. We practice rotation in lots of ways. One of the most notable is rotating around countries. We've put in a deliberate program to encourage people to spend 6-18 months in a different country. Living a good length of time in a different country does a huge amount to widen people's perspective of the world. I've benefitted personally from living both in the UK and USA, even though they are very similar cultures. This mental expansion is even greater for those that spend time in somewhere like India, where the cultural differences are greater. Geographic rotation presents lots of challanges, particular for older people with familes. One of the things we need to figure out is how to make geographic rotation easier for people, so more people do it. Already there's a...
Yorum Gönder
Devamı

Business Capability Centric

 Business Capability Centric A business-capability centric team is one whose work is aligned long-term to a certain area of the business. The team lives as long as the said business-capability is relevant to the business. This is in contrast to project teams that only last as long as it takes to deliver project scope. For example, an e-commerce business has capabilities such as buying and merchandising, catalog, marketing, order management, fulfilment and customer service. An insurance business has capabilities such as policy administration, claims administration, and new business. A telecom business has capabilities such as network management, service provisioning and assurance, billing, and revenue management. They may be further divided into fine-grained capabilities so that they can be owned by teams of manageable size. Business-capability centric teams are “think-it, build-it and run-it” teams. They do not hand over to other teams for testing, deploying or supporting what they...
Yorum Gönder
Devamı