Notlarım

 Data Mesh Principles and Logical Architecture The great divide of data What do we really mean by data? The answer depends on whom you ask. Today’s landscape is divided into  operational data  and  analytical data . Operational data sits in databases behind business capabilities served with microservices, has a transactional nature, keeps the current state and serves the needs of the applications running the business. Analytical data is a temporal and aggregated view of the facts of the business over time, often modeled to provide retrospective or future-perspective insights; it trains the ML models or feeds the analytical reports. The current state of technology, architecture and organization design is reflective of the divergence of these two data planes - two levels of existence, integrated yet separate. This divergence has led to a fragile architecture. Continuously failing ETL (Extract, Transform, Load) jobs and ever growing complexity of labyrinth of data pipel...

Monolith First  As I hear stories about teams using a  microservices architecture , I've noticed a common pattern. 1.       Almost all the successful microservice stories have started with a monolith that got too big and was broken up 2.       Almost all the cases where I've heard of a system that was built as a microservice system from scratch, it has ended up in serious trouble. This pattern has led many of my colleagues to argue that  you shouldn't start a new project with microservices, even if you're sure your application will be big enough to make it worthwhile.  .   Microservices are a useful architecture, but even their advocates say that using them incurs a significant  MicroservicePremium , which means they are only useful with more complex systems. This premium, essentially the cost of managing a suite of services, will slow down a team, favoring a monolith for simpler applications. This l...

 Data Mesh Principles and Logical Architecture Our aspiration to augment and improve every aspect of business and life with data, demands a paradigm shift in how we manage data at scale. While the technology advances of the past decade have addressed the scale of volume of data and data processing compute, they have failed to address scale in other dimensions: changes in the data landscape, proliferation of sources of data, diversity of data use cases and users, and speed of response to change. Data mesh addresses these dimensions, founded in four principles: domain-oriented decentralized data ownership and architecture, data as a product, self-serve data infrastructure as a platform, and federated computational governance. Each principle drives a new logical view of the technical architecture and organizational structure. This article is written with the intention of a follow up. It summarizes the data mesh approach by enumerating its underpinning principles, and the high level lo...

Consistent Core Maintain a smaller cluster providing stronger consistency to allow large data cluster to coordinate server activities without implementing quorum based algorithms. Problem Linearizability  is the strongest consistency guarantee where all the clients are guaranteed to see latest committed updates to data. Providing linearizability along with fault tolerance needs  consensus  algorithms like  Raft ,  Zab  or  Paxos  to be implemented on the servers. While consensus algorithm is an essential requirement to implement a Consistent Core, there are various aspects of client interaction - such as how a client finds the leader, how duplicate requests are handled, etc - which are important implementation decisions. There are also some important implementation considerations regarding safety and liveness. Paxos defines only the consensus algorithm, but these other implementation aspects are not well documented in the Paxos literature. R...