Designing Data-Intensive Applications

-

Designing Data-Intensive Applications introduces diverse landscape of technologies of storing and processing data. It covers from fundamental concepts, distributed data and data processing. This book does not tell you what software or how to build a data related application. It rather talks about how data systems work and their design concerns.


The book starts with non-functional requirements such as reliability, scalability and maintainability. Then it jumps data models which are associated with birth of NoSQL and ORM, thus query languages. The next chapter deeply dives into storage which is a common topics of traditional databases, e.g. Hash Indexes, Sorted String Tables, Log-Structured Merge-Trees, B-Trees. The topic is also extended to OLTP and OLAP, data warehouse. In the last chapter of the first section, it turns to topics of encoding formats such as JSON, XML and binary and dataflow services such as REST, SOAP and RPC.

The second section is about distributed data. It starts with scalability associated with Shared-Nothing Architectures. Then it talks about various replication problems (high availability, disconnected operation, latency, scalability, single-leader replication, multi-leader replication, leaderless replication, read-after-write consistency, monotonic read, consistent prefix reads), as well as partitioning problems (key range partitioning, hash partitioning, document-partitioned indexes, term-partitioned indexes). The next chapter is about Transactions. In the chapter, the author criticises the meaning of Atomicity, Consistency, Isolation, Durability, Serialisability, Two-Phase Locking, Serialisable Snapshot Isolation. Then it talks about troubles with distributed systems, consistency and consensus. The author also criticises Consistency, Availability and Partition Tolerance theorem unhelpful in the chapter.

The third section is about derived data. It talks about Batch Processing which includes traditional unix tools, MapReduce, Distributed FileSystems, Hadoop. It is concerned with problems of Partitioning, Fault Tolerance, Sort-merge joins, Broadcast hash joins, Partitioned hash joins. The next chapter is about Stream Processing which includes AMQP/JMS-style message broker, Log-based message broker, Stream-stream joins, Stream-table joins, Table-table joins. The final chapter is about the future of data systems which covers technologies and usages.

The coverage of this book is very wide, deep and comprehensive. Maybe some topics, especially for typical application developers, have been passed to the third-party experts such as database vendors or cloud service providers. Those topics are, however, still valuable and worth to reference when we design a data layer system. Particularly the demand of data layer systems has become increasing. A modern data layer system can consist of multiple persistence products from cache, file systems, databases, messaging brokers, stream processing, etc. Those topics can generally guide us how to select and combine them in our architectural design.


Comments

Post a Comment

Popular posts from this blog

[Event] JManc (28th June 2024)

-
Image
My Sharing about JavaFX Attending JManc , an unconference organised by Manchester Java Community and hosted at Auto Trader 's offices, was an exciting experience. This unique event demonstrating the essence of participant-driven learning and collaboration, set the stage for a series of engaging and enlightening sessions. Propose and vote for topics In the morning, we proposed topics using sticky notes. Afterward, we voted on these topics with dots, shaping the day's agenda based on collective interest. This process itself was a testament to the community's diverse interests and expertise. Breakfast Keynote by Helen Scott Keynote by Helen Scott The unconference kicked off with a keynote by Helen Scott from JetBrains . Helen's talk centered on AI, a topic so crucial and timely that, as she humorously noted, audience would have thrown her out of the window if she did not speak of AI. Her insights into AI's impact on development and the future of programming set a tho...
Read more

Tales from Earthsea

-
Image
Recommendation: please read the original novel series Earthsea before you watch the movie (Japanese: ゲド戦記, meaning the Story of Ged's Battles), otherwise you'd sleep very well in the cinema. I left some comment about this movie in other blogs , here is its English translation: I don't think that the problem of the movie is the fault of the plot, because the director Goro Miyazaki (Japanese: 宮崎吾朗) is not the author of the novel and the novel is an award-winning piece. The problem is that he selected a wrong media to express his own vision. First of all, the novel is not popular in Chinese society. Few people would spend hours to finish the books before watching the movie. The director selected to start the movie by the third book of the series, however. It would make audiences lose their way. This is because much background information has been described in previous book, say Tenar's experience in The Tombs of Atuan. Unlike his father Hayao Miyazaki (Japanese: 宮崎駿) as t...
Read more

Real View Will

-
I am a HIP (hidden, inactive and passive) member. I had looked for any suitable activity, so I attended the gathering last Saturday. There were three guests to talk about investment in personnal opinions instead of standard words. This is the irreplaceable value of a private gathering. The first guest was a broker. He began with introduction of investment theories. In his opinion, investing in stock is like multi-level marketing. When a stock releases, an investor buys and introduces it to another. The second investors buys it and the stock price increases. And so on, until no following investor buys it to support the stock price and the price drops. This figure simply explained the relationship between the cash flow of the market and the stock price. The second guest was a reporter. Being a reporter, he has experienced many issues under table. He identified unstated relationships amongst various parties such as the authories, the academics, mass media, institutional investors and smal...
Read more