Designing Data-Intensive Applications

-

Designing Data-Intensive Applications introduces diverse landscape of technologies of storing and processing data. It covers from fundamental concepts, distributed data and data processing. This book does not tell you what software or how to build a data related application. It rather talks about how data systems work and their design concerns.


The book starts with non-functional requirements such as reliability, scalability and maintainability. Then it jumps data models which are associated with birth of NoSQL and ORM, thus query languages. The next chapter deeply dives into storage which is a common topics of traditional databases, e.g. Hash Indexes, Sorted String Tables, Log-Structured Merge-Trees, B-Trees. The topic is also extended to OLTP and OLAP, data warehouse. In the last chapter of the first section, it turns to topics of encoding formats such as JSON, XML and binary and dataflow services such as REST, SOAP and RPC.

The second section is about distributed data. It starts with scalability associated with Shared-Nothing Architectures. Then it talks about various replication problems (high availability, disconnected operation, latency, scalability, single-leader replication, multi-leader replication, leaderless replication, read-after-write consistency, monotonic read, consistent prefix reads), as well as partitioning problems (key range partitioning, hash partitioning, document-partitioned indexes, term-partitioned indexes). The next chapter is about Transactions. In the chapter, the author criticises the meaning of Atomicity, Consistency, Isolation, Durability, Serialisability, Two-Phase Locking, Serialisable Snapshot Isolation. Then it talks about troubles with distributed systems, consistency and consensus. The author also criticises Consistency, Availability and Partition Tolerance theorem unhelpful in the chapter.

The third section is about derived data. It talks about Batch Processing which includes traditional unix tools, MapReduce, Distributed FileSystems, Hadoop. It is concerned with problems of Partitioning, Fault Tolerance, Sort-merge joins, Broadcast hash joins, Partitioned hash joins. The next chapter is about Stream Processing which includes AMQP/JMS-style message broker, Log-based message broker, Stream-stream joins, Stream-table joins, Table-table joins. The final chapter is about the future of data systems which covers technologies and usages.

The coverage of this book is very wide, deep and comprehensive. Maybe some topics, especially for typical application developers, have been passed to the third-party experts such as database vendors or cloud service providers. Those topics are, however, still valuable and worth to reference when we design a data layer system. Particularly the demand of data layer systems has become increasing. A modern data layer system can consist of multiple persistence products from cache, file systems, databases, messaging brokers, stream processing, etc. Those topics can generally guide us how to select and combine them in our architectural design.


Comments

Post a Comment

Popular posts from this blog

Event: Developer Productivity Engineering: What's in it for me?

-
Image
Manchester Java Community hosted another great session in Thoughtworks . The speaker is Trisha Gee , co-author of the renowned book Head First Java .  The speaker pointed out that the productivity of developers did not mainly come from individual developer themselves but from the organisation which offers productive environments or not. The factor includes work space size, quietness, privateness, any diversion or needless interruption. On technology side, productivity can be improved by consistency and reliability of builds. Very fast feedback can improve productivity and quality, thus increase revenue, reduce cost and improve brand. References: DEVELOPER PRODUCTIVITY ENGINEERING: WHAT’S IN IT FOR ME? The 2019 Tidelift managed open source survey results Peopleware: Productive Projects and Teams Software Productivity   The Goal: A Process of Ongoing Improvement Improve the Performance of Gradle Builds CI fanout Netflix Pursues Soft DevEx Goals with Hard DevProd Metrics using Test Distri
Read more

Ethical Hacking 101

-
In today's digital landscape, software security is paramount. To safeguard applications from cyber attacks, developers must embrace Ethical Hacking—an innovative approach that empowers them to uncover vulnerabilities and strengthen security measures. Ethical Hacking involves adopting the mindset of malicious hackers to identify weaknesses and fortify defences against unauthorised access, data breaches, and other threats. By integrating this practice into the coding process, developers gain valuable insights, proactively mitigating risks and creating more robust and secure software systems.  I took a basic lesson from Snyk about Ethical Hacking, which helps me consider security measures in coding.   Process   Plan & Prepare ( Reconnaissance phase)   Scan & Enumerate   Exploit   Analyse & Report Communicate & Collaborate Continuous Education & Professional Development   Tools   Nmap : network scanner   Burp Suite : security testing of web applications   Metasploi
Read more

Mark Six Analyst

-
Open Source in the world of software is amazing. Developers share and distribute source codes with each other. SourceForge is one of the well-known platforms. I have registered my first Open Source project in SourceForge. The project is called Mark Six Analyst at https://sourceforge.net/projects/marksixanalyst/ . It is a simple tool to forecast the result of Hong Kong lottery or so-called Mark Six by various statistics methods. For the success of the project, I would like to thank Java, JavaDB (Apache Derby), Java Swing, Java Persistence API, TopLink, Hibernate Entity Manager, NetBeans and CsvReader. Because of them, developing a software application becomes rapidly quicker and the developed application becomes more stable and much easily maintained. Especially for Java Persistence API or called JPA, developers are less annoyed to handle entity EJBs and Object-Relational Mapping. It is helpful for developing an application with a database of complicated entity relationships. On the ot
Read more