Understand the abstractions different technologies provide you
This is an old unpublished post. Published now.
Every technology out there is trying to simplify something. It is providing you an abstraction that offers you a simple interface to achieve complex results. Understand what the technology does. Understand why it works. This will allow you to simplify your designs. After all “Data Structures, not Algorithms, are central to programming ~ Rob Pike”.
Kafka and Kinesis provide you a log-like storage system. It’s easy to keep adding things to the end of a list. You can do this at a high speed. However, this structure makes it hard to perform any smart queries on the data.
To perform easy queries on the data, something like RDBMS and SQL would be a better fit. Adding new data is more work in these technologies (indexes have to be updated, constraints have to be validated etc.). However, this allows you to put rules to ensure data consistency and perform ad-hoc reads and aggregations on the data. This also makes them a poor choice for high speed writes. Something like Kinesis or Kafka is a better fit for that.
Originally published at https://www.floatingpoint.ca on May 13, 2018.
