Operational databases vs Warehouses
Figuring out how you keep and use your data often requires different approaches.
The world needs real-time data
Every organisation needs real-time data. Whether you are an e-commerce vendor that needs to check stock levels, a government organisation looking at the number of driving licence applications in progress or a logistics company wondering where all their trucks are.
Typically these are the typical primary transactional databases we always model. A MySQL datastore with orders, an RDS instance with work-in-progress applications, and a MongoDB shard with location data.
This works great for what we need on a day-to-day. Whole swathes of posts about microservices, domains, SAGA patterns, and views have been written about this.
Somewhere down the track, we have a Warehouse/DataLake. Our warehouse is often tasked with storing vast quantities of data, reaching back through the echelons of time that cover multiple generations of developers.
Operational data stores cover the use case between these two bastions of modern development. The fast-moving, immediate and precise requirements a business needs to respond to customer demands which are often at odds with our transactional stores.
A warehouse doesn’t make the cut
Data Warehouses have long been proposed as the end solution for all data needs. Highly scalable, heavily indexed, and flexible, they are the endless bucket that all data retires. Having fulfilled its purpose, date resides permanently for historic reports and maybe some future AI to assess and evaluate. Warehouses usually have some lag between ETL process, ingest frequency and query performance. That was acceptable in the past, but we all have higher expectations now.
What about fast-moving reports? Let’s say I wanted to fulfil the examples above. As our data is operational, a warehouse won’t do what we need.
It’ll lag behind, risk long-running queries and not give us the views we need. The data is transactionally useful but not operational.
Enter the operational datastore
An operational data store is a real-time view of your organisations data. Geared around performance and rapid queries, it will give you the rapid feedback you require without the risk of exceptionally long-running queries and out-of-sync data.
Players in this space include:
Under the hood, these databases often rely on high-performance data stores coupled with real-time indexing to bring your data back to speed. Typically, organisations would need to bring stream processing and complex technologies into their estate to manage this type of requirement. Operational data stores have seen the impact and complexity this brings and now provide this as a service.
This brings down complexity for organisations while also unlocking new capabilities.
The options are growing
Historically, you’d be chastised for growing your technology estate with new and upcoming datastores, but with an iPaas, this is derisked.
An iPaas like Kassette, allows you to experiment and move data easily around your estate. We can connect your sources to your destinations with just some small configuration changes.
Give Kassette a try today!