Developing a Large-Scale Time Series Database

Length: 60 minutes

Room: Room 7

Abstract

Cognite's customers have trillions of historical data points from industrial sensors, and they are continuously ingesting new values into millions of time series in real time. This talk will explore the architecture of Cognite's custom timeseries database, showing how it is able to query both real time data and terabytes of historical data with low latency while still providing strong consistency and high availability. The talk takes place at a conceptual level (no code), and is structured as a series of challenges that we faced during development and the solution we arrived at for each challenge.

Day & time

06 Wednesday, 3:40 – 4:40 PM

Intended audience

This talk is suitable for anyone who is interested in system architecture, distributed databases, scalability, and optimization. As an attendee, you will learn numerous tricks for working with large-scale distributed databases and developing your own solutions on top of FoundationDB, such as: setting up tiered storage to enable both high write throughput and high read throughput, introducing sharding and versioning schemes, using custom compression, reducing your code's garbage collection footprint, and doing preaggregation with segment trees. No familiarity with any particular language is needed. Some familiarity with databases in general will be helpful.

• Åsmund Eldhuset

  Åsmund Eldhuset is a geek of all trades who got a taste for teaching and public speaking when he TA'd the algorithms class at the Norwegian University of Science and Technology. After seven years in Silicon Valley, he is back home in Norway, and is now a Principal Software Engineer at Cognite.

  

  @AasmundEldhuset