Structured record data store

Petabite offers a hybrid solution for the storage of structured records that combines efficient storage in structured objects with an index on statistical metadata. The time hierarchical indexing structures allow for efficient approximate value searches, while the structured object storage allows for storage efficiency that does not compromise on fast retrieval access.

When there is a lot of data from different sources, the available options are usually the submission into databases or the direct storage in files. Database solutions become cumbersome with large data sets as queries become slow and the operational overhead may be significant without major benefits. Advanced database functionality like value searches may remain slow independent of the amount of available resources. Direct storage in files has the disadvantage of having no search cababilities while still requiring a lot of careful managing to maintain accessibility and prevent data loss. Our hybrid solution offers the best of both worlds through time hierarchical indexing.

Key features of the service offering are

• Easy to use but powerful search function
• Useful additional productivity functions (e.g stored searches)
• API access as well as secure web UI functionality.
• Agreeable use and visualisation also on mobile devices
• Optionally, datasets can be published, for example to fulfil open data requirements

The service is well integrated into the Petabite service landscape. We can support the overall data management process further through

• Hosted processing capabilities with direct and secure access to the stored data
• Operational execution of ingestion pipeline in one of our managed Kubernetes clusters
• Operation monitoring and alarm functions to ensure continuous operations of the data gathering flows.