Cost reduction due to lower resource consumption
RESTful API support allows simple interoperation with existing stack
SQL compatibility simplifies developer onboarding
Powers a real-time system that operates at enterprise network speeds
Active developer community that helps with troubleshooting
Fast turnaround time from prototype phase to production deployment
LiveAction is a cybersecurity software company offering a SaaS platform for network monitoring and security. Their encrypted traffic analysis product, ThreatEye, integrates advanced security technologies into a streaming machine learning pipeline to identify network faults, anomalies and threats at wire speed.
In this case study, VP Product Development of ThreatEye and founder of Counterflow AI (now LiveAction), Randy Caldejon describes how and why QuestDB is an important component of their SaaS platform for time-series and behavioural analytics.
The rise in encrypted traffic over HTTPS and the recent introduction of protocols such as DNS over HTTPS and TLS 1.3 means that network defenders are faced with dramatically reduced deep packet inspection capabilities and visibility. Our security offering allows LiveAction partners to extend network visibility into the nature of this traffic using Encrypted Traffic Analysis (ETA).
ETA provides techniques to gain insight into network behaviour despite encryption while protecting user privacy. It combines Deep Packet Dynamics with machine learning to identify malicious patterns in network activity. The benefit of this approach is that it can scale with continued growth in network traffic and increased use of encrypted protocols despite having no visibility into the content of the exchanges.
ThreatEye NV is powered by a streaming machine learning engine (MLE) that ingests the high-fidelity flow data generated by its software probes. We use this to provide end-to-end visibility into the nature of network traffic using real-time inferences in combination with Encrypted Traffic Analysis.
Distinct from batch processing, streaming ML is powered by analyzers designed to inspect network traffic without multiple passes over the data stream. The streaming nature of this solution means that we have to process millions of events per second. The QuestDB instances we’re running are storing billions of records with the fields which we analyze to perform our predictions. The performance of QuestDB allows us to run queries such as these without our database being the bottleneck.
We started with InfluxDB as our central time series database, but we quickly started hitting performance issues with scalability in production environments, and we needed to find a practical alternative. We’re typically executing 25k to 100k inserts per second, depending on the size of the customer and the network activity. After InfluxDB, we tried TimescaleDB, which was reasonable for performance, but the database configuration was inconvenient for us and the system footprint was not ideal.
When I first tried QuestDB using test scripts to evaluate time series databases, I initially thought I had misconfigured something because the ingestion speed seemed unrealistic. When I ran some SQL queries in the console and got near-instant results returning our full dataset, I started to get excited about QuestDB being a legitimate alternative to other systems.
Our tools export either JSON or CSV, which means that a RESTful API to import and export data allows for seamless interfacing with the rest of our technology stack. We’re now using InfluxDB line protocol over TCP for ingestion, and the performance is even better.
We’re analyzing over 150 features of network flows, and our customers want to see common aggregations such as top-n clients consuming data on the network or TLS connections with unusual entropy scores. SQL compatibility makes this easy to calculate in QuestDB and quick to verify in the web console. Even better, the Postgres interface offers our security analytics team the flexibility to dive into deeper analysis using Jupyter Hub.
Our solution runs in hybrid-cloud deployments and needs to scale up to 40Gbps worth of inspected network data. High-performance is critical to ensure scalable and reliable analytics when deploying in high-throughput scenarios such as enterprise networks.
LiveAction plans to introduce a community version of the ThreatEye analysis pipeline in Q2 2022. The pipeline includes native integration with QuestDB. The community version will be released as ThreatEye Toolkit and will be available as a Docker container on Docker Hub.
“QuestDB is impressive and stands out as a superior option. We use it as the basis of our time series analytics for network threat detection.”
Randy Caldejon, VP Product Development ThreatEye, LiveAction