ClickHouse Founder Packs 110 Databases Into One Metal Sandbox
To benchmark queries without paying $500,000 a year for cloud instances, Alexey Milovidov crammed more than 100 database engines onto a single bare-metal server.
Analytical benchmarks are notorious for selective tuning, but maintaining the tests can be even messier than running them. ClickHouse creator Alexey Milovidov has launched ClickBench Playground, an interactive sandbox that lets users execute ad-hoc queries against more than a hundred different database systems preloaded with 100 million records each.
The project evolved out of ClickBench, an open performance suite created to evaluate analytical workloads. Comparing systems meant handling wildly divergent setups: standard relational databases, embedded engines like DuckDB and DataFusion, and data manipulation libraries such as pandas and Polars. To test non-server libraries side by side with traditional servers, the benchmark team wrapped them in lightweight Python HTTP endpoints.
Keeping hundreds of deployment scripts synchronized proved fragile. Benchmark participants frequently optimized specifically for static test queries or omitted cache-clearing between runs. Standardizing the runner scripts across all projects required automating the maintenance cycle with generative tools and introducing mandatory system restarts before measuring cold queries.
Hosting that many live systems presented a massive infrastructure hurdle. Dedicated Amazon EC2 instances for over a hundred databases would have cost upwards of $500,000 annually, while on-demand provisioning was far too slow for an interactive tool. The solution was running microVMs using Firecracker and QEMU on top of an AWS EC2 Metal host.
To squeeze an estimated 100 terabytes of system snapshots into 7.5 terabytes of local storage, the platform relies on Btrfs copy-on-write deduplication and Zstandard compression. Guest instances restore from memory snapshots in under five seconds, discarding modifications after query errors or crashes to maintain an isolated baseline.
Because performance depends heavily on shared host memory and lazy allocation, running noisy interactive queries on a multi-tenant box will rarely mirror dedicated production hardware, though it does expose how differently engines handle identical syntax and corner-case calculations.
Source: habr
Comments
Help shape the next version: Add context or suggest a correction. AI review can add points toward a rewrite. Reviews and updates may take time; a full meter does not guarantee a new version.