Data exploration requires fast queries, especially when coupled with interactive visualization systems. Modern analytics database systems promise to respond quickly to queries even over large data. However, we still lack adequate infrastructure to empirically assess which of these systems provide satisfactory performance. For example, the standard database approach is to run a query benchmark (e.g., TPC-H), and compare the results for each DBMS (e.g., average response time or latency). However, DBMS benchmarks are designed for data warehousing and transactional processing, which bear little resemblance to interactive environments. In contrast, visualization benchmarks, like the Visual Analytics Benchmark Repository [6], focus on human perception and productivity (e.g., how accurately a user can analyze data), but provide limited support for performance measures or comparison of analytical operations (i.e., queries).

 

We propose that a new benchmark should combine the best of both worlds by blending methodology from the database and the visualization communities. We aim to develop a new benchmark that allows for systematic and repeatable measurement of large-scale, interactive analytics systems. Our benchmark is inspired by the different ways that the database and visualization communities evaluate systems. In this talk, we will present ongoing work to develop a new benchmark to validate the suitability of database systems for some of the most demanding needs of interactive visualization systems: crossfiltering. Our benchmark is meant to allow database designers to check if their system is compatible with low-latency interactions, but also to develop strategies to exploit the particular patterns and pacing of human interactions for further optimization.