SmartBench: Demonstrating Automatic Generation of Comprehensive Benchmarks for Question Answering Over Knowledge Graphs

Download PDF

Abstract

In recent years, a significant number of question answering (QA) systems that retrieve answers to natural language questions from knowledge graphs (KG) have been introduced. However, finding a benchmark to accurately evaluate the quality of a question answering system is a difficult task because of (1) the high degree of variations with respect to the fine-grained properties among the available benchmarks, (2) the static nature of the available benchmarks versus the evolving nature of KGs, and (3) the limited number of KGs targeted by existing benchmarks, which hinders the usability of QA systems in real deployment over different KGs. In this demonstration, we introduce SmartBench, an on-the-fly benchmarking system for QA over KGs systems that can be used to build a QA benchmark over any KG. The benchmark generated by SmartBench is guaranteed to cover all the properties of the natural language questions and queries that were encountered in the literature as long as the targeted KG’s topology includes these properties. Moreover, SmartBench generates benchmarks that follow the same standards and complexity distribution to solve the variation of the fine-grained properties among generated benchmarks.