Visual Grounding for Object Questions

Current visual grounding research remains limited for practical applications, because existing tasks primarily focus on direct visual queries (e.g., “find the red car”) or reading visible text (e.g., “what is the title of this book?”), rather than supporting general questions about objects (e.g., “how comfortable are these earbuds?”). We introduce the novel problem of Visual Grounding for Object Questions (VGOQ). Unlike previous tasks that ground only what is directly visible in images, VGOQ handles open-ended general questions about objects, including concepts such as ease and comfort of use, and aims to identify visual evidence or context that would support an answer. This unexplored problem has immediate practical value, particularly in designing and optimizing product imagery in e-commerce stores. As initial steps toward this task, we develop two automated data generation techniques, which serve to train a lightweight visual grounding model, and to evaluate visual grounding approaches on the resulting synthetic benchmarks, ABO-VGOQ and VizWiz-VGOQ. Our results provide initial evidence that VGOQ represents a meaningful research direction: current SoTA visual grounding performance decreases from 52% gIoU to 37% gIoU when questions are rephrased from visual questions (segmentation of the answer) to general object questions (segmentation of visual evidence). On our new benchmarks, our lightweight model outperforms prior models while being much smaller.

The data generation pipeline we developed to create the ABO-VGOQ data allows locating visual evidence / context for answering general object questions, but it relies on multiple models and is computationally expensive, making it impractical for real-time deployment.

We thus train a lightweight model (1.77M parameters), based on CLIPSeg, that combines CLIP image and text encoders with a lightweight grounding transformer.

The model takes as input a textual input (e.g., an object question) and one image, and outputs a segmentation map and a relevance score. The relevance score indicates how relevant the image is for answering the question, and the segmentation map highlights visual evidence or context that would support answering the question.

Synthetic datasets. Both datasets are automatically generated rather than manually created, lacking the quality and precision of human-created segmentation. The quality of individual examples varies. However, taken as a whole, the metric (gIoU) appears to be a good indication of the model's ability to locate visual evidence or relevant context that would support answering general object questions. To the best of our knowledge, there are no prior existing datasets that contain images, general questions about objects, and visual grounding of the evidence / context that would support answering those questions, rather than visual grounding of the answer directly.

Subjectivity and task definition. Visual evidence assessment for object questions is inherently subjective. What constitutes valid evidence can vary based on context and persons. For instance, for a size question, it is unclear whether showing an image where size is only conveyed relative to other elements counts as valid evidence, or if precise measurements are required. Future work could categorize question types and define what counts as valid evidence, including the distinction between direct visual evidence and visual evidence requiring external knowledge (see the Mrs Dash example above).