HLR-LLM: Hear, Localize, and Reason

Abstract

Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene. Recent spatial audio-visual studies address parts of this problem, but often focus on specific abilities, such as spatial correspondence or direction and distance reasoning. In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene. We further introduce HLR-LLM, which augments a strong audio-visual foundation model with a binaural spatial audio branch alongside its monaural semantic audio pathway. We train HLR-LLM with a curriculum that incorporates final-stage chain-of-thought supervision for multi-step spatial reasoning. Experiments show that existing AV-LLMs remain limited in spatial grounding and relational reasoning, while HLR-LLM substantially improves spatial audio-visual scene understanding without compromising semantic audio-visual perception.

Contributions

We introduce HLR-AVSceneQA, a comprehensive benchmark that evaluates spatial hearing and grounding both directly and as intermediate evidence for downstream spatial relational reasoning.
We propose HLR-LLM, a spatially aware AV-LLM that augments a strong foundation model with a spatial audio pathway to improve grounding and reasoning while preserving semantic perception.
We demonstrate that final-stage chain-of-thought curriculum training improves multi-step spatial reasoning by connecting auditory, spatial, and visual cues through intermediate reasoning steps.

HLR-AVSceneQA Benchmark

175,061

training QA pairs

21,808

test QA pairs

94,519

CoT-augmented pairs

12

tasks in 3 families

HLR-AVSceneQA includes 53,279 training and 4,869 test pairs from scenes with multiple simultaneously active sound sources. Because collecting real-world spatial audio-visual data with accurate annotations is challenging, we construct the benchmark with a two-stage simulator-based pipeline: Stage I synthesizes spatial audio-visual scenes with structured metadata, and Stage II generates QA pairs from this metadata. Unlike panoramic-input settings, HLR-AVSceneQA uses normal-field-of-view egocentric observations, requiring models to reason about visible, occluded, and off-screen sources from partial visual evidence.

Stage I — Spatial audio-visual scene synthesis

Scene and acoustic simulator. SoundSpaces 2.0 renders geometry-aware room acoustics and spatializes dry monaural clips into binaural audio. 3D environments come from HM3DSem (216 indoor spaces, 3,100 rooms with dense semantics); 145 scenes for training and 36 for testing.
Audio source curation. Non-speech sounds from VGGSound (310+ classes), manually filtered to remove classes that are not naturally grounded to a single indoor object, plus ImageBind-based text-audio similarity filtering of likely mislabeled clips. Speech comes from LibriSpeech for far-field ASR.
Visual object curation. A 3D mesh pool of sounding objects from Objaverse-XL, supplemented with off-the-shelf 2D/3D generation models for under-covered categories — roughly 81 meshes per category for training and 35 for testing, all manually checked.
Scene assembly. 2–6 candidate meshes are inserted per room and up to two are activated as sound sources. Some audio-object pairs are semantically matched, others are spatially aligned but semantically mismatched. Sources may be fully visible, occluded, in an adjacent room, or outside the field of view. The final RGB image is rendered at 1280×720 with a 90° horizontal field of view, paired with binaural audio.

Examples of rendered scenes and structured metadata

Examples of rendered scenes and metadata. Our dataset covers diverse scenarios that require spatially aware audio-visual reasoning, including (a) sound sources occluded by objects and (b) ambiguous sound sources among same-category instances. Each sample is summarized with structured scene metadata. Yellow denotes existing scene objects, pink denotes inserted objects, blue denotes sounding objects, and green denotes the agent.

Stage II — Task design and QA construction

1 Spatial hearing

Sound event recognition — identify audible event categories. · Far-field ASR — transcribe spoken content from far-field audio.

2 Spatial grounding

Egocentric direction / distance — localize the source relative to the agent's current viewpoint. · Allocentric direction / distance — localize it from a transformed viewpoint, defined by standing at a reference object and facing another object.

3 Spatial relational reasoning

Visibility — is the source visible from the current view? · Correspondence — which visible object is spatially aligned with the sound (the answer may be semantically mismatched, or None)? · Hidden-source localization — is a non-visible source hidden behind an object, in an adjacent room, or outside the field of view? · Loudness change — does the source get louder or quieter after the agent moves to a target object? · Proximity — which candidate object is nearest to the source? · Scene captioning — summarize sound events, directions, and visibility states.

Chain-of-thought supervision example

Q: Given the sound, where is its source located?

A: The sound event is a hair dryer drying, arriving from the front-left direction and 2.7 meters away. The hair dryer is not visible because it is occluded by the armchair. Therefore, the source is hidden behind the armchair.

CoT rationales are deterministically generated from structured simulator metadata using task-specific templates; GPT-4 is used only to paraphrase question wording.

Evaluation scope compared with existing benchmarks

Benchmark	A+V	Multi- source	Far-field ASR	Sound event	Ego.	Allo.	AV Corr.	Cross- room	Hidden Source	Proximity	AV Caption
BAT	✗	✓	✗	✓	✓	✗	✗	✗	✗	✗	✗
SpatialWhisper	✗	✓	✓	✗	✓	✗	✗	✗	✗	✗	✗
Hear You Are	✓	✗	✗	✓	✓	✗	✓	✗	✗	✗	✗
SAVVY-Bench	✓	✗	✗	✗	✓	✓	✗	✓	✗	✗	✗
SpatialSceneQA	✓	✓	✗	✗	✓	✗	✓	✗	✗	✗	✗
HLR-AVSceneQA (Ours)	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓

Evaluation scope of spatial audio and audio-visual benchmarks. A+V denotes audio-visual input; Multi-source denotes simultaneous sound sources; Ego./Allo. denote egocentric/allocentric reasoning; Corr. denotes audio-visual correspondence. Cross-room, Hidden Source, Proximity, and AV Caption denote reasoning about across-room sources, invisible or occluded sources, nearest-object relations, and open-ended audio-visual captioning.

HLR-LLM

Architecture and training curriculum of HLR-LLM. (a) HLR-LLM augments a strong AV-LLM with a binaural spatial audio branch, enabling the language model to jointly attend to text, visual, downmixed monaural audio, and binaural spatial audio tokens. (b) The model is trained through a three-stage curriculum: spatial audio alignment, spatial audio-visual instruction tuning, and final-stage CoT tuning for structured spatial reasoning.

HLR-LLM is built on Qwen2.5-Omni and adds a spatial audio pathway alongside the existing visual and monaural audio branches. A binaural signal is encoded by a frozen spatial audio encoder and mapped into the language model embedding space by a trainable MLP projector. A new media token, <spatial_audio>, marks the placeholder positions that are replaced with the projected spatial audio tokens, giving the input sequence X = [E(q), Z_vis, Z_aud, Z_spa]. The monaural branch preserves semantic hearing ability, while the spatial branch supplies direction- and distance-sensitive cues for grounding and reasoning.

Three-stage curriculum

Stage	Training data → Objective	# Samples	Trainable
1	Spatial audio-only QA → Spatial audio alignment	514k	Spatial projector
2	HLR-AVSceneQA → Spatial audio-visual instruction tuning	175k	Spatial projector + LoRA
3	HLR-AVSceneQA-CoT → Chain-of-thought tuning	94.5k	Spatial projector + LoRA

All visual, monaural audio, and spatial audio encoders remain frozen throughout training. LoRA uses rank 16, alpha 16, dropout 0.05, learning rate 2×10⁻⁴, and weight decay 0.01; 2 epochs on four NVIDIA A6000 GPUs with a per-device batch size of 4. Binaural audio is sampled at 32 kHz with a 1024 window, 320 hop, and 128 mel bins.

Results

All numbers are mean ± standard deviation over three runs. Best and second-best mean results are shown in bold and underlined.

Spatial hearing and spatial grounding

Model	Modality	Spatial Hearing		Egocentric Grounding		Allocentric Grounding
Model	Modality	Sound (Acc. %)	Speech (WER ↓)	Direction	Distance	Direction	Distance
Random chance	–	10.00	–	12.50	10.00	12.50	10.00
Zero-shot models
MiniCPM-o-v2.6	V+M	35.26±0.32	126.10±5.51	12.83±0.48	14.26±0.66	11.92±0.36	18.42±0.51
VideoLLaMA2	V+M	54.54±0.11	113.54±0.82	12.15±0.41	7.73±0.05	15.84±0.36	7.19±0.17
Video-SALMONN 2	V+M	32.20±0.13	13.06±0.02	17.00±0.22	7.62±0.07	17.55±0.20	6.98±0.12
Qwen2.5-Omni	V+M	57.21±0.09	14.92±0.56	17.33±0.07	7.10±0.07	21.43±0.60	7.03±0.13
Qwen3-Omni	V+M	55.84±0.03	15.27±1.29	15.64±0.07	11.20±0.06	18.09±0.21	9.17±0.26
SAVVY	V+M+B	–	–	52.36±0.46	19.87±0.15	17.81±0.38	12.37±0.60
GPT-Realtime-Mini	V+M	34.21±0.36	18.07±0.86	12.65±0.42	14.55±0.60	14.27±0.28	17.28±0.96
Gemini-2.5-Flash-Lite	V+M	38.08±0.10	21.58±0.18	19.67±0.14	12.23±0.24	19.39±0.29	8.04±0.11
Fine-tuned models
Hear You Are LLM	V+B	67.67±0.37	148.00±1.13	44.65±0.42	22.63±0.54	21.53±0.19	21.57±0.33
Qwen2.5-Omni	V+M	80.57±0.27	15.33±2.33	32.69±0.26	30.92±0.42	26.26±0.18	27.73±1.22
Dual-channel Qwen2.5-Omni	V+M_L+M_R	84.52±0.07	13.42±1.14	64.27±0.32	46.59±0.27	42.71±0.25	37.84±0.24
HLR-LLM (Ours)	V+M+B	81.03±0.11	14.43±0.54	78.21±0.34	55.56±0.27	52.03±0.26	45.08±0.85

Sound event recognition and grounding tasks are reported in accuracy (%); speech recognition is reported in WER (%, lower is better). V, M, and B denote visual input, monaural audio, and binaural audio; M_L and M_R denote independently encoded left- and right-channel audio inputs. Zero-shot AV-LLMs and commercial models show reasonable spatial hearing but perform poorly on direction and distance grounding. Dual-channel Qwen2.5-Omni improves localization by processing left and right channels separately, but HLR-LLM outperforms it across all egocentric and allocentric grounding metrics, demonstrating the benefit of explicitly modeling binaural spatial information rather than exposing the LLM to separate left- and right-channel tokens.

Spatial relational reasoning and scene captioning

Model	Modality	Spatial Relational Reasoning (Acc. %)					Scene Captioning
Model	Modality	Visibility	Correspondence	Hidden source	Loudness	Proximity	CIDEr-D	METEOR	SPIDEr
Random chance	–	50.00	9.49	20.00	50.00	28.68	–	–	–
Zero-shot models
MiniCPM-o-v2.6	V+M	75.20±0.14	22.52±1.09	11.65±0.62	47.29±0.48	26.83±0.62	0.078±0.004	0.173±0.001	0.136±0.002
VideoLLaMA2	V+M	68.37±0.22	19.77±0.14	21.25±0.26	39.18±0.20	25.52±0.30	0.112±0.002	0.131±0.001	0.079±0.000
Video-SALMONN 2	V+M	75.56±0.00	7.97±0.22	22.00±0.31	51.89±0.00	30.10±0.05	0.075±0.001	0.094±0.001	0.074±0.000
Qwen2.5-Omni	V+M	75.51±0.05	33.30±0.13	21.48±0.54	51.87±0.04	32.95±0.10	0.301±0.002	0.168±0.001	0.172±0.001
Qwen3-Omni	V+M	73.90±0.30	26.18±0.28	23.58±0.36	51.29±0.85	32.06±0.39	0.344±0.004	0.203±0.000	0.260±0.001
GPT-Realtime-Mini	V+M	69.46±0.61	12.84±0.25	20.10±0.42	50.13±0.99	26.03±0.44	0.052±0.002	0.237±0.001	0.151±0.002
Gemini-2.5-Flash-Lite	V+M	49.81±0.32	20.29±0.05	15.36±0.17	48.27±0.57	30.89±0.34	0.570±0.010	0.209±0.002	0.218±0.003
Fine-tuned models
Hear You Are LLM	V+B	71.31±0.05	54.73±0.08	43.97±0.20	65.41±0.28	33.33±0.49	4.160±0.008	0.385±0.000	2.409±0.004
Qwen2.5-Omni	V+M	62.72±0.10	55.73±0.07	51.24±0.58	68.84±0.32	47.11±0.27	4.796±0.014	0.420±0.000	2.751±0.007
Dual-channel Qwen2.5-Omni	V+M_L+M_R	77.30±0.12	69.07±0.15	60.12±0.43	72.59±0.31	55.64±0.36	5.414±0.011	0.444±0.001	3.070±0.005
HLR-LLM (Ours)	V+M+B	80.51±0.38	76.19±0.11	67.23±0.37	80.87±0.20	61.50±0.51	4.461±0.004	0.439±0.001	3.109±0.004

HLR-LLM achieves the best performance across all spatial relational reasoning tasks and consistently outperforms the dual-channel Qwen2.5-Omni baseline, supporting the benefit of explicit binaural modeling for scene-level reasoning. For scene captioning, HLR-LLM achieves the best SPIDEr score, while dual-channel Qwen2.5-Omni obtains the best CIDEr-D and METEOR.

Ablation study

Variant	Modality	Egocentric Grounding		Allocentric Grounding		Spatial Relational Reasoning
Variant	Modality	Direction	Distance	Direction	Distance	Correspondence	Hidden source	Proximity
Modality ablation
(a) Without vision	M+B	74.72±0.36	50.64±0.54	21.51±0.41	26.63±0.62	57.11±0.11	66.62±0.21	34.13±0.12
(b) Visual only	V	31.20±0.14	20.33±0.49	28.11±0.36	22.96±0.22	26.71±0.02	24.16±0.43	37.09±0.55
(c) Without binaural	V+M	32.69±0.26	30.92±0.42	26.26±0.18	27.73±1.22	55.73±0.07	51.24±0.58	47.11±0.27
Training ablation
(d) Stage-2 training	V+M+B	77.92±0.05	52.44±0.37	48.37±0.21	38.83±0.96	67.69±0.11	66.52±0.35	56.23±0.43
(e) Joint training	V+M+B	78.00±0.15	53.04±0.17	45.01±0.59	39.25±0.68	72.71±0.23	69.78±0.24	56.92±0.21
(f) HLR-LLM (Ours)	V+M+B	78.21±0.34	55.56±0.27	52.03±0.26	45.08±0.85	76.19±0.11	67.23±0.37	61.50±0.51

All metrics are accuracy (%). Binaural audio supports spatial localization, vision supports object-level grounding, and final-stage CoT tuning improves complex spatial reasoning. Notably, the model without vision (a) still performs relatively well on hidden-source reasoning, suggesting that binaural cues provide useful evidence for distinguishing non-visible source configurations. Applying CoT tuning after Stage-2 instruction tuning (f) performs best across all grounding metrics and most relational reasoning tasks, with clear gains in allocentric grounding, correspondence, and proximity, compared with mixing CoT data into Stage 2 (e).

Real-world adaptation on STARSS23

Model	Pretrain	Sound	Direction	Distance	Visibility
(a) Hear You Are LLM	–	38.66±0.16	15.25±0.57	34.87±0.35	62.35±0.07
(b) Hear You Are LLM	✓	40.80±0.76	16.15±0.36	26.93±0.45	63.80±0.48
(c) HLR-LLM	–	44.59±0.13	16.13±0.04	28.11±0.04	69.67±0.18
(d) HLR-LLM	✓	53.93±0.14	29.92±0.13	35.13±0.07	71.68±0.09

“Pretrain” denotes HLR-AVSceneQA pretraining before STARSS23 fine-tuning. “Sound” denotes sound event recognition; direction, distance, and visibility are egocentric. All metrics are accuracy (%). HLR-AVSceneQA pretraining improves most evaluated tasks, suggesting that it provides useful spatial priors for adapting to real spatial audio-visual recordings. Among pretrained models, HLR-LLM (d) outperforms Hear You Are LLM (b) across all metrics, with particularly large gains in sound recognition and direction grounding.

Qualitative Results

Qualitative examples of spatial grounding and hidden-source reasoning

Qualitative examples of spatial grounding. In the left example, multiple visually similar electric blenders are present while the sound originates from the front-right direction. The monaural Qwen2.5-Omni predicts an incorrect direction, whereas models with binaural spatial input correctly localize the source. The right example is more challenging because the sounding bird is not visible and must be localized using spatial audio and scene layout: Qwen2.5-Omni and Hear You Are LLM attribute the source to other non-visible regions, while HLR-LLM correctly identifies it as hidden behind the bed.

Limitations

HLR-AVSceneQA is built with a simulator-based pipeline, which provides controllable annotations but cannot fully capture the acoustic complexity and visual diversity of the real world. The benchmark focuses on indoor scenes, leaving dynamic agents, moving sound sources, and richer embodied interaction for future work. Future research should extend spatial audio-visual benchmarks to more realistic and dynamic settings and develop AV-LLMs that can actively perceive, move, and reason in spatial environments.

BibTeX

@inproceedings{sungbin2026hlr,
  title     = {Hear, Localize, and Reason: Spatially Aware Scene Understanding
               for Audio-Visual LLMs},
  author    = {Sung-Bin, Kim and Jung-Mok, Lee and Jung, Jinwoo and
               Hyun-Bin, Oh and Ryu, Hyeonggon and Harwath, David and
               Oh, Tae-Hyun},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

Acknowledgments

This work was supported by the National Research Foundation of Korea (NRF) funded by the Korean government (MSIT) in 2026 (Grant No. RS-2024-00451947, Interaction Framework for Connecting Users to Generative Experiences via Time-Space Expanded Multimedia: Centering on Social Communication Disorders; No. RS-2024-00358135, Corner Vision: Learning to Look Around the Corner through Multi-modal Signals), the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. RS-2025-25443318, Physically-grounded Intelligence: A Dual Competency Approach to Embodied AGI through Constructing and Reasoning in the Real World; No. RS-2026-25518317, Development of AI memory mechanism that reflects human cognitive principles), the InnoCORE program of the Ministry of Science and ICT (26-InnoCORE-01; N10250156), and the Artificial Intelligence Industrial Convergence Cluster Development Project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City.

Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs