Hospital Sign Dialogue Translation for Intake Interviews Using Visual Prompt Adaptation

Abstract

Communication barriers between deaf or hard-of-hearing patients and hearing healthcare professionals during medical intake interviews represent a critical vulnerability in global healthcare delivery. Traditional human interpretation services, while highly accurate, are frequently constrained by availability, high costs, and privacy concerns, particularly in urgent or sensitive clinical scenarios. Recent advancements in automated sign language translation offer a promising alternative; however, deploying these systems in hospital environments introduces severe domain shifts. Clinical settings are characterized by complex visual backgrounds, unpredictable lighting conditions, occlusion from medical equipment, and varying patient postures, which collectively degrade the performance of standard translation models. This paper proposes a comprehensive framework for hospital sign-dialogue translation integrated with visual prompt adaptation to mitigate these environmental challenges. By injecting learnable visual prompts into the encoding phase of a transformer-based translation architecture, the system dynamically recalibrates its spatial-temporal focus, effectively filtering out extraneous clinical noise while preserving fine-grained manual and non-manual sign features. The proposed approach specifically targets the intake interview domain, modeling multi-turn dialogues to contextualize isolated signs within broader clinical narratives. Extensive evaluations on a specialized clinical sign language dataset demonstrate that visual prompt adaptation significantly improves translation accuracy, robustness, and clinical utility compared to traditional baseline models. The findings indicate that tailored multimodal architectures can successfully bridge the communication gap in acute care settings, facilitating equitable healthcare access and improving diagnostic accuracy for marginalized patient populations.