Context aided video-to-text information fusion

Information Fusion consists of organizing a set of data for correlation in time, association over multimodal collections, and estimation in space. There exist many methods for object tracking and classification; however video tracking suffers from exact methods in object labeling, the ability to correlate tracks through dropouts, and determination of intent. Our novel solution is to fuse video data with text data for better simultaneous tracking and identification. The need for such solutions resides in answering user queries, linking information over different collections, and providing meaningful product reports. For example, text data can establish that a certain person should be visible in a video. Together, video-to-text (V2T) enhances situation awareness, provides situation understanding, and affords situation assessment. V2T is an example of hard (e.g., video) and soft (i.e., text) data fusion that links Level 5 User Refinement to Level 1 object tracking and characterization. A demonstrated example for multimodal text and video sensing is shown where context provides the means for associating the multimode data aligned in space and time.