The MIT AI Laboratory has developed as part of the DARPA ITO HCI program an "Intelligent Room" combining active vision systems, speech recognition and language understanding (provided by START). We have demonstrated the use of these combined technologies in simulated crisis management scenarios; an illustrative interaction involves pointing at a location on a map from a Web page projected on the wall and asking (in spoken English) how many planes are near this point.
We believe that the natural modality for acquiring Know-How often involves such mixed use of language and vision processing. The visual system is remarkably powerful at understanding structural relationships which explains our seemingly constant need to draw box and arrow diagrams of many sorts. Procedures are drawn as flow charts, larger software systems use architectural notations involving components and connectors, evidence interpretation processes are drawn as "influence diagrams" and the like. However, the content within this structure is much better described linguistically.
In the Intelligent Room, we have the laboratory to explore how best to use such facilities in support of knowledge acquisition. We will develop and demonstrate techniques within the Room environment for enabling extremely flexible and natural interactions between HAWK and its users. We anticipate that eventually most users will have working environments similar to that in the Intelligent Room: The processing power necessary to support it is certainly within reach, the cameras and speech recognition technology are not expensive; the major roadblock is the lack of network bandwidth necessary to support broad-area "Room-like" interactions. But this too is on the horizon; DARPA ITO's BIT program has already developed the core technologies required. DoD is likely to be one of the first institutions capable of using such distributed intelligent rooms; it has both the need and the capabilities. We think that explorations of how to exploit this technology for collaborative knowledge acquisition are well motivated at this time.