Computer Vision for Computer Games

 

 


We developed vision-based interfaces for several computer games. These allow the player to move or gesture to affect the game, instead of pressing buttons. The characters in the game may imitate those motions, or respond accordingly. These interfaces allow for engaging, exciting games.
      The system uses Mitsubishi Electric's Artificial Retina chip, a low-cost image sensor that can also perform image processing. The system was demonstrated at COMDEX in November, 1996, and it received the General Manager's Award from Headquarters R&D in Jan., 1998.
      This prototype led to a business arrangement with Nintendo where they combined MELCO's Artificial Retina chip with their GameBoy handheld game. 700,000 units have been sold in Japan; they went on sale in North America in June, 1998.
     







 

 

Background and objectives: Vision can be a powerful interface device for computers. There is the potential to sense body position, head orientation, direction of gaze, pointing commands, and gestures. Such unencumbered interaction can make computers easier to use.
      The application of vision to computer games poses special challenges. The response time must be very fast, while the total hardware cost must be very low. We sought to meet these challenges using special hardware, and fast and simple algorithms.

Technical discussion: The low-cost, real-time control required for the computer game is very challenging. Fortunately, the computer game itself makes the computer vision task easier. The game restricts the possible visual interpretations. For example, if the game context requires that the player is running in place, the vision system may only need to ascertain how fast the player runs. This is a much easier vision problem to solve than a full 3D reconstruction of a player's unknown motion. MELCO's low-cost Artifical Retina chip is well-suited to this problem, since it is both a detector and a simple image processor.
      We used a variety of simple, fast techniques to measure the desired features of the players actions. Some game applications required tracking the position or orientation of a body or hand that is prominent in the visual field of the camera. Image moments, which are fast to compute, provide a very coarse summary of global averages of orientation and position. With uncluttered backgrounds, or using temporal difference information, this allows tracking the position or orientation of the game player, or of his hand, if it is close to the camera.
      We are able to calculate moments particularly quickly using the Artificial Retina chip. One of the image processing capabilities of the chip is to calculate image projections quickly. These projections can be used in a fast calculation of the image moments.
      We also used a fast motion analysis technique, developed at Mitsubishi Electric's Advanced Technology R&D Center, to analyze gestures of the game players. Throw timing and running rate parameters of the game player can be measured from global averages of the optical flow, which the motion analysis method calculates very quickly.
     


Computer Vision for Computer Interaction
SIGGRAPH Computer Graphics magazine, November 1999
W. T. Freeman, P. A. Beardsley, H. Kage, K. Tanaka, K. Kyuma, C. D. Weissman.
Also available as MERL-TR99-36.

Computer Vision for Interactive Computer Graphics
IEEE Computer Graphics and Applications, Vol. 18, No. 3, May-June 1998
W. T. Freeman, D. B. Anderson, P. A. Beardsley, C. N. Dodge, M. Roth, C. D. Weissman, W. S. Yerazunis, H. Kage, K. Kyuma, Y. Miyake, and K. Tanaka.
Also available as MERL-TR99-02.

Computer Vision for Computer Games
In 2nd International Conference on Automatic Face and Gesture Recognition, Killington, VT, USA. IEEE.
W. T. Freeman, K. Tanaka, J. Ohta, K. Kyuma
Also available as MERL-TR96-35.