ECE 5554 Computer Vision Project

Actively Interpreted AI Controlled Game

By: Aaron Kaluszka and Rutger Thomschutz


This website presents a project that involves both computer vision and artificial intelligence to give a computer the ability to emulate a human playing a computer game.  The computer vision component deals with recognizing objects and the area of play in a game, taking input in the form of images displayed on a monitor.  The artificial intelligence (AI) component reacts to events based on the output of the computer vision component.  The project has two phases with the goal of achieving at least Phase I and, with time permitting, implementing Phase II.  In either case, the game used is restricted to Dr. Mario [1], a legacy Nintendo game that shares similarities to the game Tetris.  Previous work has resulted in a similar system for the game Tetris [2].  Provided below is a high-level outline of the steps that were taken to implement Phase I and the planned steps for Phase II.


  • The game used was Dr. Mario

 Phase I

  1. Acquire image and detect the playing field – this is the area of interest displayed on the screen
    • Find the playing field
    • Determine its dimensions
  2. Detect objects currently in the playing field
    • Find the objects that are randomly placed in the field of interest at the start of the game
    • Determine type of object by color and shape
  3. Detect changes between frames in playing field
    • Determine the location of the falling object with respect to the other already placed (static) objects
    • Determine which pieces have been eliminated
  4. React to the changes in the playing field
    • Determine a logical position to place the falling object
    • Display the button that needs to be pushed by the user

Phase II

  1. Implement Phase I using images obtained by a “web-cam” from a screen displaying Dr. Mario
  2. Develop an interface from the computer to the game system so that the AI can play independently of any human input 

At the completion of the project we had successfully accomplished Phase I and had begun progress towards Phase II, thereby meeting originally proposed goals. 



About Us | ©2003 Kaluszka and Thomschutz