ECE 5554 Computer Vision Project

Procedure

Due to the complexity of the project, the program was implemented as a set of replaceable functions allowing for an easy progression from Phase I to Phase II.  The major components involved acquiring images and detecting the playing field, detection of the objects in the playing field, detection of changes within the playing field, and reactions to the changes within the playing field. 

Acquiring Images and Detecting the Playing Field

Before detection of the playing field could be accomplished, an initial screen capture of the game upon initialization was made.  The screen capture was written to a RAM disk [3] as a Device Independent BitMap (DIB) image [5]. The program then used the “convert” utility from ImageMagick [4] to convert this image to a Tagged Image File Format (TIFF) image [6].  The RAM disk was used in order to reduce the operational overhead incurred by the screen capture conversion.  Utilizing the captured image, the program next analyzed the image to find the playing field.  To detect the playing field, a routine searched through the TIFF image and found the grayscale and threshold value [7] for each pixel image.  This information was then used by the routine to locate the largest 3-row-by-8-column-multiple rectangle.  The size was chosen based on the observation that no viruses appeared in the top 3x8 portion of the playing field, thereby providing a consistent way to find the playing field.  (For perspective purposes, each vitamin covers a 1x2 area of the playing field).  Once the playing field is determined, the program assumes that the location of the playing field does not change for the remainder of the game.  Sequential frame captures must occur at speeds near 30 frames per second, the refresh rate of the game.

Detection of Objects Currently in the Playing Field

With the playing field determined and stored, the next step was to detect objects that were in the playing field.  This detection was very important, as the objective of the game was to destroy all the static viruses that appear at random places in the playing field upon initialization of the game. (See Background Information on the Game Dr. Mario for further details).  The playing field was divided into a 16 row by 8 column matrix.  To determine what an object on the screen represented (virus or portion of a vitamin) and what color it was (red, blue, or yellow), rules obtained from analyzing the properties of all the possible image pieces that could appear in the playing field were implemented in a routine.  These rules were based on the mean values of each color in the sub-image (block in the matrix), which may potentially contain a piece.  The red, yellow, and blue values had very distinctive mean characteristics, allowing easy detection of each piece.  Furthermore, virus pieces had a lower number of colored pixels per sub-image than vitamin pieces.  Utilizing knowledge of the red, green, and blue (RGB) means of each possible piece allowed for accurate detection of viruses and vitamins.  Since the active vitamin piece is offset by 12%, the algorithm also had to ignore the overhang while properly determining the identity of the vitamin.  This was accomplished by ignoring the 12% offset.  The mean RGB values used for piece identification are listed in Table 1.  The complete set of rules for piece identification were derived from this data and are listed in Table 2.

Table 1. Mean RGB Intensities of All Possible Pieces

Figure 2. Pieces of the Playing Field

 

Table 2. Selection Rules for Piece Detection

Identification	Selection Rules
Red color	Red mean > 1.5 * Green mean Red mean > 1.5 * Blue mean
Yellow color	Red mean > 1.5 * Blue mean Green mean > 1.5 * Blue mean
Blue color	Green mean > 1.5 * Red Mean Blue mean * 1.5 > * Red Mean
Virus piece	40 <= Mean of selected values < 134
Vitamin piece	Mean of seleted values < 134

Detect Changes between Frames in the Playing Field

With the field of play determined and objects in the playing field located and recognized the next step was to detect changes between frames in the playing field.  By detecting differences between two consecutive frames, moving objects (the vitamins) can be followed and moved, to destroy rows/columns of vitamins and viruses.  The following iteration of the program would then be able to detect any changes that had occurred since the last iteration and thereby react appropriately.  This step is crucial for the AI to be effective and play the game successfully.  

React to the Changes in the Playing Field

The final goal and major portion of the program was to react to the changes that were found in the playing field.  Actions are represented by moving vitamins horizontally with the ultimate goal of eliminating all viruses present in the playing field.  As AI was not the focus of this project, only a very rudimentary AI was developed.  The AI does not take into account rotation capabilities, nor does it look for horizontal matches.  A much better algorithm would entail finding the best path from the current vitamin position to its target.The priority to which the AI reacts and places a vitamin is determined as follows with the rules shown in descending order of priority:

1.      Have vitamin fall on top of a virus of same color as vitamin

2.      Have vitamin fall on top of the biggest stack of vitamins of same color

3.      Have vitamin fall on top of the smallest stack of vitamins (preferably on top of no other pieces)

 

Results>>