Machine Learning: Pattern Recognition/Game

Machine Learning can do pattern recognition among its many uses. This blog article will show one way how that can be accomplished using just EWB and open source tools available to you.

The resulting system is a game you can play. Look in the right hand corner to see the computer’s guess.

Try game

Professor Radu Mariescu-Istodor (#Radu) collected about 7,000 drawings from his students. He asked each to draw 8 specific objects in a web browser and submit them online. The items were:

  • Car

  • Fish

  • House

  • Tree

  • Bicycle

  • Guitar

  • Pencil

  • Clock

The drawings were raster based, ie. line segments similar to using a typical Paint program with just the paint brush and a small drawing surface 400 pixels by 400 pixels wide. These images are small enough to be equally usable on a phone or a computer browser.

Some of the collected images were incredibly ornate, some were very simplisitic. The system absorbed them all and this level of variation makes it better.

However, since students are human, some of them uploaded possibly inappropriate images, and others were confused, such as drawing a horse when asked for a house or mixing up when item they were supposed to submit. There is always some junk data, you could weed it out to improve accuracy to a small degree, however I did not bother to do that.

Then the professor generously made these drawings available to the public as training data for learning and demonstration purposes like this.

The exercise goal was to create a program which could detect which of these objects a user is attempting to draw, sort of a closed-door version of Pictionary where there is a limited selection of things you can draw.

In his course, Professor Rado achieved about 80% accuracy in guesses. I’ve upgraded the resolution, and using the same test data it seems to be even more accurate. But your actual results will vary with your drawing ability. With a little practice, you will learn how to improve your score by meeting its expectations. First we trained the computer, then it trains you!

I’ve replicated the drawing environment in EWB for you to experiment.
As you add to your drawing, it uploads the image to a server in the background and sends back what it thinks you have drawn.

I’ll explain in a nut-shell how it works. The full eight hour course is available on YouTube for free at:

YouTube Course

I recommend it as an excellent introduction to AI as well as a refresher on some JavaScript topics.

In that course Radu explains how to use and buidls a KNN (Kth nearest neighbour) model with which he eventually achieves about 60% accuracy purely in JavaScript.

He later upgrades to a neural network, where he projects the 400x400 pixel image onto a small 20x20 grid and achieved almost 80% accuracy - even though the images do not look very clear. I’ve updated the model to use 32x32 pixels and achieved a bit higher accuracy, but at the cost of some speed and having to involve a server.

I used the Rubix/PHP library to generate a neural network of the scanned images and to process your uploads. This has a few effects on the program:

  • it means you don’t have to download a large model file to the browser

  • the browser remains responsive, asking the server for results in the background while you continue to draw

  • it also creates a bit of load on my server. If this becomes too popular I may have to remove it or move to a different site.

There are three routines of importance, they take mouse events and convert them into lines:

procedure TForm1.Paint1MouseDown(Sender: TObject; Button: Integer;
                                 ShiftKey: Boolean; CtrlKey: Boolean;
                                 AltKey: Boolean; X: Integer; Y: Integer);
begin
   isDrawing := True;
   setlength( lines, length(lines) +1);
   curline := 0;
   SetLength( lines[ length(lines)-1], 1 );
   lastx := x;
   lasty := y;
   lines[ length(lines) -1][curline] := [ x, y ];

end;


procedure TForm1.Paint1MouseMove(Sender: TObject; ShiftKey: Boolean;
                                 CtrlKey: Boolean; AltKey: Boolean;
                                 X: Integer; Y: Integer);
begin
   if isDrawing then begin
      if ( lastx <> x ) or ( lasty <> y ) then begin
         inc( curline);
         lines[ length(lines) - 1 ][curline] := [ x,  y ] ; // TPoint.Create(x,y);
         lastx := x;
         lasty := y;
     end;
     update;
   end;
end;

procedure TForm1.Paint1MouseUp(Sender: TObject; Button: Integer;
                               ShiftKey: Boolean; CtrlKey: Boolean;
                               AltKey: Boolean; X: Integer; Y: Integer);
begin
  isDrawing := False;  // stop drawing
end;

And each mouse movement causes the Update function to be called.

procedure TForm1.Update;
var
  i, j : integer;
  x,y, x2, y2 : integer;
begin
  try
     paint1.Canvas.Clearrect(0,0,paint1.Width, paint1.Height);
     for i := 0 to length( lines ) -1 do begin
        x := lines[i][0][0];
        y := lines[i][0][1];
        paint1.Canvas.StrokeStyle := dsColor;
        paint1.Canvas.StrokeColor := clblack;
        paint1.Canvas.LineWidth := 2;
        paint1.Canvas.LineCapStyle := csRound;
        paint1.canvas.beginpath;
        paint1.Canvas.moveto( x, y );


        for j := 1 to length( lines[i] )-1 do begin
           x2 := lines[i][j][0];
           y2 := lines[i][j][1];
           paint1.canvas.LineTo( x2, y2 );

        end;
        paint1.Canvas.Stroke;
     end;
   except on E:Exception do
     showmessage('Update Error : ' + e.message );
   end;
   timer1.Enabled := True; // Query;
end;

The timer1 calls the network server to try to guess from the submitted line segments. Putting it into a timer makes sure we don’t request a guess more often than once per second… it slows the system down to avoid server load.