Writing a bot for the game of Balls 2.0

Recently I stumbled upon a simple toy where you need to shoot a ball in groups of the same color. Although I play games very rarely, I sat with her for about 30 minutes.
I wanted to automate this process. Knowledge of the game is not required, but there are many such games.
I describe the process of writing a bot to this game.



The idea was as follows:
Step 1. we receive the picture from the screen
Step 2. Transfer the colors of the balls into the matrix
Step 3. Calculating the position and clicking the mouse to perform a move

For writing the program, I hesitated a bit between Qt and Delphi, but since I wanted to do everything quickly, I decided to do it in Delphi.
')
Now you can implement Step 1.
To get the picture, set the approximate position of the window with the game and copy this section in the TImage, previously located on the form.

procedure MakeFrame(); var bmp:TBitmap; rect:TRect; begin rect.Left := 50; rect.Top:= 150; rect.Right:=550; // Screen.Width rect.Bottom:=550; // Screen.Height bmp := TBitmap.Create; bmp.Width := Screen.Width; bmp.Height := Screen.Height; BitBlt(bmp.Canvas.Handle, 0, 0, rect.Right, rect.Bottom, GetDC(0), rect.Left, rect.Top, SRCCOPY); Form1.Image1.Width := rect.Right; Form1.Image1.Height := rect.Bottom; Form1.Image1.Picture.Assign(bmp); bmp.Free; end; 

If you put this code in the timer, moving the window with the game you can arrange it so that it will copy the image of the game.

Step2. Need to know the color of the balls. As you can see, there are only 3 options, straight RGB.
The height between the rows and the diameters of the balls are constant, so we can make a grid of the position of the balls.

 for I:=0 to rows do begin offset:=10; //  if (I mod 2) <> 0 then offset:=offset+(36 div 2); for J:=0 to cols do begin mat[i,j].Y := 10+Round(I*36); //   mat[i,j].X := offset+Round(J*36); //  (        ) if (I=rows) and (J=cols) then begin mat[i,j].Y := 519; mat[i,j].X := 262; end; //    end; end; 

To find out the color of the ball, compare its pixel with the threshold value of each of the RGB colors.

  rgb:=Form1.Image1.Canvas.Pixels[ mat[i,j].X, mat[i,j].Y-4 ]; mat[i,j].color := 0; mat[i,j].summa := 0; if GetRValue(rgb) > 230 then mat[i,j].color:=1; if GetGValue(rgb) > 230 then mat[i,j].color:=2; if GetBValue(rgb) > 230 then mat[i,j].color:=3; 

And to make sure that everything is in order, we will draw on the ball a small rectangle with the resulting color.

  if mat[i,j].color = 0 then Form1.Image1.Canvas.Brush.Color := clWhite; if mat[i,j].color = 1 then Form1.Image1.Canvas.Brush.Color := clRed; if mat[i,j].color = 2 then Form1.Image1.Canvas.Brush.Color := clGreen; if mat[i,j].color = 3 then Form1.Image1.Canvas.Brush.Color := clBlue; Form1.Image1.Canvas.Rectangle( mat[i,j].X-5, mat[i,j].Y-5, mat[i,j].X+5, mat[i,j].Y+5 ); 

It turns out such a pretty picture.


At this work with the grid ends and begins working with the resulting matrix.

Step 3.
Let's check all the balls for the opportunity to shoot into it, for this you need to check that we do not encounter another ball on the way to it.
Verification occurs through nested loops (by balls and for each one we check the others)
The verification condition itself is obtained with a system of three inequalities.

{A = 1, B = (x0-y0) / (y1-y0), C = -x0-By0}
| Ax3 + By3 + C | / sqrt (A ^ 2 + b ^ 2) <= radius

  ///      for I:=0 to rows-1 do for J:=0 to cols do begin if (mat[I,J].color = 0) and (I <> 0) then continue; mat[i,j].allow:=1; //    LineMaxX:=mat[I,J].X; LineMaxY:=mat[I,J].Y; //    LineMinX:=mat[rows,cols].X; LineMinY:=mat[rows,cols].Y; mat[I,J].dist := sqrt( sqr(mat[I,J].X-mat[rows,cols].X)+ sqr(mat[I,J].Y-mat[rows,cols].Y)); for II:=I+1 to rows-1 do for JJ:=0 to cols do begin if mat[II,JJ].color = 0 then continue; //    LineMiddleX:=mat[II,JJ].X; LineMiddleY:=mat[II,JJ].Y; //        //      ka:=1; kb:=(LineMinX-LineMaxX)/(LineMaxY-LineMinY); kc:=-LineMinX-kb*LineMinY; kz:= abs(ka*LineMiddleX + kb*LineMiddleY+kc)/ sqrt(sqr(ka)+sqr(kb)); if kz < 39 then mat[i,j].allow:=0; //Form1.Memo1.Lines.Add(FloatToStr(kz)); end; 

For clarity, we draw lines to the balls, which can be bullet.

 if mat[i,j].allow = 1 then begin Form1.Image1.Canvas.Pen.Width:= 2; Form1.Image1.Canvas.Pen.Color:= clWhite; Form1.Image1.Canvas.MoveTo(Round(LineMinX),Round(LineMinY)); Form1.Image1.Canvas.LineTo(Round(LineMaxX),Round(LineMaxY)); end; 

It is necessary to choose from the allowed ball with the same color.

  for I:=0 to rows-1 do for J:=0 to cols do begin if mat[i,j].allow = 1 then begin Form1.Image1.Canvas.Pen.Width:= 2; Form1.Image1.Canvas.Pen.Color:= clWhite; Form1.Image1.Canvas.MoveTo(Round(LineMinX),Round(LineMinY)); Form1.Image1.Canvas.LineTo(Round(LineMaxX),Round(LineMaxY)); //  if mat[i,j].color=mat[rows,cols].color //    then begin MouseX:=mat[i,j].X+50; MouseY:=mat[i,j].Y+250; //  +    break; end; end; end; 

And make a mouse click on the screen through WinApi

  if Form1.CheckBox2.Checked then begin GetCursorPos(MouseL); SetCursorPos(MouseX,MouseY); mouse_event(MOUSEEVENTF_LEFTDOWN,MouseX,MouseY,0,0);// -    mouse_event(MOUSEEVENTF_LEFTUP,MouseX,MouseY,0,0);// -    SetCursorPos(MouseL.X,MouseL.Y); end; 

It remains to do everything in the timer and go drink tea.

Video of the program


You can improve the algorithm, for example, find groups with a large number of balls or search for lines that cut off parts of the field.

Function names and variables may not be named beautifully, but this can be fixed.
If you need source , they are here .