Screentendo - Generate levels for Super Mario Bros based on screen content.

Screentendo is a desktop application that turns a selected area of ​​the screen into a playable level for Super Mario Bros.

I did not make applications for the Cocoa app and did not use the Sprite Kit before, and this was a great exercise for me. The source code is available by reference .
')

How it works?

When you start the application, a translucent window appears that can be moved and resized. After placing a window over a certain area of ​​the screen, a click inside it will cause the application to create a level based on the content under it.

Screentendo has two main stages of work: processing the image to determine the structure of the selected area, and creating a level.

Image processing

The first step is to get the selected area of ​​the screen under the window. The following description is an example of choosing a graphic inside Google Sheets inside Safari:

The application uses the CGWindowListCopyWindowInfo API (from Quartz Window Services) to get the list of windows in the current session, in the sequence in which they appear on the screen.
From this list, the window data (id, size, etc.) is taken for the Screentendo window, and for the next window in the hierarchy (we have this Safari window).
Using window information, CGWindowListCreateImage takes a screenshot from Safari
The screenshot is cropped at the borders of the Screentendo window, using the difference between the Safari window and the Screentendo window, and the height and width of the Screentendo window.


Screentendo is above the target window (Google Sheets chart)


Cropped image

The cropped picture passes several filters before it is converted into a format that Screentendo can use to create the game level.

Motion blur - reduces overall noise and visual artifacts.


Luminance filter - calculates the average threshold of the brightness of the image, and decreases to two colors.


Pixellation filter - pixelization filter simplifies the detailing of the image and prepares it to be divided into sub-blocks.


Sub-blocking - the picture is divided into sub-blocks, the default size of 10x10 pixels.




Average block color to array - an average color is calculated for each sub-block. A two-dimensional array is created, and each sub-block, mainly consisting of black, takes the value 1, and each sub-block, mostly white, takes the value 0.

Level creation and game logic

The representation of the image as a two-dimensional array is passed to the GameScene class, which creates the game level. It passes through the array, and creates blocks of values ​​1, ignoring values ​​0.

Sprites of blocks, clouds, background and player

When the array is processed, the background, clouds, and player are added to the scene. The rest of the game is based on the basic physical principles presented in the physical engine of the Sprite Kit (player physics, collision handling, animation of flying remnants of blocks, etc.).

The application has a menu for changing the block size. Downsizing increases resolution and processing time.

Restrictions

The app is proof of concept, and it has several flaws. Image processing is still very slow, splitting the image into subblocks is slow (each subblock turns into an NSImage, which is not a very effective way to solve, but fast in implementation). It also requires a fairly strong contrast in the colors of the original image. Physics is a bit crooked - I didn’t write a Super Mario Bros emulator, but just something that would work “quite well”. Therefore, sometimes there are some ghostly peaks that I have not overcome yet.

The code is available in the github repository.