Remote control of Fceux emulator using Python
In the article I will describe how to make the NES emulator managed remotely, and the server for remote sending commands to it.
Some emulators of various gaming consoles, including Fceux , allow you to write and run custom scripts on Lua. But Lua is a bad language for writing serious programs. It is rather a language for calling functions written in C. Emulator authors use it only because of the lightness and ease of embedding. Accurate emulation requires a lot of processor resources, and previously the emulation speed was one of the main goals of the authors, and the possibility of scripting actions, if remembered, is far from being the first.
')
Now the average processor power is enough to emulate NES, why not use powerful scripting languages ​​like emulator or javascript in emulators?
Unfortunately, none of the popular NES emulators have the ability to use these or other languages. I found only a little-known Nintaco project, which is also based on the Fceux core, which is for some reason rewritten in Java. Then I decided to add the ability to write scripts in Python to control the emulator myself.
My result is Proof-of-Concept emulator control capabilities, it does not pretend to speed or reliability, but it works. I did it for myself, but since the question of how to manage the emulator using scripts is quite common , I put the source code on the github .
The Fceux emulator already includes several Lua libraries included in it as compiled code . One of them is LuaSocket . It is poorly documented, but I managed to find an example of working code among the collection of Xkeeper0 scripts . He used sockets to control the emulator via Mirc. Actually, the code that opens the tcp-socket:
This is a low-level socket that receives and sends data by 1 byte.
In the Fceux emulator, the main loop of the Lua script looks like this:
And check data from the socket:
The code is quite simple - reading data from the socket is done, and if the next command was found, then it is parsed and executed. Parsing and execution are organized using Coroutine (coroutines) - this is a powerful concept of the Lua language for suspending and continuing code execution.
And one more important thing about Lua-scripting in Fceux - emulation can be temporarily stopped from the script. How to organize the continuation of the execution of Lua-code and re-launch it with a command obtained from the socket? That would be impossible, but there is a poorly documented opportunity to call the Lua code even when emulation is stopped (thanks to feos for bringing it):
Using it, you can stop and continue emulation inside passiveUpdate - this is how you can organize installation of the emulator breakpoint via a socket.
I use a very simple text based RPC protocol based on JSON. The server serializes the function name and arguments into a JSON string and sends it via a socket. Further, the code execution stops until the emulator responds with the command completion line. The response will contain the fields " FUNCTIONNAME_finished " and the result of the function.
The idea is implemented in the syncCall class:
Using this class, the Fceux emulator Lua methods can be wrapped in Python classes:
And then called verbatim the same way as from Lua:
You can register callbacks in Lua - functions that will be called when a certain condition is met. We can transfer this behavior to the server in Python using the following trick. First, we save the identifier of a function callback written in Python, and pass it to the Lua code:
The Lua code also saves this identifier and registers the usual Lua-Kolbek, which will transfer control to the Python code. Next, a separate thread is created in the Python code, which deals only with checking whether the call command from Lua was accepted:
The last step is that after the Python-kabek is executed, control returns to Lua using the " CALLBACKNAME_finished " command to inform the emulator that the callback is complete.
Everything, you can send commands from the Jupyter laptop in the browser directly to the Fceux emulator.
You can execute all the lines of the example notebook in sequence and observe the result of execution in the emulator.
Full example:
https://github.com/spiiin/fceux_luaserver/blob/master/FceuxPythonServer.py.ipynb
It contains simple functions like reading memory:
More complex examples with creating callbacks:
And a script for a particular game that allows you to move enemies from Super Mario Bros. using the mouse:
Laptop runtime video:
The script is not foolproof and is not optimized for execution speed - it would be better to use the binary RPC protocol instead of the text one and group the messages together, but my implementation does not need to be compiled. The script can switch execution contexts from Lua to Python and back 500-1000 times per second on my laptop. This is enough for almost any application, except for specific cases of pixel-by-pixel or line-by-line debugging of a video processor, but Fceux still does not allow performing such operations from Lua, so it does not matter.
Possible application ideas:
I used:
Fceux - www.fceux.com/web/home.html
This is a classic NES emulator, and most people use it. It has not been updated for a long time, and is not the best in features, but it remains the default emulator for many romhackers. Also, I chose it because the support for Lua sockets is integrated into it, and there is no need to connect it myself.
Json.lua - github.com/spiiin/json.lua
This is JSON implementation on pure Lua. I chose it because I wanted to make an example that does not require compiling code. But I still had to fork the library, because some of the libraries built into Fceux overloaded the library function tostring and broke serialization (my rejected pull request to the author of the original library).
Python 3 - www.python.org
Fceux Lua server opens a tcp socket and listens to commands received from it. The server that sends commands to the emulator can be implemented in any language. I chose Python for its “Battery included” philosophy - most modules are included in the standard library (work with sockets and JSON as well). Python also has a library of working with neural networks, and I want to try using them to create bots in NES games.
Jupyter Notebook - jupyter.org
Jupyter Notebook is a very cool environment for interactively executing Python code. With it, you can write and execute commands in a table editor inside the browser. It is also good for creating presentable examples.
Dexpot - www.dexpot.de
I used this virtual desktop manager to pin the emulator window on top of others. This is very useful when deploying a server in full screen for instantly tracking changes in the emulator window. Regular Windows tools do not allow to organize the fastening of the window on top of others.
Actually, the project repository .
Nintaco - NES emulator in Java with remote control
Xkeeper0 emu-lua collection - a collection of various Lua scripts
Mesen is a modern C # NES emulator with powerful scripting capabilities for Lua. So far, without the support of sockets and remote control.
CadEditor is my universal level editor project for NES and other platforms, as well as powerful tools for researching games. I use the script and server described in the post in order to explore the games and add them to the editor.
I would appreciate feedback, testing and attempts to use the script.
Why do you need it?
Some emulators of various gaming consoles, including Fceux , allow you to write and run custom scripts on Lua. But Lua is a bad language for writing serious programs. It is rather a language for calling functions written in C. Emulator authors use it only because of the lightness and ease of embedding. Accurate emulation requires a lot of processor resources, and previously the emulation speed was one of the main goals of the authors, and the possibility of scripting actions, if remembered, is far from being the first.
')
Now the average processor power is enough to emulate NES, why not use powerful scripting languages ​​like emulator or javascript in emulators?
Unfortunately, none of the popular NES emulators have the ability to use these or other languages. I found only a little-known Nintaco project, which is also based on the Fceux core, which is for some reason rewritten in Java. Then I decided to add the ability to write scripts in Python to control the emulator myself.
My result is Proof-of-Concept emulator control capabilities, it does not pretend to speed or reliability, but it works. I did it for myself, but since the question of how to manage the emulator using scripts is quite common , I put the source code on the github .
How does it work
On the emulator side
The Fceux emulator already includes several Lua libraries included in it as compiled code . One of them is LuaSocket . It is poorly documented, but I managed to find an example of working code among the collection of Xkeeper0 scripts . He used sockets to control the emulator via Mirc. Actually, the code that opens the tcp-socket:
function connect(address, port, laddress, lport) local sock, err = socket.tcp() if not sock then return nil, err end if laddress then local res, err = sock:bind(laddress, lport, -1) if not res then return nil, err end end local res, err = sock:connect(address, port) if not res then return nil, err end return sock end sock2, err2 = connect("127.0.0.1", 81) sock2:settimeout(0) --it's our socket object print("Connected", sock2, err2) This is a low-level socket that receives and sends data by 1 byte.
In the Fceux emulator, the main loop of the Lua script looks like this:
function main() while true do -- passiveUpdate() --, emu.frameadvance() -- end end And check data from the socket:
function passiveUpdate() local message, err, part = sock2:receive("*all") if not message then message = part end if message and string.len(message)>0 then --print(message) local recCommand = json.decode(message) table.insert(commandsQueue, recCommand) coroutine.resume(parseCommandCoroutine) end end The code is quite simple - reading data from the socket is done, and if the next command was found, then it is parsed and executed. Parsing and execution are organized using Coroutine (coroutines) - this is a powerful concept of the Lua language for suspending and continuing code execution.
And one more important thing about Lua-scripting in Fceux - emulation can be temporarily stopped from the script. How to organize the continuation of the execution of Lua-code and re-launch it with a command obtained from the socket? That would be impossible, but there is a poorly documented opportunity to call the Lua code even when emulation is stopped (thanks to feos for bringing it):
gui.register(passiveUpdate) --undocumented. this function will call even if emulator paused Using it, you can stop and continue emulation inside passiveUpdate - this is how you can organize installation of the emulator breakpoint via a socket.
Server side commands
I use a very simple text based RPC protocol based on JSON. The server serializes the function name and arguments into a JSON string and sends it via a socket. Further, the code execution stops until the emulator responds with the command completion line. The response will contain the fields " FUNCTIONNAME_finished " and the result of the function.
The idea is implemented in the syncCall class:
class syncCall: @classmethod def waitUntil(cls, messageName): """cycle for reading data from socket until needed message was read from it. All other messages will added in message queue""" while True: cmd = messages.parseMessages(asyncCall.waitAnswer(), [messageName]) #print(cmd) if cmd != None: if len(cmd)>1: return cmd[1] return @classmethod def call(cls, *params): """wrapper for sending [functionName, [param1, param2, ...]] to socket and wait until client return [functionName_finished, [result1,...]] answer""" sender.send(*params) funcName = params[0] return syncCall.waitUntil(funcName + "_finished") Using this class, the Fceux emulator Lua methods can be wrapped in Python classes:
class emu: @classmethod def poweron(cls): return syncCall.call("emu.poweron") @classmethod def pause(cls): return syncCall.call("emu.pause") @classmethod def unpause(cls): return syncCall.call("emu.unpause") @classmethod def message(cls, str): return syncCall.call("emu.message", str) @classmethod def softreset(cls): return syncCall.call("emu.softreset") @classmethod def speedmode(cls, str): return syncCall.call("emu.speedmode", str) And then called verbatim the same way as from Lua:
# : emu.poweron() Callback methods
You can register callbacks in Lua - functions that will be called when a certain condition is met. We can transfer this behavior to the server in Python using the following trick. First, we save the identifier of a function callback written in Python, and pass it to the Lua code:
class callbacks: functions = {} callbackList = [ "emu.registerbefore_callback", "emu.registerafter_callback", "memory.registerexecute_callback", "memory.registerwrite_callback", ] @classmethod def registerfunction(cls, func): if func == None: return 0 hfunc = hash(func) callbacks.functions[hfunc] = func return hfunc @classmethod def error(cls, e): emu.message("Python error: " + str(e)) @classmethod def checkAllCallbacks(cls, cmd): #print("check:", cmd) for callbackName in callbacks.callbackList: if cmd[0] == callbackName: hfunc = cmd[1] #print("hfunc:", hfunc) func = callbacks.functions.get(hfunc) #print("func:", func) if func: try: func(*cmd[2:]) #skip function name and function hash and save others arguments except Exception as e: callbacks.error(e) pass #TODO: thread locking sender.send(callbackName + "_finished") The Lua code also saves this identifier and registers the usual Lua-Kolbek, which will transfer control to the Python code. Next, a separate thread is created in the Python code, which deals only with checking whether the call command from Lua was accepted:
def callbacksThread(): cycle = 0 while True: cycle += 1 try: cmd = messages.parseMessages(asyncCall.waitAnswer(), callbacks.callbackList) if cmd: #print("Callback received:", cmd) callbacks.checkAllCallbacks(cmd) pass except socket.timeout: pass time.sleep(0.001) The last step is that after the Python-kabek is executed, control returns to Lua using the " CALLBACKNAME_finished " command to inform the emulator that the callback is complete.
How to run an example
- You must have working Python 3 and Jupyter Notebook in the system. You need to run jupyter with the command
jupyter notebook - Open the FceuxPythonServer.py.ipynb laptop and run the first line
- Now you have to run the Fceux emulator, open the ROM file in it (I use the game Castlevania (U) (PRG0) [!]. Nes in my example) and run a Lua script called fceux_listener.lua . It should connect to the server running on the Jupyter laptop.
These actions can be performed using the command line:fceux.exe -lua fceux_listener.lua "Castlevania (U) (PRG0) [!].nes" - Now switch back to Jupyter Notebook. You should see a message about successful connection with the emulator:
Everything, you can send commands from the Jupyter laptop in the browser directly to the Fceux emulator.
You can execute all the lines of the example notebook in sequence and observe the result of execution in the emulator.
Full example:
https://github.com/spiiin/fceux_luaserver/blob/master/FceuxPythonServer.py.ipynb
It contains simple functions like reading memory:
More complex examples with creating callbacks:
And a script for a particular game that allows you to move enemies from Super Mario Bros. using the mouse:
Laptop runtime video:
Limitations and Applications
The script is not foolproof and is not optimized for execution speed - it would be better to use the binary RPC protocol instead of the text one and group the messages together, but my implementation does not need to be compiled. The script can switch execution contexts from Lua to Python and back 500-1000 times per second on my laptop. This is enough for almost any application, except for specific cases of pixel-by-pixel or line-by-line debugging of a video processor, but Fceux still does not allow performing such operations from Lua, so it does not matter.
Possible application ideas:
- As an example of the implementation of similar control for other emulators and languages
- Study games
- Adding cheats or features to organize TAS passages
- Insert or retrieve data and code into games
- Empowering emulators - writing debuggers, writing scripts and viewing passages, script libraries, game editors
- Network game, game control using mobile devices, remote services, joypad or other control devices, saving and patches in cloud services
- Cross emulation features
- Using libraries of Python or other languages ​​for data analysis and game management (creating bots)
Technology stack
I used:
Fceux - www.fceux.com/web/home.html
This is a classic NES emulator, and most people use it. It has not been updated for a long time, and is not the best in features, but it remains the default emulator for many romhackers. Also, I chose it because the support for Lua sockets is integrated into it, and there is no need to connect it myself.
Json.lua - github.com/spiiin/json.lua
This is JSON implementation on pure Lua. I chose it because I wanted to make an example that does not require compiling code. But I still had to fork the library, because some of the libraries built into Fceux overloaded the library function tostring and broke serialization (my rejected pull request to the author of the original library).
Python 3 - www.python.org
Fceux Lua server opens a tcp socket and listens to commands received from it. The server that sends commands to the emulator can be implemented in any language. I chose Python for its “Battery included” philosophy - most modules are included in the standard library (work with sockets and JSON as well). Python also has a library of working with neural networks, and I want to try using them to create bots in NES games.
Jupyter Notebook - jupyter.org
Jupyter Notebook is a very cool environment for interactively executing Python code. With it, you can write and execute commands in a table editor inside the browser. It is also good for creating presentable examples.
Dexpot - www.dexpot.de
I used this virtual desktop manager to pin the emulator window on top of others. This is very useful when deploying a server in full screen for instantly tracking changes in the emulator window. Regular Windows tools do not allow to organize the fastening of the window on top of others.
Links
Actually, the project repository .
Nintaco - NES emulator in Java with remote control
Xkeeper0 emu-lua collection - a collection of various Lua scripts
Mesen is a modern C # NES emulator with powerful scripting capabilities for Lua. So far, without the support of sockets and remote control.
CadEditor is my universal level editor project for NES and other platforms, as well as powerful tools for researching games. I use the script and server described in the post in order to explore the games and add them to the editor.
I would appreciate feedback, testing and attempts to use the script.
Source: https://habr.com/ru/post/435428/
All Articles