Universal metasystem in C ++
Hi, Habrhabr!
I want to share my experience in developing a metasystem for C ++ and embedding various scripting languages.
Relatively recently, I began to write my game engine. Of course, as in any good engine, there was a question about embedding a scripting language, and even better a few. Of course, there are already enough tools for embedding a specific language (for example, luabind for Lua, boost.python for Python), and I didn't want to reinvent my own bike.
I started by building simple and nimble Lua, and used luabind for binding. And he looks really nice.
It is easy to read, the class is registered simply and without problems. But this solution is exclusively for Lua.
Inspired by the Unity scripting system, I realized that there should definitely be several languages in the system, as well as the possibility of their interaction with each other. And here such tools as luabind give up slack: for the most part they are written using C ++ templates and generate code only for a specific language. Each class must be registered in each system. At the same time, you need to add a lot of header files and manually enter everything into the templates.
')
But I want to have a common type database for all languages. And also the ability to download information about the types of plug-ins directly in runtime. For these purposes, binding libraries are not suitable. Need a real metasystem. But here, too, it was not all smooth. Finished libraries were rather cumbersome and inconvenient. There are some very elegant solutions, but they draw additional dependencies and require the use of special tools (for example, Qt moc or gccxml). There are, of course, quite nice options, such as, for example, the Camp reflection library. It looks almost the same as luabind:
True performance of such "beautiful" solutions leaves much to be desired. Of course, like any “normal” programmer, I decided to write my metasystem. This is how the uMOF library appeared.
uMOF is a cross-platform open source library for metaprogramming. Conceptually, it resembles Qt, but it is implemented using templates, which Qt itself refused at the time. They did this for code readability. And so really faster and more compact. But using the moc compiler makes it totally dependent on Qt. This is not always justified.
Let's get to the point nevertheless. To make the metadata available to the user in the class of the Object inheritor, you need to register OBJECT macros with inheritance hierarchy and EXPOSE to declare functions. After that, the class API is available, which stores information about the class, functions, and public properties.
While the definition of meta-information is invasive, an external variant is also planned for a more convenient wrapping of third-party code.
Due to the use of advanced templates, uMOF turned out to be very fast, while being quite compact. This also led to some restrictions: C ++ 11 features are actively used, not all compilers are suitable (for example, to compile on Windows, you need the latest Visual C ++ November CTP). Also, not everyone will like the use of templates in the code, so everything is wrapped up in macros. Meanwhile, the macros hide a large number of templates and the code looks pretty neat.
In order not to be unfounded I cite the results of benchmarks.
I compared the metasystems by three parameters: compile / link time, executable file size, and function call time in a loop. As a reference, I took an example with native function calls. The subjects were tested on Windows under Visual Studio 2013.
For clarity, the same thing in the form of graphs.
I also looked at a few more libraries:
But they did not fall into the role of subjects for various reasons. Boost.Mirror and XcppRefl look promising, but are still under active development. Reflex requires GCCXML, I did not find any adequate replacement for Windows. XRtti again in the current release does not support Windows.
So, how it all works. The speed and compactness of the library give templates with functions as arguments, as well as variadic templates. All meta information by type is organized as a set of static tables. There is no additional load in runtime. A simple structure in the form of an array of pointers does not give the code much swell.
An important role in efficiency is also played by the class Any, which allows you to rather compactly store types and information about them. The basis was the hold_any class from the boost spirit library. Templates are also actively used here to effectively wrap types. Types smaller than a pointer are stored in size directly in void *, for larger types the pointer refers to an object of type.
RTTI also had to be abandoned, too slowly. Type checking is done solely by comparing pointers to a type table. All type modifiers are pre-cleared, otherwise, for example, int and const int will be different types. But in fact, their size is single, and in general it is the same type.
Embedding scripting languages has become easy and enjoyable. For example, for Lua, it suffices to define a generic call function that checks the number of arguments and their types and, of course, calls the function itself. Binding is also not difficult. For each function in Lua, it is enough to save the MetaMethod in upvalue. By the way, all objects in uMOF are “thin”, that is, just a wrapper over a pointer that refers to an entry in a static table. Therefore, you can copy them without concern about performance.
An example of a Lua binding:
So, what we have:
Advantages of uMOF:
Disadvantages of uMOF:
The library is still quite raw, I would like to do a lot of interesting things - functions of variable arity (read, default parameters), non-invasive type registration, signals about changes in object properties. And all this will surely appear, because the method has shown very good results.
Thank you all for your attention. I hope the library will be useful for someone.
The project can be found on the link . Write your feedback and recommendations in the comments.
I want to share my experience in developing a metasystem for C ++ and embedding various scripting languages.
Relatively recently, I began to write my game engine. Of course, as in any good engine, there was a question about embedding a scripting language, and even better a few. Of course, there are already enough tools for embedding a specific language (for example, luabind for Lua, boost.python for Python), and I didn't want to reinvent my own bike.
I started by building simple and nimble Lua, and used luabind for binding. And he looks really nice.
See for yourself
class_<BaseScript, ScriptComponentWrapper>("BaseComponent") .def(constructor<>()) .def("start", &BaseScript::start, &ScriptComponentWrapper::default_start) .def("update", &BaseScript::update, &ScriptComponentWrapper::default_update) .def("stop", &BaseScript::stop, &ScriptComponentWrapper::default_stop) .property("camera", &BaseScript::getCamera) .property("light", &BaseScript::getLight) .property("material", &BaseScript::getMaterial) .property("meshFilter", &BaseScript::getMeshFilter) .property("renderer", &BaseScript::getRenderer) .property("transform", &BaseScript::getTransform) It is easy to read, the class is registered simply and without problems. But this solution is exclusively for Lua.
Inspired by the Unity scripting system, I realized that there should definitely be several languages in the system, as well as the possibility of their interaction with each other. And here such tools as luabind give up slack: for the most part they are written using C ++ templates and generate code only for a specific language. Each class must be registered in each system. At the same time, you need to add a lot of header files and manually enter everything into the templates.
')
But I want to have a common type database for all languages. And also the ability to download information about the types of plug-ins directly in runtime. For these purposes, binding libraries are not suitable. Need a real metasystem. But here, too, it was not all smooth. Finished libraries were rather cumbersome and inconvenient. There are some very elegant solutions, but they draw additional dependencies and require the use of special tools (for example, Qt moc or gccxml). There are, of course, quite nice options, such as, for example, the Camp reflection library. It looks almost the same as luabind:
Example
camp::Class::declare<MyClass>("FunctionAccessTest::MyClass") // ***** constant value ***** .function("f0", &MyClass::f).callable(false) .function("f1", &MyClass::f).callable(true) // ***** function ***** .function("f2", &MyClass::f).callable(&MyClass::b1) .function("f3", &MyClass::f).callable(&MyClass::b2) .function("f4", &MyClass::f).callable(boost::bind(&MyClass::b1, _1)) .function("f5", &MyClass::f).callable(&MyClass::m_b) .function("f6", &MyClass::f).callable(boost::function<bool (MyClass&)>(&MyClass::m_b)); } True performance of such "beautiful" solutions leaves much to be desired. Of course, like any “normal” programmer, I decided to write my metasystem. This is how the uMOF library appeared.
Meet uMOF
uMOF is a cross-platform open source library for metaprogramming. Conceptually, it resembles Qt, but it is implemented using templates, which Qt itself refused at the time. They did this for code readability. And so really faster and more compact. But using the moc compiler makes it totally dependent on Qt. This is not always justified.
Let's get to the point nevertheless. To make the metadata available to the user in the class of the Object inheritor, you need to register OBJECT macros with inheritance hierarchy and EXPOSE to declare functions. After that, the class API is available, which stores information about the class, functions, and public properties.
Example
class Test : public Object { OBJECT(Test, Object) EXPOSE(Test, METHOD(func), METHOD(null), METHOD(test) ) public: Test() = default; float func(float a, float b) { return a + b; } int null() { return 0; } void test() { std::cout << "test" << std::endl; } }; Test t; Method m = t.api()->method("func(int,int)"); int i = any_cast<int>(m.invoke(&t, args)); Any res = Api::invoke(&t, "func", {5.0f, "6.0"}); While the definition of meta-information is invasive, an external variant is also planned for a more convenient wrapping of third-party code.
Due to the use of advanced templates, uMOF turned out to be very fast, while being quite compact. This also led to some restrictions: C ++ 11 features are actively used, not all compilers are suitable (for example, to compile on Windows, you need the latest Visual C ++ November CTP). Also, not everyone will like the use of templates in the code, so everything is wrapped up in macros. Meanwhile, the macros hide a large number of templates and the code looks pretty neat.
In order not to be unfounded I cite the results of benchmarks.
Test results
I compared the metasystems by three parameters: compile / link time, executable file size, and function call time in a loop. As a reference, I took an example with native function calls. The subjects were tested on Windows under Visual Studio 2013.
| Framework | Compile / Link time, ms | Executable size, KB | Call time spent *, ms |
|---|---|---|---|
| Native | 371/63 | 12 | 2 (45 **) |
| uMOF | 406/78 | 18 | 359 |
| Camp | 4492/116 | 66 | 6889 |
| Qt | 1040/80 (129 ***) | 15 | 498 |
| cpgf | 2514/166 | 71 | 1184 |
Footnotes
* 10.000.000 calls
** Force no inlining
*** Meta object compiler
** Force no inlining
*** Meta object compiler
For clarity, the same thing in the form of graphs.
I also looked at a few more libraries:
- Boost.Mirror;
- XcppRefl;
- Reflex;
- XRtti.
But they did not fall into the role of subjects for various reasons. Boost.Mirror and XcppRefl look promising, but are still under active development. Reflex requires GCCXML, I did not find any adequate replacement for Windows. XRtti again in the current release does not support Windows.
What is under the hood
So, how it all works. The speed and compactness of the library give templates with functions as arguments, as well as variadic templates. All meta information by type is organized as a set of static tables. There is no additional load in runtime. A simple structure in the form of an array of pointers does not give the code much swell.
Sample Method Description Template
template<typename Class, typename Return, typename... Args> struct Invoker<Return(Class::*)(Args...)> { typedef Return(Class::*Fun)(Args...); inline static int argCount() { return sizeof...(Args); } inline static const TypeTable **types() { static const TypeTable *staticTypes[] = { Table<Return>::get(), getTable<Args>()... }; return staticTypes; } template<typename F, unsigned... Is> inline static Any invoke(Object *obj, F f, const Any *args, unpack::indices<Is...>) { return (static_cast<Class *>(obj)->*f)(any_cast<Args>(args[Is])...); } template<Fun fun> static Any invoke(Object *obj, int argc, const Any *args) { if (argc != sizeof...(Args)) throw std::runtime_error("Bad argument count"); return invoke(obj, fun, args, unpack::indices_gen<sizeof...(Args)>()); } }; An important role in efficiency is also played by the class Any, which allows you to rather compactly store types and information about them. The basis was the hold_any class from the boost spirit library. Templates are also actively used here to effectively wrap types. Types smaller than a pointer are stored in size directly in void *, for larger types the pointer refers to an object of type.
Example
template<typename T> struct AnyHelper<T, True> { typedef Bool<std::is_pointer<T>::value> is_pointer; typedef typename CheckType<T, is_pointer>::type T_no_cv; inline static void clone(const T **src, void **dest) { new (dest)T(*reinterpret_cast<T const*>(src)); } }; template<typename T> struct AnyHelper<T, False> { typedef Bool<std::is_pointer<T>::value> is_pointer; typedef typename CheckType<T, is_pointer>::type T_no_cv; inline static void clone(const T **src, void **dest) { *dest = new T(**src); } }; template<typename T> Any::Any(T const& x) : _table(Table<T>::get()), _object(nullptr) { const T *src = &x; AnyHelper<T, Table<T>::is_small>::clone(&src, &_object); } RTTI also had to be abandoned, too slowly. Type checking is done solely by comparing pointers to a type table. All type modifiers are pre-cleared, otherwise, for example, int and const int will be different types. But in fact, their size is single, and in general it is the same type.
Another example
template <typename T> inline T* any_cast(Any* operand) { if (operand && operand->_table == Table<T>::get()) return AnyHelper<T, Table<T>::is_small>::cast(&operand->_object); return nullptr; } How to use it
Embedding scripting languages has become easy and enjoyable. For example, for Lua, it suffices to define a generic call function that checks the number of arguments and their types and, of course, calls the function itself. Binding is also not difficult. For each function in Lua, it is enough to save the MetaMethod in upvalue. By the way, all objects in uMOF are “thin”, that is, just a wrapper over a pointer that refers to an entry in a static table. Therefore, you can copy them without concern about performance.
An example of a Lua binding:
Example, a lot of code
#include <lua/lua.hpp> #include <object.h> #include <cassert> #include <iostream> class Test : public Object { OBJECT(Test, Object) EXPOSE( METHOD(sum), METHOD(mul) ) public: static double sum(double a, double b) { return a + b; } static double mul(double a, double b) { return a * b; } }; int genericCall(lua_State *L) { Method *m = (Method *)lua_touserdata(L, lua_upvalueindex(1)); assert(m); // Retrieve the argument count from Lua int argCount = lua_gettop(L); if (m->parameterCount() != argCount) { lua_pushstring(L, "Wrong number of args!"); lua_error(L); } Any *args = new Any[argCount]; for (int i = 0; i < argCount; ++i) { int ltype = lua_type(L, i + 1); switch (ltype) { case LUA_TNUMBER: args[i].reset(luaL_checknumber(L, i + 1)); break; case LUA_TUSERDATA: args[i] = *(Any*)luaL_checkudata(L, i + 1, "Any"); break; default: break; } } Any res = m->invoke(nullptr, argCount, args); double d = any_cast<double>(res); if (!m->returnType().valid()) return 0; return 0; } void bindMethod(lua_State *L, const Api *api, int index) { Method m = api->method(index); luaL_getmetatable(L, api->name()); // 1 lua_pushstring(L, m.name()); // 2 Method *luam = (Method *)lua_newuserdata(L, sizeof(Method)); // 3 *luam = m; lua_pushcclosure(L, genericCall, 1); lua_settable(L, -3); // 1[2] = 3 lua_settop(L, 0); } void bindApi(lua_State *L, const Api *api) { luaL_newmetatable(L, api->name()); // 1 // Set the "__index" metamethod of the table lua_pushstring(L, "__index"); // 2 lua_pushvalue(L, -2); // 3 lua_settable(L, -3); // 1[2] = 3 lua_setglobal(L, api->name()); lua_settop(L, 0); for (int i = 0; i < api->methodCount(); i++) bindMethod(L, api, i); } int main(int argc, char *argv[]) { lua_State *L = luaL_newstate(); luaL_openlibs(L); bindApi(L, Test::classApi()); int erred = luaL_dofile(L, "test.lua"); if (erred) std::cout << "Lua error: " << luaL_checkstring(L, -1) << std::endl; lua_close(L); return 0; } Conclusion
So, what we have:
Advantages of uMOF:
- Compact;
- Fast;
- Does not require third-party tools, only a modern compiler.
Disadvantages of uMOF:
- Not supported by all compilers;
- The auxiliary macros are rather clumsy.
The library is still quite raw, I would like to do a lot of interesting things - functions of variable arity (read, default parameters), non-invasive type registration, signals about changes in object properties. And all this will surely appear, because the method has shown very good results.
Thank you all for your attention. I hope the library will be useful for someone.
The project can be found on the link . Write your feedback and recommendations in the comments.
Source: https://habr.com/ru/post/238237/
All Articles