UnnyWorld: postmortem
After the closure of our game UnnyWorld, many developers have asked friends to write a post-mortem on the game . I decided to share specific examples, of which a decent amount has accumulated during development. There will be considered errors that we have made, I will try to give a couple of useful tips.
Earlier, I published the article āThree years of developing my MMOā , where it was more about finding investments, the team and our path to āsuccessā. Unfortunately (or fortunately?), The project had to be closed. In this article I will try to consider the mistakes made and, possibly, give at least a couple of useful tips.
Conventionally, Unnyworld can be divided into two parts: City Builder and Arenas.
The part about the builder is a certain Clash of Clans. You have your own planet that you need to equip. And you can attack other planets to steal resources.
')
Attacking other players you are one of your characters and manage it yourself.
Arenas - a typical 3na3 MOBA with different modes (capture the flag, capture points, etc.).
Each character has its own spells that can be combined with other players.
Before the battle, you can change spells.
The game cycle as a whole looks like this:
That is, we tried to somehow link the regime of the planet and arenas. We probably did everything wrong.
Yes, many things were constantly discussed, but they were not realized by chance, without thoroughly analyzing what should be done in the first place.
As a result, they tried to do everything at once. Do I need a guild system when the game and a half user? Hmm, hardly.
Do you need a system that allows you to create a custom match, inviting friends and co-guards there when there is a small CCU in the game? Not sure.
In the development process, we tried to do a lot of things that we probably didnāt have to do at that stage. As a result, the really necessary things were not implemented.
Since we used to work mainly with single-player games, then we stepped on a large number of rakes when choosing this or that technology.
Let's talk a little about the technical part of the question.
Small clarification. For the most part, we are pure client developers. Of the entire team, only a couple of people had experience with server technologies. About adminstvo I generally keep quiet. I will try on specific technologies to go with a small summary of each.
Which cloud provider to use? AWS? Azure? Soft Layer?
At that time, there was no fundamental difference. Plus, we had a SoftLayer loan as a startup.
Oh, boi, if you only knew how bad everything is there:
- Saport is not particularly aware of the problem. There were cases when I addressed them about a problem on a certain virtual machine (could not connect, etc.). To which he received the answer:
- There were cases when the virtualka went up for hours . As you can see, I waited for 4 hours, but the virtual was never created.
- Frequent maintance.
- It happens that a machine is rebooted without warning, or a private network is cut off.
As a result, switched to Azure. There were no such problems. The support quickly responds and always helps in case of anything.
Good: -
Bad: did not analyze properly all possible options. But the server - this is the most important part for online games = /
So, you need to somehow run the game instances on the server, transfer it through some kind of API system after authorization of the player to the necessary instance. What are we going to do? And let's take a ready-made solution, which will itself, depending on the workload, manage this business. Wow, there's a thing called Kubernetes. True, it is in beta ... But anyway, let's try!
If we discard the fact that experience is needed to work with this technology, then even with the basic configuration of this case, it managed to fall. Some services fell off, etc.
Well, what else is there? Mesosphere and Apache Mesos! With him all the same, without experience is difficult. If something falls off, then without a tambourine you cannot fix the problem.
As a result, they wrote everything themselves. Instances are started by the supervisor, as is the small manager above them (written in Java). Java application ttl'it to the service of discovery tools about the status (the number of free rooms on the instances, etc.). When authorizing and requesting the creation of a room to the API for this information about the instances, the request goes to the desired node, which raises the room on the required instance.
Those. instances are always pre-launched. With a shortage, we are raising a new VPS.
Good: analyzed alternatives.
Bad: spent a lot of time on the prototype. For the first version, it was necessary not to think about these things at all, but simply to start the instances without any applications over the top. It was possible to directly hardwrite the addresses of the instances on the client in the prototype.
For the discovery service used www.consul.io This is probably one of those solutions, which did not regret. There, however, there are problems like these , when the rebuy breaks the config. But this is rare and with an unplanned reboot of the machine. In general, for all the time it was a pleasure to work with the consul.
Good: they took a ready-made solution, but did not begin to cut something themselves.
Initially, bash scripts were used for deployment.
Later, I transferred the whole patch to Ansible. I can not get enough to this day. There were, of course, problems at first. But the system is quite simple to learn, and indeed documentation in bulk.
Good: quickly write a bash script, no special knowledge is required.
Bad: when switching to a normal deployment system, I had to throw out almost everything previously written.
To communicate between their services tried www.rabbitmq.com . But he just couldnāt fall apart in a few days. As a result, they did it in a simple way - all services interact either through pure tcp sockets, or http requests from keep-alive, if you need to send requests in one direction only.
Good: analyzed alternatives. Chose a good solution.
Bad: lack of experience with technology. There is no need to drag things into production that you cannot fix in case of problems.
Playing online, then you need chat. Write yourself? It is unlikely to be scalable. Let's take something ready. XMPP? Ejabberd? Seems good. In general, they tried both a hedgehog and MongooseIM, but finally stopped on a hedgehog. There were some problems with raising it on its servers (jambs with timings in messages, crashing, etc.). We decided to use their cloud solution ejabberd-saas.com . Yes, it is paid. But it worked without problems.
Good: analyzed alternatives. Chose the right option.
Bad: instead of sorting out local problems, we decided to use a paid cloud solution. Rates there from 200 euros. We had several gambling regions. For an indie team, this goes into a very substantial amount that it would be better to spend on other things.
At first, we didnāt have any system for collecting metrics on servers at all. Why does the query slow down? What is wrong with the service? How many rooms are there now? Yes, we could not even see how many rooms there are at the moment!
Later came the realization that something had to be done. Tried to use Graphite + Grafana. Even an image before the docker did with all of this: github.com/Suvitruf/docker-grafana-graphite-diamond
But it did not work out. I did not want to spend time on it, decided to use something ready. The choice fell on www.datadoghq.com
All perfectly. Metrki, alerts, graphics. The client driver is almost the same as for graphite. Beauty. But ... $ 10 + for each host per month. Iiii ... it comes out at 200 + $ per month.
The realization that we pissed off a lot of money on this was too late. They decided, nevertheless, to do this on their servers. Set up www.influxdata.com . As a result, one car for a couple of tens of bucks quietly handles metrics from tens / hundreds of cars.
Good: tried it yourself in a quick way. Found an alternative ready. Realized (albeit late) that the decision was wrong. Set up a convenient system locally.
Bad: how should not understand the question. Spent a lot of money.
Regarding metrics, the same problem with performance. At first, we especially neither the client nor the server on profilers. As a result, it is too late to detect memory leaks on the server instances of the game. Identify and fix promptly failed. As a result, they wrote that after the creation of a certain number of rooms, the game instance is restarted.
I can not now build the correct order in time for all these events, I will mention some key decisions that we made.
The players requested an Asian server and a server in South America (before this, the servers were in Europe and the USA). Why not do, yes? Made. As a result, one and a half user smeared in 4 regions. Once several regions, you need to make a transfer system. Is it logical Is logical.
Good: a couple of people got better ping (ļ½” ā¢ ļøæ ā¢ Ģļ½”)
Bad: a lot of time spent on the creation of regions, transfer systems, etc.
Listen to the tips / suggestions of the players is necessary, but you should not immediately run and implement all this.
Before the planets looked like this:
And attacks:
Switching to hexes has simplified many things technically. Plus, it looked much better, it's easier to work with game elements.
It used to look like this:
The upgrade itself was carried out for the stones that fell from the chests. Everything was unclear, confusing.
As a result, the system of stones was replaced with scrolls as in Clash Royale.
To improve the spell you need a certain number of scrolls. Everything is simple and clear.
Well: noticed a problem place and remade.
Bad: initially did not analyze how the players perceive it. Many things that are obvious to developers, players perceive quite differently. Therefore, feedback should be collected at the earliest possible stages, arrange playtests, etc.
We even agreed with www.twitch.tv , so that in-game purchases could be made on the game page.
But since no one has streamlined our game, then thereās absolutely no point in such a decision.
Good: a potential place for a fair withdrawal of money from the players.
Bad: if your game is not streaming, then there is no point in this. Just wasted time.
In the wake of the HYIP, they decided to file such a regime in the game. But since There was a small online game, in this mode there were almost only bots, which eliminated everything.
In such a project it is difficult not to make bugs. There were a lot of relatively harmless GUI bugs.
There were more serious bugs, for example, when players instantly died in the center of the arena. Repeat this bug, we could not locally. It periodically occurred with the players, but we could not fix it.
Funny bug, when the character model of the previous one was not deleted when switching the character. In the end, it was possible to arrange a party hard.
There were also bugs related to the platform / engine.
For example, sometimes all GUI could just disappear. But if you go into the hierarchy of objects and just click on the object, it appears again.
We about this problem reported Unity. They answered that they could give us an employee to help for $ 10k per month (ą² _ą² į)
On the Facebook Gameroom platform, there was a problem with the scale when the game reacted to the cars in the wrong place where they were performed.
This, let alone the bugs in various libraries. For example, on some machines Steamworks.NET could fall github.com/rlabrecque/Steamworks.NET/issues/121 .
We almost did not invest in marketing, hoping that there would be an organic influx of players. The game in the end did not reach that critical mass, after which bots would not be needed and there would be an organic influx of new players.
Especially no one engaged in content management and communication with the players, there were no mailings.
A lot of time was lost during development when selecting and testing various technologies.
There was no clear plan for the implementation of features / content.
In general, most of all these problems are due to inexperience.
Unnyworld was closed. We decided to make a smaller project within the framework of current opportunities.
In one article, do not cover everything. And what he wrote about may seem like an incoherent set of facts to an outsider reader. Unfortunately, not a whale to write such texts.
If there are any questions, I will be happy to answer either in the comments or in the new article.
Earlier, I published the article āThree years of developing my MMOā , where it was more about finding investments, the team and our path to āsuccessā. Unfortunately (or fortunately?), The project had to be closed. In this article I will try to consider the mistakes made and, possibly, give at least a couple of useful tips.
Pro game in a nutshell
Conventionally, Unnyworld can be divided into two parts: City Builder and Arenas.
The part about the builder is a certain Clash of Clans. You have your own planet that you need to equip. And you can attack other planets to steal resources.
')
Attacking other players you are one of your characters and manage it yourself.
Arenas - a typical 3na3 MOBA with different modes (capture the flag, capture points, etc.).
Each character has its own spells that can be combined with other players.
Before the battle, you can change spells.
The game cycle as a whole looks like this:
- For pumping spells need scrolls that fall from the chests. Chests can be obtained in various free ways (for the league, for winning Battle Battle, etc.) or to buy.
- To pump the spell, you need to build and improve the building of the hero to a certain level.
- To improve the building of the hero need to improve other buildings (main building, altar, etc.).
That is, we tried to somehow link the regime of the planet and arenas. We probably did everything wrong.
Lack of a clear plan and strategy
Yes, many things were constantly discussed, but they were not realized by chance, without thoroughly analyzing what should be done in the first place.
As a result, they tried to do everything at once. Do I need a guild system when the game and a half user? Hmm, hardly.
Do you need a system that allows you to create a custom match, inviting friends and co-guards there when there is a small CCU in the game? Not sure.
In the development process, we tried to do a lot of things that we probably didnāt have to do at that stage. As a result, the really necessary things were not implemented.
Lack of experience
Since we used to work mainly with single-player games, then we stepped on a large number of rakes when choosing this or that technology.
Let's talk a little about the technical part of the question.
Choice of technology
Small clarification. For the most part, we are pure client developers. Of the entire team, only a couple of people had experience with server technologies. About adminstvo I generally keep quiet. I will try on specific technologies to go with a small summary of each.
Which cloud provider to use? AWS? Azure? Soft Layer?
At that time, there was no fundamental difference. Plus, we had a SoftLayer loan as a startup.
Oh, boi, if you only knew how bad everything is there:
- Saport is not particularly aware of the problem. There were cases when I addressed them about a problem on a certain virtual machine (could not connect, etc.). To which he received the answer:
We rebooted the car, now everything is fine
- There were cases when the virtualka went up for hours . As you can see, I waited for 4 hours, but the virtual was never created.
- Frequent maintance.
- It happens that a machine is rebooted without warning, or a private network is cut off.
As a result, switched to Azure. There were no such problems. The support quickly responds and always helps in case of anything.
Good: -
Bad: did not analyze properly all possible options. But the server - this is the most important part for online games = /
So, you need to somehow run the game instances on the server, transfer it through some kind of API system after authorization of the player to the necessary instance. What are we going to do? And let's take a ready-made solution, which will itself, depending on the workload, manage this business. Wow, there's a thing called Kubernetes. True, it is in beta ... But anyway, let's try!
If we discard the fact that experience is needed to work with this technology, then even with the basic configuration of this case, it managed to fall. Some services fell off, etc.
Well, what else is there? Mesosphere and Apache Mesos! With him all the same, without experience is difficult. If something falls off, then without a tambourine you cannot fix the problem.
As a result, they wrote everything themselves. Instances are started by the supervisor, as is the small manager above them (written in Java). Java application ttl'it to the service of discovery tools about the status (the number of free rooms on the instances, etc.). When authorizing and requesting the creation of a room to the API for this information about the instances, the request goes to the desired node, which raises the room on the required instance.
Those. instances are always pre-launched. With a shortage, we are raising a new VPS.
Good: analyzed alternatives.
Bad: spent a lot of time on the prototype. For the first version, it was necessary not to think about these things at all, but simply to start the instances without any applications over the top. It was possible to directly hardwrite the addresses of the instances on the client in the prototype.
For the discovery service used www.consul.io This is probably one of those solutions, which did not regret. There, however, there are problems like these , when the rebuy breaks the config. But this is rare and with an unplanned reboot of the machine. In general, for all the time it was a pleasure to work with the consul.
Good: they took a ready-made solution, but did not begin to cut something themselves.
Initially, bash scripts were used for deployment.
Later, I transferred the whole patch to Ansible. I can not get enough to this day. There were, of course, problems at first. But the system is quite simple to learn, and indeed documentation in bulk.
Good: quickly write a bash script, no special knowledge is required.
Bad: when switching to a normal deployment system, I had to throw out almost everything previously written.
To communicate between their services tried www.rabbitmq.com . But he just couldnāt fall apart in a few days. As a result, they did it in a simple way - all services interact either through pure tcp sockets, or http requests from keep-alive, if you need to send requests in one direction only.
Good: analyzed alternatives. Chose a good solution.
Bad: lack of experience with technology. There is no need to drag things into production that you cannot fix in case of problems.
Playing online, then you need chat. Write yourself? It is unlikely to be scalable. Let's take something ready. XMPP? Ejabberd? Seems good. In general, they tried both a hedgehog and MongooseIM, but finally stopped on a hedgehog. There were some problems with raising it on its servers (jambs with timings in messages, crashing, etc.). We decided to use their cloud solution ejabberd-saas.com . Yes, it is paid. But it worked without problems.
Good: analyzed alternatives. Chose the right option.
Bad: instead of sorting out local problems, we decided to use a paid cloud solution. Rates there from 200 euros. We had several gambling regions. For an indie team, this goes into a very substantial amount that it would be better to spend on other things.
At first, we didnāt have any system for collecting metrics on servers at all. Why does the query slow down? What is wrong with the service? How many rooms are there now? Yes, we could not even see how many rooms there are at the moment!
Later came the realization that something had to be done. Tried to use Graphite + Grafana. Even an image before the docker did with all of this: github.com/Suvitruf/docker-grafana-graphite-diamond
But it did not work out. I did not want to spend time on it, decided to use something ready. The choice fell on www.datadoghq.com
All perfectly. Metrki, alerts, graphics. The client driver is almost the same as for graphite. Beauty. But ... $ 10 + for each host per month. Iiii ... it comes out at 200 + $ per month.
The realization that we pissed off a lot of money on this was too late. They decided, nevertheless, to do this on their servers. Set up www.influxdata.com . As a result, one car for a couple of tens of bucks quietly handles metrics from tens / hundreds of cars.
Good: tried it yourself in a quick way. Found an alternative ready. Realized (albeit late) that the decision was wrong. Set up a convenient system locally.
Bad: how should not understand the question. Spent a lot of money.
Regarding metrics, the same problem with performance. At first, we especially neither the client nor the server on profilers. As a result, it is too late to detect memory leaks on the server instances of the game. Identify and fix promptly failed. As a result, they wrote that after the creation of a certain number of rooms, the game instance is restarted.
A little bit about conceptual and DG solutions
I can not now build the correct order in time for all these events, I will mention some key decisions that we made.
Dividing the game into regions
The players requested an Asian server and a server in South America (before this, the servers were in Europe and the USA). Why not do, yes? Made. As a result, one and a half user smeared in 4 regions. Once several regions, you need to make a transfer system. Is it logical Is logical.
Good: a couple of people got better ping (ļ½” ā¢ ļøæ ā¢ Ģļ½”)
Bad: a lot of time spent on the creation of regions, transfer systems, etc.
Listen to the tips / suggestions of the players is necessary, but you should not immediately run and implement all this.
Replacing a square grid with hexes and remaking attacks on planets
Before the planets looked like this:
And attacks:
Switching to hexes has simplified many things technically. Plus, it looked much better, it's easier to work with game elements.
Redone spell system
It used to look like this:
The upgrade itself was carried out for the stones that fell from the chests. Everything was unclear, confusing.
As a result, the system of stones was replaced with scrolls as in Clash Royale.
To improve the spell you need a certain number of scrolls. Everything is simple and clear.
Well: noticed a problem place and remade.
Bad: initially did not analyze how the players perceive it. Many things that are obvious to developers, players perceive quite differently. Therefore, feedback should be collected at the earliest possible stages, arrange playtests, etc.
Shopping at Tviche
We even agreed with www.twitch.tv , so that in-game purchases could be made on the game page.
But since no one has streamlined our game, then thereās absolutely no point in such a decision.
Good: a potential place for a fair withdrawal of money from the players.
Bad: if your game is not streaming, then there is no point in this. Just wasted time.
Battle Royale mode
In the wake of the HYIP, they decided to file such a regime in the game. But since There was a small online game, in this mode there were almost only bots, which eliminated everything.
About bugs
In such a project it is difficult not to make bugs. There were a lot of relatively harmless GUI bugs.
There were more serious bugs, for example, when players instantly died in the center of the arena. Repeat this bug, we could not locally. It periodically occurred with the players, but we could not fix it.
Funny bug, when the character model of the previous one was not deleted when switching the character. In the end, it was possible to arrange a party hard.
There were also bugs related to the platform / engine.
For example, sometimes all GUI could just disappear. But if you go into the hierarchy of objects and just click on the object, it appears again.
We about this problem reported Unity. They answered that they could give us an employee to help for $ 10k per month (ą² _ą² į)
On the Facebook Gameroom platform, there was a problem with the scale when the game reacted to the cars in the wrong place where they were performed.
This, let alone the bugs in various libraries. For example, on some machines Steamworks.NET could fall github.com/rlabrecque/Steamworks.NET/issues/121 .
Results
We almost did not invest in marketing, hoping that there would be an organic influx of players. The game in the end did not reach that critical mass, after which bots would not be needed and there would be an organic influx of new players.
Especially no one engaged in content management and communication with the players, there were no mailings.
A lot of time was lost during development when selecting and testing various technologies.
There was no clear plan for the implementation of features / content.
In general, most of all these problems are due to inexperience.
What's next?
Unnyworld was closed. We decided to make a smaller project within the framework of current opportunities.
In one article, do not cover everything. And what he wrote about may seem like an incoherent set of facts to an outsider reader. Unfortunately, not a whale to write such texts.
If there are any questions, I will be happy to answer either in the comments or in the new article.
Source: https://habr.com/ru/post/417035/
All Articles