How to permanently delete 100,000 users and stay alive?
In this post I want to share with you the experience of developing one large project. During its creation, 100,000 users have been deleted forever, and they had to be restored.
For young developers, this will be a story that will not allow me to make my mistakes in the future, and more experienced dudes can just laugh and comment that they would never allow such a situation in their lives).
The story began in April 2015. In the beginning everything was standard. The customer (I will not deliberately say the names and names of the company) wrote in VKontakte, asked what for, threw off the approximate TK, met, talked, liked each other, decided to do.
')
The idea of ​​the project is as follows:
There are companies that publish on the site discounts from themselves. Well, for example, sneakers with a discount of 20%. Users buy a monthly subscription and can enjoy discounts from all companies represented.
Also, the site has an affiliate system that allows you to invite people. If the person invited by you paid for the subscription, then the percentage goes to you. The logic of the passage of percentages is rather complicated, but the customer explained as follows: “Well, there’s nothing complicated, only small formulas to register.”
And they have already made a similar site, they just ran it in and decided to make it more beautiful, bigger, better. So, it was necessary even at the launch of the new site to transfer all the data from the old one.
It should be clarified that I had not done such projects before and for me it was something new and unknown. Therefore, I initially did not understand how much objectively I would have to do this. Given that the design has already been drawn, it was only necessary to lay out and put on the back-end, I requested 65t.r. and 2 months of work. Both of the last numbers were fatal. It was not enough time and money) I had to take more. But ... lack of experience is lack of experience ...
Now tell the key point. Considering that the “affiliate program” stage was described as “several formulas”, I decided to do it in a week. Moreover, there really wasn’t much work on the TK. Oh, how bad I then appreciated this work ... oh, how bad ...
As a result, according to the law of the genre, while I wasted up the prepayment, the work did not go and began only after a month) Therefore, night-day work was introduced and in three weeks I did everything that was required, except for the logic of the partner network. It was necessary to hand over on June 1 and just somewhere in the 20th, I took up its implementation.
During the week everything was written exactly according to the statement of work. But it was not there ... at the meeting and testing it turned out that they did not tell me much! And there are many more nuances in logic that need to be done ... Well, I think, okay ... we'll do it. And the deadline, in the meantime, was already flying to hell.
In short ... so a month went by.
All June, we ruled and completed this damn logic. That money does not go there, then the invited person becomes the wrong person ... it was hard. Now I would send them to hell and ask for additional payments ... but then I was younger, snotter and could not afford such luxury.
At the end of the month I was squeezed like a lemon. I did not want to live ... I had to rest. I told the customer that everything ... need a pause. And he dumped the country for two weeks.
When I arrived the endless series of edits began again. In general, I stopped responding to messages and calls. I'm missing. Gone missing and it was impossible to find me. Like this.
I just gave up and went underground. Yes, not manly. Now I would not do that in my life. But then I was not ready for this kind of problems. They broke me ...
After some time they stopped looking for me and calling me. I thought that everything was fine. Maybe they found another developer, maybe something else. But nobody else bothered me. Paradise.
For you to understand, these were not just some strange types. These are relatively famous personalities in our city. Behind the customers themselves there was a bigger guide, which really could make a very big stir in my life. But I learned about it only in the middle of the project.
During the lull, I got a job at a web studio. In early August, they started calling me again, and I began not to answer again. As a result, the customer himself came to me at work) Asked to just finish what was started and disperse. The format is as follows: he writes that it is necessary to finish, I do and happy end.
Also, we met with the most important man in the project, who identified the fact that under the contract I have a percentage for each day of delay. In general, I had to pay another 180 tr with a plus ... But they were ready to forgive all good-naturedly if I made all the changes until the autumn and started the project, otherwise the court. I think it is clear that I didn’t have much choice.
At the end of August there was a launch. He passed with a creak. It turned out that several errors flashed in the code, which several tens of thousands of rubles were sent to the wrong people ...
Here you need to make a clarification. The project already existed and 100,000 people were registered in it (in fact, 110,000, but I rounded off). Every day the site went about a thousand. Several hundred people a day made transactions. And these few hundreds of people were very unhappy ((((
Happenes. What to do ... Notified on the site that you can write to me on this issue in VK. I did the translation with my own hands and at the same time understood where the errors are in the code. Every day I hung about a hundred unread messages. Within a month, I woke up at 7 am, processed user requests, went to work, and processed requests in the evening again.
In general, people reacted to the situation with understanding, so it was not very smooth, but the situation was turned out. Gradually, there were fewer errors and the site began to work as it should.
But as they say, trouble does not come alone!
One evening in the office after work, I decided to file a site. I go, and he lies! mysql fell off. Prior to this, mysql sometimes crashed when the server was unable to cope with the load and I just rebooted it. But there was another case. It did not even start ... I spent the whole evening on the forums. Then I believed in Toster! Some guy just the whole evening answered my messages and helped me find a mistake! I was surprised that someone can help a complete stranger for free.
Plus, I didn’t have much experience with servers, so it was difficult. I only knew that there is a terminal and commands can be entered there. I googled, I was looking for which command I could enter, entered and got the result.
As a result, I came to the conclusion that the database is lost ... no ... Then I found out that there is an ibdata file and if it is empty, then all data is lost. As you can imagine, it was empty. Fully. 0kb
And the girl and I went to the Stand-up this evening, our friends played. I did not sit out until the end and asked to go home. It was there, in the hall, that the thought reached me that I somehow lost 100,000 users. There is not a single backup. There is not a single hope for recovery. What to do? Call and say that ass guys, I sow labor a few years of your work.
After the call and clarification of the situation, a consultation was held. At 12 at night. What to do?
I was telling the story that the server ran out of space (and that was in fact), and therefore mysql was inhibited. In fact, to the last I was sure that this was so. So, like, my fault is not here.
Thought-thought and decided to go to sleep and on the trail. day to figure out what can be done and what actually happened.
I asked for help in freelancing. Found there one system administrator who agreed to help me out. As a result, he confirmed my theory that most likely mysql fell off because of the finished space. Worse, he confirmed that the base could not be restored.
Customers asked for help from the web-studio, in which the programmer looked at the history of commands in the terminal and know what he found? The command that erases the ibdata file! Apparently on the forums, someone was amused to enter this command if mysql crashes, but I did not read between the lines in a panic and entered this command.
Well, I think everything. The end to me. But the customers realized that the problem should be solved and if they “kill” me, there will be no one to finish. Fortunately, there was a backup of the old site on which there were 80,000 people. The class. Now it remains to recover 30,000 people) A new phase of messages has begun in VK, restorations and panic.
Thanks to my girlfriend who supported me. Thanks to her for not leaving me at all) For it was not possible to communicate with me then, because every evening I screamed that it was hard for me and I no longer know what to do ... I do not like people who ache. And I didn’t love myself either.
With grief in half, everything was restored. In the end, I said that they should find a programmer on a permanent system administrator who will maintain and backup the server and live somehow without me. I no longer wanted to deal with this hellish project.
There should have been something else to fall off and break, but that was all. The site is still working. They never found a programmer for themselves (as far as I know), apparently still I do not have such crooked hands and now everything works as it should.
By the way, they didn't pay me 10 thousand. But I was somehow ashamed to ask them later ... therefore the project cost 50 thousand rubles and 6 months of suffering. After it was all over, I thought to myself: “But I got a lot of experience.” And really ... this project taught me a lot. And it will be wrong if I do not share them:
1) Carefully read the terms of reference and contracts
In fact, weighing Tusa-Dzhusa started because I was light on TK. If I had read more carefully into it, I understood that the stage with logic was far from complete and it needs to be added, then the dates would have moved and we calmly completed the project.
Be sure to read the contracts that are signed. And they must be compiled!
I now do not draw up a contract only if the customer is my friend or I have already worked with him and am confident in his good faith. In general, since I signed the letter, that every day overdue results in a lot of money, this turned out to be the situation described above.
2) Evaluate the project in terms of money.
Then I did not know how much to take for the project and for me the figure of 65 tr. was something unattainable. Now for such a project, I would have asked for 200-300 thousand. I realized that if the project has a large number of personal accounts, logical interconnections, then the project will not be a priori simple. And even if it seems that there is nothing to write there, then this will not happen.
3) Speak waste routes
You should always understand that a project can go beyond. Or the customer came up with many more different ideas that were not in the TK. Or you simply did not calculate and the deadline went yesterday, and the project is only in the middle.
Speak with customers this moment on the shore. At the crossing of horses do not change.
Now, when customers ask how much will be spent on the project, I say minimum and maximum. For example: “The project is really done in 2 weeks, but, objectively speaking, we will do it a month. Subject to revisions, additions and so on. ".
4) Make backups
Make backups, their mother! Do it! Be sure to do it!
I now have two of my working computers backed up to my hard drive. The server backed up every day, but after one situation, I had them created every hour and every day. In a nutshell, I didn’t keep track of the hard place and mysql broke down (the situation is like in this project), but I had a backup and, in general, he saved me).
After this incident, I thought ... And what if in fact really mysql broke because of the small hard disk space, but I deleted the ibdata and then it was impossible to prove anything? If this is so, then somehow even offensively ...
5) See what you enter into the terminal
See what commands, where and why enter !!! Now if I don’t know the command, I’ll google it before entering it into the terminal).
6) Separate responsibility
In fact, the situation with the server happened due to the fact that I was not in my domain. If you make a website, then make a website, and let those who know how to answer for it be responsible for the server. After this joke, they hired a system administrator, who then ruled a server).
7) Do not whine
In any not clear situation do not whine! Whining people seem miserable. You can "cry." If it is hard, go to a close person, ask to listen and pour out all the boiling water. But do not whine every night as you feel bad and hard.
From time to time I meet people who constantly whine. They can not be tolerated. With some I stopped communicating even because of this.
8) Take responsibility for your shoals
Nakosyachil - catch lyuley. If something went wrong, then call the customer and explain the situation as it is. They are people too, and they will understand, they will come into your position and even help solve the problem. Do not disappear) It will be worse only for you.
This applies not only to sites, in fact, but also to life.
9) We get high from force majeure
During all this, my friend told me one interesting thing, that all the problems come from our heads and if there are problems, we want them to be. And I realized that to some extent the buzz from this situation. I am a programmer masochist! I love force majeure and therefore attract them. What I love is not so that “Aaaaaa! I want to break deadlines! Let's make the site another 5 months for free at night. ” Just inside some small part of me sits and quietly enjoys what is happening.
Think about it if you constantly break deadlines.
10) Do not touch dubious projects.
Filter your orders. This project was initially dubious. But I needed money and experience. Now I eliminate 70% of orders and say that I don’t want to do them, they just aren’t really interesting to me. Another question that I can allow myself now, then I could not)
On the other hand, I took up this project and got a pretty decent experience. Therefore, the devil knows. Probably, after all, take on dubious projects - this is experience. But only once ...
11) Belief in Toster
I believed in Toster. I was really surprised when I realized that people simply respond to other people just like that. It was a great break for me. Now I constantly ask my questions there and they constantly help me out. And as far as possible, I answer others.
Recently there was even a situation when I was googling a question, a link to a toaster appeared in the search, and there I ask the same question and answered it myself in the commentary. I shut up.
Completion
Every item now lives with me every day. Every new order I take into account all my mistakes and no longer repeat them. After this project, I did not have short deadlines and frustrated customers. If force majeure situations arose, then they were immediately led back on track and resolved in peace negotiations.
Do not be afraid to make mistakes. Be careful not to draw conclusions from them. It was a hard lesson. But I do not regret that this happened to me.
Thanks for reading! I hope it was interesting.
For young developers, this will be a story that will not allow me to make my mistakes in the future, and more experienced dudes can just laugh and comment that they would never allow such a situation in their lives).
The story began in April 2015. In the beginning everything was standard. The customer (I will not deliberately say the names and names of the company) wrote in VKontakte, asked what for, threw off the approximate TK, met, talked, liked each other, decided to do.
')
The idea of ​​the project is as follows:
There are companies that publish on the site discounts from themselves. Well, for example, sneakers with a discount of 20%. Users buy a monthly subscription and can enjoy discounts from all companies represented.
Also, the site has an affiliate system that allows you to invite people. If the person invited by you paid for the subscription, then the percentage goes to you. The logic of the passage of percentages is rather complicated, but the customer explained as follows: “Well, there’s nothing complicated, only small formulas to register.”
And they have already made a similar site, they just ran it in and decided to make it more beautiful, bigger, better. So, it was necessary even at the launch of the new site to transfer all the data from the old one.
It should be clarified that I had not done such projects before and for me it was something new and unknown. Therefore, I initially did not understand how much objectively I would have to do this. Given that the design has already been drawn, it was only necessary to lay out and put on the back-end, I requested 65t.r. and 2 months of work. Both of the last numbers were fatal. It was not enough time and money) I had to take more. But ... lack of experience is lack of experience ...
Now tell the key point. Considering that the “affiliate program” stage was described as “several formulas”, I decided to do it in a week. Moreover, there really wasn’t much work on the TK. Oh, how bad I then appreciated this work ... oh, how bad ...
As a result, according to the law of the genre, while I wasted up the prepayment, the work did not go and began only after a month) Therefore, night-day work was introduced and in three weeks I did everything that was required, except for the logic of the partner network. It was necessary to hand over on June 1 and just somewhere in the 20th, I took up its implementation.
During the week everything was written exactly according to the statement of work. But it was not there ... at the meeting and testing it turned out that they did not tell me much! And there are many more nuances in logic that need to be done ... Well, I think, okay ... we'll do it. And the deadline, in the meantime, was already flying to hell.
In short ... so a month went by.
All June, we ruled and completed this damn logic. That money does not go there, then the invited person becomes the wrong person ... it was hard. Now I would send them to hell and ask for additional payments ... but then I was younger, snotter and could not afford such luxury.
At the end of the month I was squeezed like a lemon. I did not want to live ... I had to rest. I told the customer that everything ... need a pause. And he dumped the country for two weeks.
When I arrived the endless series of edits began again. In general, I stopped responding to messages and calls. I'm missing. Gone missing and it was impossible to find me. Like this.
I just gave up and went underground. Yes, not manly. Now I would not do that in my life. But then I was not ready for this kind of problems. They broke me ...
After some time they stopped looking for me and calling me. I thought that everything was fine. Maybe they found another developer, maybe something else. But nobody else bothered me. Paradise.
For you to understand, these were not just some strange types. These are relatively famous personalities in our city. Behind the customers themselves there was a bigger guide, which really could make a very big stir in my life. But I learned about it only in the middle of the project.
During the lull, I got a job at a web studio. In early August, they started calling me again, and I began not to answer again. As a result, the customer himself came to me at work) Asked to just finish what was started and disperse. The format is as follows: he writes that it is necessary to finish, I do and happy end.
Also, we met with the most important man in the project, who identified the fact that under the contract I have a percentage for each day of delay. In general, I had to pay another 180 tr with a plus ... But they were ready to forgive all good-naturedly if I made all the changes until the autumn and started the project, otherwise the court. I think it is clear that I didn’t have much choice.
At the end of August there was a launch. He passed with a creak. It turned out that several errors flashed in the code, which several tens of thousands of rubles were sent to the wrong people ...
Here you need to make a clarification. The project already existed and 100,000 people were registered in it (in fact, 110,000, but I rounded off). Every day the site went about a thousand. Several hundred people a day made transactions. And these few hundreds of people were very unhappy ((((
Happenes. What to do ... Notified on the site that you can write to me on this issue in VK. I did the translation with my own hands and at the same time understood where the errors are in the code. Every day I hung about a hundred unread messages. Within a month, I woke up at 7 am, processed user requests, went to work, and processed requests in the evening again.
In general, people reacted to the situation with understanding, so it was not very smooth, but the situation was turned out. Gradually, there were fewer errors and the site began to work as it should.
But as they say, trouble does not come alone!
One evening in the office after work, I decided to file a site. I go, and he lies! mysql fell off. Prior to this, mysql sometimes crashed when the server was unable to cope with the load and I just rebooted it. But there was another case. It did not even start ... I spent the whole evening on the forums. Then I believed in Toster! Some guy just the whole evening answered my messages and helped me find a mistake! I was surprised that someone can help a complete stranger for free.
Plus, I didn’t have much experience with servers, so it was difficult. I only knew that there is a terminal and commands can be entered there. I googled, I was looking for which command I could enter, entered and got the result.
As a result, I came to the conclusion that the database is lost ... no ... Then I found out that there is an ibdata file and if it is empty, then all data is lost. As you can imagine, it was empty. Fully. 0kb
And the girl and I went to the Stand-up this evening, our friends played. I did not sit out until the end and asked to go home. It was there, in the hall, that the thought reached me that I somehow lost 100,000 users. There is not a single backup. There is not a single hope for recovery. What to do? Call and say that ass guys, I sow labor a few years of your work.
After the call and clarification of the situation, a consultation was held. At 12 at night. What to do?
I was telling the story that the server ran out of space (and that was in fact), and therefore mysql was inhibited. In fact, to the last I was sure that this was so. So, like, my fault is not here.
Thought-thought and decided to go to sleep and on the trail. day to figure out what can be done and what actually happened.
I asked for help in freelancing. Found there one system administrator who agreed to help me out. As a result, he confirmed my theory that most likely mysql fell off because of the finished space. Worse, he confirmed that the base could not be restored.
Customers asked for help from the web-studio, in which the programmer looked at the history of commands in the terminal and know what he found? The command that erases the ibdata file! Apparently on the forums, someone was amused to enter this command if mysql crashes, but I did not read between the lines in a panic and entered this command.
Well, I think everything. The end to me. But the customers realized that the problem should be solved and if they “kill” me, there will be no one to finish. Fortunately, there was a backup of the old site on which there were 80,000 people. The class. Now it remains to recover 30,000 people) A new phase of messages has begun in VK, restorations and panic.
Thanks to my girlfriend who supported me. Thanks to her for not leaving me at all) For it was not possible to communicate with me then, because every evening I screamed that it was hard for me and I no longer know what to do ... I do not like people who ache. And I didn’t love myself either.
With grief in half, everything was restored. In the end, I said that they should find a programmer on a permanent system administrator who will maintain and backup the server and live somehow without me. I no longer wanted to deal with this hellish project.
There should have been something else to fall off and break, but that was all. The site is still working. They never found a programmer for themselves (as far as I know), apparently still I do not have such crooked hands and now everything works as it should.
By the way, they didn't pay me 10 thousand. But I was somehow ashamed to ask them later ... therefore the project cost 50 thousand rubles and 6 months of suffering. After it was all over, I thought to myself: “But I got a lot of experience.” And really ... this project taught me a lot. And it will be wrong if I do not share them:
1) Carefully read the terms of reference and contracts
In fact, weighing Tusa-Dzhusa started because I was light on TK. If I had read more carefully into it, I understood that the stage with logic was far from complete and it needs to be added, then the dates would have moved and we calmly completed the project.
Be sure to read the contracts that are signed. And they must be compiled!
I now do not draw up a contract only if the customer is my friend or I have already worked with him and am confident in his good faith. In general, since I signed the letter, that every day overdue results in a lot of money, this turned out to be the situation described above.
2) Evaluate the project in terms of money.
Then I did not know how much to take for the project and for me the figure of 65 tr. was something unattainable. Now for such a project, I would have asked for 200-300 thousand. I realized that if the project has a large number of personal accounts, logical interconnections, then the project will not be a priori simple. And even if it seems that there is nothing to write there, then this will not happen.
3) Speak waste routes
You should always understand that a project can go beyond. Or the customer came up with many more different ideas that were not in the TK. Or you simply did not calculate and the deadline went yesterday, and the project is only in the middle.
Speak with customers this moment on the shore. At the crossing of horses do not change.
Now, when customers ask how much will be spent on the project, I say minimum and maximum. For example: “The project is really done in 2 weeks, but, objectively speaking, we will do it a month. Subject to revisions, additions and so on. ".
4) Make backups
Make backups, their mother! Do it! Be sure to do it!
I now have two of my working computers backed up to my hard drive. The server backed up every day, but after one situation, I had them created every hour and every day. In a nutshell, I didn’t keep track of the hard place and mysql broke down (the situation is like in this project), but I had a backup and, in general, he saved me).
After this incident, I thought ... And what if in fact really mysql broke because of the small hard disk space, but I deleted the ibdata and then it was impossible to prove anything? If this is so, then somehow even offensively ...
5) See what you enter into the terminal
See what commands, where and why enter !!! Now if I don’t know the command, I’ll google it before entering it into the terminal).
6) Separate responsibility
In fact, the situation with the server happened due to the fact that I was not in my domain. If you make a website, then make a website, and let those who know how to answer for it be responsible for the server. After this joke, they hired a system administrator, who then ruled a server).
7) Do not whine
In any not clear situation do not whine! Whining people seem miserable. You can "cry." If it is hard, go to a close person, ask to listen and pour out all the boiling water. But do not whine every night as you feel bad and hard.
From time to time I meet people who constantly whine. They can not be tolerated. With some I stopped communicating even because of this.
8) Take responsibility for your shoals
Nakosyachil - catch lyuley. If something went wrong, then call the customer and explain the situation as it is. They are people too, and they will understand, they will come into your position and even help solve the problem. Do not disappear) It will be worse only for you.
This applies not only to sites, in fact, but also to life.
9) We get high from force majeure
During all this, my friend told me one interesting thing, that all the problems come from our heads and if there are problems, we want them to be. And I realized that to some extent the buzz from this situation. I am a programmer masochist! I love force majeure and therefore attract them. What I love is not so that “Aaaaaa! I want to break deadlines! Let's make the site another 5 months for free at night. ” Just inside some small part of me sits and quietly enjoys what is happening.
Think about it if you constantly break deadlines.
10) Do not touch dubious projects.
Filter your orders. This project was initially dubious. But I needed money and experience. Now I eliminate 70% of orders and say that I don’t want to do them, they just aren’t really interesting to me. Another question that I can allow myself now, then I could not)
On the other hand, I took up this project and got a pretty decent experience. Therefore, the devil knows. Probably, after all, take on dubious projects - this is experience. But only once ...
11) Belief in Toster
I believed in Toster. I was really surprised when I realized that people simply respond to other people just like that. It was a great break for me. Now I constantly ask my questions there and they constantly help me out. And as far as possible, I answer others.
Recently there was even a situation when I was googling a question, a link to a toaster appeared in the search, and there I ask the same question and answered it myself in the commentary. I shut up.
Completion
Every item now lives with me every day. Every new order I take into account all my mistakes and no longer repeat them. After this project, I did not have short deadlines and frustrated customers. If force majeure situations arose, then they were immediately led back on track and resolved in peace negotiations.
Do not be afraid to make mistakes. Be careful not to draw conclusions from them. It was a hard lesson. But I do not regret that this happened to me.
Thanks for reading! I hope it was interesting.
Source: https://habr.com/ru/post/339178/
All Articles