How to get to Hell because of Helm, but grab onto a straw

- To tired hipsters we speak the truth.

All of us (that is, I) love to pull everything new and brilliant in production in order to finally replace some problems with others. This story is dedicated to us (that is, to me).

To understand and forgive further text, you will need superficial knowledge of how to work with Kubernetes at the user level and some rumors about Helm .

Let's first abstract, and then let someone deal with it. Imagine for a moment that we are a kind of Columbus in the world of smoothies, electric scooters and Kubernetes. Our people are crowding Europe in an overpopulated old woman in one of its small, endless states, deploying deployment with Kronjobes on a daily basis under the crunch of quadcopters. But astronomers have already found an explanation for the suspicious curvature of the horizon. And there is a direct feeling that we are the same Chuzen Van of modernity. (But no one asked.) And somewhere there, beyond the semicircular ocean, is the shortest way to India! Endless spaces, freedom from the burden of routine and many free flavored spices. You just need to bring our people there and finally release them! Pushing the oceans in front of you is too much even for such magnificence as we are. Therefore, it is necessary to build ships, pack our deployments, Krondobs and other demons in them, raise services instead of sails and steer straight into the light there. To steer need a steering wheel, that is Helm. He is a helmet. A helmet would definitely be useful, if we know ahead what awaits us ahead. But there is only a steering wheel. Building ships is not easy, so we need help, as it were, but our people are always busy with something very useless. Therefore, it is necessary as if by oneself and as always. Slowly we start, we build a small boat, we row to India, we see the vast expanses with our own eyes, we take one spice, we float back, we show people. People like it, they bless our attempts. Then we are preparing a great outcome, we put in order the configurations, we are building many ships already. People come up, are interested. We show, tell, educate, promise, promise, promise ... The more ships - the more interest. Someone joins, helps. So that all this is not idle, we swim in staging nearby, there are no spices yet, it is impossible to live there, but tourists like it. And now, after weeks / months / best-years / sleeves he, THAT DAY, has come! It is time to release the flotilla and subdue the treasured terra incognita. With trepidation in our hearts, we are sailing our people to India, and there, nuttyponel, America. And everything seems to be very similar, but this is a premonition ... Our people, of course, immediately come to taste (haha, they don't care at all anyway, in fact), they begin to explore the territory, cabins there, cabbage, all kinds of services are deployed. But here and there something periodically disappears, as through the earth. That floor will disappear, the coconut will not give birth. And we are mumbling with the growing buzz “ wait a minute, wait, I'm not ready yet, just give me a day ...”. And still so we squint attentively, and there, in front, behind every bush SUDDENLY, the Indians with tomahawks appear and look so ... unkindly or something ... And the chill runs through the inter-aural ganglion. And the people are: - “And what is this over there? And where is my floor? ”And we are:“ My people! It’s not that I’ve misled you for 40 years, and no one, of course, will bury you back, but it’s urgent right away, *****, IT IS NECESSARY TO BUILD THE FREQUENCES !!!!!! ”And they are: -“ And , well, of course, as always, again. "
')
That's about how I felt when I transferred a certain number of services in production from pure Kubernetes to Helm, and then ran into it .

Well, at the end of the promised saving straw. First, the light version, but with an explanation of the above delusions. Here is a demo script:

1. Suppose I deploy a chart for project: 1.5. For the first time with Helm, and before that was just Kubernetes.
2. Then I find that there is a bug in the release, and in version 1.4 it was not. And it would be necessary to roll back, but for her and Helm the chart was not there either. Therefore, I decide to do it in the old manner: kubectl set image deployment/project project=registry.project.com/project:1.4 --record . For this and for the pack of other services that unfolded together.
3. Then it turns out that the bug is not in this service, but in the next one, and this is all well and it is necessary to return 1.5. If you call helm upgrade --install , then a big surprise awaits you ( more details ): the image will still be from 1.4, and the labels from 1.5. And Helm shows that everything is fine, generally 1.5 has been deployed there and even the pods have been restarted (CI-build is green).

How to avoid it? If any change is made to any K8s resource controlled by Helm with pure kubectl commands on top of the Helm Chart deployed, then these changes should also be canceled with the kubectl commands. Helm can deploy a new chart. But he compares the new chart with the previous one, but not with the current state of resources. And if you have edited the image, then the future version of Chart will probably contain another image and everything will be fine. But if you edited an environment variable, or launch arguments, or something else, then the new version of Chart most likely does not differ from the previous one. And your manual changes will remain in place after the update.

And for a snack, a hard version of the saving straw for those who cannot agree with this unpredictable state.

New technologies - the source of new sorrows.