Internet unprofitable things
Call the kids! Now Uncle Andrei will tell a Christmas horror story about NTP (Network Time Protocol).
Almost two years ago, on Monday, January 16, 2017, an interesting ticket from an unauthorized person entered our BitFolk bug report system . The sender introduced himself as NetThings UK Ltd. Lead Software Engineer.
What is so special about address
Here the important word was. At the end of December 2016, I brought BitFolk NTP servers out of the public pool and blocked them for outsiders.
I did this because because of the Snapchat NTP bug , they received unusually high traffic. In fact, it did not cause any huge problems, just such a volume of traffic pushed out useful information from the base of saving the Jump network network of a fixed size, and I didn’t want to deal with it during the holidays, so I simply turned off public access to the service.
This article appeared on Hacker News , and in a few comments there were asked to briefly explain what NTP is, so I added this non-technical section. If you know the technology, you can skip it.
Network Time Protocol is a tool by which a computer can use several other computers, often from all over the Internet, on completely different networks managed by different administrators to accurately determine the current time. When using several different computers, some of them give an inaccurate result, some can be broken or behave hostilely, but the protocol is able to recognize the "wrong" clock and take into account only the result from a more accurate majority.
It is assumed that NTP is used in a hierarchical order: hardware is directly connected to several servers to determine the exact time: atomic clocks, GPS, etc. They are called Stratum 1 servers. Second-level servers set their time on Stratum 1 servers, serve much more customers and so on.
In past times, it was difficult to find available NTP servers. In your own company you could have one or two such servers, but in fact you had to check with at least 3-7 servers, and better from different organizations. In a university environment, it was easier: you could talk with colleagues and share access to NTP. But as the Internet grew, the number of requests grew, including from commercial companies and individuals.
The NTP Pool project came to the rescue. Through a simple web-based interface, anyone added their own NTP servers to the pool: they were collectively served in a common DNS zone with some basic means for load balancing. A private individual was allowed to specify three names from the pool, and he would receive three different (constantly changing) NTP servers for free.
It was assumed that commercial companies would apply to a separate “vendor zone”. They donate a small fee to the project — and get the DNS zone allocated for their product, so it’s easier for the administrators of the pool to send traffic.
Unfortunately, many companies did not bother to study these subtleties and simply used the common pool area. NetThings UK Ltd. went even further in a very wrong direction - and took the IP address from the pool, simply using it directly and assuming that it will always be available. In fact, this free service was donated to the general pool by BitFolk, and due to temporary inconveniences, the service was turned off.
Going back to our story ...
The lead software engineer continued:
Hmm ...
Obviously,
Then I asked a few relevant questions to determine the scope of services that would have to be provided:
Answers to some questions were very disappointing.
Apparently, NetThings UK Ltd produced thermostats with remote control, lighting controllers for large retail spaces, etc. It seems that they recorded one of the BitFolk IP addresses in the firmware, and these devices could not be identified or remotely updated.
And yes, in any case, without an external time source, the watches of these devices will begin to noticeably shift within two weeks.
By the way, they solved their problem with a hard-coded IP address, raising the BitFolk IP address locally in their factory to set the initial date / time.
I admit, at that time there was a slight temptation to identify these devices, to give them completely wrong data - and see if some stores start to turn on and off the lights at a strange time of day.
Yes, they made each of their unidentified devices knock on a hard-coded IP address for a two-minute interval every Sunday evening.
The lead software engineer was very worried that they caused the excess flow I mentioned earlier, but I assured that this is definitely a Snapchat bug. In fact, their 500 devices never differed from the background noise. It turned out that half a thousand SNTP requests are a fairly light load. They did this for more than two years before the problem arose.
Of course, I noted their luck that we quickly noticed the problem, otherwise it could have ended as in the Netgear case versus the University of Wisconsin [when Netgear devices flood the University NTP server with requests - approx. trans.].
Forgiven. I must note that throughout this time it was a pleasure to work with the lead engineer.
In fact, BitFolk provides clients with an NTP service out of courtesy, this is not a paid service. In the end, who will pay for it if there is a public pool? For a commercial company, properly maintain a pool by subscribing to a vendor zone.
But NetThings UK Ltd. found themselves in a difficult situation, and a ban on access to the NTP server would cause them serious financial damage. Potentially, I could have asked for a lot of money at that moment, but I felt (no doubt, to the detriment of my pocket) that this is simply wrong.
For a start, I offered to pay me for two hours of consultations in order to cover the work already done on making changes to the firewall.
In addition, I offered to pay one hour of consultations per month for 12 months to cover the costs of continuing to work with the NTP server. Of course, I do not spend an hour a month fiddling with NTP, but such non-standard deviation from my usual work required a certain reward.
I really wanted to point out that this is not forever:
NetThings UK Ltd. gladly agreed to such a proposal.
In the future, I communicated with the lead software engineer only once. The rest of the correspondence was conducted with financial personnel, mainly because NetThings UK Ltd. did not like to pay bills on time.
For the year NetThings UK Ltd. paid late three of the four bills. In each case, I tried to charge them the statutory late charge fee.
When 2017 came to an end, I asked the lead software engineer how the situation with our IP address was in the firmware of their devices, how successfully did they solve the problem?
It was a little sad, because a year ago there were "about 500" devices. Despite the best efforts throughout the year, the number seems to have doubled.
This alone is enough to increase the fee, which I was still going to do because of the regular late payments on their part. For two months, I had previously notified them that the price would double.
Approximately 15 weeks after the price doubling report, NetThings UK Ltd. Chief Financial Officer asked why this happened while simultaneously reporting the transfer of one of the overdue payments:
I was very happy again to explain in detail why she doubled. The financial director in response tried to negotiate a fixed price for the year, to which I agreed, subject to payment in advance for the year.
My rationale was that the price increase was mainly due to late payments on their part: they take too much time, so if I save the quarterly payment, I need the opportunity to charge more if necessary. If they want guarantees, in my opinion, they must pay for it by making one annual payment.
No reply followed, so payments continued on a quarterly basis.
On November 20, 2018, we received a letter from Deloitte :
And then December 21:
Fortunately, their only unpaid bill was for service since November, so everything was paid.
This is the story of NetThings UK Ltd, a daring explorer of the Internet of Things, who thought that the NTP public pool is just an integral part of the Internet and everyone can use it for free, just choose one IP address at random and sew it into thousands of its devices that are distributed throughout country without the ability to remotely update.
This faith, combined with the innovative reluctance to pay for anything on time, unfortunately, was not enough to preserve solvency.
Almost two years ago, on Monday, January 16, 2017, an interesting ticket from an unauthorized person entered our BitFolk bug report system . The sender introduced himself as NetThings UK Ltd. Lead Software Engineer.
Subject: NTP request for IP85.119.80.232
')
Hello,
This may seem strange, but I need to configure the NTP server by IP address85.119.80.232.
What is so special about address
85.119.80.232 ? This is the IP address of one of the NTP servers to serve our customers. A few weeks before this ticket, the server was also part of the NTP Pool project.Here the important word was. At the end of December 2016, I brought BitFolk NTP servers out of the public pool and blocked them for outsiders.
I did this because because of the Snapchat NTP bug , they received unusually high traffic. In fact, it did not cause any huge problems, just such a volume of traffic pushed out useful information from the base of saving the Jump network network of a fixed size, and I didn’t want to deal with it during the holidays, so I simply turned off public access to the service.
NTP?
This article appeared on Hacker News , and in a few comments there were asked to briefly explain what NTP is, so I added this non-technical section. If you know the technology, you can skip it.
Network Time Protocol is a tool by which a computer can use several other computers, often from all over the Internet, on completely different networks managed by different administrators to accurately determine the current time. When using several different computers, some of them give an inaccurate result, some can be broken or behave hostilely, but the protocol is able to recognize the "wrong" clock and take into account only the result from a more accurate majority.
It is assumed that NTP is used in a hierarchical order: hardware is directly connected to several servers to determine the exact time: atomic clocks, GPS, etc. They are called Stratum 1 servers. Second-level servers set their time on Stratum 1 servers, serve much more customers and so on.
In past times, it was difficult to find available NTP servers. In your own company you could have one or two such servers, but in fact you had to check with at least 3-7 servers, and better from different organizations. In a university environment, it was easier: you could talk with colleagues and share access to NTP. But as the Internet grew, the number of requests grew, including from commercial companies and individuals.
The NTP Pool project came to the rescue. Through a simple web-based interface, anyone added their own NTP servers to the pool: they were collectively served in a common DNS zone with some basic means for load balancing. A private individual was allowed to specify three names from the pool, and he would receive three different (constantly changing) NTP servers for free.
It was assumed that commercial companies would apply to a separate “vendor zone”. They donate a small fee to the project — and get the DNS zone allocated for their product, so it’s easier for the administrators of the pool to send traffic.
Unfortunately, many companies did not bother to study these subtleties and simply used the common pool area. NetThings UK Ltd. went even further in a very wrong direction - and took the IP address from the pool, simply using it directly and assuming that it will always be available. In fact, this free service was donated to the general pool by BitFolk, and due to temporary inconveniences, the service was turned off.
Going back to our story ...
They want ... what?
The lead software engineer continued:
Recently, the NTP service was turned off, and I’m interested to know if there is any possibility to start it again at the specified IP address. Either through the current owner of the IP address, or through the migration of the current machine to another address, so that we can rent 85.119.80.232 .Hmm ...
I understand that this is a strange request, but I can assure you that it is sincere.
It will not work
Obviously,
85.119.80.232 used by all our clients as a resolver and NTP server. Ask them all to change the configuration in order to lease the address to NetThings UK Ltd. - this is not an option. So I just removed the firewall, so 85.119.80.232 began to work again for NetThings UK Ltd. until we figure out what can be done.Then I asked a few relevant questions to determine the scope of services that would have to be provided:
- How many clients do you use this server?
- Do you know their IP addresses?
- When do they need an NTP server and for how long?
- Can I get them to use the pool correctly (through the vendor zone)?
Deeper into the forest
Answers to some questions were very disappointing.
The server is partially used by our production system where the RTC was originally installed. Unfortunately, a fairly large amount of equipment (~ 500 units with weekly NTP calls) works on roaming GPRS SIM. I do not know whether in this case it is possible to rely on the initial IP address of the APN to configure the firewall (I will check). We also can not remotely update the firmware on these devices, because they have a quota for traffic of 5 MB per month. We can update them locally, but this will take months, not weeks.
Apparently, NetThings UK Ltd produced thermostats with remote control, lighting controllers for large retail spaces, etc. It seems that they recorded one of the BitFolk IP addresses in the firmware, and these devices could not be identified or remotely updated.
And yes, in any case, without an external time source, the watches of these devices will begin to noticeably shift within two weeks.
By the way, they solved their problem with a hard-coded IP address, raising the BitFolk IP address locally in their factory to set the initial date / time.
I admit, at that time there was a slight temptation to identify these devices, to give them completely wrong data - and see if some stores start to turn on and off the lights at a strange time of day.
Weekly??
NTP calls from us starts weekly crowns without load balancing on the client side. This leads to a flow of requests at the same time every Sunday at about 7:45 pm.
Yes, they made each of their unidentified devices knock on a hard-coded IP address for a two-minute interval every Sunday evening.
The lead software engineer was very worried that they caused the excess flow I mentioned earlier, but I assured that this is definitely a Snapchat bug. In fact, their 500 devices never differed from the background noise. It turned out that half a thousand SNTP requests are a fairly light load. They did this for more than two years before the problem arose.
Of course, I noted their luck that we quickly noticed the problem, otherwise it could have ended as in the Netgear case versus the University of Wisconsin [when Netgear devices flood the University NTP server with requests - approx. trans.].
I feel very, very bad because of this. I am very, very sorry if we caused your problems.
Forgiven. I must note that throughout this time it was a pleasure to work with the lead engineer.
We made a deal
In fact, BitFolk provides clients with an NTP service out of courtesy, this is not a paid service. In the end, who will pay for it if there is a public pool? For a commercial company, properly maintain a pool by subscribing to a vendor zone.
But NetThings UK Ltd. found themselves in a difficult situation, and a ban on access to the NTP server would cause them serious financial damage. Potentially, I could have asked for a lot of money at that moment, but I felt (no doubt, to the detriment of my pocket) that this is simply wrong.
For a start, I offered to pay me for two hours of consultations in order to cover the work already done on making changes to the firewall.
In addition, I offered to pay one hour of consultations per month for 12 months to cover the costs of continuing to work with the NTP server. Of course, I do not spend an hour a month fiddling with NTP, but such non-standard deviation from my usual work required a certain reward.
I really wanted to point out that this is not forever:
Finally, this is a penalty. It seems that you are now in a difficult situation, and there is a temptation to charge you the maximum amount (in any case, much more than ÂŁ 840 + VAT), but it seems to me unfair. However, the provision of NTP services to third parties is not our business, so we expect the contract to end within 12 months. If you ultimately have to extend this service, it will mean that we did not charge you enough and we will increase the price.
Does this seem reasonable?
NetThings UK Ltd. gladly agreed to such a proposal.
Thanks again for the info and help. You saved me from a lot of stupid and useless work. We have enough time to fix it.
Bumps on the road
In the future, I communicated with the lead software engineer only once. The rest of the correspondence was conducted with financial personnel, mainly because NetThings UK Ltd. did not like to pay bills on time.
For the year NetThings UK Ltd. paid late three of the four bills. In each case, I tried to charge them the statutory late charge fee.
Disappointing results of the year
When 2017 came to an end, I asked the lead software engineer how the situation with our IP address was in the firmware of their devices, how successfully did they solve the problem?
In short, most of our products managed to get rid of the use of a fixed IP address. It remains to update another project, after which the production of new units with such firmware will be completely discontinued. But we still have about 1,000 units in production that are not easy to update: they will continue to send weekly NTP requests to a fixed IP address. Therefore, answering your question: yes, we still need a service in January 2018.
It was a little sad, because a year ago there were "about 500" devices. Despite the best efforts throughout the year, the number seems to have doubled.
This alone is enough to increase the fee, which I was still going to do because of the regular late payments on their part. For two months, I had previously notified them that the price would double.
Want to argue?
Approximately 15 weeks after the price doubling report, NetThings UK Ltd. Chief Financial Officer asked why this happened while simultaneously reporting the transfer of one of the overdue payments:
Date: Wed, 21 Feb 2018 14:59:42 +0000
Already paid, but could you explain why the price doubled?
I was very happy again to explain in detail why she doubled. The financial director in response tried to negotiate a fixed price for the year, to which I agreed, subject to payment in advance for the year.
My rationale was that the price increase was mainly due to late payments on their part: they take too much time, so if I save the quarterly payment, I need the opportunity to charge more if necessary. If they want guarantees, in my opinion, they must pay for it by making one annual payment.
No reply followed, so payments continued on a quarterly basis.
That's the end of the tale
On November 20, 2018, we received a letter from Deloitte :
Netthings Limited - In Administration (“The Company”)
Company Number: SC313913
[...]
Termination
The company ceased operations from November 15, 2018.
Investigation
As part of our responsibilities as a Property Manager, we investigate which assets belong to the company and what funds can be recovered for creditors, as well as how the company’s operations were conducted.
And then December 21:
In accordance with paragraph 51 (1) (b) of the Insolvency and Bankruptcy Act of 1986, the Asset Manager is not obliged to convene a meeting of creditors unless the company has enough funds to distribute to unsecured creditors or if the meeting is required by form SADM_127 unsecured creditors with 10% or more of the cost of debt. There are no funds for distribution among unsecured creditors, so the meeting will not be convened.
Fortunately, their only unpaid bill was for service since November, so everything was paid.
This is the story of NetThings UK Ltd, a daring explorer of the Internet of Things, who thought that the NTP public pool is just an integral part of the Internet and everyone can use it for free, just choose one IP address at random and sew it into thousands of its devices that are distributed throughout country without the ability to remotely update.
This faith, combined with the innovative reluctance to pay for anything on time, unfortunately, was not enough to preserve solvency.
Source: https://habr.com/ru/post/434422/
All Articles