As I took the data from the BLE-thermometer from Xiaomi
Background: as one of my hobbies, I had a “smart home”. I want beautiful devices, but I still want freedom and privacy. Therefore, I am engaged in crossing Xiaomi uzhik with a hedgehog Home Assistant .
To maintain comfortable conditions, we need to know what is happening at home in general. In short, we need sensors. Xiaomi has many different ones, but most of all I liked the square thermometer on electronic ink. But it’s not at all clever, in the sense that it doesn’t provide any interfaces at all, except for the graphic one - neither WiFi, nor BLE, nor ZigBee. But CR2032 batteries last for several years. There is another version with bluetooth, but it is a little less elegant - a kind of thick pancake.
And in the beginning of spring a new temperature / humidity sensor was announced, on electronic ink, with BLE, and even with a clock. I don’t need a watch especially, but everything else immediately suppressed all rational arguments and the thermometer was ordered at one of the popular online stores, on preorder. It went riding, and finally arrived.
')
The sensor was added to the MiHome application without problems (I have an English-language interface everywhere, with the Russian version of MiHome, they say there were difficulties in translation). Shows current values and change history.
But with the integration into the Home Assistant complexity has happened. The existing component for the temperature sensor did not want to take any data from the device and complained about the wrong data format. Well, nothing to do, we get a shovel and start digging.
My first thought was to familiarize myself with the BLE protocol device, but after assessing the size of the documentation, it was decided to switch to the national custom method.
To begin with, open the terminal on ubunt and launch the bluetoothctl. We see the following:
MJ_HT_V1 is an old temperature sensor, LYWSD02 is a new one. The difference in the model naming format is somewhat alarming.
Then we need to somehow read, and what data we can get at all. Opened the mitemp library sources , which is used in Home Assistant to get data from the old sensor. There I found that the library used blewrap, which, in turn, is a wrapper over two Python libraries to work with BLE. So many layers to me to anything, we will use bluepy . There is documentation, there is not a lot of it and not a little, we read and write a script that goes through all the data fields that are on the device.
In general, everything is simple - the BLE device provides a set of services, each of which consists of a set of characteristics. Each characteristic can be of one of 8 types, for one characteristic it is possible to specify several types simultaneously. Services and features are identified in two ways - an address in the form of HEX values and a UUID. With UUID I work somehow familiar.
So, I counted all the characteristics for both sensors, looked at them and realized that devices from completely different manufacturers are being sold again under the Xiaomi brand. Among the values of the old sensor was found "Cleargrass Inc", and in the new - "miaomiaoce.com". The structure of services and characteristics of these two sensors are also completely different, and the list of characteristics of the new sensor is twice as long. It became clear that you need to write your own library to integrate with the sensor (no, of course I google first, maybe there is something useful at the request of LYWSD02, but I did not give anything sensible to Google).
Among the available types of characteristics, besides READ, there is also WRITE and NOTIFY. WRITE is for sending data to the device, and NOTIFY is for receiving data. There is still WRITE NOTIFY at the same time - the device will send data only after they subscribe to it, sending the required byte with the WRITE command.
Attempts to do something with my hands did not bring any result, the first line of despair was reached, but at that moment I read articles about crafts based on chips from Nordic Semiconductors and put the nRF Connect program on a smartphone. With it, I was able to subscribe to all the services provided by the device, saved the logs of the answers and began to try to understand what was in them.
These triple arrows activate the subscription.
The peculiarity of the old sensor was that the data on temperature and humidity came in the form of a UTF string, while the new one gave up everything in binary form.
To receive data from the sensor, you need to send a subscription request. In the mitemp library for this, two bytes were sent to the characteristic, but it is not clear where to get it. Then I looked at how the data structure for the old sensor in nRF Connect looks and noticed that the correct address was specified for the characteristic with the data as a kind of descriptor. Then I began to read the documentation for bluepy again and I realized that the address of the descriptor can be easily obtained from the object characteristics. It remains only to write a class with a callback method, which will receive data from the notification.
Fortunately, only three characteristics were labeled as WRITE NOTIFY, with the data coming at different frequencies and, uh ..., visual features.
The first request sent immediately a large data dispenser, and then it stuck. In this case, the first byte was a monotonically increasing number. It appears that this is an accumulated history of averages.
The second and third were sent periodically, but looking closer, I saw that one of them does not change, and in the data of the second only one byte changes. Well, it means that this is the current time (I remind you that there is a clock in this thermometer. There should be a clock in any self-respecting device for a smart home).
Suppose the third characteristic is useful data on temperature and humidity. To confirm the hypothesis, a physical experiment was carried out - went up to the sensor and rudely breathed on it. The display has dramatically increased data values, and in the terminal - the Baitics have changed. Cheers, data somewhere nearby.
I usually work with textual data (get data via HTTP in the form of JSON / xml, put it in a file or in a database), so I didn’t really know how to approach a task. Therefore, I began to try to transform the data in different ways that can be made from python. I wrote such a function-converter and began to look at how it relates to the data on the sensor screen.
Lines of varying degrees of incomprehensibility fell in the console, but the third byte was always a number, and this number coincided with the value of humidity. To be sure, I once again breathed on the sensor - and the humidity values on the screen and in the third byte changed the same way!
Here I assumed that the temperature is stored in the first two bytes. In order for the data to change - transferred the sensor to the heated towel rail in the bathroom. But, no matter how hard I tried to transform the results, the necessary numbers did not work.
At this point, I once again looked at the description of the sensor and saw that there was a Swiss Sensirion sensor inside. It was probably worth starting with this, but this is not our method. A package of sensors was found on the Swiss Sensirion website, and datasheets for them. In the datasheet, among other things, a formula was found for converting bytes transmitted via the I2C bus into a number.
But ... It turned out very strange values. Something like -34.66, and I was obviously warmer. From sadness and sadness, I even opened the sensor and checked if the sensor from Swiss Sensirion was true. It turned out that it is true, but with the SHTC3 index, and a slightly different formula is needed for it.
However, all the same, the data after the conversion was not even close to real ones. Then I became even more sad, opened the source code of the library for SHTC3 from Adafruit and began to try to adapt the transformation code from C ++ to python. Deduced all in a label - the raw data, transformed sishny structure and result.
Got something like this:
Yes ... somehow cold ... But, wait, wait, what is this 2604? This is it, 26.0 degrees on the screen! To confirm the hypothesis, again took the sensor to the battery, checked - the values are the same.
As a result, we get the following data conversion code:
A couple of evenings went on operations to connect to the sensor and search for the right transformation algorithm. I wanted to quit everything several times, but at the same time new ideas came along and I kept trying.
Now the data is transferred to the Home Assistant, then you need to finish the integration code and maybe rewrite it from bluepy to bleak, since bleak uses async / await and is better suited for Home Assistant, written by aiohttp.
To maintain comfortable conditions, we need to know what is happening at home in general. In short, we need sensors. Xiaomi has many different ones, but most of all I liked the square thermometer on electronic ink. But it’s not at all clever, in the sense that it doesn’t provide any interfaces at all, except for the graphic one - neither WiFi, nor BLE, nor ZigBee. But CR2032 batteries last for several years. There is another version with bluetooth, but it is a little less elegant - a kind of thick pancake.
And in the beginning of spring a new temperature / humidity sensor was announced, on electronic ink, with BLE, and even with a clock. I don’t need a watch especially, but everything else immediately suppressed all rational arguments and the thermometer was ordered at one of the popular online stores, on preorder. It went riding, and finally arrived.
')
The sensor was added to the MiHome application without problems (I have an English-language interface everywhere, with the Russian version of MiHome, they say there were difficulties in translation). Shows current values and change history.
But with the integration into the Home Assistant complexity has happened. The existing component for the temperature sensor did not want to take any data from the device and complained about the wrong data format. Well, nothing to do, we get a shovel and start digging.
My first thought was to familiarize myself with the BLE protocol device, but after assessing the size of the documentation, it was decided to switch to the national custom method.
The first approach to the projectile
To begin with, open the terminal on ubunt and launch the bluetoothctl. We see the following:
[NEW] Controller 00:1A:7D:DA:71:13 fett [default] [NEW] Device 3F:59:C8:80:70:BE LYWSD02 [NEW] Device 4C:65:A8:DC:0D:AF MJ_HT_V1 MJ_HT_V1 is an old temperature sensor, LYWSD02 is a new one. The difference in the model naming format is somewhat alarming.
Then we need to somehow read, and what data we can get at all. Opened the mitemp library sources , which is used in Home Assistant to get data from the old sensor. There I found that the library used blewrap, which, in turn, is a wrapper over two Python libraries to work with BLE. So many layers to me to anything, we will use bluepy . There is documentation, there is not a lot of it and not a little, we read and write a script that goes through all the data fields that are on the device.
from bluepy import btle mac = '3F:59:C8:80:70:BE' p = btle.Peripheral(mac) for s in p.getServices(): print('Service:', s.uuid) for c in s.getCharacteristics(): print('\tCharacteristic:', c.uuid) print('\t\t', c.propertiesToString()) if c.supportsRead(): print('\t\t', c.read()) In general, everything is simple - the BLE device provides a set of services, each of which consists of a set of characteristics. Each characteristic can be of one of 8 types, for one characteristic it is possible to specify several types simultaneously. Services and features are identified in two ways - an address in the form of HEX values and a UUID. With UUID I work somehow familiar.
So, I counted all the characteristics for both sensors, looked at them and realized that devices from completely different manufacturers are being sold again under the Xiaomi brand. Among the values of the old sensor was found "Cleargrass Inc", and in the new - "miaomiaoce.com". The structure of services and characteristics of these two sensors are also completely different, and the list of characteristics of the new sensor is twice as long. It became clear that you need to write your own library to integrate with the sensor (no, of course I google first, maybe there is something useful at the request of LYWSD02, but I did not give anything sensible to Google).
How to get the data?
Among the available types of characteristics, besides READ, there is also WRITE and NOTIFY. WRITE is for sending data to the device, and NOTIFY is for receiving data. There is still WRITE NOTIFY at the same time - the device will send data only after they subscribe to it, sending the required byte with the WRITE command.
Attempts to do something with my hands did not bring any result, the first line of despair was reached, but at that moment I read articles about crafts based on chips from Nordic Semiconductors and put the nRF Connect program on a smartphone. With it, I was able to subscribe to all the services provided by the device, saved the logs of the answers and began to try to understand what was in them.
These triple arrows activate the subscription.
The peculiarity of the old sensor was that the data on temperature and humidity came in the form of a UTF string, while the new one gave up everything in binary form.
Subscribe to notifications
To receive data from the sensor, you need to send a subscription request. In the mitemp library for this, two bytes were sent to the characteristic, but it is not clear where to get it. Then I looked at how the data structure for the old sensor in nRF Connect looks and noticed that the correct address was specified for the characteristic with the data as a kind of descriptor. Then I began to read the documentation for bluepy again and I realized that the address of the descriptor can be easily obtained from the object characteristics. It remains only to write a class with a callback method, which will receive data from the notification.
class MyDelegate(btle.DefaultDelegate): def handleNotification(self, cHandle, data): print(data) mac_addr = '3F:59:C8:80:70:BE' p = btle.Peripheral(mac_addr) p.setDelegate(MyDelegate()) uuid = 'EBE0CCC1-7A0A-4B0C-8A1A-6FF2997DA3A6' # , ch = p.getCharacteristics(uuid=uuid)[0] # desc = ch.getDescriptors(forUUID=0x2902)[0] # , desc.write(0x01.to_bytes(2, byteorder="little"), withResponse=True) while True: p.waitForNotifications(5.0) Separate the wheat from the chaff
Fortunately, only three characteristics were labeled as WRITE NOTIFY, with the data coming at different frequencies and, uh ..., visual features.
The first request sent immediately a large data dispenser, and then it stuck. In this case, the first byte was a monotonically increasing number. It appears that this is an accumulated history of averages.
The second and third were sent periodically, but looking closer, I saw that one of them does not change, and in the data of the second only one byte changes. Well, it means that this is the current time (I remind you that there is a clock in this thermometer. There should be a clock in any self-respecting device for a smart home).
Suppose the third characteristic is useful data on temperature and humidity. To confirm the hypothesis, a physical experiment was carried out - went up to the sensor and rudely breathed on it. The display has dramatically increased data values, and in the terminal - the Baitics have changed. Cheers, data somewhere nearby.
Parsing data
I usually work with textual data (get data via HTTP in the form of JSON / xml, put it in a file or in a database), so I didn’t really know how to approach a task. Therefore, I began to try to transform the data in different ways that can be made from python. I wrote such a function-converter and began to look at how it relates to the data on the sensor screen.
def parse(v): print([x for x in v]) print('{0:#x}'.format(int.from_bytes(data, byteorder='big'))) print('{0:#x}'.format(int.from_bytes(data, byteorder='little'))) Lines of varying degrees of incomprehensibility fell in the console, but the third byte was always a number, and this number coincided with the value of humidity. To be sure, I once again breathed on the sensor - and the humidity values on the screen and in the third byte changed the same way!
Here I assumed that the temperature is stored in the first two bytes. In order for the data to change - transferred the sensor to the heated towel rail in the bathroom. But, no matter how hard I tried to transform the results, the necessary numbers did not work.
On the way to success
At this point, I once again looked at the description of the sensor and saw that there was a Swiss Sensirion sensor inside. It was probably worth starting with this, but this is not our method. A package of sensors was found on the Swiss Sensirion website, and datasheets for them. In the datasheet, among other things, a formula was found for converting bytes transmitted via the I2C bus into a number.
But ... It turned out very strange values. Something like -34.66, and I was obviously warmer. From sadness and sadness, I even opened the sensor and checked if the sensor from Swiss Sensirion was true. It turned out that it is true, but with the SHTC3 index, and a slightly different formula is needed for it.
However, all the same, the data after the conversion was not even close to real ones. Then I became even more sad, opened the source code of the library for SHTC3 from Adafruit and began to try to adapt the transformation code from C ++ to python. Deduced all in a label - the raw data, transformed sishny structure and result.
def handleNotification(self, cHandle, data): temp = data[:2] humid = data[2] unpacked = struct.unpack('H', temp)[0] print(data, unpacked, -45 + 175 * unpacked / 2 ** 19, sep='\t') Got something like this:
b',\n2' 2604 -44.130821228027344 b'-\n2' 2605 -44.1304874420166 b'+\n2' 2603 -44.131155014038086 b',\n2' 2604 -44.130821228027344 Yes ... somehow cold ... But, wait, wait, what is this 2604? This is it, 26.0 degrees on the screen! To confirm the hypothesis, again took the sensor to the battery, checked - the values are the same.
As a result, we get the following data conversion code:
def handleNotification(self, cHandle, data): humid_bytes = data[2] temp_bytes = data[:2] humidity = humid_bytes temperature = struct.unpack('H', temp_bytes)[0] / 100 print(temperature, humidity) Epilogue
A couple of evenings went on operations to connect to the sensor and search for the right transformation algorithm. I wanted to quit everything several times, but at the same time new ideas came along and I kept trying.
Now the data is transferred to the Home Assistant, then you need to finish the integration code and maybe rewrite it from bluepy to bleak, since bleak uses async / await and is better suited for Home Assistant, written by aiohttp.
References:
Source: https://habr.com/ru/post/452558/
All Articles