What we should fix a bug, which is "no"

So, we have a task: fix a bug, a manufacturer from which it denies, customers do not notice, but I want to live. There is a camera, the flow from it to UDP just breaks hell, the stream to TCP works, but connections are constantly broken (and at every break 3-5 seconds video disappears). Everyone is guilty of the problem (both the camera and the software), but both sides claim that they are all hurt, that is, the situation is ordinary: do you see a bug? not. And he is .

Since the software is updated much more often than the camera, it makes sense to edit the place that you don’t have to touch. So, we will fix from the side of the camera.

Springboard study

First of all, we take the latest firmware (in my case, firmware_TS38ABFG031-ONVIF-P2P-V2.5.0.6_20140126120110.bin), and find out what it is:

  $ file firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin
 firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin: data

 $ du -b firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin
 15222724 firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin

 $ xxd firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin |  head
 0000000: 4649 524d 5741 5245 6481 db15 c447 e800 FIRMWAREd .... G ..
 0000010: 0300 0000 1406 0000 b0f1 1b00 4c21 815d ............ L !.]
 0000020: 5453 3338 4f45 4d41 4246 475f 4c49 4e55 TS38OEMABFG_LINU
 0000030: 5800 0000 0000 0000 0000 0000 0000 0000 X ...............
 0000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
 0000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
 0000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
 0000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
 0000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................

 $ binwalk firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin
 DECIMAL HEX DESCRIPTION
 -------------------------------------------------- -------------------------------------------------- ---
 1556 0x614 uImage header, header size: 64 bytes, header CRC: 0xB21E2C9F, created: Sun Sep 22 11:07:02 2013, image size: 1831280 bytes, Data Address: 0x80008000, Entry Point: 0x80008000, data CRC: 0x1F4EFBAB, OS : Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-2.6.18_pro500-davinci_IPNC"
 14468 0x3884 gzip compressed data, from Unix, last modified: Sun Sep 11 11:07:02 2013, max compression
 1832900 0x1BF7C4 CramFS filesystem, little endian size 13389824 version # 2 sorted_dirs CRC 0xc832a8c3, edition 0, 7334 blocks, 2607 files 

So, its format is unknown, the start label “FIRMWARE” evokes the idea that this is something of its own, the presence of a kernel and a cramfs file inside uImage suggests that in fact it is something simple. The presence of the line TS38OEMABFG_LINUX suggests that this is something that even resembles some version of the archive.
')
Since now we just need to find where to look - just pull out the file system from there, and look for the guilty module:

  $ dd if = firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin bs = 1832900 skip = 1 of = cramfs
 7 + 1 records received
 7 + 1 records sent
  13389824 bytes (13 MB), 0.161755 seconds, 82.8 MB / s
 $ fakeroot cramfsck -x fs cramfs
 $ grep LIVE555 -R fs /
 The binary file fs / opt / topsee / rtsp_streamer matches
 $ strings fs / opt / topsee / rtsp_streamer |  grep TCP
 sendRTPOverTCP
 12RTCPInstance
 sendRTPOverTCP failed, sock:% d, chn:% d
  is not a RTCP instance
 RTCPInstance :: RTCPInstance error: totSessionBW parameter should not be zero!
 RTP / AVP / TCP
 % sTransport: RTP / AVP / TCP; unicast; destination =% s; source =% s; interleaved =% d-% d
 / TCP; unicast
 Failed to create RTCP socket (port% d)
 MediaSession :: initiate (): unable to create RTP and RTCP sockets
 Failed to create RTCP instance
 Received RTCP "BYE" on "
 18RTCPMemberDatabase 

Hohoho! “SendRTPOverTCP failed, sock:% d, chn:% d” tells us that the code is translated with debug prints, which means that the amount of work is reduced by orders of magnitude!

So, we have a module that contains the desired error, a module with a bunch of debugging lines inside, which means that the process of unzipping is much easier.

Localization and fix problems

We load the module into the disassembler, search for the “OverTCP” debug string, from it look for the code that calls the printout => we found the function sendRTPOverTCP.
Looking through it, we quickly see two calls to the send () function — one with 4 bytes, one with a buffer passed to the input. So, we got the version is not the oldest, and when we have already merged the buffer, but have not yet done the sendDataOverTCP function (details about the differences in the implementation - in the past <post .

Now the question arises how to fix the bug in binary form, when there is almost no stock in place (there is no empty space inside the file).
Go to the function above, which calls sendDataOverTCP - sendPacket. Its code from version to version has not changed, it is essentially one - foreach (streams) {sendDataOverTCP (packet, stream)}.

Fortunately, the code was generously stuffed with debugging fprintf'ami, and it saves us! Here is how this cycle looks in binary form:

loop_next_5FAE4: LDR R4, [R4,#4] CMP R4, #0 BEQ loc_5FB68 loop_body_5FAF0: MOV R3, R4 MOV R1, R5 MOV R2, R7 MOV R0, R6 BL sendRTPOverTCP CMP R0, #0 BGE loop_next_5FAE4 MOV R1, #0 LDR R2, =aS_10 ; "%s(): " LDR R3, =aSendpacket ; "sendPacket" MOV R0, #STDERR_FILENO BL fprintf_0 LDRB R12, [R4,#0xC] LDR R3, [R4,#8] MOV R1, #7 LDR R2, =aSendrtpovert_0 ; "sendRTPOverTCP failed, sock: %d, chn: %"... MOV R0, #STDERR_FILENO STR R12, [SP,#0x350+var_350] BL fprintf_0 ...... 

This is actually a salvation! Simply cutting the debug piece gives us space in the amount of 12 instructions (for ARM, all instructions are exactly 4 bytes, and this is very good).

So, we have a place in 12 instructions to do something to improve the situation. But what? Fully cramming in here the sendDataOverTCP code from the latest version will be very difficult ...
Although stop. What for? I, like, described in detail that even using the correct form sendDataOverTCP is still bad ... And if you don’t see the difference, why not just wrap the call in makeSocketBlocking () .. makeSocketNonBlocking ()?

Indeed, if the place in the system buffer is - send () will be executed instantly. If there is no space, then their implementation of sendDataOverTCP will still stick (for why it will stick and not fall out immediately with zero - see the previous post).

Fine! By fast unwinding back from the fcntl function, we find makeSocketBlocking and makeSocketNonBlocking, after which we draw what should get the code:

 loop_next_5FAE4: LDR R4, [R4,#4] CMP R4, #0 BEQ loc_5FB68 loop_body_5FAF0: ;     LDR R0, [R4,#8] BL makeSocketBlocking ;      MOV R3, R4 MOV R1, R5 MOV R2, R7 MOV R0, R6 BL sendRTPOverTCP ;     STMFD SP!, {R0} ;     LDR R0, [R4,#8] BL makeSocketNonBlocking ;      LDMFD SP!, {R0} ;      CMP R0, #0 BGE loc_5FAE4 ;       NOP NOP NOP NOP NOP NOP 

In order to patch, either open the documentation on the arm and broadcast in the mind, or write the code in a separate file and broadcast it (do not forget to set exact addresses with ORGs so that all transitions (BL / BGE / ITA) are recalculated correctly) I just found suitable instructions in the code and based on them I calculated the necessary opcodes (I edit ARM for the first time, sorry).

As a result, we get rtsp_streamer with a patch imposed on it that protects the TCP stream from damage.

Explosion soldering, assembly sober

So, we have a new rtsp_streamer, and there is a firmware ... bin in which to embed it. Well, it seems, everything is simple: you need to unpack cramfs, replace the file, pack it back, replace it inside the bin:

  $ fakeroot -s .fakeroot cramfsck -x repack cramfs
 $ cp rtsp_streamer repack / opt / topsee /
 $ fakeroot -i .fakeroot mkcramfs repack newcramfs
 $ dd if = firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin of = firmware_new.bin bs = 1832900 count = 1
 $ cat newcramfs >> firmware_new.bin 

Fill the received firmware_new.bin into the camera ... 0 effect. The camera eats the firmware, but nothing happens. Unpleasant So, you need to understand the format of this .bin.

Open it in the hex editor, and start to add code:
(0) "FIRMWARE" - 100% header, 8 bytes.
(8) 64 81 DB 15 - 4 bytes, the destination is not clear. by smell - checksum
(12) 0x00E847C4 = 15222724 - yeah, 4 bytes, the size of the firmware. check _new.bin - no, the size has not changed, so he has nothing to do with it.
(16) 0x00000003 - 4 bytes xs that. header version can?
(20) 0x00000614 = 1556 - so, and this is the offset to the core inside
(24) 0x001BF1B0 = 1831344 - and this is the size of the core (1831344 + 1556 = 1832900)
(28) 4C 21 81 5D - hmm. again something similar to the checksum.
(32) "TS38OEMABFG_LINUX" and a bunch of zeros later - 100h bytes, clearly the place under the section name
(288) 0x001BF7C4 = 1832900 - aha, offset to the next section
(292) 0x00CC5000 = 13389824 - yeah, section size
(296) "TS38OEMABFG_V2.5.0.6" and a bunch of zeros - opachki. 100h bytes, clearly under the section name.
But there is no checksum before it O_O
(552-1556) - something of an unknown appearance.

So, about the essence is clear. The size of our cramfs has not changed, so this is not the size, which means, the checksums.
Extract the kernel, and consider its checksum 4 bytes long:

  $ dd if = firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin of = kernel bs = 1 skip = 1556 count = 1831344
 $ crc32 kernel 
 5d81214c 

Well, well ... at offset 28 just 0x5D81214C. Lucky - this is the standard CRC32. It was lucky because according to the standard tool it can only count it. Otherwise, I would have to launch a python and consider it “more difficult” :)

So, our checksums are crc32. And what is the checksum of the original cramfs? .. 37499eef. Well, well, well. At offset 552, 0x37499eef is written. So, for some reason, for the file system, the checksum AFTER the partition name is recorded. Well, OK, what are we, we are not proud. We update the label:
(28) 0x5D81214C - crc32 kernel partition
(552) 0x37499eef - crc32 of the FS section
(556-1556) - something of an unknown appearance

We recalculate crc32 newcramfs, hex with the editor we enter at a offset of 552 it into the binary, fill it into the camera.
And ... nothing O_O. So, the gut didn’t let down - at offset 8, it’s really crc32, but from what?
Here we act simply - we start brutally.

  $ python
 >>> from zlib import crc32
 >>> d = open ("firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin", "r"). read ()
 >>> hex (crc32 (d [12:]))
 '-0x781aca29' # none
 >>> hex (crc32 (d [12: 1556]))
 '-0x6f8f1744' no #
 >>> hex (crc32 (d [0: 8] + d [12: 1556]))
 '0x6d29f056' # none
 >>> hex (crc32 (d [0: 8] + d [12:]))
 '-0x652ac4fd' # none
 >>> hex (crc32 (d [0: 8] + "\ 0 \ 0 \ 0 \ 0" + d [12: 1556]))
 '0x15db8164' # Oops!  It! 

Excellent, quickly managed. So we update the label:
(8) 0x15DB8164 - CRC32 header (first 1556 bytes), first setting this field to zero

So, right there in python we quickly recalculate crc32 from the header of firmware_new.bin and enter it in the beginning in the hex editor.
Fill in the camera ... She goes into reboot. And does not answer ... does not answer ... still a brick ... Oh! Ping's gone! Fuuuh.

We take cam-resync.py, and again poke our camera with a stick. And ... and the flow does not break! Just like that, on the first attempt! Wiiii :)

On bread it is smeared, and there is already possible, but something is not right

The previously mentioned Andrei Syomochkin, in the meantime, assembled his firmware with rtsp_streamer corrected by me and uploaded it to one of the problem cameras from which many gaps had come. As a result, testing has shown that the stream does not break, but video artifacts began, the same as with packet loss on UDP. Since I didn’t embed anything like this, it was curious what it was - I had to look into the code again. For starters, look into the strings, we have a bunch of debug strings. “CheckBufferTimeout for% d seconds !!!”, “buffered data more than% d ms, drop all the buffered data !!!”.

Aha It turns out, manufacturers have made protection against overflow! And if for some reason while the synchronous send () hangs on more than necessary (by default - 1 second), it drops the excess. This protects against OOM and video lag if it does not fit into a thin channel. But the code obviously did not work before because of the use of non-blocking sockets and send ().

After wrapping around in Blocking ... NonBlocking - the code started working :)
However, there is a small problem: 1 sec is not enough. If the channel begins to fail, but is quite thick, then the probability of a drop becomes stronger. After any such drop, the video is restored only after the keyframe. Usually the keyframe is rarely enough (every 5-10 seconds) ... And an unpleasant situation turns out - if there was a failure, then 5-10 seconds should wait until the next keyframe for the video to be fixed. If you reduce the frequency of the keyframe - it automatically increases the amount of transmitted data, since the keyframes are quite thick, which means it increases the frequency of channel dying. Vicious circle.

In general, I additionally increased the buffering timeout to 10 seconds - this should not be enough to catch OOM, but enough to calmly experience retransmitters and lags on non-superstable channels.

By the way, the “tricky” treatment algorithm

In the last article I promised to tell you how easy it is to fix the situation. The solution is just like a boot - since we send RTP packets, we each have a timestamp. It is enough to check the packet's lifetime in sendRTPorRTCPPacketOverTCP before sending, and if it is less configured (I still think that 1 second is not enough for TCP, it is necessary just 6-10 seconds), then send, otherwise just silently not send it.

We automate assembly and disassembly

It remains the case for small: to automate the assembly and disassembly of the firmware.

unpack.sh
Since we don’t know the meaning of a piece of title, we cannot collect arbitrary ones so as not to kill the camera, therefore we save all the pieces as they are.
Since we have block and other devices inside the firmware, it’s impossible to work with it from a simple user. But here comes fakeroot, which can save state to an external file. Therefore, unpack using fakeroot.
However, there is a microproblem with it - the file should eventually be readable by the current user. If you are a “real” root, then you can easily read the file, even if it is “chmod -r”. But fakeroot breaks on such a file. Therefore, immediately after unpacking, I change the read rights for all files. But the correct access rights are stored in the fakeroot state dump, so the reassembly goes off well.

The rest of the unpacking does not have any interesting points.

pack.sh
But the packaging is already a bit more complicated. We need to pack back cramfs, update individual file lengths and total length in the header; recalculate the checksums, including the title, and only then merge everything together.
In general, the camera checks the boundary values ​​itself, however, for convenience, I added a check on the size limits of the file system and the kernel, so that if the assembly crawls out of size, I get a warning and remove the extra tails from the firmware.

The result of the works

So, I have compiled the following patched firmware of the latest version 2.5.0.6:

1. firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110-TCPFIX.bin
2. firmware_TS38CD-ONVIF-P2P-V2.5.0.6_20140126121011-TCPFIX.bin
3. firmware_TS38HI-ONVIF-P2P-V2.5.0.6_20140126121444-TCPFIX.bin
4. firmware_TS38LM-ONVIF-P2P-V2.5.0.6_20140126121913-TCPFIX.bin
5. firmware_HI3518C-V4-ONVIF-V2.5.0.6_20140126124339-TCPFIX.bin

If suddenly you need to fix on any other module of the same manufacturer - write in the comments, I will look if possible.

Ps: to everyone who needs something else from these firmware, put the scripts in a more convenient place - on the githab .