Exploring OpenWRT: what is the difference between the uImage and sysupgrade images

In the comments to the article “We are flashing the Upvel UR-313N4G router on OpenWRT” between your humble servant and the respected Maysoft there was a dispute about the differences in the structure of the uImage and sysupgrade images of the OpenWRT firmware. I promised Maysoft to sort out the problem, and here is this article.

As you know, in the download directory of OpenWRT, there are, for the most part, two types of firmware, uImage and sysupgrade, for example:

openwrt-15.05-rc3-ramips-rt305x-dir-320-b1-initramfs-uImage.bin
openwrt-15.05-rc3-ramips-rt305x-dir-320-b1-squashfs-sysupgrade.bin
')
The official FAQ writes very briefly about their differences:

What is the difference between the different image formats?
a factory image is one built for the bootloader flasher or stock software flasher
a sysupgrade image (previously named trx image) is designed to be flashed from within openwrt itself
Whatever is the platform needs. Generally speaking, it can be used with the OEM GUI or OEM flashing tool. After that, use the sysupgrade images.

According to the documentation, the content of the images is identical, except that the image of the factory contains additional headers so that this image can be flashed through the web interface of the original firmware.

Ok, let's compare the size of the firmware:

openwrt-15.05-rc3-ramips-rt305x-dir-320-b1-initramfs-uImage.bin - 3253035 bytes.

openwrt-15.05-rc3-ramips-rt305x-dir-320-b1-squashfs-sysupgrade.bin - 3407876 bytes.

Wow, the sysupgrade firmware is almost 140 Kb larger than the uImage, and according to the documentation they should be about the same size, and the uImage at the expense of this “extra header information” itself is a bit larger.

Of course, it is enough to look at the build scripts to understand the difference between uImage and sysupgrade-images, but this, you see, is unsportsmanlike. Today we will analyze the firmware “head on”, as if we do not have the source code, and at the end we will look at the assembly scripts to confirm our guesses.

The main tool for analyzing the firmware at the moment is the binwalk utility, available under Linux. Rename the firmware files shorter for us to be more convenient, and begin the analysis.

> binwalk ./uImage.bin DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 uImage header, header size: 64 bytes, header CRC: 0x19DE1499, created: Fri Jul 3 22:16:00 2015, image size: 3252971 bytes, Data Address: 0x80000000, Entry Point: 0x80000000, data CRC: 0x886ADE01, OS: Linux, CPU: IPS, image type: OS Kernel Image, compression type: lzma, image name: "MIPS OpenWrt Linux-3.18.17" 64 0x40 LZMA compressed data, properties: 0x6D, dictionary size: 8388608 bytes, uncompressed size: 5479932 bytes 

It seems that the entire firmware is an uImage image - at the beginning there is a header 64 (0x40) bytes in length, and after us is the data stream compressed by the LZMA algorithm, 3252971 bytes in size. Fold 64 and 3252971, we get 3253035 bytes, that is, the size of the downloaded image. Consequently, apart from the uImage image, there is nothing else in the file. Binwalk can unpack found LZMA streams. In principle, you can manually cut off the first 64 bytes from the file and unpack the rest with the lzma -d command, but why, when there is such a handy tool?

 > binwalk -e ./uImage.bin 

Let's see what we have

 > ls -l ./_uImage.bin.extracted/  8532 -rw-r--r-- 1 user user 5479932  8 23:10 40 -rw-r--r-- 1 user user 3252971  8 23:10 40.7z 

The file named 40 (offset in the source file) is the unpacked stream. Set binwalk on it:

 binwalk ./_uImage.bin.extracted/40 DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 2812692 0x2AEB14 Linux kernel version "3.18.17 (buildbot@builder1) (gcc version 4.8.3 (OpenWrt/Linaro c version 4.8.3 (OpenWrt/Linaro GCC 4.8-2014.04 r46018) ) #2 Fr" 2932132 0x2CBDA4 LZMA compressed data, properties: 0x5D, dictionary size: 16777216 bytes, missing uncompressed size 2936592 0x2CCF10 xz compressed data 3400392 0x33E2C8 LZMA compressed data, properties: 0x6D, dictionary size: 1048576 bytes, uncompressed size: -1 bytes 

And here we have something, at first glance, incomprehensible - binwalk discovered the Linux kernel at offset 0x2AEB14 and three compressed data streams following the kernel. The fact is that binwalk uses heuristics for analysis and what comes out of it at the output is not the ultimate truth, but information for consideration.
Common sense dictates that the kernel must start at offset 0, and the compressed stream must be one and contain initramfs the initial file system loaded into RAM. The documentation on the core says the same thing:

What is initramfs?
- All 2.6 Linux kernels contain a gzipped "cpio" format archive, which is the root boots up. If you’re running a PID 1.

and further

Populating initramfs:
- The 2.6 kernel build process will be created. By default, this archive is empty (consuming 134 bytes on x86).

The stream format is also mentioned here - CPIO.

Let's see what binwalk can extract from our image:

 > binwalk -e ./_uImage.bin.extracted/40 ls -l ./_uImage.bin.extracted/_40.extracted/  14028 -rw-r--r-- 1 user user 2547808  8 23:25 2CBDA4.7z -rw-r--r-- 1 user user 2543340  8 23:25 2CCF10.tar -rw-r--r-- 1 user user 7186944  8 23:25 33E2C8 -rw-r--r-- 1 user user 2079540  8 23:25 33E2C8.7z 

So, only the stream at offset 33E2C8 was successfully unpacked. If we do everything right, then this should be a CPIO container with the file system:

 > binwalk _uImage.bin.extracted/_40.extracted/33E2C8 DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 ASCII cpio archive (SVR4 with no CRC), file name: "dev", file name length: "0x00000004", file size: "0x00000000" 116 0x74 ASCII cpio archive (SVR4 with no CRC), file name: "dev/console", file name length: "0x0000000C", file size: "0x00000000" 240 0xF0 ASCII cpio archive (SVR4 with no CRC), file name: "lib", file name length: "0x00000004", file size: "0x00000000" 356 0x164 ASCII cpio archive (SVR4 with no CRC), file name: "lib/netifd", file name length: "0x0000000B", file size: "0x00000000" 480 0x1E0 ASCII cpio archive (SVR4 with no CRC), file name: "lib/netifd/netifd-wireless.sh", file name length: "0x0000001E", file size: "0x00001638" *********** *********** 7186416 0x6DA7F0 ASCII cpio archive (SVR4 with no CRC), file name: "dev/urandom", file name length: "0x0000000C", file size: "0x00000000" 7186540 0x6DA86C ASCII cpio archive (SVR4 with no CRC), file name: "dev/pts", file name length: "0x00000008", file size: "0x00000000" 7186660 0x6DA8E4 ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000" 

At the end of the archive we see a file named TRAILER !!! .. which. according to the documentation , is the end of archive label.

Hence, the structure of the firmware initramfs-uImage is as follows:



Now let's take on the image of squashfs-sysupgrade. The name implies that the image contains (except for the kernel) the squashfs file system. Let's see if this is so:

 > binwalk -e ./sysupgrade.bin DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 uImage header, header size: 64 bytes, header CRC: 0x66CC90D2, created: Mon Jul 6 21:54:35 2015, image size: 1142606 bytes, Data Address: 0x80000000, Entry Point: 0x80000000, data CRC: 0x91B77696, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "MIPS OpenWrt Linux-3.18.17" 64 0x40 LZMA compressed data, properties: 0x6D, dictionary size: 8388608 bytes, uncompressed size: 3396940 bytes 1142670 0x116F8E Squashfs filesystem, little endian, version 4.0, compression:lzma (non-standard type definition), size: 2221946 bytes, 1132 inodes, blocksize: 262144 bytes, created: Mon Jul 6 21:54:02 2015 

Let's take up arithmetic: 64 + 1142606 (image size) = 1142670, the squashfs image begins exactly at this offset, and it ends at the offset 1142670 + 2221946 = 3364616. The image size, meanwhile, is 3407876 bytes, which means we still have 3407876 - 3364616 = 43260 bytes of unidentified information. Let's see what's in there, hexdump

 > hexdump -s 3364616 ./sysupgrade.bin 0335708 ffff ffff ffff ffff ffff ffff ffff ffff * 0335ff8 ffff ffff ffff ffff adde dec0 ffff ffff 0336008 ffff ffff ffff ffff ffff ffff ffff ffff * 0337ff8 ffff ffff ffff ffff adde dec0 ffff ffff 0338008 ffff ffff ffff ffff ffff ffff ffff ffff * 033fff8 ffff ffff ffff ffff adde dec0 0340004 

There is clearly some kind of stub. We return to it later.

Let's see what we have in the directory with the unpacked image:

 > ls -l _sysupgrade.bin.extracted/  8820 -rw-r--r-- 1 user user 2221946  8 23:40 116F8E.squashfs -rw-r--r-- 1 user user 3396940  8 23:40 40 -rw-r--r-- 1 user user 3407812  8 23:40 40.7z 

Extract the LZMA stream at offset 40:

 binwalk -e _sysupgrade.bin.extracted/40 DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 2812644 0x2AEAE4 Linux kernel version "3.18.17 (buildbot@builder1) (gcc version 4.8.3 (OpenWrt/Linaro c version 4.8.3 (OpenWrt/Linaro GCC 4.8-2014.04 r46018) ) #1 Fr" 2932068 0x2CBD64 LZMA compressed data, properties: 0x5D, dictionary size: 16777216 bytes, missing uncompressed size 2936444 0x2CCE7C xz compressed data 3396424 0x33D348 ASCII cpio archive (SVR4 with no CRC), file name: "dev", file name length: "0x00000004", file size: "0x00000000" 3396540 0x33D3BC ASCII cpio archive (SVR4 with no CRC), file name: "dev/console", file name length: "0x0000000C", file size: "0x00000000" 3396664 0x33D438 ASCII cpio archive (SVR4 with no CRC), file name: "root", file name length: "0x00000005", file size: "0x00000000" 3396780 0x33D4AC ASCII cpio archive (SVR4 with no CRC), file name: "TRAILER!!!", file name length: "0x0000000B", file size: "0x00000000" 

Here we have a Linux kernel and a small initramfs image with four files. The rest, apparently, moved to the image of squashfs:

 > unsquashfs -i ./_sysupgrade.bin.extracted/116F8E.squashfs Parallel unsquashfs: Using 4 processors 1033 inodes (1034 blocks) to write squashfs-root squashfs-root/bin squashfs-root/bin/ash squashfs-root/bin/board_detect squashfs-root/bin/busybox *********** *********** squashfs-root/www/luci-static/resources/load.svg squashfs-root/www/luci-static/resources/wifirate.svg squashfs-root/www/luci-static/resources/wireless.svg squashfs-root/www/luci-static/resources/xhr.js created 848 files created 99 directories created 184 symlinks created 0 devices created 0 fifos 

Indeed, the main file system is contained in the squashfs image.

Now let's deal with the stub at the end of the image. There is a suspicion that it somehow relates to the JFFS2 file system, because when you flash the sysupgrade image and then boot into dmesg, the lines appear:

 [ 29.550000] jffs2_scan_eraseblock(): End of filesystem marker found at 0x0 [ 29.580000] jffs2_build_filesystem(): unlocking the mtd device... done. [ 29.590000] jffs2_build_filesystem(): erasing all blocks after the end marker... 

and when flashing and loading the uImage image, there are no these lines. A search in the “vanilla” core along these lines does not give anything, but there are such lines in the OpenWRT source tree :

 --- a/fs/jffs2/build.c +++ b/fs/jffs2/build.c @@ -114,6 +114,16 @@ static int jffs2_build_filesystem(struct dbg_fsbuild("scanned flash completely\n"); jffs2_dbg_dump_block_lists_nolock(c); + if (c->flags & (1 << 7)) { + printk("%s(): unlocking the mtd device... ", __func__); + mtd_unlock(c->mtd, 0, c->mtd->size); + printk("done.\n"); + + printk("%s(): erasing all blocks after the end marker... ", __func__); + jffs2_erase_pending_blocks(c, -1); + printk("done.\n"); + } + dbg_fsbuild("pass 1 starting\n"); c->flags |= JFFS2_SB_FLAG_BUILDING; /* Now scan the directory tree, increasing nlink according to every dirent found. */ --- a/fs/jffs2/scan.c +++ b/fs/jffs2/scan.c @@ -148,8 +148,14 @@ int jffs2_scan_medium(struct jffs2_sb_in /* reset summary info for next eraseblock scan */ jffs2_sum_reset_collected(s); - ret = jffs2_scan_eraseblock(c, jeb, buf_size?flashbuf:(flashbuf+jeb->offset), - buf_size, s); + if (c->flags & (1 << 7)) { + if (mtd_block_isbad(c->mtd, jeb->offset)) + ret = BLK_STATE_BADBLOCK; + else + ret = BLK_STATE_ALLFF; + } else + ret = jffs2_scan_eraseblock(c, jeb, buf_size?flashbuf:(flashbuf+jeb->offset), + buf_size, s); if (ret < 0) goto out; @@ -561,6 +567,17 @@ full_scan: return err; } + if ((buf[0] == 0xde) && + (buf[1] == 0xad) && + (buf[2] == 0xc0) && + (buf[3] == 0xde)) { + /* end of filesystem. erase everything after this point */ + printk("%s(): End of filesystem marker found at 0x%x\n", __func__, jeb->offset); + c->flags |= (1 << 7); + + return BLK_STATE_ALLFF;```` + } + /* We temporarily use 'ofs' as a pointer into the buffer/jeb */ ofs = 0; max_ofs = EMPTY_SCAN_SIZE(c->sector_size); 

Yeah, that's our stub 0xDEADCODE. If the JFFS2 driver finds this label, then it considers it to be the end of the previous file system and erases everything from the zero byte of the label to the end of the drive. Thus the label itself is also overwritten.
After that, the driver creates a new instance of JFFS2 in this place.
So, we have the following image structure:



In this way:

1. uImage contains the minimum necessary functionality for running OpenWRT and due to this, its structure can be easily changed so that it passes validation checks in the Web interfaces of the original firmware.
2. Sysupgrade is more complex and uses Linux-specific tools - SquashFS and JFFS2.
3. Both types of images do not contain a bootloader (and should not)
4. When flashing through the bootloader (via UART or disaster recovery), you can immediately sew sysuprade.
5. With firmware via mtd_write, if this utility is available via telnet from the official firmware, you can also immediately sew sysupgrade.

In conclusion, let's take a look at the OpenWRT build scripts to make sure of your conclusions. Let's start with the file / target / linux / ramips / Makefile . But if you think that this is a regular GNU Makefile, it is not. Here is how the developers themselves describe the improved Makefile:

Looking at one of the package makefiles This is a pattern that has been transformed, making it easier to use it.

It seems that the image build runs on line 564:

 Image/Build/Profile/DIR-320-B1=$(call BuildFirmware/Default8M/$(1),$(1),dir-320-b1,DIR-320-B1) 

The goal of BuildFirmware / Default8M is described in lines 195 and 196:

 BuildFirmware/Default8M/squashfs=$(call BuildFirmware/OF,$(1),$(2),$(3),$(ralink_default_fw_size_8M),$(4)) BuildFirmware/Default8M/initramfs=$(call BuildFirmware/OF/initramfs,$(1),$(2),$(3),$(4)) 

It consists of two positions: squashfs and initramfs. The goals of BuildFirmware / OF and BuildFirmware / OF / initramfs are described in the same file just above

 149 # $(1), Rootfs type, eg squashfs 150 # $(2), lowercase board name 151 # $(3), DTS filename without .dts extension 152 # $(4), maximum size of sysupgrade image 153 # $(5), uImage header's ih_name field 154 define BuildFirmware/OF 155 $(call MkImageLzmaDtb,$(2),$(3),$(5)) 156 $(call MkImageSysupgrade/$(1),$(1),$(2),$(4),$(6)) 157 endef 

and

 169 # $(1), squashfs/initramfs 170 # $(2), lowercase board name 171 # $(3), DTS filename without .dts extension 172 # $(4), ih_name field of uImage header 173 define BuildFirmware/OF/initramfs 174 $(call MkImageLzmaDtb,$(2),$(3),$(4),-initramfs) 175 $(CP) $(KDIR)/vmlinux-$(2)-initramfs.uImage $(call imgname,$(1),$(2))-uImage.bin 176 endef 

By the name of the target MkImageLzmaDtb, you can guess that it creates an uImage image from the description from the DTS file. For the purpose of BuildFirmware / OF / initramfs, the initramfs partition with the main file system is added to this image, after which the image is copied to the output directory. For the purpose of BuildFirmware / OF, the source image is processed by the MkImageSysupgrade function, which with the help of the “cat” command attaches the squashfs section to it, and then calls the prepare_generic_squashfs function defined in the image.mk file.

 # pad to 4k, 8k, 16k, 64k, 128k, 256k and add jffs2 end-of-filesystem mark 110 define prepare_generic_squashfs 111 $(STAGING_DIR_HOST)/bin/padjffs2 $(1) 4 8 16 64 128 256 112 endef 

And the padjffs2 utility written in C writes the 0xDEADCODE mark to the end of the image file:

 static unsigned char eof_mark[4] = {0xde, 0xad, 0xc0, 0xde}; *** static unsigned char *pad = eof_mark; *** /* write out the JFFS end-of-filesystem marker */ t = write(fd, pad, pad_len); if (t != pad_len) { ERRS("Unable to write to %s", name); goto close; } 

In general, the world of OpenWRT is beautiful and amazing, especially deliver comments like:

# The real magic happens inside these templates

or

/ * vodoo from original driver * /

In conclusion, I would like to say that when building OpenWRT from source codes, you need to pay attention to the build environment, for example, in this repository on my Debian 8 + Sid, even the cross-compilation tool is not going to. But on Debian 7, all this wealth is going to be amazing, if you do not edit the initial configuration. If it is corrected, it may turn out that some Web sources, from which the assembly script downloads source codes, are already “spoiled”, you need to look for new sources, correct the script, and so on.