The first set of random musings ended up being far longer than planned, and rather a lot of material ended up on the cutting room floor. Some of it has been swept into this article.
Incremental Patches
The Gentle Reader may have noticed that using a read-only file system like squashfs has an important repercussion: it is impossible to copy in a modified file as an incremental patch. No matter how small the change, delivering a bug fix means delivering a complete new filesystem. Realistically, this means a complete new software image.
Personally I don't consider this to be a drawback for an embedded system. Microsoft, Apple, and most Linux distributions offer incremental patches because it suits their needs: when your software image is gigabytes in size and you have gazillions of users, it is important to deliver fixes in a bandwidth-efficient way. Embedded software is less affected by this problem, as it tends to be much smaller and have fewer downloads (either because the product sells in low volume, or has a relatively low fraction of users who will ever apply the updates).
Additionally, incremental patches come with their own set of costs:
- perhaps obviously, you need management software to apply, remove, and list installed patches
- tech support has a larger number of variations to deal with
- test load increases rather dramatically: if you have 5 independent patches you may need to test the combinations: up to 2^5=32 variations to test, not just 5.
Again, for companies the size of Microsoft/Apple/RedHat these issues can be handled, but for a small company it can greatly add to the support and QA load for little real benefit.
So personally, I favor not delivering incremental patches. For each major release of the product, maintain a patch branch containing all accumulated fixes to date. When a new customer issue comes in, the fix is added to the tip of that branch and a complete new build released with an incremented version number. In exceptional cases (an issue for one specific customer which may be deleterious for other customers) a new branch can be made and a private image delivered, but for your own sanity you want to make this an infrequent occurrence.
General purpose OS vendors also use incremental patches to deliver fixes for specific problems without annoying their customers with superfluous changes or breakage. This is an issue for embedded vendors as well: there is no customer quite so pissed off as one where a software patch made things worse. However in the end I think this boils down to being disciplined about what changes are allowed in a maintenance release, and what needs to be pushed out to the next major release.
How to Implement Incremental Patches
If you really want to deliver incremental patches, it can still be done using the filesystem choices recommended here. The key observation is that even though there is a file in the filesystem, you don't have to use it. If a newer binary is present elsewhere, you can use it instead.
To implement this scheme it is easiest to segregate the locally developed applications into their own directory, for example /mycompany/bin instead of directly in /bin. For this example I'll assume that incremental patch binaries will be installed in a flash filesystem in /flash/mycompany. When assembling the squashfs/cramfs/etc for a release, place the application binaries in /ro/mycompany/bin and make /mycompany be a softlink to the writeable ramdisk:
# ls -l / drwxr-xr-x 1 root root 472 Jun 6 11:55 bin drwxr-xr-x 1 root root 836 Jun 6 11:55 dev lrwxrwxrwx 1 root root 7 Jun 6 11:55 etc -> /rw/etc drwxr-xr-x 1 root root 555 Jun 6 11:55 lib lrwxrwxrwx 1 root root 9 Jun 6 11:55 mycompany -> /rw/mycompany drwxr-xr-x 1 root root 12 Jun 6 11:55 ro drwxrwxrwt 9 root root 180 Jun 6 12:17 rw lrwxrwxrwx 1 root root 7 Jun 6 11:55 tmp -> /rw/tmp lrwxrwxrwx 1 root root 7 Jun 6 11:55 var -> /rw/var
At boot, you would use a script to populate softlinks in the /mycompany directory structure to point to each binary in /ro/mycompany. If an updated version of the application binary is available in /flash, the softlink will be updated to point to it instead.
# First populate links to the release binaries
cd /ro
for dir in `find mycompany -type d -noleaf`; do
mkdir -p /$dir
done
for file in `find mycompany -not -type d -noleaf`; do
ln -s -f /ro/$file /$file
done
# Now replace the softlinks for binaries which we've incrementally patched
cd /flash
for dir in `find mycompany -type d -noleaf`; do
mkdir -p /$dir
done
for file in `find mycompany -not -type d -noleaf`; do
ln -s -f /flash/$file /$file
done
What you'll end up with is a series of softlinks:
# ls -l /mycompany/bin/ lrwxrwxrwx 1 root root 22 Jun 24 16:49 appNum1 -> /ro/mycompany/bin/appNum1 lrwxrwxrwx 1 root root 22 Jun 24 16:49 appNum2 -> /ro/mycompany/bin/appNum2 lrwxrwxrwx 1 root root 25 Jun 24 16:49 appNum3 -> /flash/mycompany/bin/appNum3
Flash filesystems
These two articles have focussed on in-memory filesystems. If the total size of the libraries and binaries is reasonably small an embedded system can get by with just the in-memory filesystem, but at some point it makes more sense to store the application binaries in a flash filesystem to be paged into memory as needed. The Gentle Reader might be interested in an overview of flash filesystem technology at the IBM developerWorks site, written by M. Tim Jones.
For now, I'll leave it at that.