Ryan Tomayko

Special Sauce in GTK+ 2.4?

I finally got around to pulling Fedora Core 2 Test 1 last weekend because I'm a good contributing citizen and all that. It looks nice thus far, no problems that I wouldn't expect. Feeling bleeding edge I went ahead and decided to sync up with rawhide, which was also pretty painless. Today I had 50 or so updated packages come down including the just released GTK+ 2.4.0. I restart X and come back to find what seems to be an enormous increase in perceived responsiveness of the entire GNOME environment. I usually run with the stock Bluecurve theme because I really just don't care that much about themes as long as they are tolerable and don't bog the system down. However, on this day, with GNOME acting all balls to the wall, I decide to load up the Milk 2.0 Theme [screenshot].

I've been envying a guy at work who lugs his PowerBook G4 17" into the office everyday--the panther GUI really does have amazing style and polish. I take the scenic route, passing his cube, to the coffee two or three times a day; sometimes, when he's not around, I admit I walk up and rub it softly and promise to come back for her when I have an extra $3,000 laying around.

Milk is pretty heavy on the pixmaps. I tried it for about 10 minutes about a week ago and was just completely disappointed at how much it slowed things down. Not now. I'm back around the same responsiveness I got with Bluecurve under GTK+ 2.2, which wasn't bad by a long shot. It's just kind of amazing how you can get serious system-wide increases by optimizing in sweet spots. We should be looking for more of these.

Minimal System Backups with rdiff-backup and Yum

I'm falling in love with rdiff-backup. This tool gives you the best of both incremental and mirror backups, uses rsync/rdiff libraries to increment modifications to files, and best of all is written in python. rdiff-backup has made backups sexy again (again?). I'm just completely all about backups now. Not so much for functional reasons--I've only had to go to the backups once or twice to restore something, and it was a beautiful experience--but because the tool just rocks and makes me want to back stuff up. If Ben Escoto (primary author, rdiff-backup) could hack together some tax return software I might not have to deal with an audit this year. I could wax about rdiff-backup for a couple of pages but to make a long story short, it has inspired me to believe that backups are an old concept that still has lots of room for innovation and can be fun to code for.

I'm already in a serious love affair with Yum and have been trying to contribute to this project in anyway I can. This tool filled a huge gap in the Red Hat offering by providing a customizable package management system similar to the apt-get tool Debian users have been enjoying for years. Yum is extremely easy to use - to the point that you might think that it's light on functionality. No, it's not like that. Yum is the Bruce Lee of system utilities in that it resembles some silly little Chinese guy that you would think would be no match for, say, Kareem Abdul-Jabbar, and then you get the flex, and it pulls a full distro upgrade with a single command (yum upgrade).

I should note here that apt-get can be used on RPM based systems as well but I am apt-get ignorant and will be refering to Yum exclusively for the rest of this entry. It is extremely possible that apt-get or even up2date could provide the the functionality needed to restore a system from RPM.

More pertinent to the topic at hand is the fact that Yum takes care of pulling RPMs from remote repositories, working out dependencies, and performing installations given a list of package names. What this means to the backup artist is that in order to be able to restore a wrecked system to it's previous set of packages you need only backup your yum config file (usually /etc/yum.conf), which contains your repository configuration, along with the names of all packages installed on your system. We can get the list of packages easily enough:

$ rpm -q --all > list-of-packages

Once we have our yum.conf and package-list backed up, a fresh machine with just the minimum requirements to run Yum, should be able restore to it's previous state with something like the following:

$ cat list-of-packages | xargs yum install

You have to swish the concept around in your head a little bit but you can think of the many RPM repositories that are getting thrown up nowadays as shared backups for common stuff. This greatly reduces the amount of files that need to be backed up by each person because they can always be obtained from a publicly available source.

So we can backup every single RPM managed file [that hasn't been modified] with a very small footprint. Now we get into the more tricky part, which is backing up all the stuff RPM either doesn't manage or manages but determines has been modified. We will use rdiff-backup to increment the files for all the benefits stated previously, but there are a few things missing from the current rdiff-backup/yum toolset.

• There is no straightforward method of obtaining a list of files that are not managed by RPM. You can rpm -ql --all to get a list of files that are managed by RPM and feed them into rdiff-backup as an exclude list but there are issues with this. Most notably is the fact that the file list will contain directories managed by RPM, which may contain files/directories not managed by RPM and rdiff-backup will exclude all files in a directory if supplied in an exclusion list.
• rpm -ql -all outputs full filenames relative to the root of file system. This means that rdiff-backup would always have to be run on / as file paths in the exclude list are interpreted relative to the directory being backed up.
• While not impossible, it is fairly annoying to get a list of files that are managed by RPM but have been modified from their original versions. The following is RPM's verify output, which can be parsed to find modified files:

  $ rpm --verify --all --nomtime --nodigest \
       --nosignature --nodeps --nordev \
      --noscripts --nouser --nogroup --nomode
S.5..... c /etc/ant.conf
S.5..... c /etc/krb.conf
S.5.....   /usr/lib/rpm/rpmrc
S.5..... c /etc/X11/gdm/gdm.conf
...

  This has some serious drawbacks including the fact that it will take a very long time to run on even the fastest machine because it's performing MD5 checksums on thousands files, which also pretty much makes the machine unusable. You can speed things up a bit by passing --nomd5 into rpm --verify, which will cause checking to occur on sizes only but there's a pretty good chance you will miss something.
• Lastly, it would be nice if files known to have no value in a backup could be excluded by default. e.g. /var/run, /var/lock, /tmp, /home/*/.mozilla/*/*/Cache, etc.

What we need is a library that, given a directory, will tell us what files and sub-directories are unmanaged or managed-but-modified. This tool should be able to utilize some kind of database or cache of file information (possibly slocate's database or rpm's).

Once we have a mechanism for determining what files are unmanaged, a nice framework for defining backup sets should be provided. I'm thinking something along the lines of having an /etc/backup.d directory that would have config files for each backup set. The config files would specify the root directory to backup, where the backups should be stored, and a list of additional exclusions and/or forced inclusions relative to the backup directory.

# sample backup config file
[info]
name=home directories
source=/home
dest=root@backup-host:/backups/$hostname/home
backuptype=rdiff-backup
frequency=1d                # how often should we backup?
retain=5d                   # how long should increments be
                            # kept around?

[files]
- .phoenix/**/Cache         # exclude mozilla firebird cache
- .Xauthority               # we don't need that either..
+ .Xresources               # forcibly include .Xresources

A couple of concepts to point out here. Source and dest are pretty self explainatory. backuptype would allow other backup tools to be used in place of rdiff-backup. Maybe we just want to straight rsync the files, or maybe somebody feels that incremental tarballs are still relevant <g>. Lastly, the files list contains a list of files to include or exclude. I really like rdiff-backups file selection syntax ; it is simple and powerful.

There should also be a sane default backup set containing all unmanaged/modified files on the system minus cruft like /var/{run,lock}, cache directories, and anything else that doesn't have backup value. The only aspect that should require configuration is the destination of the backup. Reasonable defaults for all other aspects should be provided.

A Minimal Backup System would provide a simple-as-in-easy-to-configure yet powerful tool that could act as an almost turnkey backup solution for most small scale GNU/Linux installations that use RPM for package management. A positive side-effect of having RPM aware backups is that it further promotes the use of RPM to package common/unchanging files as the more that is managed by RPM the less space is required for backups.

Experimental Firebird Extension RPMs Available

Building RPMs for Mozilla Firebird Extensions is moving along nicely. The following extensions have been packaged up without much problem.

• uptime
• nukeimage
• useragentswitcher
• popupcount
• autohide
• rssreader

At this point I'm fairly confident that any extension can be packaged in RPM. I'm planning on soliciting feedback from the fedora and freshrpms lists soon and should be publishing something in the interim.

If you want to take a look at building these yourself, I have viewcvs setup here with the tarball option on.

RPMifying Mozilla Firebird Extensions

So I was able to package up some simple Firebird extensions into RPMs. Here are my initial findings and some thoughts on moving forward.

1. Under %prep, extract .xpi file to BUILD dir.
2. Under %install, extracted contents of the .xpi file should be installed under /usr/lib/MozillaFirebird/chrome with the exact structure from the xpi file (the exception to this rule is the install.js, which comes with each xpi file and is only used during the install). I've found that most simple extensions contain a single .jar file that has all XUL content.
3. Now comes the tricky part - we need to register the extension/chrome with one of the ugliest goddamned hacks I've ever seen. We add one or more line to /usr/lib/MozillaFirebird/chrome/installed-chrome.txt that points to the files we just laid down. The lines required will vary from extension to extension; it is usually possible to deduce the format but some extensions do some special stuff with locales and whatnot. I think the only surefire way to know what these lines need to look like is to run Firebird as root, install the extension through the browser, and then grab the lines out of installed-chrome.txt. Anyway, I had to add a bunch of crap to %post that appends the lines to installed-chrome.txt and then to executes /usr/lib/MozillaFirebird/regchrome. This updates chrome.rdf (and possibly some other files).
4. For %preun, you need to remove the lines added to installed-chrome.txt and exec regchrome again. I hacked this up with an error-proned grep -v extension-name, which seems to work so far but definitely needs refined.

Moving forward there are a few things I need to figure out. If anyone has info on these, please let me know.

I haven't seen a single reference to redistribution policy anywhere. Are most of these covered under the standard Mozilla license?

I'd like to throw together some RPM packaging guidelines for Firebird extensions. Right now I have all of the extensions I've packaged in a single SRPM and I'm using %package to split them out. This saves some time when I'm adding new extensions but isn't very maintainable. I also have 4 function macros defined at the top of the spec file for extracting the xpi, adding/removing installed-chrome.txt entries, running regchrome, etc. It might be nice to move these into a macros file somewhere.

Another thing to standardize on is a package name prefix. I've been using MozillaFirebird-ext-name. I'm starting to think that the -ext may not be necessary. On pure aesthetics, I prefer MozillaFirebird-googlebar to MozillaFirebird-ext-googlebar.

Firebird Extension / Theme RPMs

If someone can provide a mechanism for programmatically installing Mozilla Firebird extensions, I would be more than happy to put a night aside for packaging a bunch of them for RPM to push up to fedora. From what I'm reading here, it looks like the xpinstall stuff is limited/non-existent from the command line. Lots of people wanting to know how to do unattended/cli installs but no one replying seems to understand why you would want to go cli instead of using the browser. Trying to roll Firebird out to a large number of machines w/ a standard set of extensions is a nightmare.

Right now I'm thinking I may have to resort to installing the most common extensions on one box and blow tarballs. These boxes are somewhere around 99% managed by RPM so slapping unmanaged crap into /usr/lib just kills me.

I have a few things I would like to try before giving up. This post looks promising but it's dated 2000-01-02. If any mozilla-savy person out there could hack up a nice little xpi install app it would be much appreciated.