rpmlint is a tool for checking common errors in rpm packages. It can be used to test individual packages and spec files before uploading or to check an entire distribution. By default all applicable checks are processed but specific checks can be performed by using command line parameters. 

I love man pages; they always have the best descriptions. rpmlint is one of the most sensitive tools I’ve ever used on Linux. One little thing and it warns you even if its commented out. In the users perspective that’s a good thing since it sort of forces developers to create better installation scripts; however developing an RPM to get warnings for some pretty useless things gets not necessarily frustrating but really annoying.

Ive built 3 RPM packages recently:

1. Installation of a yum repository (special thanks to rpmfusion)
2. 64-bit Firefox nightly
3. 32-bit Firefox nightly

The amount of time i spent on making these RPMs was actually fairly short considering the emo tools that have to be used to validate them. Both Firefox nightly RPMs are the same (save for the source tar ball).

Mock is  a simple program that will build source RPMs inside a chroot.
It doesn’t do anything terribly fancy other than populate a chroot with
the  contents  specified  by a configuration file, then build any input
SRPM(s) in that chroot.

I haven’t worked with Mock too much however from what i can tell so far; it is a fairly strong tool however just as annoying as rpmlint if not more. The tool just compiles a program with the default packages just like the description above says however what it doesn’t tell you is that it also checks for really small things such as the use of wild cards.

Now lets see how mock works; you must build a package and its source RPM (rpmbuild -ba ~/rpmbuild/SPECS/file.spec) and then run mock for a preferred operating system (mock -r fedora-13-x86_64 ~/rpmbuild/SRPS/file.src.rpm) then see the output for any errors; Heres the kicker; for you to create a source RPM the package must build correctly; therefore you assume that everything is fine minus any non-default packages.

File not found by glob: /builddir/build/BUILDROOT/minefield-4.0b8pre-1.fc14.x86_64/usr/lib64/minefield-4.0b8pre/*

I got this error every time i used mock; I tried to fix it about a dozen times and i just couldn’t figure out where this issue was in the spec file. After a couple frustrating hours i figured out that it was the wild card being used in the %files section of the spec file.

After fixing that ridiculous issues; mock successfully completes; but just because it completes doesn’t mean everything works; you now have to comb through a log file and attempt to spot any errors.

Repository RPM:

• Bumped version to 0.4
• Cleaned up .repo file
• Removed invalid gpg key
• Fixed a typo

Repository:

• Includes drpm and srpm directories which hold Delta RPMs and Source RPMs respectively.

Minefield RPM:

• Passes rpmlint; I did say that it passed in my release 0.2 however that was only for the spec file. The rpms output 2 warnings: no documentation; no manual
• Both 32 bit and 64 bit source RPMs are now in the repository

Minefield Spec file:

• Removed a buildrequires (tree)
• Removed use of the “install” command because of permission issues and replaced with “cp” which can copy files/directories with their file permissions

Automation:

• Script now takes URL as the argument; version number and architecture are taken from the URL.
• Script validates configuration
• Removes partial wget downloads
• Performs local jobs first before going out to download minefield

The main goal for this release was getting automation to a point where it can be implemented into a system and not have to worry about any issues caused by future changes. Achieving this is relatively difficult however Ive made it so that the only thing that needs to change is the URL to access the latest minefield. Ive talked about how to set up the server side of our program here.

don’t worry about integrating anything into Mozilla’s systems this
quarter as I pretty much have to switch gears to finish another couple
of goals.

Although our project page said that our goal for this release would be to have our system implemented into Mozilla’s infrastructure; it looks like it wont. I have blogged about everything to get the system working on their side so i have done my part. It is now up to them to implement it. I will be happy to assist them in doing so and to make any improvements to what we have developed.

Future updates to be done:

• Complete automation scripts
  • validation is incomplete
  • only tested for nightlies
• Implement RPM signing
• Fix any outstanding bugs. Ive talked about bugs here

• Version number changes:
  Old: minefield-4.0b8pre-20101211.fc13.rpm
  New: minefield-20101211.1-4.0b8pre.i686.rpm

  The versioning system was changed to make the rpm version number valid. The version number is not allowed to have any letters so i just switched the version and the release numbers. At the end of the version i added a .1 to add support for releasing multiple nightlies in one day. The .1 will increment if there are more than 1 release on the same day. The fc13 part was removed as this release isn’t tied to any release of Fedora; since the program we packaged is pre-compiled it should theoretically work on most if not all versions of Fedora.

• Delta RPMs:
  drpm files will now be generated; this will reduce the download size for users updating from an older nightly.
• Source RPMS:
  src.rpm files are placed in a sub directory under the correct architecture.
• Spec File:
  The use of the command “install” was removed; it has been replaced by the “cp” and “mkdir” commands. The reason I did this was because cp is capable of keeping file permissions the same without having to specify it.

  A buildRequires was removed as a result of switching out the tree command with cp. The spec file no longer installs one file at a time; it simply copies the entire directory to the correct location.

  Although it was fast enough; the above changes have significantly improved the install speed of the application.

All of the older nightlies with the old versioning has been removed. To take advantage of the new system; remove your current minefield and install the latest one with YUM.

• Current and previous Minefield nightly show up under “Add/Remove Software”

Ive been looking into this after reading Armen’s blog and I’m not quite sure what is causing it. I am fairly confident to say that it isn’t on our side. It may be a bug in the “Add/Remove Software” application or a way Fedora implemented their filters. I updated another application on my system that is on the official repositories and it too shows up twice.

• Setting Minefield as the default browser seems to mess up the “Preferred Web Browser” icon.

• Using the “Preferred Web Browser” launcher opens the users home directory.

Using the “Default Browser” dialog simply doesn’t work; I highly recommend not using that dialog. If you wish to set Minefield as the default browser navigate to System -> Preferences -> Preferred Applications. To fix the above 2 issues, it should match the following:

When using the “Default Browser” dialog, the command is set to: /usr/lib/minefield-4.0b8pre/firefox “%s”; The quotation marks cause the browser to open the home directory; removing the quotations will fix that issue alone. The other issue which is the “Preferred Web Browser”‘s icon; the “Command” field must not be an absolute path.

After changing this; The “Preferred Web Browser” will automatically change to the same exact command as the one that opens the menu item.

• Should we be listed under “debug and development software sources” on the Software Sources manager?

• Minefield gets unpacked under /usr/lib/minefield-4.0b8pre. Should this be instead /usr/lib/minefield?

The version number is in the unpacked directory’s name to keep some level of consistency with other packages on the system as well as Firefox itself. The Firefox executable points to /usr/local/lib/firefox-4.0b8pre. Firefox itself is compiled with that directory. The reason files are not placed in /usr/local/lib is so that we follow at least some of the Fedora guidelines for packaging software.

• Minefield when launched from Applications->Internet->Minefield starts with the ProfileManager and allows running the nightly build with the other instances of Firefox (“minefield –no-remote –ProfileManager); should this not be a feature?

Personally I don’t like this feature as I can imagine it would get really annoying. However it was suggested by Chris to be implemented to help out new users with using the nightly. I leave this option to Armen whether to keep or not.

To remove this change:

• edit /usr/share/applications/minefield.desktop
• remove “–no-remote –ProfileManager” from the end of the line which starts with “Exec”

• Since /usr/lib/minefield-4.0b8 is owned by “root” Firefox’s update system doesn’t work. If we change ownership then updates work again and I wonder what bug would I hit if I can receive updates through yum and through AUS. Only one way to find out!

Updating the nightly from yum will remove the directory and recreate the directory. This will ensure that even if the version bumps up, the old directory will disappear and the new directory will appear in its place. From the best of my knowledge using AUS should not cause any issues unless the version number changes. I would highly recommend using one or the other and not both.

• I also believe that there is not a way to offer partial updates through yum as we do through our normal update system

Delta RPMs will allow for partial updates. It is roughly the equivalent to the way patch files are with the diff command. I will implement this into the repository as well as the automation scripts as soon as i can.

• Minefield is being listed in the wrong group. It is listed as “Other” however it should be listed under “Internet”

No idea whats causing this; The spec file specifies the group exactly the same way as other packages but it still shows up under “Other”.

Ive uploaded a copy of the automation scripts to “http://tarin.freehostia.com/files/scripts.tar.gz”. The steps to actually getting this to work is as follows:

1. Create a new user. (Not required but highly recommended; the scripts remove files in the home directory / repository)
2. Download the scripts from http://tarin.freehostia.com/files/scripts.tar.gz
3. Extract the tar ball into the home directory of the new user (or current user if u decided to use it)
4. Verify that the directory structure is as follows:
   • ~ (home directory)
     • UpdateRepo.bash
     • includes
       • BuildDesktopFile.bash
       • BuildSpecFile.bash
     • …
5. edit UpdateRepo.bash; The top of the file holds all the configuration (there’s only 1 thing to configure)
   Change the path to the parent directory of your repository. (do not end with a trailing slash)
6. Run the script with the following commands:
   ~/UpdateRepo.bash http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-4.0b8pre.en-US.linux-i686.tar.bz2
   ~/UpdateRepo.bash http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-4.0b8pre.en-US.linux-x86_64.tar.bz2

Troubleshooting:

Q: I run the script but i get “Bad Configuration: * unwritable”. Help!
A: Verify that the new user has access to write to your repository.

Q: I run the script but i get “Missing Dependency: rpmdevtools”. Help!
A: To fix this, login as root and type in: “yum install rpmdevtools -y”. This should install required packages.

Q: I have a different problem!
A: The scripts are currently experimental, any issues or errors should be posted here.

EDIT: Fixed links

The command used to create RPM packages is “rpmbuild”; we used the “-ba” option to make rpmbuild generate all required packages for a standard Fedora release; This includes .rpm, .src.rpm and debuginfo. debuginfo is disabled inside the spec file since we are not building the nightly; we are simply extracting the binary to a location.

The .src.rpm package is used to add support for the “yumdownloader –source” command and to follow the the general guidelines of a Fedora YUM repository. A .src.rpm package can be extracted and viewed like a tar ball, that is one of the benefits of using it; another benefit is that anyone who installs the SRPM and has the “rpmdevtools” package installed will have an organized, structured and fixed location where the source is stored.

A .rpm is the equivalent to a windows installer;  the .src.rpm is a “how to” for the OS to build that .rpm. The reason it isn’t stored in a SVN is simply because of the design of how an RPM is built; When a .src.rpm package is installed, it installs everything to create the .rpm however not the .rpm itself.

When viewing the .src.rpm and .rpm with an archiving tool the only difference is the access to the SPEC file. A spec file is a file which holds instructions for the OS to install the an application.

* Any reason that the RPM to add the “Firefox Nightly” repo to Fedora is provided as one of the directories (“noarch”) inside of the repo as a whole? I would have expected this file to provided from another website

The repository RPM is there for convenience; all it does is adds the .repo file and the gpg key to the system. All directories in the repository are simply the architecture and the repository meta data; “noarch” means that no matter which architecture you are using, this package is installable.

You can provide this file from another website, or any of the files in the repository. That is simply your choice. Normally most users don’t care about the back end of a repository and wont even know about how it functions. The only reason it is included in the repository is for convenience.

If you were to change something in the RPM such as the server location or gpg key, how would you make the user update their system with the latest repository, or better yet, make it convenient? Adding this package to the repository was meant to answer this question, all we need to do is bump the version or the release number and the user just needs to do a yum update and with a simple system function they are up to date.

Tom ‘spot’ Callaway, Manager of the Fedora Engineering team at Red Hat created a repository for Google Chromium here. The directory structure is slightly different than what we’re using however the general idea is the same. I posted this here so you have something to compare the repository that we built to other repositories built for Fedora.

Any other questions or concerns feel free to leave a comment.

To remove this change:

• edit /usr/share/applications/minefield.desktop
• remove “–no-remote –ProfileManager” from the end of the line which starts with “Exec”

This change will remain in the final version unless requested to be removed by Mozilla.

Ive made some changes to the automation scripts to remove any potential issues that may arise when the Minefield version number changes. The scripts were slightly redesigned as a result which actually benefits us in the long run.

• The script to update the repository takes an argument; That argument before the change was the architecture to build Minefield; after the changes I made today the argument is now the URL to the Minefield tar ball. The version number and architecture are taken off the URL. This little change assumes that the naming convention of Minefield remains relatively the same. However it makes the scripts a lot more robust as it requires very little to be changed should it break.
• The order in which the script processes stuff has changed. The script now does all local things first such as building the spec file and desktop file before downloading the tar ball. This was done to ensure that bandwidth isn’t wasted on either side just to find out there’s something wrong locally.
• The output of the script was changed from “Neat” (only showed errors should they arise) to a little more convenient. If the script is run manually the script should output additional information  to show the status of the update. (shows download process)

Issues remaining:

• When i gave the script to my colleague he was unable to run the script; the issue lied with the use of the function “dirname”; the script will only run when called with an absolute path or directly(your sitting in your home directory and the script is sitting in a directory called project; call it using either ~/project/UpdateRepo.bash or project/UpdateRepo.bash. Using ./project/UpdateRepo.bash or any other relative path will not work.)
• RPM signing; I did plan on doing it yesterday however moving our scripts over to a new server introduced some additional work. We have to recreate gpg keys. Since this isn’t absolutely vital for the system to work it hasn’t really received my full attention however it should be done soon.

The end is nearing! Unfortunately I am completely swamped with work however I should be able to finalize most of the automation sometime later this week. I am currently working on another post  on instructions on how to set up the server side of things, including gpg keys, script configuration, etc.

The structure of our Repository:

http://chile.proximity.on.ca/ffrepo

• i686
  • *.rpm
    This is the actual 32bit Minefield installation RPM; using the command “yum install minefield” will download and install the latest version of this package; Or if a manual installation was to be done; This package must be installed.
  • *.src.rpm
    Source RPM files contain the source code to build the above RPM; It includes the spec file, desktop file and the minefield tar ball. This package is optional and the only reason it is included is to allow outside users to see how we built the RPM and to meet the expectations of leaving the RPM open source.
• noarch
  • Similar to the i686 directory; The src.rpm is the source code for the .rpm file. The RPM in this directory is not related to minefield. Its sole purpose is to allow end users to install our yum repository easily. It is included inside the repository should we ever need to update the end user. “noarch” indicates that the package is not limited to any system architecture. The RPM here is updated on a need to need basis and not generated daily therefore it uses a different naming convention.
• repodata
  • This directory is automatically generated when the repository is created/updated.
• x86_64
  • This directory is exactly the same as i686 however the tarball inside these RPMs contain the 64 bit version of minefield.

Automation:

The automation script is a block of code which automatically creates both .src.rpm and .rpm files for i686 and x86_64. The following contains a basic order of how this script works:

1. Download the latest tarball from mozilla’s ftp server
2. Creates a fresh spec file with unique properties(version/release numbers) and a desktop file
3. Update the repository

This script is designed to build either 32 bit or 64 bit minefield and not both at the same time; This means that to build both 32 bit and 64 bit minefield; the script must be run twice with a different argument.

Other potential changes:

• With respect to Armen’s request; The RPM does not create a profile however after today’s small presentation to Chris; he suggested that every time minefield is opened, the profile manager should pop up.

• Include Automatic RPM signing inside the automation scripts using the “expect” packages.

Other Concerns:

Armen left a comment regarding the naming convention of minefield (minefield-4.0b8pre-20101207). His comment was about how multiple nightlies may be released a day and the automation script should be able to handle that. Currently The scripts are designed to run at the top of the hour after the nightly has been been released; 7 AM EST for 32 bit and 8 AM EST for 64 bit. However the scripts can also be run manually whenever any updates are needed. However the naming convention does not consider the fact that more than one nightly can be released on a single day therefore if the scripts were run manually; they will override the one that was automatically built on the same day.

The structure of the automation scripts (the structure is dynamic and designed to work from any directory inside a users home directory):

• ~ (home directory)
  • project
    • BuildSpecFile.bash
    • BuildDesktopFile.bash
  • rpmbuild
    • RPMS
    • SRPMS
    • BUILD
    • BUILDROOT
    • SOURCES

#dependency check
if rpmdevtools is not installed
output error
exit script

#prepare environment
create directory where RPMs are stored
wipe rpmbuild directory
(re)create rpmbuild directory

#build (if anything fails here; script exits with error)
download latest binary tarball
move tarball to correct location
call external script (builds spec file)
call external script (builds desktop file)
build RPM and SRPM
move RPM to correct location
remove old SRPM(if this fails; script does not exit)
move SRPM to correct location

#update repo
wipe rpmbuild directory
update repository

This script is used to build the spec file:

#set variables
version=4.0..
release=date

#create spec file
cat >output <EOF
contents of spec file…
EOF

This script is used to build the desktop file:

cat > output <EOF
contents of desktop file
EOF

• Automatic rpm signing:
  This is fairly easy from what ive read. A package called “expect” is capable of providing exactly what we need. The only problem is that the password must be inside the automation scripts in plain text.
• Dynamically downloading source code:
  Currently the source code is downloaded from Mozilla’s FTP server however the URL used to get the code is: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-4.0b8pre.en-US.linux-x86_64.tar.bz2
  Curious to see the amount of errors that come out when the version number changes.

Our Proof of Concept server’s Apache is currently down, honestly not surprised, when you give root password to people that may not know exactly what their doing; usually results in disaster. Ive attempted to fix Apache however fixing something is just too much work especially if you don’t know what exactly changed. I just moved the automation scripts to another server and that server has been running fine. The server i will be using from now on is: http://chile.proximity.on.ca/ffrepo/

Unfortunately moving stuff is easy for me however anyone with the repository will have to do some work to get it working again:

• Download and install the latest RPM from here
• Your back!

Moving servers also means that the key included with the RPM is now invalid; although that really doesn’t matter at this point since gpgcheck is still disabled and the packages still haven’t been signed.

I will try to get some time to post some pseudo code for the automation scripts very soon.