zfs storage pool tuning for performance

ZFS Storage pool tunning tips

- Tuning Open Solaris

After tuning the following Result : 45+MB/s
root@10:/mnt/vxstor2_1001# dd if=/dev/zero of=100M bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 22.5591 s, 46.5 MB/s
in OpenSolaris SNV_134 create  the following files.
append the following lines to /etc/system
* "fsflush" tuning
 set tune_t_fsflushr = 5
 set autoup = 300
* Disable the Automatic Interrupt Adjustment
 set dld:dld_opt = 2
* Disable "soft rings"
 set ip:ip_squeue_fanout = 0
 set ip:ip_soft_rings_cnt = 0

create a file in /etc/rc3.d/S99ndd   
 #!/bin/bash

NDD=/usr/sbin/ndd

$NDD -set /dev/tcp tcp_xmit_hiwat 1048576
$NDD -set /dev/tcp tcp_recv_hiwat 8388608
$NDD -set /dev/tcp tcp_max_buf 8388608
$NDD -set /dev/udp udp_xmit_hiwat 1048576
$NDD -set /dev/udp udp_recv_hiwat 8388608
$NDD -set /dev/udp udp_max_buf 8388608
$NDD -set /dev/tcp tcp_conn_req_max_q 65536
$NDD -set /dev/tcp tcp_conn_req_max_q0 65536
$NDD -set /dev/tcp tcp_fin_wait_2_flush_interval 67500

Note: This NFS tunning will work only with 2 nics with Aggregate like bonding.


ZFS ZIL on Separate Device
- Seperate Device of ZIL (ZFS Intend log )
Generally by default the ZIL and the data is written on the same zpool, Wherein the disk overhead will come into picture. If we use separate device for ZIL to write in the performance will boost up
Note: Recommended SSD device can be used and SSD ideally can be half of the RAM size in the Storage subsystem


JUMBO Frames on NICs
- We can also use JUMBO Frames in both end of the NIC associated with Storage Subsystem
Note: Jumbo frames are not supported by all the nics.
How to figure out, if the existing nic support jumbo frames as follows
# ifconfig mtu 9000
If the Nic accepts the value without throwing any error, it means it support or else it is not.


Storage Separate Network
- Storage separate private network only for " Data Writes" will definitely boost the performance of the storage

Cloud Computing Hierarchy (IAAS, PAAS, SAAS)

## Cloud Technologies ##
http://www.cloud.com
http://www.hexagrid.com
http://www.rackspace.com
http://aws.amazon.com/ec2
http://www.microsoft.com/windowsazure/pricing
http://www.apple.com/icloud
http://www.gogrid.com/cloud-hosting

#Thin Client
http://www.ncomputing.com

Procedure to insert a module into initrd

How to work with initrd and procedure to insert modules into initrd

initrd
The term « initrd » is short for « initial ramdisk ».  It refers to a file of the same name, used by Linux systems during the boot process in order to load certain modules (drivers, and the like) prior to the full system being brought online.  An example usage of the initrd would be to load a module for a RAID card, so that the actual filesystem – and all of the delicious data contained within – can be accessed properly.
There are (perhaps unfortunately) a number of ways in which initrds can be constructed and implemented.  Generally speaking, the actual file is a compressed archive, which may contain further archives, filesystems, configuration files, and so forth.  Over the years, different distributions have handled the initrd in different ways, such that one distro’s method of constructing an initrd may be totally incompatible with that of another.

ramdisk

A « ramdisk » is, as the name implies, a sort of filesystem which has been loaded into RAM.  Although it does not physically exist, it appears for all intents and purposes to be a normal disk, which has the advantage of being seamless to the user (and to the system).  Ramdisks are handy because, compared to physical disks, they are fast.  Within the context of the initrd, however, speed isn’t a concern – it’s the versatility of the ramdisk that is key.  As in the module example i mentioned initially, the various bits and peices of hardware in a given computer often require special drivers to make them function properly.  These drivers can be built directly into the kernel, but often, for reasons of ease of use and cross-compatibility, it’s easier to put them into the initrd, and then load them via the initial ramdisk that gets built when the machine boots.

The basic idea behind a « loadable kernel module » (or just module for short) is, again, to add support for hardware, special filesystems, and the like, which the kernel doesn’t natively understand.  Modules can be loaded at any time, but for the most part, they are initialised at boot time, so that the kernel can interact with things like RAID, audio, or video cards, non-Linux filesystems (NTFS, for example), or even more esoteric items like robotic controllers and (the dreaded) software modems.

gzip (and tar, too !)
Gzip is, along with « tar », arguably the most ubiquitous archival format pairing in the Linux world.  Generally speaking, a gzip archive is a single file which has been compressed using the gzip binary.  Commonly, gzip is used to compress a « tar » archive, which normally does not use compression, but which can contain any number of files in a single archive.  Together, they are used to create « .tar.gz » or « .tgz » files, which are more or less the standard way to create compressed archives in Linux ;

cpio
It is a rare and glorious thing when ancient technology is updated and used constantly from day one.  Take the lightbulb, for example.  Or fire.  Or « cpio » .  It’s basically like tar (mentioned above), in that it can be used to take any number of given files, and create a single archival file containing said files.  Like tar, compression isn’t part of the deal, and gzip can be used to compress a cpio archive as normal.

While cpio has a long and storied history (it was first specified way back in 1977), over the years, it has increasingly replaced by tar in many implementations.  Some time ago, Red Hat decided to use cpio (instead of tar) for their ultra-popular packaging format « RPM ». 

Add Module into intird
step1:
$ cp /tmp/
$ cd /tmp
$ file initrd.img
initrd.img: gzip compressed data, from Unix, last modified: Fri Nov  2 15:54:12 2007, max compression
step2:
$ gzip -d initrd.img
gzip: initrd.img: unknown suffix -- ignored
Uh oh ! By default, gzip doesn’t handle files with suffixes it doesn’t understand (i.e. « .gz » ).  The easiest way to deal with this is simply to add the appropriate suffix, then try again :
step3:
$ mv initrd.img initrd.img.gz
$ gzip -d initrd.img.gz
This gives us a single file (as we expected) : initrd.img .  Though it has the same name as the file we started with, it’s not the same data at all.  This initrd.img is larger than the original, and most importantly, it’s a different type of file
$ file initrd.img
initrd.img: ASCII cpio archive (SVR4 with no CRC)
step4:
Unpack the initial Cpio Archive
As i mentioned above, cpio has a long and storied history, which is reflected in the myriad of arguments and options that can be provided during normal usage.  The command we’ll be using at this stage will be :
$ cpio -i -d -H newc -F --no-absolute-filenames
The arguments are, in order :

    * -i : « copy-in » is more or less like saying « extract » in the parlance of cpio.
    * -d : « make directories » allows cpio to re-create the directory structure in the archive, instead of just unpacking everything to the same place.
    * -H newc : « format type » defines the particular format used by this cpio archive.  Over the course of cpio’s history, numerous formats have been used for various reasons : « newc » is an SVR4-based format that allows for large files and filesystems.  Remember the archive type from above ?  This is why we needed to make note of it.
    * -F : « file filename » simply indicates the file we want to work with.
    * –no-absolute-filenames : as the name suggests, this forces cpio to unpack the contents relative to where we executed the command.  Basically speaking, it prevents the possibility of cpio trying to write to our root filesystem, which could have disastrous results.
step5:
First we must make a directory in which to unpack the archive, then we can proceed :
$ mkdir /tmp/initrd
$ cd /tmp/initrd
$ cpio -i -d -H newc -F ../initrd.img --no-absolute-filenames
16442 blocks
The number of « blocks » refers to the amount of data that was unpacked – it is a measurement, like bytes.  This value may be different for you, as not all initrd’s are created equal.
step6:
Just push in the required drivers to appropriate locations
step7:
$ cd /tmp/initrd
$ find . | cpio -o -H newc | gzip > /tmp/initrd.img
$ file /tmp/initrd.img
/tmp/initrd.img: gzip compressed data, from Unix, last modified: Wed Jun 10 19:39:51 2009

How to work with Squashfs filesystem

### Remaster the Squashfs-filesytems ###
Prepare the host
apt-get install squashfs-tools chroot
step1: mkdir /tmp/livecd
step2: mount -o loop *.iso /tmp/livecd & cp -r /tmp/livecd /tmp/live-cd
step3: mkdir squashfs
step4: mount -t squashfs -o loop filesystem.squashfs squashfs
step5: To Access the internet
step6: cp /etc/resolv.conf /etc/hosts ./squashfs/etc
step7: chroot ./squashfs & mount -t proc none /proc & mount -t sysfs none /sys & export HOME=/root
step8: apt-get install * & update sources.list
step9: apt-get clean & rm -rf /tmp & rm -f /etc/resolv.conf & rm -f /etc/hosts
step10: umount /proc/ & umount /sys/ & exit
step11: mksquashfs ./squashfs filesystem.squashfs
step12: update the md5sum & rebuild the Cd by mkisofs
step13:
IMAGE=/opt/Gyani/router_custom_i386.iso
BUILD=/opt/Gyani/router
mkisofs -r -V "VxGrid Enterprise Cloud" \
            -cache-inodes \
            -J -l -b isolinux/isolinux.bin \
            -c isolinux/boot.cat -no-emul-boot \
            -boot-load-size 4 -boot-info-table \
            -o $IMAGE $BUILD
step14: Done
Note: Make sure When we unpack and pack with Squashfs tools, If the live Cd is little old then it leads to some problem like the CD won't boot.
Reason: If the Squashfs tools version-4 and the kernel & initrd module version it support version 3 it leads to problem
Overcome: Either the squashfs tools can be installed downgradded one (3.*) or insert a module in initrd with new version of Squashfs tools

Note: Once you get the Chroot prompt, its nothing but another linux operating system.

Building Solaris Package(pkg) from Scratch

### Building Solaris Packages Quickly ###
ftype Description
d directory
x exclusive directory (package specific)
p named pipe
c character special
b block special
s symbolic link
f executable or data file
e editable file
v volatile file (content may change)
l linked file
i meta-file

To add a package
# pkgadd -d
# pkgadd -d . , for example pkgadd -d . SFWsnort
To remove a package
# pkgrm
To get short description (info) on a package
# pkginfo -x
# pkginfo -l
To list all installed packages
# pkginfo
To list the files that constitute the package
• pkgchk -l
• pkgchk -l | grep Pathname # lists files only.
• pkgchk -d -l
To what package the file /usr/bin/ls belongs:
pkgchk -lp /usr/bin/ls


To package any software you need to know the files that are to be included in the package.
Creating Package Configuration Files
Once the software is installed you need to create a table of contents suitable to be used for the package creation tool.
% cat /tmp/{Custom-Folder} | pkgproto > /tmp/Prototype
This will create a /tmp/Prototype file with the following in it:

After manually reviewing the /tmp/Prototype file you may notice the permissions are not to your liking. Simply edit the file to specify what permissions you want on each inode. I manually alter each file in the package archive to have all files world readable, root owned, root group and all directories read and execute. (drwx-r-xr-x or -rw-r--r--).
So to the front of the Prototype file I add:
i pkginfo
If you want checkinstall, preinstall and postinstall scripts run simply include them at the top.
checkinstall is a script that is run at install time, by the user "nobody". With it you can check for dependancies and problems and if needed, exit the pkgadd gracefully. Care is needed when building the package that the permissions on the directories are set up so the user "nobody" can get to the script.
preinstall is run as root when the package is actually installing. An abort here will require the user to pkgrm the half installed package. You are asked by pkgadd if you want to run this script.
postinstall is a script that runs after the rest of the package is installed. With it you can do things like utilize the just installed software, install kernel drivers, create devices and other post-install things. This script also runs as root. You are asked by pkgadd if you want to run this script.
To add them to the Prototype file put in a line like:
i preinstall
i postinstall
i checkinstall
These scripts are run after installation starts, so if something goes wrong with the install in those scripts and they exit with non-zero then a pkgrm will be required to clean up the half installed package

pkginfo
PKG="GNUbison"
NAME="GNU bison 1.24 SPARC Solaris 2.6"
VERSION="1.24"
ARCH="sparc"
CLASSES="none"
CATEGORY="utility"
VENDOR="GNU"
PSTAMP="12thAugust2004"
EMAIL="request@gnu.ai.mit.edu"
ISTATES="S s 1 2 3"
RSTATES="S s 1 2 3"
BASEDIR="/"


Prototype
i pkginfo
i checkinstall
d none /usr/local/bin 0755 root root
d none /usr/local/man 0755 root root

checkinstall
#!/bin/sh
#
# expected_release="5.6"
expected_platform="sparc"
#
release=`uname -r`
platform=`uname -p`
#
if [ ${platform} != ${expected_platform} ]; then
echo "\n\n\n\tThis package must be installed on a ${expected_platform} architecture\n"
echo "\tAborting installation.\n\n\n"
exit 1
fi
# if [ ${release} != ${expected_release} ]; then
# echo "\n\n\n\tThis package must be installed on a ${expected_release} machine\n"
# echo "\tAborting installation.\n\n\n"
# exit 1
# fi
exit 0

There are also several command that permit creation of your own custom packages or modification of existing packages:
• pkgmk Makes a new package from the specifications contained within a prototype file
• pkgproto Develops a new package specification to be stored within a prototype file
• pkgtrans Translates all of the files in a directory into a single package archive
Making New Packages
You need to create a specification for how an archive is to be built (called a prototype file), by using the pkgproto command, before using the pkgmk command to actually build the package. This material needs to be included in a pkginfo file. Here's a list of the most commonly used parameters that can be specified in the pkginfo file:
ARCH: The architecture (sparc or intel) for which the package is designed
BASEDIR: The target directory into which the files will be unpacked
CATEGORY: The archive type (system or application)
EMAIL: The contact details for the package builder
NAME: The package name
PSTAMP: The name of the package builder
VENDOR: The creator of the software being archived
VERSION: The release level of the software which is to be packaged

Once the pkginfo and prototype files have been created, it's very easy to build a package using pkgmk.
The pkginfo file should define all the necessary parameters (such as ARCH, BASEDIR, and CATEGORY), which were outlined previously. For the Borland Application Server, these elements would look like this:
PKG="BAS_LOCAL"
NAME="Borland App Server"
ARCH="sparc"/”all”
VERSION="4.5"
CATEGORY="application"
VENDOR="Borland"
EMAIL="support@borland.com"
PSTAMP="Paul Watters"
BASEDIR="/usr/local/inprise/bas45"
CLASSES="none"
We can interpret these entries in the following way:

*The package name is BAS_LOCAL (indicating a local installation of the BAS).
*The package title is Borland Application Server.
*The target architecture is SPARC.
*The version number is 4.5.
*The category of BAS is an application.
*The vendor of BAS is Borland.
*The e-mail address of the vendor is support@borland.com.
*The name of the package builder is the author of this book.
*The base installation directory is /usr/local/inprise/bas45.
Package Dependencies:
If you want your package to depend on another package, you need to create a 'depend' file and add to prototype file
i depend=”path”

Once the pkginfo file is created, we should use the following command to create the prototype file:
#touch prototype
Next, we need to specify the location of the pkginfo file, by inserting the following line into the prototype file:
i pkginfo=/usr/local/borland/bas45/pkginfo
In the base installation directory specified in the pkginfo file, you can use the following command to generate a list of relative pathnames and files that is then piped through the pkgproto command. The output is then appended to the prototype file:
# find . -print | pkgproto >> prototype
We should then be able to examine the contents of the newly constructed prototype file:
bash-2.03# head /usr/local/borland/bas45/prototype
i pkginfo=/usr/local/borland/bas45/pkginfo
f none install.idb 0644 bas borland
d none bin 0775 bas borland
f none bin/JdsExplorer 0775 bas borland
f none bin/JdsExplorer.config 0775 bas borland
f none bin/JdsServer 0775 bas borland
f none bin/JdsServer.config 0775 bas borland
f none bin/jdbce 0775 bas borland
f none bin/jdbce.config 0775 bas borland
f none bin/jsql 0775 bas borland
f none bin/jsql.config 0775 bas borland
Every file (f) and directory (d) underneath the root directory is represented by a unique entry in the prototype file.
After the prototype file has been modified to your satisfaction, create the package with the following command:
# pkgmk -o -r /usr/local/borland/bas45
The files to be packaged have been transferred to the /var/spool/pkg/BAS_LOCAL directory at this point. Next, you simply need to use the pkgtrans command to bundle the files into a single archive file:
# pkgtrans -s /var/spool/pkg /backup/BAS_LOCAL.pkg

Each Solaris package is represented as a directory that has the same name as the package. The package contains many files and directories out of which only pkginfo and pkgmap are mandatory. All other items shown in the figure are optional.
The pkginfo and pkgmap files are meta-information files. The pkginfo file acts like a label on a box. It contains information such as name, identifier, build date, who to contact if there's a problem, and so on.
The pkgmap file is a packing list, where each item in the package is defined and its size and checksum are stored. The size and checksum help in determining that the package has not been tampered with. The various items in the pkgmap are recognized by their ftype entry as shown here.

When a package is installed (using pkgadd), first directories (d), devices (c/b), and symlinks (s) are created. Then the files of type f, e, or v are copied into the file system. Hard links (ftype l) are created after all the other files have been installed. The install meta-files (ftype i) are used to direct the installation and removal of software and are not installed on the system.
A Solaris package typically contains reloc and install directories. The reloc directory contains all the files that are to be installed relative to a base directory, i.e. it contains the relocatable files. An optional root directory contains files that are to be installed relative to the root directory. Another optional directory called archive may also be present, which contains general-purpose files whose functions are defined by the package developer. The install directory contains scripts and other auxiliary files. All files of ftype i except pkginfo and pkgmap go into install.
The install directory contains a copyright file, an optional depend file, and scripts like checkinstall, preinstall, postinstall, preremove, and postremove. This directory may also contain a set of some special scripts known as Class Action Scripts. All the meta-info files are Bourne shell scripts.
The checkinstall script performs a detailed analysis of the system and gathers data; it doesn't change anything on a file system. The script may create or alter environment variables to control the installation. The script can cleanly halt the package installation, if necessary.
preinstall runs just before the actual file installation begins. This script is involved in things such as initializing the prototype and meta-files for the backout package, which will be constructed later by the postinstall script.
If a package is removed (using pkgrm), the preremove and postremove scripts, if present, run before and after the removal of the package to do some tasks like deleting the contents of some directories.
All the objects in a package fall under some class that is defined during package creation. If an object doesn't fall under any class, none class is used by default. All the classes to be installed are listed in the parameter CLASSES, which is defined in the pkginfo file. The installation occurs in the order they are listed in pkginfo. Objects with class none are always installed first. So none, if present in pkginfo, is always the first class entry. If a class isn't listed in the pkginfo, the objects in that class won't get installed even if they are defined in pkgmap.
Class Action Scripts (CAS) redefine how to add or remove an object from the package. That is, instead of a simple copy of files from the package to the appropriate file system, CAS may allow merging entries with an existing file, or saving a copy of an existing file. The Class Action Scripts work similarly during removal of a package. The install and removal Class Action Scripts are named i. and r., respectively, for . Files with ftype e and v are expected to have Class Action Scripts.
All the dependencies that a package might have on other packages are listed in the depend file.
As mentioned previously, pkginfo and pkgmap are the only required files in a package. One can create a package with just these two files. Such a package is perfectly acceptable, though it doesn't provide any important functionality per se.
Solaris packages can also be delivered in Package Datastream format. In this format there is a file containing one or more packages. This file is different from the directory format explained previously, which had one directory containing a package. Package datastreams are easier to distribute. We can convert from one format to another using pkgtrans.

Packaging Software with RPM/How to build the RPM Package

Packaging Software with RPM/How to build the RPM Package

The principal benefit of open source software is, as its name implies, access to the inner workings of an application. Given the source, you can study how an application works; change, improve, and extend its operation; borrow and repurpose code (per the limits of the application's license); and port the application to novel and emergent platforms.

In general, a pre-built, open source application is called a package and bundles all the binary, data, and configuration files required to run the application. A package also includes all the steps required to deploy the application on a system, typically in the form of a script. The script might generate data, start and stop system services, or manipulate files and directories. A script might also perform operations to upgrade existing software to a new version.

We will go with a example to build a “wget” rpm package

1.      Get the source code for wget

2.       $ tar xzf wget-latest.tar.gz

$ cd wget-1.12

$ ./configure

configure: configuring for GNU Wget 1.12

checking for a BSD-compatible install... /usr/bin/install -c

checking whether build environment is sane... yes

checking for a thread-safe mkdir -p... build-aux/install-sh -c -d

checking for gawk... no

$ make

$ sudo make install

./configure queries the system and sets compilation options suitable for the hardware and software detected. make compiles the code, and sudo make install installs the code in system directories. By default, the directories are rooted at /usr/local, although you can change the target root with the --prefix=/some/full/path/name option to ./configure.

To convert this process to RPM, you place the source in a repository and write a configuration file to dictate where to find the source to be compiled and how to build and install the code. The configuration file, called a spec file, is the input to a utility called rpmbuild. The spec file and the binaries are packaged by rpmbuild into an RPM. When another user downloads your RPM, the rpm utility reads the spec file and installs the package per your prewritten instructions.

Building the RPM

To build an RPM, you must:

• Set up a directory hierarchy per the rpmbuild specifications.
• Place your source code and supplemental files in the proper locations in the hierarchy.
• Create your spec file.
• Build the RPM. You can optionally build a source RPM to share your source code with others.

To begin, build the hierarchy. In a directory in your home directory—say, $HOME/mywget—create five subdirectories:

• BUILD. BUILD is used as scratch space to actually compile the software.
• RPMS. RPMS contains the binary RPM that rpmbuild builds.
• SOURCES. SOURCES is for source code.
• SPECS. SPECS contains your spec file or files—one spec file per RPM you want to build.
• SRPMS. SRPMS contains the source RPM built during the process.

At a minimum, you need source code in SOURCES and a spec file in SPECS.

Copying your source

$ cd ~

$ mkdir mywget

$ cd mywget 

$ mkdir BUILD RPMS SOURCES SPECS SRPMS

$ cd SOURCES

$ cp wget-latest.tar.gz .

$ mv wget-latest.tar.gz wget-1.12.tar.gz

$ cd ..

Next, create the spec file. A spec file is nothing more than a text file with a special syntax.Sample spec file as follows: 

# This is a sample spec file for wget

%define _topdir                              /home/strike/mywget

%define name                                                      wget 

%define release                               1

%define version          1.12

%define buildroot %{_topdir}/%{name}-%{version}-root

BuildRoot:                    %{buildroot}

Summary:                                         GNU wget

License:                         GPL

Name:                                                %{name}

Version:                         %{version}

Release:                         %{release}

Source:                           %{name}-%{version}.tar.gz

Prefix:                            /usr

Group:                                               Development/Tools

%description

The GNU wget program downloads files from the Internet using the command-line.

%prep

%setup -q

%build

./configure

make

%install

make install prefix=$RPM_BUILD_ROOT/usr

%files

%defattr(-,root,root)

/usr/local/bin/wget

%doc %attr(0444,root,root) /usr/local/share/man/man1/wget.1

Let's walk through the spec file from top to bottom. Lines 1-5 define a set of convenience variables used throughout the rest of the file. Lines 7-15 set a number of required parameters using the form parameter: value. As you can see in line 7 and elsewhere, variables can be evaluated and combined to produce the value of a setting.

The parameter names are largely self-evident, but BuildRoot merits some explanation to differentiate it from the BUILD directory you already created. BuildRoot is a proxy for the final installation directory. In other words, if wget is ultimately installed in /usr/local/bin/wget and other subdirectories in /usr/local, such as /usr/local/man for documentation, BuildRoot stands in for /usr/local during the RPM build process. Once you set BuildRoot, you can access its value using the RPM_BUILD_ROOT environment variable. You should always set BuildRoot in your spec file and check the contents of that directory to verify what is going to be installed by the package.

Here are a few tips:

• Do not use ./configure --prefix=$RPM_BUILD_ROOT. This command builds the entire package, assuming that the final location of the files is the build root. It is likely that this would cause any program that needs to locate its installed files at run time to fail, because when your RPM is finally installed on a user's system, the files aren't under the build root anymore—that's just a temporary directory on your build system.
• Do not include a path in the definition of Source.
• Version and Release are especially important. Each time you change your application's code or data and make a new RPM available, be sure to increment the values of Version and Release to reflect major and minor changes, respectively. You may find it helpful to bump the release number each time you build an RPM, even if for your own use, to keep attempts separate.

The next section starts with %description. You should provide a concise but clear description of the software here. This line is shown whenever a user runs rpm -qi to query the RPM database. You can explain what the package does, describe any warnings or additional configuration instructions, and more.

The %prep, %build, and %install sections are next, consecutively. Each section generates a shell script that is embedded into the RPM and run subsequently as part of the installation. %prep readies the source code, such as unpacking the tarball. Here, %setup -q is a %prep macro to automatically unpack the tarball named in Source.

The instructions in the %build section should look familiar. They are identical to the steps you used to configure and launch the build manually. The %install section is identical, too. However, while the target of the manual build was the actual /usr/local directory of your system, the target of the %install instruction is ~/mywget/BUILD.

%files lists the files that should be bundled into the RPM and optionally sets permissions and other information. Within %files, you can use the %defattr macro to define the default permissions, owner, and group of files in the RPM; in this example, %defattr(-,root,root) installs all the files owned by root, using whatever permissions found when RPM bundled them up from the build system.

You can include multiple files per line in %files. You can tag files by adding %doc or %config to the line. %doc tells RPM that the file is a documentation file, so that if a user installs the package using --excludedocs, the file is not be installed. %config tells RPM that this is a configuration file. During upgrades, RPM will attempt to avoid overwriting a user's carefully modified configuration with an RPM-packaged default configuration file.

Be aware that if you list a directory name under %files, RPM includes every file under that directory.

Revving the RPM

Now that your files are in place and your spec file is defined, you are ready to build the actual RPM file. To build it, use the aptly named rpmbuild utility:

$ rpmbuild -v -bb --clean SPECS/wget.spec

This command uses the named spec file to build a binary package (-bb for "build binary") with verbose output (-v). The build utility removes the build tree after the packages are made (--clean). If you also wanted to build the source RPM, specify -ba ("build all") instead of -bb. (See the rpmbuild man page for a complete list of options.)

rpmbuild performs these steps:

• Reads and parses the wget.spec file.
• Runs the %prep section to unpack the source code into a temporary directory. Here, the temporary directory is BUILD.
• Runs the %build section to compile the code.
• Runs the %install section to install the code into directories on the build machine.
• Reads the list of files from the %files section, gathers them up, and creates a binary RPM (and source RPM files, if you elect).

If you examine your $HOME/mywget directory, you should find a new directory named wget-1.12-root. This directory is the proxy for the target destination. You should also find a new directory named RPMS/i386, which should in turn contain your RPM, named wget-1.12-1.i386.rpm. The name of the RPM reflects that this is wget version 1.12 for the i386 processor.

Verifying the RPM Contents

$ rpm -Vp RPMS/i386/wget-1.12-1.i386.rpm

missing     /usr/local/bin/wget

.M....G.    /usr/local/etc

missing   c /usr/local/etc/wgetrc

.M....G.    /usr/local/share

missing     /usr/local/share/info

missing   d /usr/local/share/info/wget.info

missing     /usr/local/share/locale

missing     /usr/local/share/locale/be

missing     /usr/local/share/locale/be/LC_MESSAGES

missing   d /usr/local/share/locale/be/LC_MESSAGES/wget.mo

The command rpm -Vp RPMS/i386/wget-1.12-1.i386.rpm verifies the package against the files on the system. Although there are seemingly lots of errors, each is a clue that the contents of the RPM file are correct. If you are expecting a file to be installed and it does not appear in the output, it was not included in the package. In that event, review the spec file and make sure the file is enumerated in the %files section.

Another Example : How to Create Binary RPM Package

There is one important rule the reader should take in mind always - never run "rpmbuild" as root. This might end up in an empty root directory sooner as the reader might expect. Therefore the first configuration step to use "rpmbuild" should always be to create a suitable environment in normal user space. We are using the "sandbox" folder in our "Projects" tree for this prupose:

cd ~
mkdir -p Projects/sandbox
cd Projects/sandbox
mkdir SPECS
mkdir SOURCES
mkdir BUILD
mkdir RPMS
mkdir SRPMS

Afterwards we create the file ".rpmmacros" in our home directory:

cd ~
vi .rpmmacros

After the editor launced, please enter the following text according to local environment constraints:

%_topdir /home/$user/Projects/sandbox
%_builddir %{_topdir}/BUILD
%_rpmdir %{_topdir}/RPMS
%_sourcedir %{_topdir}/SOURCES
%_specdir %{_topdir}/SPECS
%_srcrpmdir %{_topdir}/SRPMS

The Spec File

"rpmbuild" needs a ".spec" file to build a ".rpm" file. Create such a spec file in your project directory (where the makefile is residing).

Summary: Key info on software
Name: name-of-the-package
Version: 1.2.3.4
Release: 1
Copyright: Copyright info
Group: Applications/System
%description
Brief description of software package.
%prep
%build
%install
%clean
%files
%defattr(-,root,root)
%doc
/opt/target/bin/programfile
/opt/target/bin/libshared.so
/opt/target/etc/config.cfg
%changelog

Finally – Get the RPM

rpmbuild -bb specfile.spec

Another Example: How to build RPM package

Creating a RPM for your own software requires three basic steps:

1. Making sure the rpm-build software is installed.
2. Perhaps modifying your software install process to be rpm "aware".
3. Writing a spec file.

This last step is the heart of the matter.  The "spec" file tells rpm-build everything that it needs to know to make the software and install it.

The rpm build process basically works as any software build works.  First it compiles the software, then it installs it.  The difference is that rpmbuild generally installs files in a subdirectories of a temporary directory - that way it doesn't interfere with software already installed and it is easy to identify the files that belong to the software package.  After the software is "installed", rpm-build compresses all the files in the subdirectory into one file, and the information in the spec file is included.  This way, when you go to really install the resulting binary rpm, the files will end up in the correct place and rpm knows which files are part of the package.

First you will want to create the following directories:

$ mkdir ~/rpm ~/rpm/BUILD ~/rpm/RPMS ~/rpm/RPMS/i386 ~/rpm/RPMS/i686 ~/rpm/RPMS/noarch ~/rpm/SOURCES ~/rpm/SPECS ~/rpm/SRPMS ~/rpm/tmp

These are the directories where the build process will take place.  You might have to add more directories if your system architecture is different from i386 or i686.

Then you need to create a file ~/.rpmmacros with at least the following lines:

%_topdir               /home/rpmbuild/rpm
%_tmppath              /home/rpmbuild/rpm/tmp

  In a spec file, anything that begins with a % is a macro.  rpmbuild will expand these.  These macros are typically defined in /usr/lib/rpm/macros and nearby files.

  Some macros are particularly important as they break the installation process into sections.  For example, %build tells rpmbuild to begin generating a simple bash script that will be run to build the software package.

  If the build process ends in an error, you can take a look at the bash scripts in ~/rpm/tmp/rpm-tmp.*.  Looking at these scripts will tell you exactly what variables are being defined and what is being done.  Often times you can fix an error in the spec file just by looking at the scripts.

  These scripts will be deleted upon termination of a section, preventing you from looking at them.  You can cheat and get a look at them by forcing an error in that section.  Just add "exit 1" to your spec file in the appropriate section.  Another useful thing is to "set; exit 1" in a spec file.  This will exit and spit out a list of defined environment variables.  Finally, you can place "bash -i" in a spec file to stop rpmbuild and drop to a shell where you can take a look around.

  Use %config(noreplace) - this will save configuration files from being replaced or over-written on update

Yet another Example : Howto Create RPM

First of all, we need to create a directory structure which is going to host the different elements that are required when building an RPM. As a normal user, type:

$ mkdir ~/packages
$ cd ~/packages
$ mkdir BUILD RPMS SOURCES SPECS SRPMS

The different directories will be used for:

• BUILD : The place where rpmbuild will build the package
• RPMS : The binary package created by rpmbuild will be stored here
• SOURCES : The place where to keep the original software sources
• SPECS : This is where the .spec will be stored
• SRPMS : The source rpm from your build will be stored there

# yum install rpm-build

In the first place, we need to get the software source available on our system. As said earlier, the directory to host the sources is: SOURCES ;), therefore we are going to cd to this directory and download the source there.

$ cd ~/packages/SOURCES/
$ wget http://kitone.free.fr/subtitleeditor/files/subtitleeditor-0.20.0alpha4.tar.gz

Spec file

$ cd ~/packages/SPECS
$ vi subtitleeditor.spec

Summary: Subtitle editor

Name: subtitleeditor

Version: 0.20.0alpha4

Release: 1.rb7

License: GPL

Group: Applications/Multimedia

URL: http://kitone.free.fr/subtitleeditor/

Source: http://kitone.free.fr/subtitleeditor/files/subtitleeditor-%{version}.tar.gz

BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root

Patch: gtk_update_icon_cache.patch

BuildRequires: gstreamer-plugins-base-devel, gtk2-devel, gtkmm24-devel, libglademm24-devel

BuildRequires: gstreamer-devel, pcre-devel, libxml++-devel, enchant-devel, iso-codes-devel

Packager: chantra

%description

Subtitle Editor is a GTK+2 tool to edit subtitles for GNU/*. 

It can be used for new subtitles or as a tool to transform, edit,

correct and refine existing subtitle. This program also shows sound 

waves, which makes it easier to synchronise subtitles to voices.

Subtitle Editor is free software released under the GNU General Public License (GPL).

%prep

%setup -q

%patch -p1 -b .buildroot

%build

%configure --disable-debug

%{__make}

%install

%{__rm} -rf %{buildroot}

%makeinstall

%find_lang %{name}

%clean

%{__rm} -rf %{buildroot}

%files -f %{name}.lang

%defattr(-, root, root, 0755)

%doc AUTHORS ChangeLog INSTALL NEWS TODO README COPYING

%{_bindir}/subtitleeditor

%{_datadir}/subtitleeditor/

%{_datadir}/icons/hicolor/*

%exclude %{_datadir}/icons/hicolor/icon-theme.cache

%{_datadir}/applications/subtitleeditor.desktop

%changelog

* Fri Aug 3 2007 - chantra AatT rpm-based DdOoTt org 0.20.alpha4-1.rb

- Initial release.

This file basically contains 7 sections: introduction, prep, build, install, clean, files and changelog. I will go through each section individually.

The first section, up to %prep is pretty explicit by itself. It mainly contains information as seen by rpm -qi subtitleeditor. Here is a short description of each parameters:

• Summary: a short description of the package
• Name: the name of the package
• Version: the version of the package we are building. Here subtitleeditor-0.20.0alpha4
• Release: the package release number. Here 1.rb7
• License: the license under which the software is released, here GPL
• Group: the group your package belongs to. See /usr/share/doc/rpm-/GROUPS
• URL: software homepage
• Source: the location of the sources
• BuildRoot: where the package is going to be built
• Patch: if you need to apply patch, enter the patch file name and copy the file to the SOURCES directory
• BuildRequires: packages requires in order to build this packages
• Packager: the name of the packager
• %description: a longer description of the package

Prep Section

The prep section prepares the package for the build process. This is the section taking care of untarring the package as well as patching if required.
%setup -q is a rpm macro which knows how to unarchive the source.
%patch will patch the source tree.

Build Section

This section will configure and make the package. Here we use the %configure to which we append the --disable-debug as we don't want to compile this feature.
Then we compile the package with the %{__make} macro.

“The meaning of the macros can be found in /usr/lib/rpm/macros”

Install Section

This section take care of cleaning up the BuildRoot and installing the application.

Clean Section

Cleans up files created by other sections.

Files Sections

defines the files that have to go in the binary RPM. Permission are defined and special attributes can be used.
This section starts with %files, additionally, the list of files that have to be included can be in a separate file, here subtitleeditor.lang.
%defattr(-, root, root, 0755) define the file attributes.
Files can be marked as document files by prepending their name with %doc.
You can exclude a file from a package using the %exclude macro.

Change log Section

You should add a new changelog entry anytime you repackage the RPM. Changelogs entry are defined as:

* Date - email name

- Action taken

Building the package

At this point, in ~/packages we can see the directories:

$ ls ~/packages/

BUILD RPMS SOURCES SPECS SRPMS

SPECS contains subtitleeditor spec file, SOURCES contains our source tarball and eventually, patches.

$ cd ~/packages/SPECS

In order to build a rpm package, We are going to use rpmbuild command

rpmbuild syntax is: rpmbuild -b . So, depending on which build stage you choose, you are going to build either th binary RPM, source RPM, none of them or both of them. Here is the list of the different switches that can be used:

• -ba : build all, both a binary and source RPM
• -bb : build only the binary RPM
• -bc : only compile the program. Does not make the full RPM, stopping just after the %build section
• -bp : prepare for building a binary RPM, and stop just after the %prep section
• -bi : create a binary RPM and stop just after the %install section
• -bl : check the listing of files for the RPM and generate errors if the buildroot is missing any of the files to be installed
• -bs : build aonly the source RPM

Make sure the ~/.rpmmacros pointing to Correct location

%_topdir /home//packages

$ rpmbuild -ba subtitleeditor.spec (Source rpm and Binary rpm)

$ rpmbuild -bb subtitleeditor.spec (Build Binary RPM)

Signing your pacakage

Signing a RPM with rpmbuild is pretty easy. First, you need to have a GPG key. I suppose that you already have a GPG key. Now, go and edit ~/.rpmmacros and add:

%_signature gpg

%_gpg_path /path/to/.gnupg

%_gpg_name name lastname (comment) 

%_gpgbin /usr/bin/gpg

You might want to use the output of gpg --list-keys to find out we value to enter for %_gpg_name .

$ rpmbuild -ba --sign subtitleeditor.spec