 | Level: Introductory Martin Brown (questions@mcslp.com), Freelance writer and consultant
08 Mar 2004 If
you run Perl across many different computers of any sort, you know how
frustrating it can be to install Perl extension modules across those
machines. The administrative process is even worse if you have a Web
server farm and need to keep each machine up to date with a set suite
of extension modules for your installation. CPAN helps, but there are
issues with CPAN that make it an unwieldy solution for use on a
network. This article provides possible solutions before covering the
final system. The main goals are a unified installation/module set, a
single download, and a guaranteed unified set of version numbers across
all the computers in the network.
Perl module installation
As soon as you install Perl, you start to realize just how many
modules make up the "base" installation of the language. As time goes
on, you install more and more modules to extend and improve the
functionality of your Perl installation. Installing modules is not a
difficult process. Most modules are packaged as a simple compressed
tarball. The tarball contains the necessary files -- including C/C++
source code to integrate with a third-party library, if necessary.
Most Perl modules use the MakeMaker system -- another Perl module
installed with the base Perl installation -- which provides information
about the Perl installation as well as a mechanism for building and
installing components in the right place. MakeMaker works by
translating a simple configuration file into a standard makefile that
can be used with the familiar make command.
Using CPAN
If you don't know the Comprehensive Perl Archive Network (CPAN)
already, refer to the main page that comes with each installation. CPAN
can work in two ways -- either through an interactive shell or through
a series of functions that can be used as part of a wider Perl
script-based solution. At its simplest, you can use CPAN to install a
module simply by typing perl -MCPAN -e "install modulename", where modulename is the name of the package, bundle, or full Perl module you want to install. CPAN will do the rest for you.
Automating CPAN
Most people are familiar with CPAN as an interactive shell for
building and installing Perl modules. What many don't realize is that
the CPAN module that provides this functionality is actually just a
wrapper around an internal application program interface (API). For
example, to generate a list of installed modules that are out of date
compared with the versions available on CPAN, you can simply call CPAN::Shell->r. Listing 1 provides an example of the command and its output.
Listing 1. Code to generate a list of outdated modules
$ perl -MCPAN -e 'CPAN::Shell->r'
CPAN: Storable loaded ok
Going to read /export/build/solaris/cpan/Metadata
Database was generated on Sun, 25 Jul 2004 02:10:00 GMT
Package namespace installed latest in CPAN file
B::Assembler 0.06 0.07 N/NW/NWCLARK/perl-5.8.5.tar.gz
Cwd 2.19 2.20 K/KW/KWILLIAMS/Cwd-2.20.tar.gz
DBD::mysql 2.9003 2.9004 R/RU/RUDY/DBD-mysql-2.9004.tar.gz
DateTime 0.2101 0.22 D/DR/DROLSKY/DateTime-0.22.tar.gz
DateTime::TimeZone 0.27 0.28 D/DR/DROLSKY/DateTime-TimeZone-0.28.tar.gz
File::Scan 1.23 1.25 H/HD/HDIAS/File-Scan-1.25.tar.gz
File::Spec 0.87 0.88 K/KW/KWILLIAMS/File-Spec-0.88.tar.gz
Image::Magick 5.5.7 6.0 J/JC/JCRISTY/PerlMagick-6.02.tar.gz
Mail::ClamAV 0.08 0.11 S/SA/SABECK/Mail-ClamAV-0.11.tar.gz
Module::CoreList 1.94 1.95 R/RC/RCLAMP/Module-CoreList-1.95.tar.gz
POE 0.2802 0.29 R/RC/RCAPUTO/POE-0.29.tar.gz
XML::RSS::Parser 2.11 2.15 T/TI/TIMA/XML-RSS-Parser-2.15.tar.gz
htmlop v0.2.6 0.26 J/JA/JANL/w3mir-1.0.10.tar.gz
5 installed modules have a version number of 0
890 installed modules have no parseable version number
|
To install these modules automatically so that you are up to date, embed the call to CPAN::Shell->r in a call to the CPAN::Shell->install function:
$ /usr/local/bin/perl -MCPAN -e 'CPAN::Shell->install(CPAN::Shell->r)'
This code dumps the information you see in Listing 1, then installs each module to bring it up to date.
Automating CPAN across a network
CPAN is an excellent tool, but the system has its limitations. One
major drawback is that it's a single-system solution, which is fine if
you want to manage your Perl installations on a single-computer basis.
But if you have a set of Perl modules spread across your Web server
farm or even across your lab on a range of platforms, trying to keep up
to date across all the computers can be a time-consuming process --
even if you use the automation techniques I discussed above.
CPAN also consumes a lot of Internet bandwidth in that for each
computer on which you use CPAN, CPAN expects to download a copy of the
files from one of the central CPAN repositories. CPAN also relies on a
configuration that has to be tailored to each computer: You can copy a
configuration file, which resides in
/perlinstalldirectory/CPAN/Config.pm, only if you can configure your
platforms identically.
An obvious route to take when using CPAN is to use the autobundle
function. This function generates a list of every installed module in a
Perl installation, then dumps the list into a format that you can
execute to re-install the selection either on the same computer (after
an update to the latest Perl version) or onto another computer. The
code in Listing 2 shows you how to create a new bundle.
Listing 2. Code to create a CPAN autobundle
$ perl -MCPAN -e autobundle
...
Package namespace installed latest in CPAN file
AnyDBM_File 1.00 1.00 N/NW/NWCLARK/perl-5.8.5.tar.gz
Apache::ASP 2.57 2.57 C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::ApacheCommon undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::Application undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::CGI undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::CGI::Table undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::CGI::Test undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::Collection undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::CollectionItem undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
Apache::ASP::Date undef undef C/CH/CHAMAS/Apache-ASP-2.57.tar.gz
...
Wrote bundle file
/export/build/solaris/cpan/Bundle/Snapshot_2004_08_08_00.pm
|
The file that is generated is created within your CPAN installation
directory. You can copy it from that location or, if you're upgrading
to a later version of Perl, you can simply run the bundle module to re-install everything:
$ perl -MCPAN -e 'install Bundle::Snapshot_2004_08_08_00'
Although the bundle function works fine, I have to
admit that I'm not a huge fan of the bundling system as a network
solution for a few reasons. First, bundling relies on having an
identical configuration on each computer, which is fine if everything
is identical. But if you have different platforms and environments, you
can easily run into problems. Second, because the autobundle feature
lists every module installed, it lists multiple packages (individual
packages may contain multiple modules). However, the feature can also
list references to later versions of Perl (as the source for some
module updates).
The final problem, however, is that bundling relies on having a
single computer responsible for containing the latest version of all
the modules. You can't use the feature to upgrade a standard suite up
to the latest versions -- whatever they may be -- and this stipulation
makes the system complex to use in a network environment.
if you want to build a custom solution, two are available. You can use a central CPAN installation directory, or you can use a central CPAN source directory.
Use a central CPAN installation directory
A relatively straightforward way of configuring CPAN across a
network -- and one that works well if all your computers are identical
-- is to create a single CPAN directory that is shared over Network
File System (NFS), copy the CPAN/Config.pm file onto each computer,
then use the CPAN::Shell techniques discussed earlier to
install the modules on each computer. You install the "base" module set
on the main computer (usually the one sharing the CPAN directory over
NFS). For every other computer, the process of building and installing
each required module should be quick, because the source tarballs
already exist. Therefore, you don't need to download the tarball again,
saving precious Internet bandwidth and significant time.
The main problem with this technique is that for computers that are
in all other ways identical, it's wasteful on processor time: You've
already compiled it once, and it's wasteful to compile it all again.
You also run into a dangerous situation in which the same build
directory is being used for multiple computers, which really isn't a
good idea because different computers could be compiling the source at
the same time, causing a combination of timing errors and, in a
cross-platform environment, bad object files when linking.
Use a central CPAN source directory
An adaptation of the previous solution is to use the same source
directory and basic CPAN configuration but manually change the
configuration file so that the bulk of the configuration information
for each host is host-specific. In this way, you solve the build
directory problem but still use a local (that is, on your network)
cached version of the tarballs. Listing 3 provides such a configuration
file.
Listing 3. Example of a CPAN::Config file
# This is CPAN.pm's systemwide configuration file. This file provides
# defaults for users, and the values can be changed in a per-user
# configuration file. The user-config file is being looked for as
# ~/.cpan/CPAN/MyConfig.pm.
$CPAN::Config = {
'build_cache' => q[200],
'build_dir' => q[/export/build/solaris/cpan/build],
'cache_metadata' => q[1],
'cpan_home' => q[/export/build/solaris/cpan],
'dontload_hash' => { },
'ftp' => q[/usr/bin/ftp],
'ftp_proxy' => q[proxy.mcslp.pri:8080],
'getcwd' => q[cwd],
'gpg' => q[],
'gzip' => q[/usr/bin/gzip],
'histfile' => q[/export/build/solaris/cpan/histfile],
'histsize' => q[100],
'http_proxy' => q[proxy.mcslp.pri:8080],
'inactivity_timeout' => q[0],
'index_expire' => q[1],
'inhibit_startup_message' => q[0],
'keep_source_where' => q[/export/build/solaris/cpan/sources],
'lynx' => q[],
'make' => q[/usr/local/bin/make],
'make_arg' => q[-j3],
'make_install_arg' => q[UNINST=1],
'makepl_arg' => q[],
'ncftp' => q[],
'ncftpget' => q[],
'no_proxy' => q[],
'pager' => q[/usr/bin/less],
'prerequisites_policy' => q[follow],
'scan_cache' => q[atstart],
'shell' => q[/usr/local/bin/bash],
'tar' => q[/usr/local/bin/tar],
'term_is_latin' => q[1],
'unzip' => q[/usr/bin/unzip],
'urllist' => [q[ftp://cpan.etla.org/pub/CPAN], q[ftp://cpan.teleglobe.net/pub/CPAN],
q[ftp://ftp.demon.co.uk/pub/CPAN/], q[ftp://ftp.flirble.org/pub/languages/perl/CPAN/],
q[ftp://ftp.mirror.ac.uk/sites/ftp.funet.fi/pub/languages/perl/CPAN/],
q[ftp://ftp.mirror.anlx.net/CPAN/], q[ftp://ftp.plig.org/pub/CPAN/],
q[ftp://usit.shef.ac.uk/pub/packages/CPAN/], q[http://cpan.hambule.co.uk/]],
'wget' => q[],
};
1;
__END__
|
You'll need to change all the configuration directory parameters to
something local on each computer -- for example, change the line 'build_dir' => q[/export/build/solaris/cpan/build],
to point to a directory on the local machine. Meanwhile, share the CPAN
source directory on your main computer, then change the keep_source_where parameter to point to this NFS-mounted directory.
Unfortunately, a few issues remain with this solution. First, you
have no way of conveniently recording which modules you want to install
on each computer. You also run into the problem of keeping the
information in the source directory (and the rest of the CPAN
directories) correct and current without updating or downloading a
newer version of the tarball and upsetting the status quo among
computers. This issue can lead to minor but nearly fatal differences
between the installed versions of the modules on each system, which is
the key issue that we are trying to resolve.
Use CPAN to create a rigid installation
The problem with both of the previous solutions is that they rely on
using CPAN and some manipulation of the configuration file to get CPAN
to do the work. CPAN is not an ideal solution within a local area
network (LAN), even with the above tricks. Therefore, you need to
bypass those system limitations. To do so, I'll show you how to
configure the master distribution, create an installation set for each
platform, and then run the installation on each computer.
Configure the master distribution
For this first step, you're going to use some of the elements we've
already covered. Begin by configuring one computer to use CPAN as
normal and setting up your default set of modules. Next, use NFS to set
up a shared directory that you'll use to distribute the necessary files
to the other computers in the network. Now, edit the $destdir variable you see in Listing 4, then run the Listing 4 script. This script does three things:
- Identifies the modules
- Downloads the source files
- Copies the source files to the distribution directory
The first stage determines the list of installed modules by
examining the content of the perllocal.pod file. This file is updated
each time a module is installed using the MakeMaker system. By using
this file rather than the built-in modules in CPAN, you get a list of
third-party modules installed with your base Perl installation. The
second stage uses CPAN to download the source tarball by whatever
methods are configured within CPAN itself before finally copying the
tarball or .zip file. As part of that process, you also create a list
of the modules that were copied, which you'll use when you install the
files on the other computers.
Listing 4. Code to create a base distribution
use CPAN;
use Config;
use File::Copy;
use File::Spec::Functions;
# Set up the directories to store the module packages
my $basedestdir = catfile('export','cpaninst');
mkdir($basedestdir);
my $destdir = catfile($basedestdir,'srcs');
mkdir($destdir);
# Extract a list of the third party modules installed on this machine
my $files = {};
my $podfile = catfile($Config{'archlibexp'},'perllocal.pod');
open(DATA,$podfile) or die "Can't open module list ($podfile): $!";
while(<DATA>)
{
if (m/.*C<Module> L<(.*)>/)
{
my ($module) = split /\|/,$1,0;
my $mod = expand('Module',$module);
next unless $mod;
$mod->get(); #Make sure to download the version again, in case we no
#longer have it locally
$files->{$mod->{RO}->{CPAN_FILE}} = 1; #Save the location of the file
}
}
close(DATA);
# Now copy each source installer to the source destination directory
# We save a copy of the module file, for reference, as part of the process
my $modulelist = catfile($basedestdir,'modules.lst');
open(DATA,">$modulelist") or die "Can't open the module list file ($modulelist): $!";
foreach my $file (keys %{$files})
{
my $src = catfile($CPAN::Config->{'keep_source_where'},$file);
my ($vol,$path,$filename) = splitpath($file);
my $dest = catfile($destdir,$filename);
copy($src,$dest) or warn "Copy of $src failed: $!\n";
print DATA "$filename\n";
}
close(DATA);
|
You now have a directory containing all the source installation
packages that you can use to build a suite of ready-compiled
distributions for each platform to which you need to distribute the
corresponding Perl module. How you proceed from there will depend on
your environment. If you have a homogeneous environment, you need to
perform the next stage just once for your entire network. If, however,
you have a heterogeneous network with a variety of different platforms,
you'll need to repeat the next step for each different environment.
Create an installation set for each platform
For each platform, you need to extract the source file, run
MakeMaker, then run the installer. To perform this task automatically,
use the script in Listing 5. Unfortunately, not all MakeMaker
installations are completely automatic, so you'll need to follow the
interactive systems -- for example, to set directory locations or
default options -- each time you build the distribution.
Listing 5. Code to build the modules on each platform
srcdir="/export/cpan"
srcfile="$srcdir/module.lst"
platformdir="/export/cpan/solaris-x86"
for file in `cat $srcfile`
do
cd $platformdir
tar zxf $srcdir/$file
dir=`echo $file|sed -e "s/\.tar\.gz//g"`
cd $dir
perl Makefile.PL
make
done
|
Run the installation on each computer
When the script in Listing 5 has run, you'll have a single directory
that contains all the "ready-to-install" modules for a given platform.
To install these modules on any other computer using the same platform,
simply change into each directory and run make install. Again, you can use a script such as the one in Listing 6 for this process.
Listing 6. Code to install the files on each host
srcdir="/export/cpan"
srcfile="$srcdir/module.lst"
platformdir="/export/cpan/solaris-x86"
for file in `cat $srcfile`
do
cd $platformdir
tar zxf $srcdir/$file
dir=`echo $file|sed -e "s/\.tar\.gz//g"`
cd $dir
make install
done
|
Conclusions
Despite the functionality provided by Perl, CPAN, and the MakeMaker
system, you can't completely automate the installation and distribution
of Perl modules. But you can make the process simpler across a range of
computers. Individual modules have their own configurations, and
individual platforms have their own nuances making them more complex
than they need to be. By using the scripts in this article, you can
eliminate much of the hardship while reducing the time overhead that
might normally be associated with the process.
Not all issues are covered by this solution. For example,
third-party modules often make use of third-party libraries that are
not installed using this method. See my other developerWorks articles
for help here.
Resources
- "Automate your build and distribution process" (developerWorks, September 2004) shows you how to simplify the installation of any open-source software project.
- If you're interested in squeezing more performance from your Perl code, read "Optimize Perl" (developerWorks, October 2004).
- Perl has long been an important scripting tool for Linux developers, and developerWorks has an extensive collection of Perl articles to pick from.
-
Comprehensive Perl Archive Network is the home of the CPAN network system.
- Find more resources for Linux developers in the developerWorks Linux zone.
- Get involved in the developerWorks community by participating in
developerWorks blogs.
- Purchase Linux books at discounted prices in the Linux section of
the Developer Bookstore.
- Order the no-charge SEK for Linux, a two-DVD set containing the latest IBM trial software for Linux from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
- Innovate your next Linux development project with IBM trial software, available for download directly from developerWorks.
 |
About the author | | Martin
C. Brown is a former IT Director with experience in cross-platform
integration. A keen developer, he has produced dynamic sites for
blue-chip customers, including HP and Oracle, and is the Technical
Director of Foodware.net. Now a freelance writer and consultant, MC (as
he is better known) works closely with Microsoft as an SME, is the LAMP
Technologies Editor for LinuxWorld magazine, is a core member of the
AnswerSquad.com team, and has written several books on topics as
diverse as Microsoft certification, iMacs, and open source programming.
Despite his best attempts, he remains a regular and voracious
programmer on many platforms and numerous environments. Contact MC at questions@mcslp.com or through his Web site. |
Rate this content
| |
