GridLab logo
* About
* News
* Download
* Documents
* Collaborations
* Meetings
* Links
* Mailing List
* Management
* Yellow Pages
* Our Eyes Only
Information Society Technologies  
| Home | Products & Technologies | Support & Downloads | Contact us |  
GridLab WP-5

WP5: Step-by-step guide for site admins

Table of Contents

General Comments

  • Please check our Testbed status page and make sure that you have as many green boxes as possible.

Site setup

This part describes the steps needed to setup a site.
  • Software installation: Install software as described in the next section.
  • Add CA certificates: Untar this tarball into /etc/grid-security/certificates directory.
    For details see Testbed Certification Authorities document.
  • Create user accounts
    • Copy content of master grid-mapfile into /etc/grid-security/grid-mapfile.
    • Then create operating system's user accounts named glab001 ... glabNNN for all entries in grid-mapfile.
  • Register GRIS into GIIS on It means edit one configuration file and restart MDS. Details are described in Information Services Setup.
  • Open firewall if you have it for machines in the testbed (a list of them is on the status page) and for the machine which does periodical tests. You need to open following ports:
    • gsi-gatekeeper - 2119 (tcp)
    • GRIS - 2135 (tcp)
    • gsissh - 2222 (tcp)
    • gsiftp - 2811 (tcp)
    • mercury - 3570 (tcp),3571 (tcp)
    • cactus thorn - 5555 (tcp)
    • mds-webservice - 21000 (tcp)
    • iGrid - 19000 (tcp)
    • adaptive
      • Incoming connections from other Pythia's or Delphoi - 9889 (tcp)
      • Network capacity measurements (pathrate) - 8161 (tcp), 8163 (udp)
      • Available bandwidth measurements (pathchirp) - 8165 (udp), 8167 (udp)
    • GLOBUS_TCP_PORT_RANGE - define an interval for GridFTP etc. and open it

Requirements On Installed Software


Globus 3.2 pre-webservices

Important update! Globus 3.2-pre webservices is now required instead of 2.2.4 used previously. No Java tools will be required during installation. All examples are for Linux, flavor must be changed on different platforms.

  • Download gt3.2.1-preogsi-source-installer.tar.gz. Follow installation documentation
    • tar zvxf gt3.2-preogsi-source-installer.tar.gz
    • cd gt3.2-preogsi-source-installer
      directory contains standard globus bundles, GPT and gram schedulers and reporters in directory schedulers
    • set GLOBUS_LOCATION and GPT_LOCATION and run install-gt3
      It will install basic bundles (globus-data-management, globus-information-services, globus-resource-management)
    • $GLOBUS_LOCATION/sbin/gpt-postinstall
    • if you choose set authorization callbacks (I would suggest yes), run as root setup-globus-gaa-callouts and then comment all lines in /etc/grid-security/gsi-authz.conf
  • Configure installed services according setup documentation. Setup is very similar to previous version. Make sure that changes you did in previous versions of Globus are still in place:
    • job-manager scripts submitting to local batch system ($GLOBUS_LOCATION/lib/perl/Globus/GRAM/JobManager/*) is using correct version of MPI and supports you local cluster setup
    • globus job-manager is started with correctly set environment
  • install threaded version of globus-resource-management-sdk
    $GLOBUS_LOCATION/sbin/gpt-build globus-resource-management-sdk-3.2-src_bundle.tar.gz gcc32dbgpthr
  • install gram reporters for local batch systems, to support jobmanager-fork and PBS, you have to download globus_gram_reporter-2.0.tar.gz (missing in pre-webservices bundle) and install these packages. Make sure that correct mpirun and PBS commands are in your PATH when installing:
    • $GLOBUS_LOCATION/sbin/gpt-build globus_gram_reporter-2.0.tar.gz gcc32dbg
    • $GLOBUS_LOCATION/sbin/gpt-build schedulers/globus_gram_job_manager_setup_pbs-1.7.tar.gz gcc32dbg
    • $GLOBUS_LOCATION/sbin/gpt-build schedulers/gram-reporters/globus_gram_reporter_setup_fork-1.0.tar.gz gcc32dbg
    • $GLOBUS_LOCATION/sbin/gpt-build schedulers/gram-reporters/globus_gram_reporter_setup_pbs-1.0.tar.gz gcc32dbg
    • GLOBUS_LOCATION/sbin/gpt-postinstall

Globus 2 compatibility

Globus 3 servers are compatible with Globus 2 clients. Globus 3 clients can access Globus 2 servers only of old type of proxies (grid-proxy-init -old) is used.

GSI-enabled SSH

  • Download latest version form NCSA web pages
  • $GPT_LOCATION/sbin/gpt-build -verbose gsi_openssh_bundle-3.2-src.tar.gz gcc32dbg
  • $GPT_LOCATION/sbin/gpt-postinstall
  • configuration files are in the directory $GLOBUS_LOCATION/etc/ssh
  • update your startup scripts to start new ssh daemon:
    GLOBUS_LOCATION=... LD_LIBRARY_PATH=$GLOBUS_LOCATION/lib $GLOBUS_LOCATION/sbin/sshd -f /var/globus/3.2/etc/ssh/sshd_config

Monitoring system by WP11

  • source package: mercury-monitor-2.4.1.tar.gz
  • full installation instructions: Monitoring System Reference Manual
  • quick installation instructions:
    • prerequisites: pkg-config, glib2, gmake, gcc (version 3) and java (SDK, version 1.4 or 1.5)
    • building instructions:
      (set up your GridLab environment e.g. . /etc/gridlab.conf)
      $ ./configure
      Check the output for warnings and errors. You can ignore errors saying Gtk+ is missing as it is optional and only used for an example. Things to look for:
      • "GridLab-specific functionality" should say enabled. If it isn't then it's probably a problem with your /etc/gridlab.conf.
      • "Java support is disabled". If you see this check the lines above this message to see what failed. You should have java tools in your path. If jni.h is not found specify it's location to configure via --with-java-includes. If jni.h is still not found you may have to create a symlink in the same directory where jni.h is to jni_md.h which may be in a platform dependent subdirectory.
      • example without GSI support: ./configure --with-gridlab --prefix=/opt/gridlab/mercury-2.3.1 --enable-debug --with-java --with-service-name=host
      • example with GSI and PBS support: ./configure --with-gridlab --prefix=/opt/gridlab/mercury-2.3.1/ --with-gssapi=gsi --with-globus-flavor=gcc32dbg --with-pbs=/software/pbs-5.2.2/ --enable-debug --with-java --without-openssl --with-service-name=host
      $ make
      If this fails make sure that gmake and gcc >= 3 is used.
      $ make install
      (if you are having problems send a bug report to monitoring_(at)
    • testing:
      edit /etc/gridlab.conf and add GRIDLAB_MONITORING_LOCATION=$GRIDLAB_LOCATION/mercury-monitor-2.3.1, then do:
      $ mkdir -p var/run
      $ sbin/lm -d
      in another shell:
      $ $GRIDLAB_MONITORING_LOCATION/bin/monclient -n host.loadavg -p host=<your FQDN> monp://localhost
  • configuration:
    • A local monitor (lm) should be running on each host of the resource that has to be monitored or jobs to be monitored can run on (by default the lm uses port 3570). For clusters a main monitor (mm) should also run on the front-end node. The main monitor needs a list of lm URLs in @sysconfdir@/cluster_hosts filled by system administrators.
      In the configuration installed by default, both the local and the main monitor tries to use port 3570 which causes a conflict. So hosts running both the local and the main monitor (e.g. cluster front-end nodes) should reconfigure the local monitor (.../etc/lm.conf) and change
         Listen "monp://*"
         Listen "monp://localhost:3571"

      The format of the cluster_hosts file has changed in version 2.0. Previously it was one URL per line; now it has the format of
              <hostname>        <url of the local monitor on that host>
      frontend-node.cluster   monp://localhost:3571
      node1.cluster           monp://node1.cluster
      node2.cluster           monp://node2.cluster
      The default configuration included in the release should work for most cases however, the default ACLs installed at @sysconfdir@/acls disallow all access. If you want to see a green box at the GridLab Testbed status page and support Gridlab migration scenario, add at least
      # Default ACLs for lm/mm
      # Allow monitoring from localhost
      allow monitor * from localhost;
      allow monitor * from <FQDN_OF_YOUR_FRONT_END>;
      # Allow setting low-latency mode
      allow control monitor.conn.lowlatency to everyone;
      # GridLab defaults: enable job monitoring and the logging service
      allow monitor app.* to everyone;
      allow control app.* to everyone;
      allow monitor gridlab.logging to everyone;
      allow monitor monitor.* to everyone;
      # Limit the number of messages can be queued on a connection
      limit queued_messages 10000 to everyone;
      allow monitor host.loadavg to everyone;
      # Default rules if nothing else matched
      deny monitor * to everyone;
      deny control * to everyone;
      If you want to allow all access which was the default in previous versions simply change deny to allow.
    • Example of startup script can be found in directory $MERCURY_LOCATION/etc/init.d.
    • The generic configuration documentation of the monitoring system can be found at

iGrid, the GridLab Information Service

  • source package: gridlab-igrid-3.3b.tar.gz
  • full installation instructions: GridLab-10-5-0001-igrid_3.2b.pdf
  • quick installation instructions:
    When installing iGrid first time, it's strongly suggested to follow full documentation, which describes complete setup including PostgresSQL etc.
    • prerequisites: Globus Toolkit v3.2.1 pre-ogsi, gSOAP Toolkit v2.6, Mercury 2.3.X, Libxml2, PostgresSQL, libgtop
    • building instructions:
      • create unix account igrid
      • configure - you may need to specify path for includes or libraries for libgtop, libxml2, postgres, mercury. You may also specify full path to authorization file (--with-authorized-dn), disable ssl for connection to postgres database if running on the same host (--enable-ssl=no) and you should specify, that iGrid is listening on port 19000 (--with-listening-port=19000) e.g.
            export IGRID_LOCATION=$GRIDLAB_LOCATION/igrid-3.3
            ./configure --prefix=$IGRID_LOCATION
            --with-listening-port=19000 --enable-libgtop=yes --enable-ssl=no
      • copy soapcpp2 from gSOAP to bin directory
      • make && make install
      • Postgres instalation - make sure that postgres server is running, create postgres user igrid, create database igrid owned by user igrid, make sure that user igrid can access this database - see full documentation for more info
      • edit $IGRID_LOCATION/etc/igrid_conf.xml - make sure that all path descriptions are correct, including path to Globus instalation.
        Set iStore location to
        Make sure that you host is allowed to store information to your iServe server:
      • edit $IGRID_LOCATION/etc/authrorized_dn - either copy all users from /etc/grid-security/grid-mapfile and add DN of your host or you may allow access to all people using keyword all
      • copy host certificate and key (/etc/grid-security/host*pem) to different directory and change owner to igrid
      • prepare startup script, which should be run under igrid priviledges, e.g.
         . /etc/gridlab.conf
         cd /var/igrid/log
         X509_USER_CERT=/etc/grid-security/igrid/igrid.cert \
         X509_USER_KEY=/etc/grid-security/igrid/igrid.key \
         IGRID_CONF_FILENAME=$GRIDLAB_LOCATION/igrid-beta/etc/igrid_conf.xml \
         IGRID_DEBUG=1 IGRID_AUTH=local \
         $GRIDLAB_LOCATION/igrid-beta/server/igrid -p 19000 >>igrid.log 2>&1&
        make sure that igrid is started in directory, which is writeable for igrid user - logs of igrid producers are stored here.
      • In example above, authorisation using GAS is switched off (IGRID_AUTH=local). You may need set this variable in /etc/gridlab.conf too.
  • Service registration example: How to integrate fork jobmanager"
  • We register on the machine, which runs the iGrid server on the port 19000, a "jobmanager-fork" available on the same machine. the url to access this service is x-gram:// (<service protocol>://<service hostname>:<service port>); the service default port is 2119. We don't specify a service validity time so it will be infinite.
    ./igrid-service-register-client: Usage:
            -h web_service_hostname: FQDN of the server hosting the IGrid web service
            -p web_service_port: port number where the iGrid web service is listening on
            -s : set service name
            -n service hostname: set FQDN of the server hosting the service
            -r : set service protocol
            -t : set service port
            -f : set service default port
            -d service description: set service description
            -k service keywords: set service keywords
            -l service validity time: set service validity time
            -u usage
            -v verbose mode
    ./igrid-service-register-client -h -p 19000 -s jobmanager-fork -n -r x-gram -t 2119 -f 2119 -d "Scheduler type: fork" -k "scheduler, gram, fork"
    [Fri Jun 17 15:37:40 2005] GSI plugin for gSOAP v2.5:  Established security context with: /C=IT/O=INFN/OU=Personal Certificate/L=HPCC University of Lecce/CN=Daniele Lezzi
    Your service instance has been registered
    Here we search for the services on the machine:
    [lezzi@gridsurfer bin]$ ./igrid-service-lookup-client 
    [Fri Jun 17 15:51:36 2005] GSI plugin for gSOAP v2.5:  Established security context with: /C=IT/O=INFN/OU=Personal Certificate/L=HPCC University of Lecce/CN=Daniele Lezzi
    service name = jobmanager-fork
    description  = Scheduler type: fork
    keywords     = scheduler,gram,fork
    default port = 2119
    access URLs #1
    hostname  =
    url       = x-gram://
    publisher = /C=IT/O=INFN/OU=Personal Certificate/L=HPCC University of Lecce/CN=Daniele Lezzi
    created   = 1119019060
    valid     = 0

Adaptive components

GridLab administrators do *not* need to install Pythia more that once. The Pythia installations are be kept up-to-date automatically, so manual 'upgrades' are not necessary!
  • source package: Pythia-1.0.tar.gz
  • full installation instructions: README included in Pythia-1.0.tar.gz.
  • quick installation instructions:
    • prerequisites: Java (JRE is sufficient), Mercury
      Pythia should be running under non-root account, preferable glab030 (to allow automatic updates)
    • building instructions:
      • set up your GridLab environment e.g.
        . /etc/gridlab.conf
      • set GRIDLAB_ADAPTIVE_LOCATION to directory, where Pythia should be installed and GRIDLAB_ADAPTIVE_USER to account name under which Pythia will be running e.g.
        export GRIDLAB_ADAPTIVE_USER=glab030
      • install Pathrate and Pathchirp - both packages are included in Pythia tarball and can be installed using
         ./configure -with-gridlab
         make install
      • copy directories bin, logs, jars from Pythia tarball to $GRIDLAB_ADAPTIVE_LOCATION
      • change owner of all files in $GRIDLAB_ADAPTIVE_LOCATION to $GRIDLAB_ADAPTIVE_USER e.g.
    • configuration:
      • copy pythia.config to $GRIDLAB_ADAPTIVE_LOCATION, edit path to Pathrate, Pathchirp and Mercury and change owner to $GRIDLAB_ADAPTIVE_USER
      • if your site is using firewall, check README and documentation above for list of ports which must be opened
      • set GRIDLAB_ADAPTIVE_LOCATION, GRIDLAB_ADAPTIVE_USER and GRIDLAB_ADAPTIVE_PID_FILE (file, where pid of running pythia is stored) in /etc/gridlab.conf
      • $GRIDLAB_ADAPTIVE_LOCATION/bin/control-pythia can be used to start/stop/restart of Pythia
    • testing:


We also plan tests of MPICH-G2 jobs on Gridlab testbed. Install version or 1.2.6 from Make sure that you have up-to-date globus_io and globus_nexus packages - see Globus Advisories and install at least versions globus_io-5.5 and globus_nexus-6.5. For testing, simple MPICH-G2 instalation with non-mpi globus flavor is OK. For example, on our Linux cluster we configured MPICH-G2 by

RSHCOMMAND=rsh FC=g77 ../configure \
--prefix=/software/mpich-1.2.5/build/LINUX/ch_g2_gcc \
--with-common-prefix=/software/mpich-1.2.5 \
--with-device=globus2:-flavor=gcc32dbg  --enable-g
Globus and MPICH-G2 test instructions:
  • in this example we are testing fork and pbs jobmanager on host
  • /home/ruda/tmp/cpi.p4 is simple cpi example compiled by standard MPICH over sockets (P4)
  • /home/ruda/tmp/cpi.g2 is simple cpi example compiled by MPICH-G2
  • machinefiles mg2 and mg22 contains two servers capable running MPICH-G2 jobs, in our case
           skirit:/jobmanager 1
           skirit:/jobmanager 1
    for fork jobmanager
    for pbs jobmanager
  • Fork jobmanager / simple + mpi + multiple + mpich-g2 jobs
    • globus-job-run skirit:/jobmanager /bin/uname -a
    • globus-job-run skirit:/jobmanager -np 2 -x '(jobtype=mpi)' /home/ruda/tmp/cpi.p4
    • globus-job-run skirit:/jobmanager -np 2 -x '(jobtype=multiple)' /bin/uname -a
    • mpirun -np 2 -machinefile mg2 /home/ruda/tmp/cpi.g2
  • PBS jobmanager / simple + mpi + multiple + mpich-g2 jobs
    • globus-job-run skirit:/jobmanager-pbs /bin/uname -a
    • globus-job-run skirit:/jobmanager-pbs -np 4 -x '(jobtype=mpi)' /home/ruda/tmp/cpi.p4
    • globus-job-run skirit:/jobmanager-pbs -np 4 -x '(jobtype=multiple)' /bin/uname -a
    • mpirun -np 4 -machinefile mg22 /home/ruda/tmp/cpi.g2

GridLab: Grid Application Toolkit and Testbed is co-funded by the European Commission under the Fifth Framework Programme (IST-2001-32133).
Web admin: Petr Holub, web design: Radoslaw Strugalski

Last update on Monday, 22-Aug-2005 12:01:18 CEST.