 |
WP5: Step-by-step guide for site admins
Table of Contents
- Please check our Testbed status
page and make sure that you have as many green boxes as
possible.
This part describes the steps needed to setup a site.
- Software installation: Install software as
described in the next section.
- Add CA certificates: Untar this tarball into
/etc/grid-security/certificates directory.
For details see Testbed Certification
Authorities document.
- Create user accounts
- Copy content of master
grid-mapfile into
/etc/grid-security/grid-mapfile.
- Then create operating system's user accounts named
glab001 ... glabNNN for all entries in
grid-mapfile.
- Register GRIS into GIIS on mds.gridlab.org.
It means edit one configuration file and restart MDS. Details are
described in Information Services
Setup.
- Open firewall if you have it for
machines in the testbed (a list of them is on the status page) and for
the machine
loni.ics.muni.cz which does periodical
tests. You need to open following ports:
- gsi-gatekeeper - 2119 (tcp)
- GRIS - 2135 (tcp)
- gsissh - 2222 (tcp)
- gsiftp - 2811 (tcp)
- mercury - 3570 (tcp),3571 (tcp)
- cactus thorn - 5555 (tcp)
- mds-webservice - 21000 (tcp)
- iGrid - 19000 (tcp)
- adaptive
- Incoming connections from other Pythia's or Delphoi - 9889
(tcp)
- Network capacity measurements (pathrate) - 8161 (tcp), 8163
(udp)
- Available bandwidth measurements (pathchirp) - 8165 (udp),
8167 (udp)
GLOBUS_TCP_PORT_RANGE - define an interval for
GridFTP etc. and open it
Important update! Globus
3.2-pre webservices is now required instead of 2.2.4 used
previously. No Java tools will be required during installation. All
examples are for Linux, flavor must be changed on different
platforms.
- Download
gt3.2.1-preogsi-source-installer.tar.gz. Follow installation
documentation
- tar zvxf gt3.2-preogsi-source-installer.tar.gz
- cd gt3.2-preogsi-source-installer
directory contains standard globus bundles, GPT and gram
schedulers and reporters in directory schedulers
- set GLOBUS_LOCATION and GPT_LOCATION and run install-gt3
It will install basic bundles (globus-data-management,
globus-information-services, globus-resource-management)
- $GLOBUS_LOCATION/sbin/gpt-postinstall
- if you choose set authorization callbacks (I would suggest
yes), run as root setup-globus-gaa-callouts and then comment all
lines in /etc/grid-security/gsi-authz.conf
- Configure installed services according
setup documentation. Setup is very similar to previous version.
Make sure that changes you did in previous versions of Globus are
still in place:
- job-manager scripts submitting to local batch system
($GLOBUS_LOCATION/lib/perl/Globus/GRAM/JobManager/*) is using
correct version of MPI and supports you local cluster setup
- globus job-manager is started with correctly set
environment
- install threaded version of globus-resource-management-sdk
$GLOBUS_LOCATION/sbin/gpt-build
globus-resource-management-sdk-3.2-src_bundle.tar.gz
gcc32dbgpthr
- install gram reporters for local batch systems, to support
jobmanager-fork and PBS, you have to download globus_gram_reporter-2.0.tar.gz
(missing in pre-webservices bundle) and install these packages.
Make sure that correct mpirun and PBS commands are in your PATH
when installing:
- $GLOBUS_LOCATION/sbin/gpt-build globus_gram_reporter-2.0.tar.gz
gcc32dbg
- $GLOBUS_LOCATION/sbin/gpt-build
schedulers/globus_gram_job_manager_setup_pbs-1.7.tar.gz
gcc32dbg
- $GLOBUS_LOCATION/sbin/gpt-build
schedulers/gram-reporters/globus_gram_reporter_setup_fork-1.0.tar.gz
gcc32dbg
- $GLOBUS_LOCATION/sbin/gpt-build
schedulers/gram-reporters/globus_gram_reporter_setup_pbs-1.0.tar.gz
gcc32dbg
- GLOBUS_LOCATION/sbin/gpt-postinstall
Globus 3 servers are compatible with Globus 2 clients. Globus 3
clients can access Globus 2 servers only of old type of proxies
(grid-proxy-init -old) is used.
- Download latest version form NCSA web
pages
- $GPT_LOCATION/sbin/gpt-build -verbose
gsi_openssh_bundle-3.2-src.tar.gz gcc32dbg
- $GPT_LOCATION/sbin/gpt-postinstall
- configuration files are in the directory
$GLOBUS_LOCATION/etc/ssh
- update your startup scripts to start new ssh daemon:
GLOBUS_LOCATION=... LD_LIBRARY_PATH=$GLOBUS_LOCATION/lib
$GLOBUS_LOCATION/sbin/sshd -f
/var/globus/3.2/etc/ssh/sshd_config
- source package:
mercury-monitor-2.4.1.tar.gz
- full installation instructions:
Monitoring System Reference Manual
- quick installation instructions:
-
prerequisites: pkg-config, glib2,
gmake, gcc (version 3) and java
(SDK, version 1.4 or 1.5)
-
building instructions:
(set up your GridLab environment e.g. .
/etc/gridlab.conf)
$ ./configure
Check the output for warnings and errors. You can ignore errors
saying Gtk+ is missing as it is optional and only used for an
example. Things to look for:
- "GridLab-specific functionality" should say enabled. If it
isn't then it's probably a problem with your
/etc/gridlab.conf.
- "Java support is disabled". If you see this check the lines
above this message to see what failed. You should have java tools
in your path. If jni.h is not found specify it's location to
configure via --with-java-includes. If jni.h is still not
found you may have to create a symlink in the same directory where
jni.h is to jni_md.h which may be in a platform dependent
subdirectory.
- example without GSI support: ./configure --with-gridlab
--prefix=/opt/gridlab/mercury-2.3.1 --enable-debug --with-java
--with-service-name=host
- example with GSI and PBS support: ./configure
--with-gridlab --prefix=/opt/gridlab/mercury-2.3.1/
--with-gssapi=gsi --with-globus-flavor=gcc32dbg
--with-pbs=/software/pbs-5.2.2/ --enable-debug --with-java
--without-openssl --with-service-name=host
$ make
If this fails make sure that gmake and gcc >= 3 is used.
$ make install
(if you are having problems send a bug report to monitoring_(at)_gridlab.org)
- testing:
edit /etc/gridlab.conf and add
GRIDLAB_MONITORING_LOCATION=$GRIDLAB_LOCATION/mercury-monitor-2.3.1,
then do:
$ cd $GRIDLAB_MONITORING_LOCATION
$ mkdir -p var/run
$ sbin/lm -d
in another shell:
$ $GRIDLAB_MONITORING_LOCATION/bin/monclient -n host.loadavg
-p host=<your FQDN> monp://localhost
- configuration:
- A local monitor (lm) should be running on each host of the
resource that has to be monitored or jobs to be monitored can run
on (by default the lm uses port 3570). For clusters a main monitor
(mm) should also run on the front-end node. The main monitor needs a
list of lm URLs in @sysconfdir@/cluster_hosts filled by system
administrators.
In the configuration installed by default, both the local and the
main monitor tries to use port 3570 which causes a conflict. So
hosts running both the local and the main monitor (e.g. cluster
front-end nodes) should reconfigure the local monitor
(.../etc/lm.conf) and change
Listen "monp://*"
to
Listen "monp://localhost:3571"
The format of the cluster_hosts file has changed in version 2.0.
Previously it was one URL per line; now it has the format of
<hostname> <url of the local monitor on that host>
Example:
frontend-node.cluster monp://localhost:3571
node1.cluster monp://node1.cluster
node2.cluster monp://node2.cluster
The default configuration included in the release should work for
most cases however, the default ACLs installed at @sysconfdir@/acls
disallow all access. If you want to see a green box at the GridLab
Testbed status page and support Gridlab migration scenario, add at
least
#
# Default ACLs for lm/mm
#
# Allow monitoring from localhost
allow monitor * from localhost;
allow monitor * from <FQDN_OF_YOUR_FRONT_END>;
# Allow setting low-latency mode
allow control monitor.conn.lowlatency to everyone;
# GridLab defaults: enable job monitoring and the logging service
allow monitor app.* to everyone;
allow control app.* to everyone;
allow monitor gridlab.logging to everyone;
allow monitor monitor.* to everyone;
# Limit the number of messages can be queued on a connection
limit queued_messages 10000 to everyone;
allow monitor host.loadavg to everyone;
# Default rules if nothing else matched
deny monitor * to everyone;
deny control * to everyone;
If you want to allow all access which was the default in previous
versions simply change deny to allow.
- Example of startup script can be found in directory
$MERCURY_LOCATION/etc/init.d.
- The generic configuration documentation of the monitoring
system can be found at
http://www.gridlab.org/WorkPackages/wp-11/monitor-manual/config.html
- source package:
gridlab-igrid-3.3b.tar.gz
- full installation instructions:
GridLab-10-5-0001-igrid_3.2b.pdf
- quick installation instructions:
When installing iGrid first time,
it's strongly suggested to follow full documentation, which describes complete
setup including PostgresSQL etc.
- prerequisites: Globus Toolkit v3.2.1 pre-ogsi,
gSOAP Toolkit v2.6, Mercury 2.3.X, Libxml2, PostgresSQL, libgtop
- building instructions:
- create unix account igrid
- configure - you may need to specify path for includes or libraries
for libgtop, libxml2, postgres, mercury. You may also specify full path
to authorization file (--with-authorized-dn), disable ssl for
connection to postgres database if running on the same host
(--enable-ssl=no) and you should specify, that iGrid is listening on
port 19000 (--with-listening-port=19000) e.g.
export IGRID_LOCATION=$GRIDLAB_LOCATION/igrid-3.3
./configure --prefix=$IGRID_LOCATION
--with-authorized-dn=$IGRID_LOCATION/etc/authorized_dn
--with-listening-port=19000 --enable-libgtop=yes --enable-ssl=no
--with-mercury-libs=$MERCURY_LOCATION/lib
--with-mercury-includes=$MERCURY_LOCATION/include
--with-libgtop-includes=/opt/gnome/include/libgtop-2.0
--with-libgtop-libs=/opt/gnome/lib64
- copy soapcpp2 from gSOAP to bin directory
- make && make install
- Postgres instalation - make sure that postgres server is running, create
postgres user igrid, create database igrid owned by user igrid, make sure
that user igrid can access this database - see full documentation for
more info
- edit $IGRID_LOCATION/etc/igrid_conf.xml - make sure that all path
descriptions are correct, including path to Globus instalation.
Set iStore location to mds.gridlab.org:
<Registered_iStore>
<iStore_WS>httpg://mds.gridlab.org:19000</iStore_WS>
</Registered_iStore>
Make sure that you host is allowed to store information to your iServe
server:
<Allowed_iNodes>
<iNode_DN>DN_of_your_machine</iNode_DN>
</Allowed_iNodes>
- edit $IGRID_LOCATION/etc/authrorized_dn - either copy all users from
/etc/grid-security/grid-mapfile and add
DN of your host or you may allow
access to all people using keyword all
- copy host certificate and key (/etc/grid-security/host*pem) to
different directory and change owner to igrid
- prepare startup script, which should be run under igrid priviledges, e.g.
#!/bin/sh
. /etc/gridlab.conf
cd /var/igrid/log
X509_USER_CERT=/etc/grid-security/igrid/igrid.cert \
X509_USER_KEY=/etc/grid-security/igrid/igrid.key \
IGRID_CONF_FILENAME=$GRIDLAB_LOCATION/igrid-beta/etc/igrid_conf.xml \
IGRID_DEBUG=1 IGRID_AUTH=local \
$GRIDLAB_LOCATION/igrid-beta/server/igrid -p 19000 >>igrid.log 2>&1&
make sure that igrid is started in directory, which is writeable for igrid
user - logs of igrid producers are stored here.
- In example above, authorisation using GAS is switched off
(IGRID_AUTH=local). You may need set this variable in
/etc/gridlab.conf too.
- Service registration example: How to integrate fork jobmanager"
We register on the machine gridsurfer.unile.it, which runs the iGrid server on the port 19000, a "jobmanager-fork" available on the same machine. the url to access this service is x-gram://gridsurfer.unile.it:2119/ (<service protocol>://<service hostname>:<service port>); the service default port is 2119. We don't specify a service validity time so it will be infinite.
./igrid-service-register-client: Usage:
-h web_service_hostname: FQDN of the server hosting the IGrid web service
-p web_service_port: port number where the iGrid web service is listening on
-s : set service name
-n service hostname: set FQDN of the server hosting the service
-r : set service protocol
-t : set service port
-f : set service default port
-d service description: set service description
-k service keywords: set service keywords
-l service validity time: set service validity time
-u usage
-v verbose mode
./igrid-service-register-client -h gridsurfer.unile.it -p 19000 -s jobmanager-fork -n gridsurfer.unile.it -r x-gram -t 2119 -f 2119 -d "Scheduler type: fork" -k "scheduler, gram, fork"
[Fri Jun 17 15:37:40 2005] GSI plugin for gSOAP v2.5: Established security context with: /C=IT/O=INFN/OU=Personal Certificate/L=HPCC University of Lecce/CN=Daniele Lezzi
Your service instance has been registered
Here we search for the services on the gridsurfer.unile.it machine:
[lezzi@gridsurfer bin]$ ./igrid-service-lookup-client
[Fri Jun 17 15:51:36 2005] GSI plugin for gSOAP v2.5: Established security context with: /C=IT/O=INFN/OU=Personal Certificate/L=HPCC University of Lecce/CN=Daniele Lezzi
service name = jobmanager-fork
description = Scheduler type: fork
keywords = scheduler,gram,fork
default port = 2119
access URLs #1
hostname = gridsurfer.unile.it
url = x-gram://gridsurfer.unile.it:2119/
publisher = /C=IT/O=INFN/OU=Personal Certificate/L=HPCC University of Lecce/CN=Daniele Lezzi
created = 1119019060
valid = 0
GridLab administrators do *not* need to install Pythia more that
once. The Pythia installations are be kept up-to-date
automatically, so manual 'upgrades' are not necessary!
- source package:
Pythia-1.0.tar.gz
- full installation instructions: README included in
Pythia-1.0.tar.gz.
- quick installation instructions:
- prerequisites: Java (JRE is sufficient), Mercury
Pythia should be running under non-root account, preferable
glab030 (to allow automatic updates)
- building instructions:
- configuration:
- testing:
We also plan tests of MPICH-G2 jobs on Gridlab testbed. Install
version 1.2.5.3 or 1.2.6 from http://www3.niu.edu/mpi/. Make sure
that you have up-to-date globus_io and globus_nexus packages - see
Globus
Advisories and install at least versions globus_io-5.5 and
globus_nexus-6.5. For testing, simple MPICH-G2 instalation with
non-mpi globus flavor is OK. For example, on our Linux cluster we
configured MPICH-G2 by
RSHCOMMAND=rsh FC=g77 ../configure \
--prefix=/software/mpich-1.2.5/build/LINUX/ch_g2_gcc \
--with-common-prefix=/software/mpich-1.2.5 \
--with-device=globus2:-flavor=gcc32dbg --enable-g
Globus and MPICH-G2 test instructions:
|
