buildbot is a continuous integration system. The server called buildmaster defines and schedules processes of building and testing of software to be performed by so called builders on machines called buildslaves (or buildbots). One buildslave can have several builders associated (for example a given operating system version buildslave performs testing of several software branches builders). One builder can also be associated with several buildslaves (several, “identical” machines running the same operating system), in which case buildmaster will schedule the given builder to switch periodically between the buildslaves. At the time of writing (2012) buildbot does not offer satisfactory security mechanisms, and due to clear text passwords stored/transferred both on buildmaster and buildslaves external security measures like firewall, proxy, ssh/stunnel should be used.
The sections below describe the configuration of buildmaster and buildslaves.
The buildmaster should be running as unprivileged user (usually called buildmaster). The user and group need to be created.
In order to install buildbot on RHEL6 system you need:
yum install python-twisted python-jinja2 python-tempita
Additional dependencies + buildbot itself need to be installed manually:
wget https://downloads.sourceforge.net/project/sqlalchemy/sqlalchemy/0.7.9/SQLAlchemy-0.7.9.tar.gz
wget http://sqlalchemy-migrate.googlecode.com/files/sqlalchemy-migrate-0.7.2.tar.gz
It is sufficient to unpack the files and set PYTHONPATH, and PATH accordingly. Buildbot may need to be patched for twisted compatibility http://www.npcglib.org/~stathis/blog/2012/10/20/bug-buildbot-dies-with-exceptions-importerror-cannot-import-name-noresource/
Proceed with buildbot configuration:
create the master:
buildbot create-master --relocatable python-ase
reconfigure the master with doc/development/master.cfg:
cp master.cfg python-ase
buildbot checkconfig python-ase/master.cfg
buildbot start python-ase
consider setting up a crontab which starts buildmaster after the server reboot (this solution is not reliable, deploy rather init scripts - see below):
# run every 5 minutes
*/5 * * * * sh ~/python-ase-restart.sh
with the following python-ase-restart.sh:
bot=python-ase
# no pid file - assume buildbot is not running
if ! test -f $bot/twistd.pid;
then
buildbot restart $bot || buildbot start $bot
else
# pid file exists but buildbot is not running
pid=`cat $bot/twistd.pid`
if test -z `ps -p $pid -o pid=`;
then
buildbot restart $bot
fi
fi
or create a system V init script under /etc/init.d
#!/bin/sh
#
# python-ase: python-ase buildmaster
#
# chkconfig: 345 98 02
# description: python-ase buildmaster
# LSB init-info
### BEGIN INIT INFO
# Provides: python-ase
# Required-Start: $network
# Required-Stop: $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: python-ase buildmaster
### END INIT INFO
# Source function library.
if [ -e /etc/init.d/functions ]; then
. /etc/init.d/functions
fi
# LSB functions
. /lib/lsb/init-functions
# Check that networking is configured.
[ "${NETWORKING}" = "no" ] && exit 0
RUN_AS=buildmaster-username
PYTHON_ASE_HOME=/home/$RUN_AS/python-ase
test -d $PYTHON_ASE_HOME || exit 5
LOGFILE=$PYTHON_ASE_HOME/../python-ase.log
PIDFILE=$PYTHON_ASE_HOME/../python-ase.pid
LOCKFILE=$PYTHON_ASE_HOME/../python-ase.lock
start() {
echo -n $"Starting python-ase buildmaster: "
dostatus > /dev/null 2>&1
if [ $RETVAL -eq 0 ]
then
echo -n $"python-ase buildmaster already running"
log_failure_msg
RETVAL=1
return
fi
#su - $RUN_AS -s /bin/sh -c "exec nohup /bin/sh $PYTHON_ASE_HOME/../python-ase-start.sh >> $LOGFILE 2>&1 &"
# don't produce log
su - $RUN_AS -s /bin/sh -c "exec nohup /bin/sh $PYTHON_ASE_HOME/../python-ase-start.sh >> /dev/null 2>&1 &"
RETVAL=$?
if [ $RETVAL -eq 0 ]
then
sleep 5
su - $RUN_AS -s /bin/sh -c "cat $PYTHON_ASE_HOME/twistd.pid > $PIDFILE"
su - $RUN_AS -s /bin/sh -c "touch $LOCKFILE"
log_success_msg
else
log_failure_msg
fi
return $RETVAL
}
stop() {
echo -n $"Shutting down python-ase buildmaster: "
kill $(su - $RUN_AS -s /bin/sh -c "cat $PIDFILE 2>/dev/null") > /dev/null 2>&1
RETVAL=$?
sleep 5
if [ $RETVAL -eq 0 ]
then
su - $RUN_AS -s /bin/sh -c "rm -f $PIDFILE $LOCKFILE"
log_success_msg
else
log_failure_msg
fi
return $RETVAL
}
restart() {
stop
start
}
condrestart() {
[ -f $LOCKFILE ] && restart || :
}
dostatus() {
kill -0 $(cat $PIDFILE 2>/dev/null) > /dev/null 2>&1
RETVAL=$?
if [ $RETVAL -eq 0 ]
then
echo "python-ase buildmaster (pid $(cat $PIDFILE 2>/dev/null)) is running..."
else
if [ -f $PIDFILE ]
then
echo "python-ase buildmaster dead but pid file exists"
RETVAL=1
return
fi
if [ -f $LOCKFILE ]
then
echo "python-ase buildmaster dead but subsys locked"
RETVAL=2
return
fi
echo "python-ase buildmaster is stopped"
RETVAL=3
fi
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
status)
dostatus
;;
restart|reload)
restart
;;
condrestart)
condrestart
;;
*)
echo $"Usage: $0 {start|stop|status|restart|reload|condrestart}"
exit 1
esac
exit $RETVAL
The service is added with:
chkconfig --add python-ase-buildmaster
started with:
service python-ase-buildmaster start
end enabled for boot time with:
chkconfig python-ase-buildmaster on
See also an example of systemd script in the section below.
consider protecting the buildmaster web interface by, e.g. apache reverse proxy (http://httpd.apache.org/docs/2.4/mod/mod_proxy.html). The basic configuration file may look like
# EEE.EEE.EEE.EEE is the IP of your External apache server:
# ase-buildbot.fysik.dtu.dk
<VirtualHost EEE.EEE.EEE.EEE:80>
ServerName ase-buildbot.fysik.dtu.dk
ServerAlias ase-buildbot ase-buildbot.fysik.dtu.dk
Redirect / https://ase-buildbot.fysik.dtu.dk/
</VirtualHost>
<VirtualHost EEE.EEE.EEE.EEE:443>
ServerName ase-buildbot.fysik.dtu.dk
ServerAlias ase-buildbot ase-buildbot.fysik.dtu.dk
# important http://httpd.apache.org/docs/current/mod/mod_proxy.html
ProxyRequests Off
# III.III.III.III:8080 the IP and port of the Internal buildbot web server
ProxyPass / http://III.III.III.III:8080/
ProxyPassReverse / http://III.III.III.III:8080/
<Proxy http://III.III.III.III:8080/*>
Options -Indexes
Order deny,allow
Deny from all
Allow from 127.0.0.1
Allow from ::1
#Allow from all
</Proxy>
SSLProxyEngine on
# provide an authority signed certificate
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
</VirtualHost>
# vim: filetype=apache shiftwidth=4 tabstop=4 wrap!
Some examples are given below. The configuration should be performed under an unprivileged buildslave, or buildbot user account.
Install with:
yum install buildbot-slave
Perform Configuration now.
After configuring buildbot-slave, you can configure systemd service by creating python-ase-fedora-18-x86_64-gcc-2.7.service file under /usr/lib/systemd/system.
[Unit]
Description=buildslave python-ase-fedora+18+x86_64+gcc+2.7 stock
After=network.target
[Service]
Type=forking
User=buildslave-username
Group=buildslave-groupname
ExecStart=/usr/bin/sh /home/buildslave-username/python-ase-fedora+18+x86_64+gcc+2.7-start.sh
Restart=always
[Install]
WantedBy=multi-user.target
python-ase-fedora+18+x86_64+gcc+2.7-start.sh simply exports the necessary environment variables (if needed) and starts buildslave (use the full path), e.g.:
#!/bin/sh
bot=/home/buildslave-username/python-ase-fedora+18+x86_64+gcc+2.7
buildslave start $bot
Choose User and Group under which buildslave will be running. The service is started with:
systemctl start python-ase-fedora-18-x86_64-gcc-2.7.service
In order to force the service to be started at boot time create a link:
cd /etc/systemd/system/multi-user.target.wants
ln -s /usr/lib/systemd/system/python-ase-fedora-18-x86_64-gcc-2.7.service .
Configure Homebrew, and:
brew install subversion
Configure a virtualenv, and then:
pip install numpy
pip install buildbot-slave
Perform Configuration now.
Install recent python-setuptools in order to get easy_install:
mkdir ~/buildbot-slave-el5
export PATH=$HOME/buildbot-slave-el5:${PATH}
export PYTHONPATH=$HOME/buildbot-slave-el5:${PYTHONPATH}
wget http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c11-py2.4.egg
sh setuptools-0.6c11-py2.4.egg --install-dir=$HOME/buildbot-slave-el5
then:
easy_install --install-dir=$HOME/buildbot-slave-el5 zope.interface==3.6.7
easy_install --install-dir=$HOME/buildbot-slave-el5 twisted==9.0.0 # ignore errors
easy_install --install-dir=$HOME/buildbot-slave-el5 buildbot-slave
Perform Configuration now.
Install build-slave and dependencies:
mkdir ~/buildbot-slave-el6
export PATH=$HOME/buildbot-slave-el6:${PATH}
export PYTHONPATH=$HOME/buildbot-slave-el6:${PYTHONPATH}
easy_install --install-dir=$HOME/buildbot-slave-el6 buildbot-slave
Perform Configuration now.
Install build-slave and dependencies:
mkdir ~/buildbot-slave-el7
export PATH=$HOME/buildbot-slave-el7:${PATH}
export PYTHONPATH=$HOME/buildbot-slave-el7:${PYTHONPATH}
easy_install --install-dir=$HOME/buildbot-slave-el7 buildbot-slave
Perform Configuration now.
build-slave can be installed and configured to start as a service on Windows http://trac.buildbot.net/wiki/RunningBuildbotOnWindows. This involves several steps:
install Python(x,y) from https://code.google.com/p/pythonxy/wiki/Downloads
configure distutils to use mingw. First enable showing file extensions:
Open a folder with IE -> Folder and search options-> View -> Folder Options:
Check: Show hidden files, ...; uncheck: Hide extensions for known file types.
Then, in notepad, create C:\python27\lib\distutils\distutils.cfg, containing:
[build]
compiler=mingw32
install build-slave on the command line:
C:\python27\scripts\easy_install.exe buildbot-slave
create a local (domain computer-name) user that will run the buildbot service:
control panel->administrative tools->computer management->local users and groups->users->new user: buildslave-username.
Click the created user: member of: administrators->check names
grant buildslave-username the ability to run the services. Login as computer-name\buildslave-username: Run secpol.msc on the command line as administrator (task bar->cmd->right click: run as administrator):
Select the correct “from this location” (may require login as the current domain administrator) and Enter the object names: computer-name\buildslave-username.
on the command line install the buildbot service:
buildbot_service.py --user computer-name\buildslave-username --password thepassword --startup auto install
start the service (for the moment it does not start any buildslave, because they are not configured yet):
Start->Control Panel> Administrative Tools->Services->Buildbot (Start)
There are additional steps mentioned in the buildbot wiki, but it seems just to work on Windows 7.
run regedit as administrator (type “regedit” on the command line) and add “directories” parameter of the String Value type, containing paths to all your buildslave instances (they will be configured in the Configuration section below):
HKEY_LOCAL_MACHINE\System\CurrentControlSet\services\Buildbot->paramaters->new (String Value): directories C:\python-ase-windows+7+AMD64+msc+2.7;C:\proj2
configure buildslave instance as described in the Configuration section below and start the service again (point 7.). Test that buildslave comes online, and verify that the service starts after reboot.
After having installed the buildbot create a name which will identify your buildslave. Obtain the first part of the name for your buildslave by running doc/development/master.cfg:
python master.cfg
This will return a string like redhat+6+x86_64+gcc+2.6 (OS+OSversion+Bitness+Ccompiler+PythonVersion). Append it with something that identifies your buildslave, for example gpu. For a very special system you can use a name like mycluster+x86_64+open64+2.5 gpu. Note that ASE buildbot will perform verification of python version based on the buildslave name so stay consistent!
Generate a strong buildslave password with Creating an encrypted password for SVN access. Don’t reuse any valuable passwords. You don’t need to remember it, buildbot stores it plain text!
Create the buildslave with:
buildslave create-slave python-ase-redhat+6+x86_64+gcc+2.6 buildbot.fysik.dtu.dk:ASE_BB_PORT "redhat+6+x86_64+gcc+2.6 gpu" "password"
ASE_BB_PORT is the port ASE buildbot uses for communication with the buildslave. You will have to tell us the name and password of your buildslave. Please contact ase-developers list Mailing Lists, but don’t send the name and password there!
Edit the python-ase-redhat+6+x86_64+gcc+2.6/info/{admin,info} files: describe your buildslave configuration relevant for the builder process in the info file.
Note that before starting the slave you need to perform an temporary svn checkout of ASE in order to accept the certificate permanently.
Start the buildslave with:
buildslave start python-ase-redhat+6+x86_64+gcc+2.6
Don’t forget to configure a crontab job or a service as described in the previous sections.
By default all slaves run the continuous integration for the trunk. If you prefer your buildslave works also on one of the branches, write this in the email to ase-developers Mailing Lists.