Application backup – Scripted pre- and postbackup actions

Posted on 25.10.201014.02.2011 by looke

Many of today’s businesses rely heavily on their application servers. The times of simple fileshares and single-document based processes are over and with them the time of simple filecopy as a method of backing up is over.

In this article I want to describe a method to backup the two most common components of a modern application service: filesystem and database.
No matter how you solve your backup, the approach should always make sure, that the database integrity is given and that the filesystem is in sync with the database state.

The following two scripts are deployed as a Pre- and a Post backup script. The Pre- Backup script stops the application, dumps the database contents, creates a LVM snapshot and re-starts the application. Post- Backup removes the snapshot.

The fileset for the backup application would then look like this:
Database dumps: /dbdump
Filesystem snapshot: /volume-snapshot

appbackup-run-before.sh


#!/bin/bash
#
# Application Service Backup - Part 1 of 2
#
# Pre-Backup script
# - Stops Service
# - Takes a MySQL Dump
# - Creates a LVM Snapshot
# - Restarts Service
# - Mounts the LVM Snapshot
#
# 2010, Looke
#

# Which service to mess with
SERVICE="service"

# LVM Stuff
LVMVOLUME="/dev/lvm/volume"
LVMSNAPSHOT="volume-snapshot"
LVMSNAPSHOTSIZE="50G"

# MySQL Properties
DBDUMPDIR="/dbdump"
DBNAME="aaa"
DBHOST="zzz"
DBUSER="xxx"
DBPASSWORD="yyy"

echo "Shutting down Service..."
/etc/init.d/${SERVICE} stop

while ps ax | grep -v grep | grep ${SERVICE} > /dev/null;
do
  echo "...stopping..."
  sleep 5
done

echo "Creating MySQL Dump..."
if [ ! -d "${DBDUMPDIR}" ]; then
  mkdir -p ${DBDUMPDIR}
fi
mysqldump --host=${DBHOST} --user=${DBUSER} --password=${DBPASSWORD} ${DBNAME} > ${DBDUMPDIR}/${DBNAME}.sql

echo "Creating LVM Snapshot..."
modprobe dm-snapshot
lvm lvcreate --size ${LVMSNAPSHOTSIZE} --snapshot --name ${LVMSNAPSHOT} ${LVMVOLUME}
sleep 5

echo "Restarting Service..."
/etc/init.d/${SERVICE} start

while ! ps ax | grep -v grep | grep ${SERVICE} > /dev/null;
do
  echo "...starting..."
  sleep 5
done

echo "Mounting LVM Snapshot..."
if [ ! -d "/${LVMSNAPSHOT}" ]; then
  mkdir -p /${LVMSNAPSHOT}
fi
mount -o ro /dev/lvm/${LVMSNAPSHOT} /${LVMSNAPSHOT}

exit 0

appbackup-run-after.sh


#!/bin/bash
#
# Application Service Backup - Part 2 of 2
#
# Post-Backup script
# - Unmounts the LVM Snapshot
# - Destroys the LVM Snapshot
#
# 2010, Looke
#

# LVM Stuff
LVMSNAPSHOT="volume-snapshot"

echo "Unmounting LVM Snapshot..."
umount /${LVMSNAPSHOT}

echo "Destroying LVM Snapshot..."
lvm lvremove -f /dev/lvm/${LVMSNAPSHOT}

exit 0

One backdraw of this method is, that the service has to be stopped in order to get a consistent state of the data. If the service has to be online 24/7 you would have to consider clustering (anyways, you would have to come up with something to cover unplanned downtimes).

Here is a small excerpt to show you how to configure the Pre- and Post backup scripts with the open source backup software Bacula. I assume if you use some other backup software, you can click your way through the GUI yourself :)



Job {
  Name = "Appbackup"
  ...
  Client Run Before Job = "/opt/bacula/scripts/appbackup-run-before.sh"
  Client Run After Job = "/opt/bacula/scripts/appbackup-run-after.sh"
  ...
}

Useful links:
Bacula Documentation – Job Ressource
Ubuntuusers Wiki – LVM (german)

Moving a XEN guest to a new Dom0

Posted on 04.10.2010 by looke

I assume, you use LVM volumes for your XEN guests. I’m not going to use “xm migrate” here, the method used works by dd’ing the LVM volume over to the new Dom0, so make sure, you have a fitting LVM volume in place on your destination system.
I recommend you to stop the machine you’re going to move (or you could consider to create a LVM snapshot). Anyways, if you know nothing will change you can try it with the running machine (I did this once, and it resulted in a fsck upon boot but without any further problems).

With this one you can dd the LVM volume to the new host:


dd if=/dev/x bs=1M | ssh username@remote-server "dd of=/dev/y bs=1M"

To check the status of the copyjob, open a new console and issue (note: the USR1 signal lets dd print some infos):


watch -n 5 "killall -USR1 dd"

To finish the move, copy the XEN host config file to the new system:


scp /etc/xen/hostconfig username@remote-server:/etc/xen/hostconfig

Links
http://en.wikipedia.org/wiki/Dd_(Unix)

Deploying the open-source backup solution Bacula

Posted on 01.04.201023.02.2011 by looke

It’s now about two years ago, that I wondered “Why the … are we paying support and license subscriptions, if the only benefit is that you can listen to the support line music and get a new logo in the softwares main window after each update?”. Ok, the software works so far. But for every new client, you have to relicense and especially support for linux hosts can be a real pain.

I don’t want to call any names here nor start an argument with any fanboys. But being tired of all this commercial “corporate” softwares, I want to share my approach to installing the free and open source backup software Bacula.

Please feel free, to write me if you find possible errors or misconfigurations. I plan to extend this how-to with more detailled instructions.

Well, back to bacula: This overview visualizes the interactions of all bacula modules (taken from the bacula.org wiki)

To keep things simple, I start with a small but expandable test installation, consisting of one server and one or maybe two clients. In this case:

Hosts

bacula-server (Debian Lenny) – Director, Storage Daemon, File Daemon
mysql-server – MySQL Catalog
bacula-client-linux (Debian Lenny) – File Daemon
bacula-client-win (WinXP) – File Daemon

Following, I note all commands that are necessary to install the mentioned scenarion.

Installation of bacula-server (Director, Storage Daemon, File Daemon)


bacula-server:~# aptitude install build-essential libpq-dev libncurses5-dev libssl-dev psmisc libmysqlclient-dev mysql-client
bacula-server:~# cd /usr/local/src
bacula-server:~# wget http://downloads.sourceforge.net/project/bacula/bacula/5.0.1/bacula-5.0.1.tar.gz
bacula-server:~# tar xzvf bacula-5.0.1.tar.gz
bacula-server:~# cd bacula-5.0.1

To simplify the configure process, I used a shellscript with all the options (also the ones recommended by the Bacula project)


#!/bin/sh
prefix=/opt/bacula
CFLAGS="-g -O2 -Wall" \
  ./configure \
    --sbindir=${prefix}/bin \
    --sysconfdir=${prefix}/etc \
    --docdir=${prefix}/html \
    --htmldir=${prefix}/html \
    --with-working-dir=${prefix}/working \
    --with-pid-dir=${prefix}/working \
    --with-subsys-dir=${prefix}/working \
    --with-scriptdir=${prefix}/scripts \
    --with-plugindir=${prefix}/plugins \
    --libdir=${prefix}/lib \
    --enable-smartalloc \
    --with-mysql \
    --enable-conio \
    --with-openssl \
    --with-smtp-host=localhost \
    --with-baseport=9101 \
    --with-dir-user=bacula \
    --with-dir-group=bacula \
    --with-sd-user=bacula \
    --with-sd-group=bacula \
    --with-fd-user=root \
    --with-fd-group=bacula

Paste the code above in a file, make it executable (chmod +x) and run it.

If everything worked fine, type:


bacula-server:~# make && make install

Now to the setup of baculas catalog database. In my case, I use MySQL as catalog background, because I already have some knowledge about it. Other databases are supported as well (i.e. Postgres).
Bacula comes with all necessary scripts to create the initial catalog database on a local MySQL instance (I recommend you to apt-get the MySQL server and leave the root PW empty during the bacula setup phase). To have it setup on a remote server, you just need to check out the scripts, strip away the shell stuff and copy&paste the statements to your DB server (Thats what I did).


bacula-server:~# groupadd bacula
bacula-server:~# useradd -g bacula -d /opt/bacula/working -s /bin/bash bacula
bacula-server:~# passwd bacula
bacula-server:~# chown root:bacula /opt/bacula
bacula-server:~# chown bacula:bacula /opt/bacula/working
bacula-server:~# mkdir /backup2disk && chown -R bacula:bacula /backup2disk
bacula-server:~# touch /var/log/bacula.log && chown bacula:bacula /var/log/bacula.log
bacula-server:~# chown bacula:bacula /opt/bacula/scripts/make_catalog_backup /opt/bacula/scripts/delete_catalog_backup
bacula-server:~# cp /opt/bacula/scripts/bacula-ctl-dir /etc/init.d/bacula-dir
bacula-server:~# cp /opt/bacula/scripts/bacula-ctl-sd /etc/init.d/bacula-sd
bacula-server:~# cp /opt/bacula/scripts/bacula-ctl-fd /etc/init.d/bacula-fd
bacula-server:~# chmod 755 /etc/init.d/bacula-*
bacula-server:~# update-rc.d bacula-sd defaults 91
bacula-server:~# update-rc.d bacula-fd defaults 92
bacula-server:~# update-rc.d bacula-dir defaults 90

The following configfiles contain my example config (rename bacula-server-bacula-fd.conf to bacula-fd.conf):

bacula-dir.conf
bacula-sd.conf
bacula-server-bacula-fd.conf
bconsole.conf

Installation of Bweb and Brestore on bacula-server

If you like to actually see whats happening with your backups whithout hacking away on the console, I recommend you to install Bweb.


bacula-server:~# aptitude install lighttpd ttf-dejavu-core libgd-graph-perl libhtml-template-perl libexpect-perl libdbd-pg-perl libdbi-perl libdate-calc-perl libtime-modules-perl
bacula-server:~# /etc/init.d/lighttpd stop
bacula-server:~# update-rc.d -f lighttpd remove
bacula-server:~# cd /var/www
bacula-server:~# wget http://downloads.sourceforge.net/project/bacula/bacula/5.0.1/bacula-gui-5.0.1.tar.gz
bacula-server:~# tar xzvf bacula-gui-5.0.1.tar.gz
bacula-server:~# ln -s /var/www/bacula-gui-5.0.1 /var/www/bacula-gui
bacula-server:~# cd /var/www/bacula-gui/bweb

This is my httpd.conf, which contains logging and authentication support:
bweb-httpd.conf


bacula-server:~# touch /var/log/lighttpd/access.log /var/log/lighttpd/error.log
bacula-server:~# chown -R bacula:bacula /var/log/lighttpd
bacula-server:~# ln -s /opt/bacula/bin/bconsole /usr/bin/bconsole
bacula-server:~# chown bacula:bacula /opt/bacula/bin/bconsole /opt/bacula/etc/bconsole.conf
bacula-server:~# chown -R bacula:bacula /var/www/bacula*
bacula-server:~# cd /var/www/bacula-gui/bweb/script
bacula-server:~# mysql -p -u bacula -h mysql-server bacula < bweb-mysql.sql
bacula-server:~# ./starthttp

After we start lighttpd for the first time, it creates the bweb.conf configfile, which we own to the bacula user:


bacula-server:~# chown bacula:bacula /var/www/bacula-gui/bweb/bweb.conf

Now, open up a browser and navigate to the bweb page (lighttpd tells you where you can reach it after you start the service). Check out the following screenshot to see how to configure the Bweb instance:

If you also like to run restore jobs in a graphical manner, you can install the Brestore addon to your new Bweb interface.


bacula-server:~# aptitude install libdbd-pg-perl libexpect-perl libwww-perl libgtk2-gladexml-perl unzip
bacula-server:~# cd /var/www/bacula-gui/brestore
bacula-server:~# mkdir -p /usr/share/brestore
bacula-server:~# install -m 644 -o root -g root brestore.glade /usr/share/brestore
bacula-server:~# install -m 755 -o root -g root brestore.pl /usr/bin
bacula-server:~# cd /var/www/bacula-gui/bweb/html
bacula-server:~# wget http://www.extjs.com/deploy/ext-3.1.1.zip
bacula-server:~# unzip ext-3.1.1.zip
bacula-server:~# rm ext-3.1.1.zip
bacula-server:~# mv ext-3.1.1 ext
bacula-server:~# chown -R bacula:bacula ext
bacula-server:~# nano /etc/mime.types

Add a new MIME type:


text/brestore                                   brestore.pl

Restart the lighttpd server:


bacula-server:~# killall lighttpd
bacula-server:~# /var/www/bacula-gui/bweb/script/starthttp

Installation of bacula-client-linux (File Daemon)

I assume, you have a Debian Lenny system up and running.


bacula-client-linux:~# aptitude install build-essential libssl-dev
bacula-client-linux:~# cd /usr/local/src
bacula-client-linux:~# wget http://downloads.sourceforge.net/project/bacula/bacula/5.0.1/bacula-5.0.1.tar.gz
bacula-client-linux:~# tar xzvf bacula-5.0.1.tar.gz
bacula-client-linux:~# cd bacula-5.0.1

I also use a shellscript to configure our File Daemon, to make it more comfortable to deploy on multiple clients.


#!/bin/sh
prefix=/opt/bacula
CFLAGS="-g -O2 -Wall" \
  ./configure \
    --sbindir=${prefix}/bin \
    --sysconfdir=${prefix}/etc \
    --docdir=${prefix}/html \
    --htmldir=${prefix}/html \
    --with-working-dir=${prefix}/working \
    --with-pid-dir=${prefix}/working \
    --with-subsys-dir=${prefix}/working \
    --with-scriptdir=${prefix}/scripts \
    --with-plugindir=${prefix}/plugins \
    --libdir=${prefix}/lib \
    --enable-smartalloc \
    --with-openssl \
    --enable-client-only


bacula-client-linux:~# make && make install
bacula-client-linux:~# cp /opt/bacula/scripts/bacula-ctl-fd /etc/init.d/bacula-fd
bacula-client-linux:~# chmod 755 /etc/init.d/bacula-fd
bacula-client-linux:~# update-rc.d bacula-fd defaults 90

Finally, the configfile for our linux client (rename bacula-client-linux-bacula-fd.conf to bacula-fd.conf):

bacula-client-linux-bacula-fd.conf

Installation of bacula-client-win (File Daemon)

Get the windows binaries from the bacula page and make your way through the install dialogue:

Testing

Starting the Bacula services on bacula-server:


bacula-server:~# /etc/init.d/bacula-sd start
bacula-server:~# /etc/init.d/bacula-fd start
bacula-server:~# /etc/init.d/bacula-dir start

Starting the File Daemon on bacula-client-linux


bacula-client-linux:~# /etc/init.d/bacula-fd start

Bconsole Commands on bacula-server


bconsole
status
list clients
quit

Extended scenario – Tapelibrary on a separate server called “bacula-storage”
In this case, you don’t need to build the whole package. Apt-get the same packages as mentioned in the installation of bacula-server, get the bacula tarball, unpack and configure with the following script:

#!/bin/sh
prefix=/opt/bacula
CFLAGS="-g -O2 -Wall" \
  ./configure \
    --sbindir=${prefix}/bin \
    --sysconfdir=${prefix}/etc \
    --docdir=${prefix}/html \
    --htmldir=${prefix}/html \
    --with-working-dir=${prefix}/working \
    --with-pid-dir=${prefix}/working \
    --with-subsys-dir=${prefix}/working \
    --with-scriptdir=${prefix}/scripts \
    --with-plugindir=${prefix}/plugins \
    --libdir=${prefix}/lib \
    --enable-smartalloc \
    --with-mysql \
    --with-openssl \
    --with-smtp-host=localhost \
    --with-baseport=9101 \
    --disable-build-dird \
    --with-sd-user=bacula \
    --with-sd-group=bacula \
    --with-fd-user=root \
    --with-fd-group=bacula

2be continued with
– Bweb ssh remote command execution to show library status (reminder: don’t forget chmod g-w /opt/bacula/working)
– Extended configfiles

Hints
An Issue, that I noticed was, that brestore didn’t allow you to graphically drill down to the files you wanted to restore. You couldn’t click your way through the path but had to enter the path to the desired file by hand. It seems, that as soon as you back up another host, this problem resolves itself.

Links

Main manual: http://www.bacula.org/5.0.x-manuals/en/main/main/index.html

Detecting rogue WLANs with Kismet

Posted on 03.03.200921.10.2009 by looke

In a corporate environment, where you have several IT- security related regulations, it is critical to know what kind of wireless networks are in range of your facilities to avoid the bypassing of corporate security infrastructure (such as proxies, firewalls…)

This is where Kismet comes into play and assists you in finding rogue wireless LANs, using the monitor mode of your WLAN card.

You can get the software here: http://www.kismetwireless.net/
Kismet is open-source and also included in several security related linux distros.

The usage is quite simple. Just press “h” while the program runs to display a list of available keyboard shortcuts.

Featues
(taken from kismetwireless.net)

Ethereal/Tcpdump compatible data logging
Airsnort compatible weak-iv packet logging
Network IP range detection
Built-in channel hopping and multicard split channel hopping
Hidden network SSID decloaking
…

Here are some screenshots of Kismet in action

Hint (as fas as I know while using it under Ubuntu 8.10):
After closing the program, you might have to get your WLAN card back to managed mode.

ifconfig ethXX down
iwconfig ethXX mode managed
ifconfig ethXX up

Arduino – building open-source electronics prototypes

Posted on 04.02.200925.03.2011 by looke

Arduino is an open-source microprocessor platform to build electronics prototypes. Not only the development tools are open-source, but also the hardware itself. You can download all the necessary plans for it and build your own one, if you want to.

The really cool thing about the Arduino is, that you can program it with the open-source programming language Wiring, which is very straightforward to use.

Of course, being a complete newbie in electronics engineering, I ordered my own Arduino board as a prebuilt base-board – the Arduino Duemilanove…
…and finally, it arrived :)

As a first act of familiarization, I started with the very basic “push a button to power the LED” program


int ledPin = 13; // choose the pin for the LED
int inPin = 2;    // choose the input pin (for a pushbutton)
int val = 0;      // variable for reading the pin status

void setup() {
  pinMode(ledPin, OUTPUT);  // declare LED as output
  pinMode(inPin, INPUT);      // declare pushbutton as input
}

void loop(){
  val = digitalRead(inPin);   // read input value
  if (val == HIGH) {           // check if the input is HIGH (button released)
    digitalWrite(ledPin, LOW);  // turn LED OFF
  } else {
    digitalWrite(ledPin, HIGH);  // turn LED ON
  }
}

looke.ch

Just a geeks tech blog…

Open Source

Application backup – Scripted pre- and postbackup actions

Moving a XEN guest to a new Dom0

Deploying the open-source backup solution Bacula

Detecting rogue WLANs with Kismet

Arduino – building open-source electronics prototypes