Virtual Machine Images

WORK IN PROCESS.

Information on rebuilding / updating the Foswiki Virtual Machine

Updating The Image

• Update the OS with the latest Ubuntu LTS build.
  • For minor updates, apt-get update && apt-get dist-upgrade
• Install the new Foswiki image.
  • Preferable to install new rather than upgrade, so that any removed files are cleaned.
  • Copy LocalSite.cfg from the old install
  • Be sure to run configure and update the JQuery versions to the latest default

Cleaning the image

The virtual machine image is currently created as a Vmware.vmdk image. After updating the system, it's important to reclaim unused space from the disk image files to minimize the size of the archives.

There is a script in the /root directory which will perform all of these steps.

1. Boot the virtual machine in single user mode
   • Hit escape at first indication of power on, to get the grub menu
   • Select advanced options for Ubuntu, then (recovery mode), and select root.
2. Stop apache, remove all log files, and return the machine to as clean as possible.
   • /var/log/syslog
   • /home/foswiki/.bash_history
   • /var/log/apache2/*
3. Clean up the foswiki installation. In /var/www/foswiki
   • Remove everything except README in working/logs/*
   • Remove everything except README in working/tmp/*
   • Remove everything except README in working/work_areas/*
   • Empty out data/.htpasswd and remove working/htpasswd.lock
4. Clean up user accounts
   • Remove /root/.ssh/known_hosts and /home/foswiki/.ssh/known_hosts
5. Remove any unused packages or kernels
   • apt-get clean
   • apt-get autoclean
   • apt-get autoremove
   • dpkg --get-selections | grep linux-image (Lists all installed kernels)
   • apt-get remove --purge linux-image-x.xx.xx...... (Remove unused kernels)
   • If warnings about invalid grub file, run sudo update-grub2
6. Zero out the swap space
   • Verify the actual device names using lvdisplay
   • swapoff /dev/foswiki/swap_1
   • dd if=/dev/zero of=/dev/foswiki/swap_1
   • mkswap /dev/foswiki/swap_1
   • swapon /dev/foswiki/swap_1
7. Zero out all unused space by copying zeros to a temporary file, allow it to run until the system runs out of disk space. Then remove the temp file.

    dd if=/dev/zero of=/tempfile 

8. Shutdown the VM
9. qemu-img convert -O qcow2 Foswiki.vmdk Shrink.qcow2
10. qemu-img convert -o subformat=twoGbMaxExtentSparse -O vmdk ../Shrink.qcow2 Foswiki.vmdk
11. Compress the image with 7z: 7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on Foswiki-2.1.2-vmware.7z Foswiki.vmwarevm

Notes

Other cleanup options

libguestfs tools should be able to do this in one step, however I've had issues getting it to work.

• virt-sparsify will remove unallocated space and shrink the files in one step
• virt-sysprep can prepare the image for publication (remove mac information, utmp, logs, etc)

The vmdk format currently used is twoGbMaxExtentSparse, which separates a control file and the actual data extents:

• Foswiki.vmdkContains configuration information and pointers to the following files:
  • Foswiki-s001.vmdk Extent 1
  • Foswiki-s002.vmdk Extent 2

It's not possible to access the disk images if they are renamed, unless the configuration file is edited, to point to the new extent names.

Persistent network names

As of the upgrade to Ubuntu 16.04, the network interfaces change when the VM is moved to different configurations. Solution was to modify grub.conf

sudo vi /etc/default/grub

Change line:
GRUB_CMDLINE_LINUX=""

to:
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"

This restores the old default first-come-first-serve assignment of eth0, eth1, etc. However in a VM environment with a single NIC, this is probably desired. Reference:

• http://www.itzgeek.com/how-tos/mini-howtos/change-default-network-name-ens33-to-old-eth0-on-ubuntu-16-04.html
• https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

Foswiki boot information

The IP address reported during login is generated by /usr/sbin/foswiki-ip. It is run from /etc/rc.local and from /home/foswiki/.bashrc

Passwords

Reset passwords if necessary:

• Foswiki account: foswiki
• webmin account foswiki: foswiki

Mounting the image on linux

The best way to "clean" the image, eliminating all logfiles, etc. is to mount it locally on linux. Unfortunately the system has been build using LVM, so it cannot be mounted on a simple loopback or even qemu-nbd mount.

Load the nbd driver, specifying max partitions. (default is 0) and attach the vmdk file.

# modprobe nbd max_part=8
# qemu-nbd -c /dev/nbd0 Foswiki.vmwarevm/Foswiki.vmdk
# fdisk /dev/nbd0 -l

Disk /dev/nbd0: 4 GiB, 4294967296 bytes, 8388608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0004dc8b

Device      Boot     Start       End  Blocks  Id System
/dev/nbd0p1 *         2048    499711  248832  83 Linux
/dev/nbd0p2         501758   8386559 3942401   5 Extended
/dev/nbd0p5         501760   8386559 3942400  8e Linux LVM

Scan the system for new volume groups. Mount the volume if not auto-mounted. Note that if the vgscan fails to find any volume groups, check the /etc/lvm/lvm.conf for any filter= options. Gentoo automatically filters nbd devices by default to minimize kernel messages.

# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "foswiki" using metadata type lvm2

# ls /dev/foswiki/
root  swap_1

# vgchange -ay
  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
  2 logical volume(s) in volume group "foswiki" now active

# mount /dev/foswiki/root /mnt/foswiki

Clean up the system

• Remove *.log and *.gz files from /var/log
• If you remove wtmp / utmp / btmp, wtmp must be recreated for "last" logging to be active.
• Remove logs from the foswiki install
• Remove bash history, .ssh info,
• Remove any persistent net rules

Once done,

# umount /mnt/foswiki

# vgchange -an foswiki
  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
  0 logical volume(s) in volume group "foswiki" now active

# qemu-nbd -d /dev/nbd0
/dev/nbd0 disconnected