First a bit about the urban75 boards. Based in Brixton, London, UK, urban75 is one of the busiest, non-profit community sites around, featuring huge photography sections, bulletin boards, live chat, drug information, football, direct action info and more. The boards are entirely non-profit and carry no advertising. The site is primarily funded by donations from board users (there are no subscriptions).

As a result, we’re not running on particularly flashy hardware, upgrading the server using spare equipment our hosting provider has available.

vBulletin Days and Nightmares

We have been running vBulletin for almost a decade and it served us well for much of that time.

In August 2010 we upgraded to vBulletin 4. That upgrade was, frankly, a nightmare.

I spent many, many hours trying to optimise the database, tracing slow queries and following bug reports. I also ended up installing Sphinx search, even though vBulletin seemed very slow at handling the results it got back from Sphinx. At first there were many complaints, some about things which we could fix, and others about things which eventually people just got used to.

xenForo is announced

When xenForo was announced, several of the moderators on the boards went over to the community on xenforo.com and were impressed by what they saw.

I speculatively purchased a xenForo license when they first became available, thinking I could use it myself even if it turned out to be unsuitable for Urban75.

Following the purchase of the license, I installed a test copy on our little test server, and invited the moderators over for a look.

I think it’s fair to say everyone was blown away by the functionality and there was soon a consensus that we should look to move the main Urban75 boards to xenForo.

Testing Times

We weren’t going to go live with a pre-1.0 version so we waited. We invited a number of users over from the main boards to take a look at xenForo and feedback their thoughts. That lasted a couple of weeks with lots of positive feedback. When 1.0 was released I started further investigations into the migration, checking what functionality was missing and what add-ons would be useful.

Using Paul M’s vBulletin 4 Importer  I ran through an import from a year-old backup of the main boards, importing around 9 million posts without a hitch. On our test server the import took just over 24 hours.

There was no way I was going to use the xenForo built-in search on that size of message base so I installed the Sphinx executable and mlx’s Sphinx Search Engine add-on, followed by a few hours configuring and indexing using Sphinx. Running some test searches showed results from the xenForo boards on the test server coming back much faster than they did on the vBulletin live server. Both were using Sphinx so that indicated something seriously bad was going on with vBulletin displaying search results.

I tried a few other extras, including Darkimmortal’s vB4 URL Redirection add-on, the Tapatalk and ForumRunner mobile add-ons, and the Stop Forum Spam and Multiple Account Detector add-ons.

Preparing for the Migration

Work got in the way of planning a full migration for a while, but eventually I put a stake in the ground, choosing the weekend of the anniversary of our vBulletin 4 upgrade. With the help of other moderators, we got a couple of styles sorted out, one very similar to the default xenForo style and another variation without avatars which had been a particular request.

In preparation, I installed the xenForo software on the live server under a different directory from the existing vBulletin, and did some of the config under a single Administrator user.

Here We Go!

The actual night of the migration arrived and I closed the vBulletin site at 23:59 on 5th August 2011. We have a small phpBB installation which we open for times when the main boards are down and I put a link to that on the “Site Closed” message.

First things first, a full MySQL dump of the vBulletin database and the embryonic xenForo database, just in case.

Then the first step of the migration, using the latest version of the vBulletin 4 importer.

The user importer fell over quite quickly, complaining about “Duplicate entry for key 2″ for a user named “Ęllie”. Scratched by head for a few minutes before gently renaming the user and re-starting the import. A few minutes later I hit another user with the same issue, this time called “‘Mått”. Same fix for that one and the rest of the users imported fine.

I started the largest part of the import, copying the Threads and Posts at 01:55 on Saturday morning. By 02:30 it was obvious this was going to be a very long job so I went to bed hoping that the import might speed up while I was asleep. Rising at 06:30 I found that it hadn’t got any faster and that my original estimate of a much faster import on the live system was sadly wrong.

With me monitoring progress through the day, and updating the users via the phpBB boards and the “Site Closed” message, the posts finally finished importing at about 21:30. The importer then runs through a cache update for users and threads.

A dash through configuring Friendly URLs, moving the Sphinx config to index the xenForo boards instead of the vBulletin ones, checking basic user permissions and configuring a few more add-ons took me until 23:30 when we finally re-opened the boards.

So 23.5 hours start to finish for 10.3 million posts and I understand that we’re now the largest UK site running xenForo.

Aftermath

It took me a little while afterwards to get the vB4 Redirection working properly, and we’ve had a slow-query issue with another add-on which the author is hoping to fix. My thanks to Kier for helping track down where that query was coming from despite my still-frazzled support request.

The users seem generally pleased with the results, though there are a number of complaints about relatively minor pieces of functionality missing, dislike of change, dislike of the colours along with some genuine issues with useful functions missing that were previously available. Some of these are already addressed in xenForo 1.1, other might be later, and some may never get fixed.

From a tech admin point of view, xenForo is much lighter on the hardware, we’ve hit more than 800 users online already which is close to the highest we ever had under vBulletin and the only issue has been with the add-on I mentioned earlier.

My thanks to all the folks at xenForo for coming up with what has to be the neatest bit of forum software around. Also thanks to the authors of all the add-ons we’re using, and to all the xenForo community members for sharing their experiences and supporting such a great product.

It’s been a long journey, but we’ve reached our first xenForo milestone. May there be many more in the future!

Board Stats as of 8/8/2011

Discussions: 278,346
Messages: 10,346,335
Members: 40,492

Test System

CPU: One core of 2.133GHz Xeon X3210
RAM: 1.5GB
OS: Ubuntu Linux 10.10 virtual machine running on vSphere 4.1 host
Disks: Virtual disk stored on 3ware-controlled RAID1 pair of 250GB SATA drives

Live System
CPU: Intel Quad Core Q9300 @ 2.50GHz
RAM: 4GB
OS: FreeBSD 7.2
Disks: Adaptec-controlled RAID1 pair of 150GB WD Raptor SATA drives

The first task was copying the actual files, which can be done via any machine that has both datastores mounted (with read-write access to the destination store).

The more significant job is reconfiguring the VMs to use the copies of the ISOs in the new datastore.

Here’s the PowerCLI script I used:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#
# Disconnect all CD/DVD ISOs on a particular datastore and reconnect them to the new datastore
#
$myvcenter = "myvcenter.host.net"
$mydatacenter = "BigDC1"
$originalISOstore = "MyFirstISOstore"
$finalISOstore = "MuchBetterISOstore"
 
Connect-VIServer $myvcenter
 
(Get-VM -Location: (Get-Datacenter $mydatacenter)) | ForEach ( $_ ) 
	{ 
	$myDrive = Get-CDDrive $_
	if ($myDrive.IsoPath -match "\["+$originalISOstore+"\]") 
		{
		Set-CDDrive -CD $myDrive -IsoPath ($myDrive.IsoPath -replace $originalISOstore, $finalISOstore) -Confirm:$false  
		} 
	}
Disconnect-VIServer $myvcenter -confirm:$false

It’s fairly self-explanatory.

The biggest caveat with this is that it assumes that the source and destination stores have the same structure, but it wouldn’t be difficult to amend it to change the destination path slightly.

• If a VM has nothing in the “Notes” annotation, vCenter displays the “Notes” from the previously selected VM instead.
  So if you have a machine with a note saying “Delete after 1st Jan 2011″, and you then view the summary of a machine with no note set, it’ll display the “Delete after 1st Jan 2011″. That could be bad…
  The problem only occurs if the VM has never had any Notes annotation. If you set one and then remove it, it shows the blank note correctly.
  **UPDATE** – This appears to be fixed in 4.1 U1 – it only seems to affect VMs which were deployed without notes under VirtualCenter 2.x.
• When deploying a VM from a template, the Tasks & Events history doesn’t correctly name the template from which the VM was deployed.
  As you can see in the example image, vCenter lists the deployed VM name instead of the template. 

  

Click for full size

** UPDATE ** VMware have acknowledged this as a bug but it will not be fixed until vSphere 5, later this year.

And there’s a cosmetic thing which winds me up. I haven’t reported it as a bug but if anyone from VMware reads this, maybe they can have a word. It’s really trivial….

“Send Ctrl-Alt-del“. What has “del” done to deprive it of a capital “D”?

VMware Fusion also has a “Send Ctrl-Alt-Del” menu option but it gets the capitalisation right. I can only offer this as proof that Macs are better than PCs… or something.

The slot size for a cluster can be seen by going to the Summary Page for the cluster and clicking the “Advanced Runtime Info” link in the HA box.

If none of the VMs have CPU or RAM reservations, a default of 256MHz and 0GB is used.

The slots per host is derived by taking the total available CPU/RAM for the host and dividing by the slot size. Some CPU is reserved for the system so it will usually be a little lower than the full amount. So a host with 2xquad-core 2.4GHz CPUs (total 19.2GHz) and no VM CPU or RAM reservations has 73 slots and will only allow 73 VMs to be powered on if the cluster has two hosts and is set to protect against a single host failure.

Obviously this allows a very minimal amount of resource for each VM, so either reservations should be set for each VM, or slots size can be manually adjusted (see the VMware vSphere Availability Guide (pdf) for full details).

Note that the slot size is used for admission control calculations only. It has no direct effect on the resources available to VMs should an HA event occur.

There is a VMware Knowledgebase article (1010594) which  has some details of the difference in VI3 and vSphere 4.x.

Logging on to the Service Console on an affected host, “esxcfg-nas -l” also results in the same error.

The cause is invalid entries in the /etc/vmware/esx.conf file. Fortunately, it seems to be possible to remove the bad entries and the host then starts working properly without a reboot.

In our case, the bad entries looked like:
/nas/./enabled = "false"
/nas/./host = "1"
/nas/./share = "0"
whereas valid entries are recognisable:
/nas/vmdesktop/enabled = "true"
/nas/vmdesktop/host = "10.0.0.100"
/nas/vmdesktop/readOnly = "false"
/nas/vmdesktop/share = "/vol/vmdesktop"

As I mention above, fixing it on ESX isn’t too hard, just edit /etc/vmware/esx.conf using nano or vi as root, being careful not to affect any other lines.

On ESXi, it’s a little more tricky as you can’t readily log on and edit files without enabling Technical Support mode.

It is, however, possible to edit esx.conf via the Remote CLI (Linux or Windows) or using the vMA.

Below are some example commands which grabs the esx.conf file, edit it using ‘sed’, and then put the altered file back on the host.

vifs -get /host/esx.conf work.conf
cat work.conf | sed -e '/\/nas\/\./d' > fixed.conf
vifs.pl -put fixed.conf /host/esx.conf


The ‘sed’ command will probably need to change for you, depending on what the invalid lines look like. You can just use nano or vi on Linux or the vMA to do the edit, but if you’re using Windows you may find that Notepad and Wordpad either don’t display the file clearly or convert the line endings from Unix format to DOS. Using the free VIM for Windows (http://www.vim.org/download.php) will let you keep the file in the same format.

After making those changes, it was possible to add NFS datastores as normal.

I needed to add some datastores to all the clustered hosts in a single datacenter, but to skip our non-clustered standalone hosts which are used for backups.

A bit of PowerCLI which should be fairly self-explanatory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

#
#  Add Datastores to all hosts in all clusters in a specified datacenter
#  If a host isn't in a cluster it won't get the datastore
#  Easy enough to change to do all hosts in a datacenter, all in vCenter etc
#
#  Change the value below to point it to your own vCenter
$myvCenter = "vcenter.example.com"
 
# Array of arrays below holds NFS-hostname, NFS-path and Datastore name, should be easy to add to
$nfsArray = @(
			  @("nfsserver1","/vol/nfspath1","VMstore1"),
			  @("nfsserver2","/vol/nfspath2","VMstore2"),
			  @("nfsserver3","/vol/nfspath3","VMstore3")
			 )
 
$datacenter = Read-Host -Prompt "Datacenter name"
 
Write-Host "Datacenter is $datacenter"
 
connect-viserver -server $myvCenter
 
#
# Get all hosts in all clusters in the named datacenter
$ObjAllHosts = get-datacenter -name $datacenter | Get-Cluster  |  Get-VMHost 
 
ForEach($objHost in $ObjAllHosts){
	ForEach($nfsItem in $nfsArray) {
 
      Write-Host "Adding datastore" $nfsItem[2] "with path" $nfsItem[0]":"$nfsItem[1] "to" $objHost
      New-Datastore -Nfs -NfsHost $nfsItem[0] -Path $nfsItem[1] -Name $nfsItem[2] -VMHost (Get-VMHost $objHost)
    }
}
disconnect-viserver -server $myvCenter -Confirm:$false

Under 4.1, USB is no longer listed in the documentation as a bootable device, though boot from SAN devices via HBAs is now supported.

I’ve not checked whether you can actually install ESXi Installable 4.1 to a USB drive. It may well be possible, but I suspect it’s dropped off the “Supported” list of options.

The only reason I can see to drop support for putting Installable onto USB is to encourage people to purchase and use ESXi Embedded from their hardware supplier instead.

**UPDATE 2nd March 2011**

VMware have posted a clarification Knowledgebase article about the support for USB and SD for booting ESXi.

You can install ESXi 4.x to a USB or SD flash storage device directly attached to the server. This option is intended to allow you to gain experience with deploying a virtualized server without relying on traditional hard disks. However, VMware supports this option only under these conditions:

• The server on which you want to install ESXi 4.x is on the ESXi 4.x Hardware Compatibility Guide.

  AND

• You have purchased a server with ESXi 4.x Embedded on the server from a certified vendor.

  OR

• You have used a USB or SD flash device that is approved by the server vendor for the particular server model on which you want to install ESXi 4.x on a USB or SD flash storage device.

If you intend to install ESXi 4.x on a USB or SD flash storage device while ensuring VMware support for it and you have not purchased a server with embedded ESXi 4.x, consult your server vendor for the appropriate choice of a USB or SD flash storage device.

So as I suspected, it’s only supported if you install either using “Embedded” or on a device approved by the server vendor.

It does work absolutely fine on a normal USB stick, the host that this web server runs from boots from such an item in fact. It’s just not supported by VMware.

Worth noting that an approved 2GB USB stick from HP will cost you approx £75, about 15 times the going rate…

As well as the new features which have been covered extensively (see What’s New), the release notes list some future changes for the product range. They’re not really hidden but haven’t been given much publicity.

• ESX will be dropped in future releases, with ESXi being the hypervisor product for vSphere.
• Future versions of vCenter Update Manager will not scan or remediate guest OSes. I presume the cross-licensing costs of using Shavlik were outweighing any benefit. UM will continue to scan and update ESXi hosts. and presumably aid in conversion of ESX hosts to ESXi.
• VMware vCenter Converter plugin and VMware vCenter Guided Consolidation are also going away in future versions. Converter will continue in a standalone format rather than a vCenter plugin.
• Web Access isn’t available on ESXi so that’ll be going away when ESX is dropped too

There are a few other items being dropped such as support for some versions of Linux in guests, VMI paravirtualization support, and MSCS in Windows 2000 but they aren’t as widely used.

From the ESX Service Console of the host you can use
vmware-cmd <cfgfile> setguestinfo <variable> <value>
to set a value and vmware-cmd <cfgfile> getguestinfo <variable> to read the value. Note that you don’t need to use the “guestinfo.” prefix when using these commands.

Within the VM guest OS the values can be set/read using:
(Windows)
vmtoolsd.exe --cmd "info-set guestinfo.<variable> <value>"
vmtoolsd.exe --cmd "info-get guestinfo.<variable>"(vmtoolsd.exe is usually in C:\Program Files\VMware\VMware Tools)

(Linux)
vmware-guestd --cmd "info-set guestinfo.<variable> <value>"
vmware-guestd --cmd "info-get guestinfo.<variable>"(vmware-guestd is usually in /usr/sbin)

It wasn’t immediately obvious where the guestinfo variables live with regard to the PowerCLI vmConfig properties. Googling didn’t reveal much useful info, so I though I’d try the wonders of new technology and use Twitter. I’ve been following Carter Shanklin of VMware on Twitter since I attended a London UK VMware User Group meeting a couple of months ago, and as he’s the Product Manager for PowerCLI and the SDK, I thought I’d ask.

He quickly pointed me at the VMware SDK documentation for the ConfigInfo object and the extraConfig object in particular.

A bit of further reading led me to some experiments let me to try the following code:
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$extra = New-Object VMware.Vim.optionvalue
$extra.Key="guestinfo.test"
$extra.Value="TestFromPCLI"
$vmConfigSpec.extraconfig += $extra
$vm = Get-View -ViewType VirtualMachine | where { $_.name -eq "MyVMName" }
$vm.ReconfigVM($vmConfigSpec)
That will set the guestinfo.test property to “TestFromPCLI”. Once that’s been set it can be read by the VM.

The guestinfo property can have multiple Key/Value pairs so you can pass quite a few variables through to a VM. These can only be set when a VM is powered up and running VMware Tools as the value is stored in the VMs memory, and as far as I can tell, the contents are lost when the VM reboots.

However, there is another extraConfig object which can also be set which is the machine.id. Again this can be read from within the VM (replace guestinfo. with machine.id in the above code snippets), but this one gets written to the VMs VMX config file and will thus survive reboots.

You could squish several bits of info into that one object/variable, for example a unique identifier and the VM name so that the VM can self-configure.

There were various suggestions on the web, mostly involving stopping and starting the Virtual Center and License Server services. None of them worked. The only other suggestion was to remove all the hosts from the Virtual Center, add the license, then re-add the hosts back in. This would lose any nice filing of VMs in the “Virtual Machines & Templates” view, and would generally be a bit of a pain.

As a last resort, I thought I’d have a Google round for possible registry settings which may affect licensing.

I found a blog posting about changing the license type that VC looks for within the registry, which also mentioned a “LicensePath” key.

Looking in the registry of my VC server there was no “LicensePath” key, so I created one, stopped and restart the Virtual Center service and found that VC now found the license server correctly and didn’t complain about “Not enough licenses”.

So to recap, open regedit, navigate to
HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter

Create a new String key calledLicensePath and enter the info for your license server, e.g. 27000@vc01.example.com