Skip to main content

Converting a “Big Board”: vBulletin to xenForo Migration

Migrating what?

First a bit about the urban75 boards. Based in Brixton, London, UK, urban75 is one of the busiest, non-profit community sites around, featuring huge photography sections, bulletin boards, live chat, drug information, football, direct action info and more. The boards are entirely non-profit and carry no advertising. The site is primarily funded by donations from board users (there are no subscriptions).

As a result, we’re not running on particularly flashy hardware, upgrading the server using spare equipment our hosting provider has available.

vBulletin Days and Nightmares

We have been running vBulletin for almost a decade and it served us well for much of that time.

In August 2010 we upgraded to vBulletin 4. That upgrade was, frankly, a nightmare.

I spent many, many hours trying to optimise the database, tracing slow queries and following bug reports. I also ended up installing Sphinx search, even though vBulletin seemed very slow at handling the results it got back from Sphinx. At first there were many complaints, some about things which we could fix, and others about things which eventually people just got used to.

xenForo is announced

When xenForo was announced, several of the moderators on the boards went over to the community on xenforo.com and were impressed by what they saw.

I speculatively purchased a xenForo license when they first became available, thinking I could use it myself even if it turned out to be unsuitable for Urban75.

Following the purchase of the license, I installed a test copy on our little test server, and invited the moderators over for a look.

I think it’s fair to say everyone was blown away by the functionality and there was soon a consensus that we should look to move the main Urban75 boards to xenForo.

Testing Times

We weren’t going to go live with a pre-1.0 version so we waited. We invited a number of users over from the main boards to take a look at xenForo and feedback their thoughts. That lasted a couple of weeks with lots of positive feedback. When 1.0 was released I started further investigations into the migration, checking what functionality was missing and what add-ons would be useful.

Using Paul M’s vBulletin 4 Importer  I ran through an import from a year-old backup of the main boards, importing around 9 million posts without a hitch. On our test server the import took just over 24 hours.

There was no way I was going to use the xenForo built-in search on that size of message base so I installed the Sphinx executable and mlx’s Sphinx Search Engine add-on, followed by a few hours configuring and indexing using Sphinx. Running some test searches showed results from the xenForo boards on the test server coming back much faster than they did on the vBulletin live server. Both were using Sphinx so that indicated something seriously bad was going on with vBulletin displaying search results.

I tried a few other extras, including Darkimmortal’s vB4 URL Redirection add-on, the Tapatalk and ForumRunner mobile add-ons, and the Stop Forum Spam and Multiple Account Detector add-ons.

Preparing for the Migration

Work got in the way of planning a full migration for a while, but eventually I put a stake in the ground, choosing the weekend of the anniversary of our vBulletin 4 upgrade. With the help of other moderators, we got a couple of styles sorted out, one very similar to the default xenForo style and another variation without avatars which had been a particular request.

In preparation, I installed the xenForo software on the live server under a different directory from the existing vBulletin, and did some of the config under a single Administrator user.

Here We Go!

The actual night of the migration arrived and I closed the vBulletin site at 23:59 on 5th August 2011. We have a small phpBB installation which we open for times when the main boards are down and I put a link to that on the “Site Closed” message.

First things first, a full MySQL dump of the vBulletin database and the embryonic xenForo database, just in case.

Then the first step of the migration, using the latest version of the vBulletin 4 importer.

The user importer fell over quite quickly, complaining about “Duplicate entry for key 2” for a user named “Ęllie”. Scratched by head for a few minutes before gently renaming the user and re-starting the import. A few minutes later I hit another user with the same issue, this time called “‘Mått”. Same fix for that one and the rest of the users imported fine.

I started the largest part of the import, copying the Threads and Posts at 01:55 on Saturday morning. By 02:30 it was obvious this was going to be a very long job so I went to bed hoping that the import might speed up while I was asleep. Rising at 06:30 I found that it hadn’t got any faster and that my original estimate of a much faster import on the live system was sadly wrong.

With me monitoring progress through the day, and updating the users via the phpBB boards and the “Site Closed” message, the posts finally finished importing at about 21:30. The importer then runs through a cache update for users and threads.

A dash through configuring Friendly URLs, moving the Sphinx config to index the xenForo boards instead of the vBulletin ones, checking basic user permissions and configuring a few more add-ons took me until 23:30 when we finally re-opened the boards.

So 23.5 hours start to finish for 10.3 million posts and I understand that we’re now the largest UK site running xenForo.

Aftermath

It took me a little while afterwards to get the vB4 Redirection working properly, and we’ve had a slow-query issue with another add-on which the author is hoping to fix. My thanks to Kier for helping track down where that query was coming from despite my still-frazzled support request.

The users seem generally pleased with the results, though there are a number of complaints about relatively minor pieces of functionality missing, dislike of change, dislike of the colours along with some genuine issues with useful functions missing that were previously available. Some of these are already addressed in xenForo 1.1, other might be later, and some may never get fixed.

From a tech admin point of view, xenForo is much lighter on the hardware, we’ve hit more than 800 users online already which is close to the highest we ever had under vBulletin and the only issue has been with the add-on I mentioned earlier.

My thanks to all the folks at xenForo for coming up with what has to be the neatest bit of forum software around. Also thanks to the authors of all the add-ons we’re using, and to all the xenForo community members for sharing their experiences and supporting such a great product.

It’s been a long journey, but we’ve reached our first xenForo milestone. May there be many more in the future!

Board Stats as of 8/8/2011

Discussions: 278,346
Messages: 10,346,335
Members: 40,492

Test System

CPU: One core of 2.133GHz Xeon X3210
RAM: 1.5GB
OS: Ubuntu Linux 10.10 virtual machine running on vSphere 4.1 host
Disks: Virtual disk stored on 3ware-controlled RAID1 pair of 250GB SATA drives

Live System
CPU: Intel Quad Core Q9300 @ 2.50GHz
RAM: 4GB
OS: FreeBSD 7.2
Disks: Adaptec-controlled RAID1 pair of 150GB WD Raptor SATA drives

Forum Software, ,

vSphere PowerCLI: Moved your ISO datastore ? Reconnecting your CD-ROM drives

We recently moved our ISO store from a legacy NFS server to our main NFS filer.

The first task was copying the actual files, which can be done via any machine that has both datastores mounted (with read-write access to the destination store).

The more significant job is reconfiguring the VMs to use the copies of the ISOs in the new datastore.

Here’s the PowerCLI script I used:

  1. #
  2. # Disconnect all CD/DVD ISOs on a particular datastore and reconnect them to the new datastore
  3. #
  4. $myvcenter = "myvcenter.host.net"
  5. $mydatacenter = "BigDC1"
  6. $originalISOstore = "MyFirstISOstore"
  7. $finalISOstore = "MuchBetterISOstore"
  8.  
  9. Connect-VIServer $myvcenter
  10.  
  11. (Get-VM -Location: (Get-Datacenter $mydatacenter)) | ForEach ( $_ ) 
  12. 	{ 
  13. 	$myDrive = Get-CDDrive $_
  14. 	if ($myDrive.IsoPath -match "\["+$originalISOstore+"\]") 
  15. 		{
  16. 		Set-CDDrive -CD $myDrive -IsoPath ($myDrive.IsoPath -replace $originalISOstore, $finalISOstore) -Confirm:$false  
  17. 		} 
  18. 	}
  19. Disconnect-VIServer $myvcenter -confirm:$false

It’s fairly self-explanatory.

The biggest caveat with this is that it assumes that the source and destination stores have the same structure, but it wouldn’t be difficult to amend it to change the destination path slightly.

VMware, , ,

vSphere Bugs & Minor Irritations

I’ve recently reported a couple of annoying bugs to VMware. Both have been around for a long time, almost certainly since the days of Virtual Center 2.0, maybe even earlier.

  • If a VM has nothing in the “Notes” annotation, vCenter displays the “Notes” from the previously selected VM instead.
    So if you have a machine with a note saying “Delete after 1st Jan 2011”, and you then view the summary of a machine with no note set, it’ll display the “Delete after 1st Jan 2011”. That could be bad…
    The problem only occurs if the VM has never had any Notes annotation. If you set one and then remove it, it shows the blank note correctly.
    **UPDATE** – This appears to be fixed in 4.1 U1 – it only seems to affect VMs which were deployed without notes under VirtualCenter 2.x.
  • When deploying a VM from a template, the Tasks & Events history doesn’t correctly name the template from which the VM was deployed.
    As you can see in the example image, vCenter lists the deployed VM name instead of the template. 

Click for full size

** UPDATE ** VMware have acknowledged this as a bug but it will not be fixed until vSphere 5, later this year.

And there’s a cosmetic thing which winds me up. I haven’t reported it as a bug but if anyone from VMware reads this, maybe they can have a word. It’s really trivial….

“Send Ctrl-Alt-del“. What has “del” done to deprive it of a capital “D”?

VMware Fusion also has a “Send Ctrl-Alt-Del” menu option but it gets the capitalisation right. I can only offer this as proof that Macs are better than PCs… or something.

VMware

vSphere HA Slot Sizes

vSphere HA slot sizes are used to calculate the number of VMs that can be powered on in an HA cluster with “Host failures cluster tolerates” selected. The slots size is calculated based on the size of reservations on the VMs in the cluster. HA Admission Control then prevents new VMs being powered on if it would not leave any slots available should a host fail.

The slot size for a cluster can be seen by going to the Summary Page for the cluster and clicking the “Advanced Runtime Info” link in the HA box.

If none of the VMs have CPU or RAM reservations, a default of 256MHz and 0GB is used.

The slots per host is derived by taking the total available CPU/RAM for the host and dividing by the slot size. Some CPU is reserved for the system so it will usually be a little lower than the full amount. So a host with 2xquad-core 2.4GHz CPUs (total 19.2GHz) and no VM CPU or RAM reservations has 73 slots and will only allow 73 VMs to be powered on if the cluster has two hosts and is set to protect against a single host failure.

Obviously this allows a very minimal amount of resource for each VM, so either reservations should be set for each VM, or slots size can be manually adjusted (see the VMware vSphere Availability Guide (pdf) for full details).

Note that the slot size is used for admission control calculations only. It has no direct effect on the resources available to VMs should an HA event occur.

There is a VMware Knowledgebase article (1010594) which  has some details of the difference in VI3 and vSphere 4.x.

VMware, , ,