Vl4dim1r's picture

Hell TKL Community! My community and I have been using TKL apps for a little over a year now, but this is my first post here on the forums.

We just finished migrating out network and apps from VMware ESXi to XenServer 6.2. Our host boards have 2nd gen AMD Opterons, 8224's, that do not support HVM, so we are stuck with Xen's Paravirtualization. My migration method from VMware to XenServer was to

  1. Export VM from VMware ESXi
  2. Import VM to VirtualBox running locally - I had to do all of this within VirtualBox for some reason, XenServer refused to load my imports directly from VMware, when the OS was fully configured for PV in VMware instead of VirtualBox
  3. Install xe-guest-utilities from apt
  4. change startup console to use hvc0 instead of tty1 (this breaks the stock TKL Management Console, but gets a console working in Xen
    1.  nano /etc/inittab
      add this line:
          co:2345:respawn:/sbin/getty 38400 hvc0
      immediate above: 1:2345:respawn:/sbin/getty 38400 tty1
  5. Uninstall vmware-tools from TKL App
  6. Export VM from VirtualBox to an OVA package
  7. Import VM to XenServer
  8. Change bootloader from HVM to PyGrub for PV
    1. xe vm-list power-state=halted (write down UUID for the VM)
    2. xe vm-param-set uuid=UUID HVM-boot-policy= (leave blank after equals sign)
    3. xe vm-param-set uuid=UUID PV-bootloader=pygrub
    4. xe vm-disk-list uuid=UUID (find VDB disk's UUID of root partition)
    5. xe vbd-param-set uuid=VBD's UUID bootable=true

On a side note, Debian Wheezy installs without issues Paravirtualized in XenServer, without any additional adjustments. 

This is a lot of hassle to get our apps running in XenServer. On the other hand, we have seen a 10-20% performance increase with our apps being paravirtualized in XenServer.

For fresh TKL Apps, I will download the .iso format, then use pretty much the same procedure, but cutting out anything related to VMware. Install in Virtualbox, install xe-guest-utils, change startup console in inittab, etc.

Is there a simpler way to do this that I am missing? The "Xen Optimized" builds give me a filesystem that appears to be optimized for an older version of Xen.

Jeremy Davis's picture

This will probably be very useful for others...

As for fresh install from ISO , I lifted this from ServerFault for our docs. It looks pretty straight forward to install directly into XenServer, but I haven't tested it...

Oops! I just realised that is for HVM; not PV... So that won't apply to you. Hmm... I just had a quick google and it seems that there is a bit of info floating about (no doubt you looked too) although much of it seems quite old.

Regardless, the docs are a wiki so please feel free to add to them as you see fit...

Also we have just released RC1 of TurnKey v14.0 (just ISOs of Core and TKLDev at this point) so if you get a chance then perhaps it would be useful to test how that goes installing?

Vl4dim1r's picture

I did download the isos for both the Core and TKLDev for 14.0 RC1, but I havn't had a chance to test installing them yet. My community and I plan to make some custom service images for an automated network management system, so we've been looking into TKL a bit more in depth.

I thought I could get off easy using a debian wheezy install profile for TKL apps, but they refuse to bootup with the correct bootloader. Maybe it'll work for the 14.0 RC1 core.

One thing I did find out while researching an easier way to install these apps. If I am not mistaken, Amazon EC2 uses PV images, and is largely based on XenServer. Is there any way we can get those EC2 images? Last time I looked into it, the only method I could find was to rent EC2 space, load the TKL image and then download an export for Amazon

I will post my results after I attempt to install the new Core later tonight.

Jeremy Davis's picture

TBH this is all AFAIK - please don't take everything I say as verbatim. I've likely got some details wrong and am more than happy to be corrected! :)

My understanding is that AWS use customised Xen for their VPS service. AFAIK newer versions of XenServer now run vanilla Xen (or at least closer to vanilla than it used to be when XenServer wasn't fully open source). TBH I have no idea whether they are close enough to work together and be friendly...

AWS last generation servers are/were mostly PV AMIs only. Some of them also support HVM AMIs though I think... All their new gen servers support HVM AMIs (although some still support PV builds).

[update] I found an AWS table that spells out which EC2 instance types are supported by which AMI types. See here.

So all our current AMIs are PV; but we will be transitioning to HVM in the future. We will probably release v14.0 AMIs as HVM only... So that won't be much help to you sorry :(

However what might be of value is our BuildTasks. They are the scripts which we use to convert the ISOs to the various other build targets, inducing the EC2 AMIs. This could be a good start to building a PV Xen/XenServer image? Also the Xen buildcode might also be of some value? The Xen build images are tarred FS (with no kernel AFAIK) that are designed for use as Xen PV VMs

Vl4dim1r's picture

Like I said in my first post, I checked out the Xen build images. I am still somewhat new to XenServer at the moment, but from what I could tell is that the pre-Open Xen would emulate a filesystem, while post-Open Xen's PV emulates an entire machine with a PV kernel. During my first migration, I couldn't manage to import the tarred FS in any way, as ovf/ova and vmdk were my only options.

It is slightly disheartening to hear that 14.0 may be HVM only, but it won't really make anything more difficult as I'm already doing HVM to PV conversions. after 10 VMs, I pretty much have the procedure memorized. it just gets confusing running multiple operations at once.

We will probably end up just customizing the ISO's we need to install the xe-guest-utilities package, a post-init script to fix inittab, and an external manager taking advantage of XAPI to handle bootloader conversion. I will likely make the custom PV images available to the public. I already have exports for fresh installs of my currently running apps, but I'd have to change some passwords/hostnames to release those.

In this day and age, we really should be using HVM. Our boards have fast clocks, many cores, and a lot of memory, but are old, power hungry,  and don't support AMD-V.

On a side note. How can I bring up the stock TKL console? Telling inittab to use hvc0 (xen console) seems to prevent the TKL gui from running.

Jeremy Davis's picture

Xen is a technology which allows the running of guest virtual machines on host OS. It has always been open source. The Xen images that we provide were designed in partnership with a VPS provider that uses Xen. They run PV guests, although in theory they could also be used with HVM guests...

XenServer (by Citrix) is a (formerly proprietary; now open sourced) hypervisor product (somewhat like VMware ESX/vSphere) which use the Xen technology and packages it in a nice, user friendly way. My understanding is that the newer (open sourced) version actually leverages a lot of upstream Xen tools.

Why not look at developing an automated script to do the conversion for you (if possible). We could then add it to our BuildTasks. So even if we don't officially release a XenServer PV image at least the code is there all in the one place!? I imagine that you could use one/some of the BuildTasks scripts (EC2 AMI script?) within TKLDev as a basis to make the desired images... Anyway, just thoughts...

Re HVM vs PV; I'm sure that I read recently somewhere that Amazon have stated that historically PV was faster but current HVM improvements have meant that performance/resource usage is now similar. Obviously it's not an option if your hardware doesn't support AMD-V (or Intel's equivalent VT-x) though.

As for TKL's default console, that's a little program called confconsole. You should be able to adjust your console so it starts confconsole on your XenServer instances...

Phillip Moore's picture

I just installed on my XenServer 6.5 the ISO file and then just installed it w/o any troubles.  I do have a weird GPU thing that I have a work around for all my VMs.

I just choose the Other template, and my GPU work around is that I have to turn on the GPU from the XenServer (which makes it so I can use the HDMI output), then I use GPU pass-through on the XenServer and on the Pool.  Then during the new VM wizard I attach the GPU from the XenServer.  After the install is finished, I detach the GPU and reboot the machine.

One thing I would do to increase performance was I would go to the console of the XenServer and use the "ethtool" command to set TX=off on the virtual ethernet port for the virtual server.  This would disable packet checksums and speed things up for me.




Add new comment