Udo's picture

I've quickly set up an Observium VM using the v16 ova and in parallel to it I've deployed a Core VM using also the v16 ova.

Platform is ESXi 6.5.

Core works nicely, but with Observium I have a no-go situation, still problems finding a way to boot.

I've attached a screenshot-comparison of both VMs, deployed in the same ESXi server, in the same Datastore.

Differences I've noticed:

a) Observium has no vmtools (of course)
b) Observium is running CPU at max
c) Observium uses almost no memory
d) Core uses almost no CPU
e) Core uses some memory
f) Observium uses about 1 GB more storage
g) In the settings I've only seen some differences related to vmtools

But overall I couldn't detect any main difference.

 

Weird...

Forum: 
Udo's picture

Again I've uploaded files but they don't show up.

Are they anywhere?

Udo's picture

Link to png files:

https://www.dropbox.com/sh/bo7m49he74ysnwv/AACHV_5i0Jk55_McQoKUPzrWa?dl=0

In this folder you'll find the screen captures of both systems, side-by-side

Jeremy Davis's picture

Thanks for your persistence! The only OVA that I have specifically/explicitly tested was the Core one and that worked fine (and as detailed more below; if Core works, and each ISO works, in theory all other OVAs should work fine). However, I have just tested the Observium OVA and I can reproduce the issue you've reported! So - as you have reported - there is something wrong with the Observium OVA!

Apologies that I assumed it was something wrong on your end... Seeing that the Core OVA worked, I assumed that it was something specifically wrong on your end (perhaps your virtual environment?). Anyway, that's clearly not the case, so I'll rebuilt it ASAP.

FWIW, all our appliances share the Core base. We use a common script to generate the ISOs from the rootfs (so in theory, all appliances should share the same working base install). I do my initial development on KVM and once I have a Core ISO that I'm happy with, I do a manual test of the Core ISO on VMware Player and VirtualBox.

I also do a manual basic test of every appliance ISO myself (install to a KVM VM). So each appliance ISO should always boot and the base app should generally work (at least the basics). I am human, so some issues may get past me, but they all should at least boot and the base applications should basically work.

We then use another common script to generate the OVAs from the ISOs (in part; that uses VMware's OVFTool to generate the final OVA). Initially I test the Core OVA in VMware Player and VirtualBox. And after that we just produce the rest, without individually testing each OVA. The theory goes that if Core OVA works and the individual appliance ISO works, then all OVA appliances should also work. That has served us well for many years, until now! So I'll need to be a little more careful going forward...

Thanks again...

Jeremy Davis's picture

I wonder what has gone wrong with the OVA build?! It appears that it wasn't just a "one off" as I've just rebuilt the Observium OVA and it's still suffering the same issue. :(

This time I tested it in VirtualBox too and same issue there, so it's not at all specific to VMware. I'm investigating now... I should also test some other OVAs. My suspicion is that it doesn't just affect Observium, but possibly others of the same batch...

Udo's picture

Since v14, I've probably built and destroyed some 100 TKL VMs.

Testing, learning, creating and also recommending to partners and customers.

So I'm happy to help, don't worry!

I've started using v16 Core to use in a couple of private mail servers and it worked well.

Then I went for LAMP and Observium. LAMP v16 OVA is working ok.

Just Observium was not ok.

I wanted to start using SNMP in my environment.

I've successfully deployed the ISO and applied vmtools, but I do prefer to use the OVA.

 

Jeremy Davis's picture

I've worked out the issue and it's really dumb...

FWIW there is a weird bug in Buster (Grub? LVM? udev? Something else perhaps?) that means if we generate the grub config on the fly, there is no way to unmount the filesystem afterwards - except via a reboot. As rebooting during the build is impractical for us, for v16.x I swapped the dynamic grub.cfg generation for a static grub.cfg overlay file that has the relevant parts (e.g. LVM UUID) updated dynamically. So far so good...

But the gross oversight I've made is that the kernel and initramfs that grub loads is hard coded in the grub.cfg and isn't updated. That wasn't an issue in the first few batches of v16.x appliances. But more recently, Debian has delivered a kernel update, thus breaking the hardcoded kernel/initramfs files...

The fix will be fairly straightforward. I just need to dynamically update the kernel string too. I'll fix it and rebuild the relevant OVAs ASAP. Hopefully they should be ready by early next week at the latest.

Once I'm done (today or tomorrow), I'll rebuild all the OVAs that need rebuilding. I'm happy to upload an Observium one for you if you want? It won't be on the mirror (because I don't have the key to be able to sign the hash files) but it will be the same file as will be published next week.

Udo's picture

It's ok, I can wait, so no hurry.

But I'm glad the issue was found and is being corrected!

Thanks

BR

Jeremy Davis's picture

I have double checked and the master mirrors appear to have synced the updated OVA builds. So you should be good to go. Please note that I haven't actually checked every single image that we updated (28 in total) but I did check a random selection, plus I explictly checked the Observium one.

Thanks again for reporting the issue and your patience during the delay getting the updated images uploaded.

Please do not hesitate to get in touch if you have any further concerns, issues or questions.

Add new comment