Jeff Watkins's picture

Hello I have a Rackable Systems Server ... Intel s5000psl dual Xeons with a LSI 8888elp Megaraid card... Sas/sata plus two channel network
there are 9 bays total which houses 18 terabytes ....

I have currently installed Proxmox 3.2v to the Bare metal hardware...... the installation resides on sata 0 which is also 2 terabytes..
the remaining 16 terabytes reside on the megaraid hardware... currently set as a raid 5 configuration.......

My question is how do i get Proxmox to see the megaraid and be able to load Vm' and Ct's to it... the only storage that I show is the sata 0 drive..
when booting the proxmox software or debian linux kernal recognizes the megaraid ... but when the system is up . not sure how i mount this extra storage in side of proxmox...

I am new to proxmox and to linux in general ... but hey I have gotten this far I am sure if pointed the right direction .. I can complete the mission 
of using this awesome Proxmox Hypervisor setup ... 

If some one has any Ideals 
Please dont hesitate...

and i am not scared of the command line or terminal.. Just not real fluent yet ...
I have plenty of books and i do read them ...
and I have looked on google and just havnt found the right solution yet..

Please advice...

Jeff 

Forum: 
Jeremy Davis's picture

IIRC Proxmox will install itself to the first drive it finds (and will use all of it...). Any other drives (i.e. the RAID array in your instance) is probably working but just not integrated with Proxmox.

First thing to double check is that the OS sees your array.

fdisk -l

Should give you a heap of stuff but look for physical devices, you can safely ignore the ones that mention /dev/mapper entries. Assuming that your fist disk is /dev/sda and your array is ready to go then AFAIK you should also see an entry relating to sdb (which should be your RAID array).

I can't give you too detailed response ATM but the bottom line is that you have a couple of options. You can add the RAID to the existing LV (Logical Volume - read about LVM generally or Linux implementation) or (preferably IMO) create a new LV for the RAID array. If you google about setting up LVM then you should be able to find plenty of instructions/tutorials. FWIW Proxmox 3.x is based on Debian Wheezy (same as TurnKey) so anything that applies to Debian Wheezy will in essence generally apply to Proxmox (same as TurnKey - with some exceptions of course...). This step will need to be done via the commandline.

So assuming that you go the separate LVM route (as I would suggest) you then use the Proxmox WebUI add your new LV as "storage". Then when you launch a new VM via the UI you can choose your array as the storage location....

Good luck! :)

Jeff Watkins's picture

Sounds like fun..

So I should find out how to add new lvm from command line based on how one would do it using debian wheezy.... and then once this is occamplished . probably a reastart and the the web ui of proxmox should see this as a new volume to manage as storage ....ok ... thanks jeremy your leading me down a path of jump in way over your head and see what happens...lol I can handle that...

Jeff

Jeremy Davis's picture

But IMO the rewards are worth it...! :)

FWIW I started just like you ~6-7 years ago. And I didn't have me here helping! :)

You'll be fine... Just make sure you take your time and read through stuff. I also suggest taking notes, so in a worst case scenario and you need to bin it all and start from scratch it'll be much easier.

Also you'll have developed a great resource that you can publish in our docs if you want. Chris Musty made a start a while ago, but life got in his way getting it done... Perhaps you could add to that?

Jeff Watkins's picture

I would love too

Actually I will take pictures and even add those to the project .. actually I already have some from when I put the system together...

I am a robotics engineer .. I own a company called "Robopro" I put the brains in robotic machinery.
Plc ladder , Controlled text , function block , Vb  , Karel...........

So I figured I would just build my own server .. I have played with linux throughout the years.. But truly I am a beginner....

When I bought the server the company I got it from , they were going to upgrade to newer stuff..
I seen this serial number of mine as the brains of an entire racked wall that was in a bank of 64 identical units all cable to the one I have..... 

And my buddy said you want to try it... And I told him yeah why not ...They removed thier drives...

And I brought her home.....So I know if i do it right ...I can get more lol????????

See in Centos 6.5 ... I was able to see the vg0 group of the raid 5 configuration and use it imediately

I can see that when proxmox starts up .. it shows megasas -int done or ok...

So not sure what command line command to give it.. I have read about cli managers but not sure if this is truly what I am supposed to do .....

 

But yes 

Once I get this past this stage .. you point me to where I put the build info ..and I will take time to write it out ..and also give cutos to you and the site in general ...because you are actually answering questions that are Concerning ProxMox.. they havnt even responded...which says alot for this comunity... 

Thanks for everything

are you in australia?

I actual use Krypton Suite 4.0 for Vb.net apps..they are in australia

well any hows thanks 

Still havnt found the answer..

Jeff

Jeremy Davis's picture

Hmmm... I'm assuming that you got a good price!? I wonder what the freight might be to Australia!? (Yep I'm in Australia!) :)

Jeff Watkins's picture

I have about 550 in it.. Came with 8g memory ..in the rackable systems case . with both prcessors .
of course I walked out with it... .... Go to unix surplus.... They have thousands of the same systems
I am happy thus far... I am moving right along... as i stated the machine ran centos for 3 months just playing aroung.....
 

Jeff

Jeremy Davis's picture

Compared to prices in Australia they are so cheap! However I'll need to double check on freight as that might be a killer...

Jeff Watkins's picture

If you get one .. we can try a long distance network sync..

That would be awesome to just pull it off..lol

Jeff

Jeremy Davis's picture

Yeah that'd be cool! :)

Although my upload speed is pretty crap... I was in the pipeline to get FTTP (Fibre To The Premises) within the next year, but the conservatives (ironically called the Liberal party - only in Australia...) won the last election and the change of Federal govt bought a change in policy. Now I'll only get FTTN (FTT Node) which will be only marginally better than my current ADSL2+... :(

Bloody conservative Luddites! They don't seem to appreciate how important upload speed is (and even moreso in the future), nor how their policy will cost much more in the long run (because we'll need to upgrade to FTTP in the future anyway)... Sorry to rant...

Jeff Watkins's picture

Ok the procedure I had to use to format the raid 5 on the LSI Card ..

  1. You can find it with fdisk -l
  2. But you must use GNU parted if over 2tb,which is the case 

So I have went through this procedure...from the command line using putty..
I have found this easier form than using vnc . because vnc doesnt allow paste..
Because I am new to command line terminals .
I have chosen to use putty to allow this...this was the out come after using "Parted"

Before the procedure was finished ..Parted claimed that the drive were not aligned properly?
so I used check align and told it to align "optimal" This was the result

Model: LSI MegaRAID 8888ELP (scsi)
Disk /dev/sda: 3998GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

I have elected to only use 3 *2 terabytes =6tb to start so not to be so time consumming...

There are a few things to investigate as of yet...

1.Building the file structure correctly.. ext2,ext4,???????? presently ext2...
2.Also i could only format 3998gb of the 6tb.. I assume this is because of the raid 5 config?
which make sense ......

So the next step is to find out how to get this into proxmox as directory or what?

And then once I have found all steps involved in this operation ..
I will do it all over from scratch step by step to prove this completel for this situation...

any Ideas are greatly appreciated...
but so far all is going ok .. lots of reading though?


 

 

 

Jeremy Davis's picture

I'm sure that I have added unpartitioned whole disks as a PV (LVM - Physical Volume) before. FWIW this is quite a good tutorial IMO (sorry I probably should have given you that link earlier...). It's a little dated but should still be totally relevant. Like I suggested, I'm sure that you can skip the initial partitioning step (towards the bottom of page one) and instead of allocating /dev/sdb1 as your PV; just allocate /dev/sdb (assuming your array is /dev/sdb). I did that with the Proxmox (built as v2.3 since updated to v3.0) server at my old work and it has been running fine for ages...

But to get back to the order of your email...:

Thanks for the heads up on parted. I wasn't aware of that limitation to fdisk.

+1 PuTTY! :) I mostly use Linux now so it is somewhat irrelevant (you can use SSH direct from terminal) but from Windows it's the go!

TBH I'm not sure about Parted's complaints... And I suspect that you are right. Using 6GB of raw drive space within a RAID5 would be about right I would think.

Also FWIW unless you anticipate the need for use of all 18TB (and ~9TB would be adequate) personally I'd consider going for RAID10. It will give far better performance and will give similar (or better) redundancy. From what I've read, RAID5 is generally considered to be a poor option these days, due to slow write speeds and the 'data hole' phenomenon. You could just do RAID1 but if you're mirroring then IMO you may as well also stripe and reap the performance benefit...

As for filesystem, I would recommend ext4 although ext3 is ok.

One other point to consider too is that when you set up your LV, ensure that you leave some space (10%+) for snapshots - that is if you intend to use Proxmox's snapshot function. Also, even if you do use snapshots within Proxmox I would also still recommend backups; at least for important data you wouldn't want to lose. If your snapshot backups get corrupted then you won't know until it's too late (i.e. when you try to restore...)

Jeff Watkins's picture

Yes I have learn alot just trying things to find out...Ofcourse this method wouldnt want to be followed if there was actual data that needed to be preserved...

  1. Important note :: newbies trying this such as myself..Need to get "Magic Parted"
    this is an awesome tool with a GUI that helps you understand what is going on...
  2. But first do it using the command line.. because after all you want to learn linux from the ground up.
  3. then you can verify your partition by using Magic Parted. Basically it uses GNU parted but with GUI Interface......
  4. After I verified my partion with ext2 file system . it was actually extremely close to the whole virtual disk array..So GNU did a good job.. and I am sure it would have done a perfect job if I would have understood things better....
  5. After getting Magic Parted , I re did the partition electing to go to an ext4 ...
  6. After conversing with you and a night of reading and understinding what these file structures meant...It stands to reason I want no limitations upon the raid 5 config I will be staying with..

I have taken snapshots of the build as I progress by camera.. I will eventually make all the procedures I fumbled my way through . to the ones that brought me success... at each step . I think we should have a discussion concerning all possible methods to some degree with out getting of base...

Because I tried to load a gui setup inside of PROXMOX ..I will be reloading it so to have nothing other than what comes with the setup Package.. this way if updates come .. I wont be at risk for the things I tried to modify.....

So everything i do from this point will be with standard ProxMOx 3.2 and any external tools such as putty and Magic Parted....They will all be standalones..

Jeff

Jeremy Davis's picture

Except for the space it takes up. IIRC so long as you don't also install a dm (display manager e.g. gdm for use with Gnome) then the GUI won't start on boot, only when you login (at the commandline) and manually start it. Also as long as you install only from the Debian repos (using apt/aptitiude) then you shouldn't have too many issues with it. When the time comes and Jessie (the next version of Debian - after Wheezy) is released and you'll need to upgrade your Proxmox base, the Proxmox upgrade script will uninstall all additional packages (beyond default Proxmox) automatically (IIRC it asks first but default and recommended answer is 'yes').

Sounds like you are going well! :)

Jeremy Davis's picture

I just read the post you put on the Docs page that Chris started...

Just to clarify: Proxmox won't see the VG, it will only see LVs. So IIRC you'll need to do it in an order such as this:

  • Create PV
  • Create VG
  • Create LV (remember to leave some space for snapshots - 10%+)
  • Partition LV? (Don't recall if you need to do this or not...?)
  • Format LV/partition (ext4 IMO, otherwise ext3)
  • Add LV into Proxmox as storage (via WebUI)
  • Start loading VMs! :)
Jeff Watkins's picture

Ok I have taken care of the physical volumes within the raid configuration at boot this is hardware raid.

the Volume group is VG0

Create lv leave some space?????

The entire raid array is ext4 ...Partition LV and Format ..Completed

How do I add LV into ProxMOx storage. through the web ui..

What are those steps or where can I find that info..
also do I have to mount thisLV some how. or does ect/fstab/ no its there ??????

Jeff

Jeremy Davis's picture

Hang on let me check...

Ok looks like you do need to add it to fstab. Here's my current Proxmox fstab:

cat /etc/fstab
#      
/dev/pve/root / ext3 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext3 defaults 0 1
UUID=8a31839f-b638-48e1-a4f5-6e35e4d60c6a /boot ext3 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
/dev/pve2/data2 /media/pve2 ext4 defaults 0 1

Note that the the pve VG (with root, data and swap LVs) are the original ones that Proxmox (v2.x) auto setup for me on install (to a single drive). It's still ext3 as that's what Proxmox did by default back then and I've never bothered to change it. The pve2 PV (with data2 LV on it) is one that I added (as you are now...).

Jeff Watkins's picture

Ok 

There is always 1meg left after partioning with ext4 format..
Is this correct?it show unallocated ..

If this is correct .. Then we should move to mount this in PROXMOX..

Then once we achieve this.. I will repeat the exact step to acomplish this the easiest way possible..

  1. Step one will be to start completely over.. Download ProxMox and burn to iso
  2. Use the iso too install Proxmox 
  3. access web Ui at given Ip address.
  4. Go back and configure drive array.. or do this before we start step one...
  5. then add drive array to  proxmox...
  6. then load everything up as a test lab...
  7. Then go through each one of the turnkey products and learn whats what????
  8. setup lamp and use say joomla to install website
  9. setup the domain controller.

see jeremy why I bought this server was to learn it all..not just say i have a server and it runs.
I want to look at whats out there and see how it is used and what benifits it may have to me personally and business wise... as a small business owner I definately support open source..

Plus I want to give back to the community as well....
I can definately attest to the fact that this can be learned by anyone with patience and the willingness to learn it....google and you tube can make you very smart if you go in search of knowledge...LOL

we are almost there . how do they yehaa in australia..........Mate...

Jeff

 

Jeremy Davis's picture

So I just hope that I haven't lead you astray anywhere along the line...! I think that we're going ok though! :)

Firstly, not sure if you noticed but I updated my last post... Plus AFAIK it is normal for 1MB to be left over...

Once you have your LVM setup you should be able to relatively easily add it to Proxmox. TBH I had to google to find some info. The Proxmox wiki. Once you have an LV setup then you can add it to Proxmox via the WebUI. In the left hand pane, click on the 'Datacenter' folder (above where your server should be listed). Then there should be a 'Storage' tab along the top of the right pane. Click 'Add' >> LVM and follow the prompts...

Jeff Watkins's picture

Ok as I mentioned before there is always 1 meg unallocated
 

   Partion        File System      Mount Point        Label        Size         Used            Flags
 /dev/sdb1/        ext4             /media/sdb1        raid        3.64Tb    58.62Gib         ----
 unallocated       -----              unallocated                         1mb        -- ---  ---         ---- 

ok this is the info given inside of Magic Parted after partioning and adding filesystem ext4
Not sure why there is 58.62Gib showing used????? could this be by the raid controller itself??
Not sure ? But it definately is trying to format whats available to be formated....

Also I believe the reason it is dev/sdb1/ vs dev/sdb/ is because this is the first partition possibly in the raid array///// I believe if i added say 3 more drives and added them to the same volume group.

but didnt extend the partion and include them.. they would have a seperate partion. and would become
dev/sdb2/ .. Correct me if I am wrong as I believe the number refers to the position in the physical volume of the partion in question??????

I have tried this twice by wiping it completely out 2 times same result..
I believe this to be a working setup...

I will try and research how to add it to ProxMox....

Booting up ProxMox Now.;...

Jeff

 

Jeremy Davis's picture

The ~58GB may be something to do with the RAID but I would have imagined that the RAID data parity (required/used by RAID5) would not be visible to the OS. I just had a quick look at a RAID calculator and according to that you should end up with ~3.7TB - which matches what you have... So TBH I have no idea about the ~58GB.

Again TBH I haven't played with RAID first hand much so although I know a bit about it in theory, my practical experience is virtually non-existent. So I'm not sure about your question regarding adding drives to a RAID array. I would think that you are probably right. If you were using the complete array then you could use just /dev/sdb...

Your understanding of the partitions is correct. But when you start mixing it in with RAID I'm not sure how it works. In relation to physical drives /dev/sdX is the whole drive, then the partitions are /dev/sdX1, /dev/sdX2 etc. Where I become unsure is when you introduce RAID. If you extend a RAID array you are essentially making /dev/sdX bigger. So my guess is that partitioning as you have is probably the best way to go, then when you add to the array it just becomes additional unpartitioned space. You can then make a new partition, make that a PV, add it to the existing VG and then extend your existing LV into the new space...

Jeff Watkins's picture

Ok I failed at adding the array... Not sure why?

So I am going to create the logical volume with out any formatting..

and then install proxmox on the drive that is present on sata 0... I just tried to install to the megaraid

and for what ever reason it failed ...it said tat the disk was readonly...so I am going to change this in the web bios of the Megaraid and start all over.... and I am going to wipe out the sata 0 and use it just to make a backup of promox its 2tb.... I think starting over will make things more clearer..lol

Jeff

Jeff Watkins's picture

Ok 

I believe the mission of the first tutorial is complete.

We need to assemble the data in a chronological order.
how do you recomend proceding and where I have pages of good notes
and pictures revealing the project and how it became a success...

Jeff

 

Jeremy Davis's picture

And should be editable by anyone (at least logged in users).

IMO you have 2 options:

  1. Build on the start that Chris made
  2. Start your own fresh page, probably under the Tutorial/How-to section

TBH I would lean towards 1 (as Chris has already started and with your input then perhaps when he gets time he can build on what you've done). Either way, you'll want to click the 'add child page' link towards the bottom.

Don't get too worried about doing the right or wrong thing. As I said it's a wiki so you do what you think works/looks best and if others disagree then we can edit it! :)

Add new comment