Chris Musty's picture

Hi all,

In another thread it was mentioned that a syslog server would be a cool idea. I use them alot to monitor anything from routers to servers and they can be a powerful tool to keep you up to date when you manage many sites or even if you have many devices to monitor.

I am contemplating creating one from TKL LAMP stack (if I use a database) and would like to hear from anyone interested in this feature. Please add your RFF here and I will start researching!

Features:

  • Highly configurable email alerts
  1. Delivery rules based on IP address
  2. alert level
  3. increase in volume
  4. abnormal conditions
  • and SMS alerts (would require a provider or GSM modem)
  • Customisable error levels
  • auto archiving features
  • auto report generation
  • up and downlink speeds
  • Tons of reports
  • Design web GUI from scratch
  • Heaps more but I want to hear from others
Forum: 
Liraz Siri's picture

Hi Chris, great of you to take the initiative on this. I think a TurnKey log collection appliance is a great idea, though I haven't really thought too much about this problem yet so I don't have too many specifics to add. Come to think of it, even if I had thought about that might not even matter because the ideal configuration may depend on the circumstances of your usage scenario anyhow (e.g., how many systems you are collecting logs from and what you are collecting logs for, etc.). Or not. Good designs can be generic.

Regarding the web GUI, log.io looks pretty sweet. Even if the implementation doesn't fit for the usage scenario you are thinking about, there might be a few interesting ideas in there...

Chris Musty's picture

I must admit I had only considered my usage scenario when deciding to do this, mainly bacuase I had already planned this a while ago. I am not sure how you would even define my usage scenario as I have particular needs. Saying that when I get into it I will design for what I require and if anyone else finds it useful then great. Failing that there could be several versions.

I think the greatest point with this is that there is no easy "turnkey" solution for this. The first one  I created (still running!) took over a day to get together and now it would probably take between 4-8 hours to complete. This is a screaming need for a standard appliance that can be just launched from an ISO then mabey a quick config screen and your running.

Eventually it may evolve into a multiple type appliance that you can choose during setup. Lets see what happens.

Thanks for the link Liraz, interesting...

Chris Musty

Director

Specialised Technologies

Jason Lehman's picture

We have a ton of Cisco hardware that needs monitored.

Just wondering, any word?

Thanks.

Jason

Jason

Jeremy Davis's picture

But I have just created a blueprint for it so it doesn't get forgotten.

Chris Musty's picture

I have researched a little but have not done any real dev work. As with so many other people here - I am very busy. I have 2 major projects on at the moment so when they are delivered I may delve into this. Sorry if I got anyones hopes up! 

Chris Musty

Director

Specialised Technologies

Chris Musty's picture

Found this on my web travels - http://www.balabit.com/network-security/syslog-ng/opensource-logging-system does anyone have any experience with it?

Another project I am toying with is monitoring for windows desktops eg Smart Data, Event logs, Software Installs etc any ideas or thoughts?

Chris Musty

Director

Specialised Technologies

Jeremy Davis's picture

But just as a replacement for the default logger in it's default Ubuntu configuration (apparently it has much better performance under OpenVZ). I haven't tried to take to take advantage of any of the other advanced features (nor did I even realise that it had them)!

Jeremy Davis's picture

In my random online travels I just came across this interesting piece of work called PartyLog2. Looks like it's like a log collection/monitoring server just like you were/are looking for (built on top of TKL Core v11 already!)

The software it uses is called Graylog2 which has a catchy byline of "Manage your logs in the dark and have lasers going and make it look like you're from space."!!! I like it already! :) It also has a (separate) WebUI and even a custom log format thingy called GELF (which I won't even pretend to fully comprehend, but I'm sure it's good!)

I don't know anything more than that about it (which obviously isn't much!) but IMO seems worth a look. I've just posted on the devs SF page so hopefully we'll hear from him over here soon! I'm really hoping that he'll want to work with us on this one.

Chris Musty's picture

I knew someone had already done it, I just knew it!

Now to find some time to play around with it...

Chris Musty

Director

Specialised Technologies

Jason Lehman's picture

My initial thoughts...

Install is a little different, as Partylog2 is only available as an ISO download.

I'm setting this up in VMware's vCenter, so since its an ISO; I have to give vCenter all the details about this operating system. (which I dont know all the details or requirements)

Had to take some guesses whether to tell VMware if this was to be a 32 or 64 bit server.

I guessed 64 bit.

The build of Graylog2 is slightly outdated (its running 0.9.5P2), (the new build has some major benefits) & there is no simple way to update it. I tried following an upgrade guide to get to the latest version of Graylog2 here... http://andreas-lehr.com/blog/archives/556-upgrading-graylog2-from-0-9-5p...

But I ended up trashing my Partylog2 / Graylog2 server & had to start over. Not sure what i did wrong. It may just be a little too complex for me. Better update/upgrade method needed.

So, now Im back to the original Partylog2 / Graylog2 build that Jeremy linked above.

I will setup some devices / servers to log to it & will see how it works.

I will update my findings.

Thanks.

Jason

Jason Lehman's picture

After starting over again (this time for the latest release 0.9.6) I had issues getting the graylog2-server service to run. I restarted the machine twice, I would get the message in the the Graylog2 web interface, "It seems like your Graylog2 server is not running." Odd, that I could hit the webinterface.

I manually stopped & started the graylog2-server service & it now seems fine.

I hope that a TKLPatch is created & an official TKL appliance is created for this. There is potential here. I can't really say too much else at this point; as i have to let it collect data & see what I can get out of the appliance.

Keep up the good work Jose. I will update my process.

Jason

Jason Lehman's picture

I still got the same warning message imediately after configuration. Maybe I should have given it more time, but I restarted the server. After the restart everything was fine. No more need to manually stop/start any services.

I'm still looking through documentation & searching the web for the best way to get this configured.

I am getting devices to log to it with no problem. Thats the easy part. Now the hard part, get something out of all these logs. Hopefully the community will share what they are doing, so we dont all have to reinvent the wheel.

We are interested in monitorying Active Directory servers, SQL, Web (IIS & Apache), Cisco Network devices and a few others.

Thanks for fixing

Jason

Jason Lehman's picture

Sorry for the late update.

But after allowing a few servers (5 active directory servers) to log to the Partylog server for 2 weeks, I decided it was time to try to figure out what I could get out of these logs. The 50GB root partition I gave this server was full & Partylog was no longer functioning. At the same time, our network admin (the one who requested a syslog server) came in & said he found another solution that is working well for him. I ended up deleted the Partylog server. I will keep it on my radar for future requests & revisit some time in the future to see how it has improved. Keep on working on this, there is still a lot of potential.

Jason

Jeremy Davis's picture

But is it possible for you to provide a TKLPatch though (as I detailed on your SourceForge forum). Even if you haven't got time or energy to do that, if you could share your install/config notes that would be enough for someone else to build the patch.

Then hopefully we could get this into the next official TKL appliance release.

Jeremy Davis's picture

Perhaps it's documented somewhere but I hadn't seen it previously... Have a look here for link and usernames/passwords for a public/demo Graylog2 instance. Not sure how official it is, but regardless I think it could be handy for those that want a sneak peak.

Sean McGerty's picture

Heya,

Big fan of the project thanks everyone. Yes I'm deploying / testing PartyLog2_0.9.6_r1.iso, and I'm seeing in almost all cases that the graylog2 web service starts but the graylog2 service doesn't. Have been getting in and doing it manually at the moment, but I'd like to roll one of these out soon :)

Thanks


Snickasaurus's picture

Hello my fellow nerds/geeks. I came across this thread this morning and as I read through I hoped to see a more recently dated post at the bottom but did not. Is this something that was forgotten or In my joyous times of insanity have I been stuck trying to many other flavors of TKL that I skipped over the logging one?

Hope I don't offend anyone by resurrecting this old thread!

 

What I'm trying to log:

(7) tkl vm's running on XenServer

(2) boxes running FreeNAS 9.2.1.2

(2) hardware dedicated debian  servers

(2) Asus routers running DD-WRT

(1) Linksys 48 port gigabit switch

(1) 30 year old girlfriend that thinks she's a queen*

(1) 148lb gratedane that things he's a puppy*

 

*=lulz

Jeremy Davis's picture

FWIW I have lodged a Candidate Request for Graylog2 on the TKL Tracker Dev Wiki. Although TBH I'm not sure whether it meets your specs (I don't know enough about Graylog2).

I'm fairly sure that your last 2 will require significant tweaking! :)

Snickasaurus's picture

Hopefully it will get some traction and turn into another addition to the ever growing TKL library. I think this weekend I'll download the Core and setup my own logging system just to see what I can accomplish. Currently I have a Debian vm running and will try logstash first then move around a list I've compiled from Google searches. Perhaps I can contribute something here instead of lurking in the shadows on the forum.   ;-)

Jeremy Davis's picture

It'd be great if you wanted to lead the effort on this! If you come up with something that fulfills your needs then it could be the basis for an appliance! That Graylog2 software looks like it could be a goer? But IMO it doesn't have to be that...

If you can get your head around TKLDev then perhaps you could even build the appliance - or at least the bones of it?!

Snickasaurus's picture

I'm interested, years later, to see if anyone of you that posted above ever found a solution to this and can we agree a TKL solution would still be great for logging?

Jeremy Davis's picture

AFAIK this never went anywhere unfortunately. However, I would love to see an appliance in this space.

I certainly agree that a centralised logging server would be great!

If you have any further up to date feedback and input, I'd love to hear about it. I'm not sure if/when we'll get a chance to push this ahead internally. But if you have any success, then I'd love to hear about it.

As per always, I'm more than happy to coach you (or anyone) if you want to have a go at creating a new appliance. Even if you don't create an appliance, even documenting any success you have/had would be awesome!

Snickasaurus's picture

I consider myself an above average *nix user but I learn by doing and not so much reading. Some 'man pages have actually put me to sleep while trying to interpret them. However, I do have two servers that aren't doing anything right now and would love to get something going in the way of a log server and web front end that is stable and easily updateable. I'll spin up several TurnKey Core's and go as far as I can (documenting along the way). Should I post back here with issues or start a new thread and link back to this one or just keep everything here?

Thanks for the reply.

Jeremy Davis's picture

If you get something up and running on TurnKey v15 that you're happy with, and only includes open source software, that would certainly lower the bar to getting it added as a new appliance.

Seeing as this thread is so old, I suggest that you start a new thread for your "log server" adventures. Although a post here with a link to the the new thread might be nice for any that would like to follow the progress?!

If you keep in mind, that once you have things running on Core as you'd like, then the final step is creating build code, then that might help the transition from PoC on Core, to appliance build code. Essentially, every step needs to achievable via a non-interactive script (or an overlay file).

When it comes time to create the new appliance, you'll need a TKLDev server (our dev environment). A local VM is fine (FWIW that's what I use). It's fairly well documented (there's also a "meta-doc page" with links to all the various resources). Although you probably won't want to read all that! :)

Also, some of the docs relate to the previous v14.x version and haven't been updated. For instance, the sandbox doesn't quite work as it used to. Also the deck layers also don't work the same as they used to unfortunately. I need to at least update the docs to note those changes, or ideally fix our v15.x TKLDev tools so they work like they used to... (although not sure when I'll get a chance to do that).

Also, something worth noting is that there are lots of "best practice" things and conventions that we use that aren't very well documented at all. So please feel free to keep me in the loop on what you're up to and I'm more than happy to provide advice along the way.

Also, as I know the appliance library quite well, so if you hit an issue, then chances are there is already a known way to work around it (or I'll at least have some ideas). Also there may be code snippets which you could rob from an existing appliance. So please keep in touch and I'll help out where possible.

Rob tisdell's picture

I'm in agreement that a Turnkey Syslog Server Appliance is needed. I am currently in the process of finding a good fit. I'm willing to help out in any way that I can with the development of creating a Turnkey solution. Let me know what I can do.

Jeremy Davis's picture

Be great to have others involved too Rob.

From my perspective, I don't have time ATM to be too heavily involved myself, but am more than happy to try to assist where I can. I suggest that everyone try to work publicly as possible so that we can all bounce ideas around and gain from others' experiences.

I guess for now, just sharing thoughts and ideas is as good a place to start?!

Snickasaurus's picture

Tomorrow I'm going to start going through all the documentation of setting up the dev environment. I just hooked up a Dell server I had laying around to test with. Rob, I'll get with you over the next day or two so we can get started.

And as we discussed earlier Jeremy I'll start a new thread.

Jeremy Davis's picture

I look forward to hearing how you go with it all. :)

Chris Musty's picture

Hey all,

It seems a lot happens in 9 years!

SInce then I have deployed Graylog a billion times and it is awesome. Absolutely no point reinventing the wheel here. 

https://www.graylog.org/

Chris Musty

Director

Specialised Technologies

Jeremy Davis's picture

Hey Chris, I think both of those recent posts were spammers. Thanks for dropping in anyway. Seeing as this thread is so old, I'm going to lock it for now. Take care mate.