TurnKey Linux Virtual Appliance Library

syslog server appliance Request For Features (RFF)

Chris Musty's picture

Hi all,

In another thread it was mentioned that a syslog server would be a cool idea. I use them alot to monitor anything from routers to servers and they can be a powerful tool to keep you up to date when you manage many sites or even if you have many devices to monitor.

I am contemplating creating one from TKL LAMP stack (if I use a database) and would like to hear from anyone interested in this feature. Please add your RFF here and I will start researching!

Features:

  • Highly configurable email alerts
  1. Delivery rules based on IP address
  2. alert level
  3. increase in volume
  4. abnormal conditions
  • and SMS alerts (would require a provider or GSM modem)
  • Customisable error levels
  • auto archiving features
  • auto report generation
  • up and downlink speeds
  • Tons of reports
  • Design web GUI from scratch
  • Heaps more but I want to hear from others
Liraz Siri's picture

log.io a nice source for Web GUI ideas

Hi Chris, great of you to take the initiative on this. I think a TurnKey log collection appliance is a great idea, though I haven't really thought too much about this problem yet so I don't have too many specifics to add. Come to think of it, even if I had thought about that might not even matter because the ideal configuration may depend on the circumstances of your usage scenario anyhow (e.g., how many systems you are collecting logs from and what you are collecting logs for, etc.). Or not. Good designs can be generic.

Regarding the web GUI, log.io looks pretty sweet. Even if the implementation doesn't fit for the usage scenario you are thinking about, there might be a few interesting ideas in there...

Chris Musty's picture

Different Uses

I must admit I had only considered my usage scenario when deciding to do this, mainly bacuase I had already planned this a while ago. I am not sure how you would even define my usage scenario as I have particular needs. Saying that when I get into it I will design for what I require and if anyone else finds it useful then great. Failing that there could be several versions.

I think the greatest point with this is that there is no easy "turnkey" solution for this. The first one  I created (still running!) took over a day to get together and now it would probably take between 4-8 hours to complete. This is a screaming need for a standard appliance that can be just launched from an ISO then mabey a quick config screen and your running.

Eventually it may evolve into a multiple type appliance that you can choose during setup. Lets see what happens.

Thanks for the link Liraz, interesting...

Chris Musty

Director

Specialised Technologies

any luck with this project?

I was wondering if you ever setup a syslog turnkey appliance?

 

thanks

 

Selim

Jason Lehman's picture

I to am interested in this...

We have a ton of Cisco hardware that needs monitored.

Just wondering, any word?

Thanks.

Jason

Jason

Jeremy's picture

Not sure whether there has been any progress or not

But I have just created a blueprint for it so it doesn't get forgotten.

Chris Musty's picture

Same old story

I have researched a little but have not done any real dev work. As with so many other people here - I am very busy. I have 2 major projects on at the moment so when they are delivered I may delve into this. Sorry if I got anyones hopes up! 

Chris Musty

Director

Specialised Technologies

Chris Musty's picture

Candidate for Logging (Syslog) Server

Found this on my web travels - http://www.balabit.com/network-security/syslog-ng/opensource-logging-system does anyone have any experience with it?

Another project I am toying with is monitoring for windows desktops eg Smart Data, Event logs, Software Installs etc any ideas or thoughts?

Chris Musty

Director

Specialised Technologies

Jeremy's picture

I've used syslog-ng

But just as a replacement for the default logger in it's default Ubuntu configuration (apparently it has much better performance under OpenVZ). I haven't tried to take to take advantage of any of the other advanced features (nor did I even realise that it had them)!

Jeremy's picture

This looks worth a look!

In my random online travels I just came across this interesting piece of work called PartyLog2. Looks like it's like a log collection/monitoring server just like you were/are looking for (built on top of TKL Core v11 already!)

The software it uses is called Graylog2 which has a catchy byline of "Manage your logs in the dark and have lasers going and make it look like you're from space."!!! I like it already! :) It also has a (separate) WebUI and even a custom log format thingy called GELF (which I won't even pretend to fully comprehend, but I'm sure it's good!)

I don't know anything more than that about it (which obviously isn't much!) but IMO seems worth a look. I've just posted on the devs SF page so hopefully we'll hear from him over here soon! I'm really hoping that he'll want to work with us on this one.

Chris Musty's picture

Sweeeeeeeeet!

I knew someone had already done it, I just knew it!

Now to find some time to play around with it...

Chris Musty

Director

Specialised Technologies

Jason Lehman's picture

I am trying Partylog2 out now, thanks for the find

My initial thoughts...

Install is a little different, as Partylog2 is only available as an ISO download.

I'm setting this up in VMware's vCenter, so since its an ISO; I have to give vCenter all the details about this operating system. (which I dont know all the details or requirements)

Had to take some guesses whether to tell VMware if this was to be a 32 or 64 bit server.

I guessed 64 bit.

The build of Graylog2 is slightly outdated (its running 0.9.5P2), (the new build has some major benefits) & there is no simple way to update it. I tried following an upgrade guide to get to the latest version of Graylog2 here... http://andreas-lehr.com/blog/archives/556-upgrading-graylog2-from-0-9-5p...

But I ended up trashing my Partylog2 / Graylog2 server & had to start over. Not sure what i did wrong. It may just be a little too complex for me. Better update/upgrade method needed.

So, now Im back to the original Partylog2 / Graylog2 build that Jeremy linked above.

I will setup some devices / servers to log to it & will see how it works.

I will update my findings.

Thanks.

Jason

partylog2

Hello everyone.

Glad you like it.

Im updating Partylog to Graylog2_0.9.6 and tk11.3

I will announce once its ready :)

partylog2 updated to 0.9.6

Greetings everyone,

Partylog2 has been updated with the latest and more enterprise edition
of Graylog2.

Features of Partylog2:

   Graylog2 Server (v0.9.6)
   Graylog2 Web Interface (v0.9.6)
   mongodb v2.0.2
   elasticsearch v0.18.7
   ruby v1.9.3

You may download and test this release from:
http://sourceforge.net/projects/partylog2/files/0.9.6/PartyLog2_0.9.6.iso/download


And you can find more information in:
https://sourceforge.net/p/partylog2/home/Home/

 

 

 

Jason Lehman's picture

Tried the new version of partylog2, currently testing

After starting over again (this time for the latest release 0.9.6) I had issues getting the graylog2-server service to run. I restarted the machine twice, I would get the message in the the Graylog2 web interface, "It seems like your Graylog2 server is not running." Odd, that I could hit the webinterface.

I manually stopped & started the graylog2-server service & it now seems fine.

I hope that a TKLPatch is created & an official TKL appliance is created for this. There is potential here. I can't really say too much else at this point; as i have to let it collect data & see what I can get out of the appliance.

Keep up the good work Jose. I will update my process.

Jason

Thanks for the

Thanks for the info.

Updated the release. I will apreciate if you look into it one more time if you have the time.

http://sourceforge.net/projects/partylog2/files/

Jason Lehman's picture

Ok, new build its better.

I still got the same warning message imediately after configuration. Maybe I should have given it more time, but I restarted the server. After the restart everything was fine. No more need to manually stop/start any services.

I'm still looking through documentation & searching the web for the best way to get this configured.

I am getting devices to log to it with no problem. Thats the easy part. Now the hard part, get something out of all these logs. Hopefully the community will share what they are doing, so we dont all have to reinvent the wheel.

We are interested in monitorying Active Directory servers, SQL, Web (IIS & Apache), Cisco Network devices and a few others.

Thanks for fixing

Jason

Jason Lehman's picture

Update

Sorry for the late update.

But after allowing a few servers (5 active directory servers) to log to the Partylog server for 2 weeks, I decided it was time to try to figure out what I could get out of these logs. The 50GB root partition I gave this server was full & Partylog was no longer functioning. At the same time, our network admin (the one who requested a syslog server) came in & said he found another solution that is working well for him. I ended up deleted the Partylog server. I will keep it on my radar for future requests & revisit some time in the future to see how it has improved. Keep on working on this, there is still a lot of potential.

Jason

Same problem

Indeed very odd.. The webservice is running, but no messages coming in. I don't have this problem on another server, which contains exactly the same version installation.

After a complete reinstall of the iso, i get the same error.

 

A service stop / start doesn't help here.

Jeremy's picture

Thanks for providing an updated ISO Jose

But is it possible for you to provide a TKLPatch though (as I detailed on your SourceForge forum). Even if you haven't got time or energy to do that, if you could share your install/config notes that would be enough for someone else to build the patch.

Then hopefully we could get this into the next official TKL appliance release.

Jeremy, I will look into

Jeremy, I will look into it.

Thanks for your support.

Jeremy's picture

Stumbled across an online demo

Perhaps it's documented somewhere but I hadn't seen it previously... Have a look here for link and usernames/passwords for a public/demo Graylog2 instance. Not sure how official it is, but regardless I think it could be handy for those that want a sneak peak.

Testing- Partylog2 ISO on Virtualbox

I've been testing the Partylog2 for a few days on my test server (OS X Host, Virtualbox) and I can confirm some of the issues identified here.  Occasionally on boot the graylog service does not start even though you can log in, you need to restart the service or the VM (sometimes multiple times) before it works.  I've also experienced the mongodb lock file issue, where you get 502 error.  I attempted a mongod --repair of the database without success, I reinstalled the VM.

Overall I am very excited about this project, I would like to have a small, easy to deploy syslog server in a nice VM package such as turnkey linux, I really hope this test project goes live and becomes part of your library of great VMs.  Keep up the good work.

Side note- I am not overly experienced with syslogd within your other Turnkey VMs.  Can you steer me towards a cheat sheet to set your up Turnkey Core or Turnkey Torrent server to report their syslog to Partylog2? Is there a Webmin option or do I need to commandline edit some config files? Thanks for your help.

Rsyslog.conf

Just a follow-up question, I have successfully configured 2x FreeNas devices, and 1x PFSense box to report to Graylog2/Partylog2. However I have 2x Turnkey VMs running 1 Core, 1 Torrent server and I cannot figure out how to configure the rsyslog.conf file to report to the syslog server. Some details below:

1. IPs of all servers are known

2. Rsyslog.conf- I have attempted to add a line

*.* @@192.168.1.xx:514 (send the syslogs to that IP, on that port using UDP)

3. Restarted rsyslog server, and rebooted VM

4. However the syslogs are not reporting

Any assistance is appreciated, or plse steer me towards another forum post.

Hi Kristoffer, be so kind to

Hi Kristoffer, be so kind to explain briefly how you have done to lead the syslog from pfSense to GaryLog? I try to implement it with Mikrotik, I point this to the syslog GaryLog did not get any information that can be displayed on the webgui of PartyLog2.

Of course I appreciate your response.

Pfsense

I set up pfsense by adding my graylog server as a remote syslog server using the webgui under system logs settings

Sean McGerty's picture

Thanks for this work, can confirm the server not start bug

Heya,

Big fan of the project thanks everyone. Yes I'm deploying / testing PartyLog2_0.9.6_r1.iso, and I'm seeing in almost all cases that the graylog2 web service starts but the graylog2 service doesn't. Have been getting in and doing it manually at the moment, but I'd like to roll one of these out soon :)

Thanks


Partylog2 works great!

My first two builds of Graylog were working ok for testing, collecting SIP and ISDN logs from media gateways until a large volume of traffic was moved to those devices, one test box only had 1Gb of ram, the other had 2Gb and the mongoid.yml index settings batch size was at default 4000/1 sec, we were only trying to match the output against what we have, if it didn't match there was no need to load test, both test boxes crashed with Graylog ooops something went wrong error.
Failed to save the recipe and was unable to get Graylog working exactly as it was the first time, the media gateways stuff 1220 bytes/packet into UDP 514 by default and can send 5,000 messages/second.

Decided to load Partylog2, same two test boxes were up in 10 minutes collecting logs, test boxes are only Dell 320's w/E4400, added 4Gb ram (only room for 2 sticks, won't mix ram sizes, whether it uses the top 1Gb or not).
In the past 48 hours the primary box has collected 1.5 million logs, after we dropped the level to 1, flow only messages, still not seeing certain messages and looking into the possibility of Graylog dropping messages with such a large packet size. RFC 3164 states packet size must be 1024 bytes or less, unless it's been revised or the equipment manufacturer is making a false claim that its RFC 3164 compliant I don't know yet.

At any rate, Partylog2 worked flawlessly right out of the box under intense conditions in underpowered test devices that should be 8 cpu cores and 8Gb of ram, the only alternative that works for so much data is Splunk, which is awesome but comes with a pricetag I'm not sure will get approved for funding.

We are going ahead with testing for 7 days of heavy traffic, then setting the logging level of the media gateways back to 5, a lot more data. When its all done, we'll put a final build for Graylog on ESXi hosts for which getting server space is going to be a premium since this solution can't reside with any other servers under the same load.
My hats off and many many thanks a lot to TurnKey and Partylog2!




 

no log recieve in webgui

Those who has it working. Can you tell me if there is some trick to this! I have tried many times with no luck.

After I install using the latest ISO and can log into the webgui and see the messages for localhost. I set of of our other systems to send the log over to partylog2. However I'm not seeing anything show up in the webgui!

I have tried installing IPTraf and it is showing logs hitting the partylog2 vm yet they do not show up???

What I'm I missing???

Partylog2 0.9.6_r2 released

CHANGELOG: Changes from r1 to r2:

  • Service startup fixed.
  • Graylog2 Server and Graylog2 Web Interface updated to version 0.9.6p1-RC2
  • Change upstart scripts to sysvinit (due to Turnkey Core 12 being based on Debian instead of Ubuntu)

I'd be happy if you guys can test this version and report any problems.

Thanks for your feedback, its always appreciated.

I accidentaly a link:

new verison working good

Thank you for the new version, it's looking good so far. I can see the logs coming in unlike the previous version.

Bumpity bump bump!

Hello my fellow nerds/geeks. I came across this thread this morning and as I read through I hoped to see a more recently dated post at the bottom but did not. Is this something that was forgotten or In my joyous times of insanity have I been stuck trying to many other flavors of TKL that I skipped over the logging one?

Hope I don't offend anyone by resurrecting this old thread!

 

What I'm trying to log:

(7) tkl vm's running on XenServer

(2) boxes running FreeNAS 9.2.1.2

(2) hardware dedicated debian  servers

(2) Asus routers running DD-WRT

(1) Linksys 48 port gigabit switch

(1) 30 year old girlfriend that thinks she's a queen*

(1) 148lb gratedane that things he's a puppy*

 

*=lulz

Jeremy's picture

No worries on the bump

FWIW I have lodged a Candidate Request for Graylog2 on the TKL Tracker Dev Wiki. Although TBH I'm not sure whether it meets your specs (I don't know enough about Graylog2).

I'm fairly sure that your last 2 will require significant tweaking! :)

Thanks Jeremy

Hopefully it will get some traction and turn into another addition to the ever growing TKL library. I think this weekend I'll download the Core and setup my own logging system just to see what I can accomplish. Currently I have a Debian vm running and will try logstash first then move around a list I've compiled from Google searches. Perhaps I can contribute something here instead of lurking in the shadows on the forum.   ;-)

Jeremy's picture

Please document your experience!

It'd be great if you wanted to lead the effort on this! If you come up with something that fulfills your needs then it could be the basis for an appliance! That Graylog2 software looks like it could be a goer? But IMO it doesn't have to be that...

If you can get your head around TKLDev then perhaps you could even build the appliance - or at least the bones of it?!

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)