Dan's picture

Ok... I have been using Turnkey LAMP stack in production for years now.

This has happened in the past but not often enough for me to even warrant looking into until today.

Sometimes after the applicance has been running for several days I will get reports from my users that "things seem slow"...   We are using MySQL and PHP for several applications and procedures that run on scheduled basis, etc.    Well I will bring up my LAMP Stack Applicances Virtual Console window and see that its no longer sitting at the confconsole screen, but rather at a login screen.

I usually just log in and then tell it to reboot BUT today... I am having an odd issue with one of my new scripts where it opens a port on the system and its failing with an error that the port is in use.  Even though I have double checked and that port is not in use...  I tried using a different port and its basically failing with "port is in use" no matter what port I try.

I hate rebooting the box in the middle of the day, but I am thinking if I do it will resolve this issue...

Im sorry if this seems confusing... Im just trying to determine if there is a relationship to confconsole failing and something being "wrong" with my server.

Im going to wait until lunch and reboot the box then, and try my port opening script to see if it indeed resolves the issue.

I will report back my findings to this post.

Thanks for any insight anyone may have.

Forum: 
Dan's picture

Ok... I rebooted the box... didn't work...   I am still unable to bind to ANY ports using an existing PHP script that worked a few months ago.

I dont know whats changed... but Ill start digging...  the firewall is set to allow everything I think.... this is an internal server with no external access so Im good with that.... but I dont know whats going on.

 

Jeremy Davis's picture

As far as the sluggish server goes, I'd recommend that you do some investigation of what free resources you have when it's doing that. My guess is that you have a memory leak somewhere and it's running out of RAM, but I'm only guessing...

To check memory, use the free command. I.e. Something like this:

free -h

Keep in mind that Linux reports RAM used for caching too. You can ignore that as if your system needs that RAM it can (it's used to improve performance but it can be flushed and won't harm things significantly if a process required it).

To dig in a bit deeper and see resources in use and which processes are using what, try top. I.e.:

top

Better still install htop:

apt-get update && apt-get install htop

Then launch with

htop

To check what may be running on the port you are trying to bind to, I suggest using netstat. E.g. to check port 80:

netstat -ltpn | grep ":80 "

Although it sounds like there may be something else going on perhaps? Can you share which ports you are trying? Which user is the script running under? Is the script "self serving" or is it being served by something else (e.g. by a webserver such as Apache)?

To give you the context of my questions, without some trickery only root can bind to "privileged" ports (i.e. ports below 1024) and running an network available process as root is almost always a really bad idea. If the script is being served (or proxied) by something else, then perhaps there is some configuration you are missing?

So bottom line, if you can share some more specific info about your environment, how things are configured then perhaps I can give some more specific feedback. Please feel free to post specific output of you need a hand interpreting it.

PS apologies on lack of formatting, I'm on my phone...

Dan's picture

Jeremy..... thank you so much for your informative and detailed response. Your support is amazing considering the price ;)

I will use the commands and tools you mentioned many times in the future as I work my way to becoming a better linux sysadmin.

I have found the issue and a possible solution. The issue was that I had a scheduled task running my script that opens the socket port once a minute.  For some reason while that task was enabled I was unable to get my PHP script to stop running. As soon as I disabled the task... The script started working again.

I should explain my script and what I am trying to accomplish as you may have a better solution that what Im doing.

I have a PHP application call the "Queue Processor" that was running in an infinite loop, checking for the existance of certain records in a MYSQL database, then doing stuff as soon as it found a record then removing that record when the stuff was done.   This was my "Queue" a table of records that would always be cleared out once the task completed... but it needed to run all the time.  So I created an infinite loop in the PHP script, and used the open port to determine if I already had the QueueProcessor running. 

So the Code runs by first trying to open port 7600, if it cant, it exits... if it can it ran in an infinite loop. I then had this script called from a scheduled task using WGET once a minute.

Something must have gone wrong, and in hindsight I realize now that HTTP techonology probably isnt designed to keep a PHP script running indefinently.   SO my new plan is to code the QueueProcessor to run for 5 minutes at a time and exit...   then set the scheduled task to kick it off again once a minute so that its always going to be running.

This all seems very "hacked together" to me... but I have yet to determine a better method. I would love ANY and all insight or opinion you may have on all this.

Again.. thank you SO much for your help... I love the TurnkeyPlatform and it has kept us in business for some time now.

 

Jeremy Davis's picture

Firstly thanks for your kind words! :)

So essentially, it sounds like you've been using the port binding as something of a process lock?! Assuming I'm right, whilst I agree that it's quite hacky, it's also quite inventive! :) It's not something that I would have considered.

I'm unfamiliar with anything but the most rudimentary PHP so I can't really comment on the "best" way of acheiving your ends. But I can think of a couple of pathways which would be less hacky.

option 1: triggered by a cron job, leveraging a PID file

First up, if I were creating a script to do what you're doing, I'd be inclined to generate a "PID" file (a file which stores your script's "Process ID" - every running process get's assigned a PID). Ideally you'd want to trap errors too, so that even if the script fails, it would clean up the PID file on exit (so the file definitely only exists when the process is running; or the edge case of a complete server crash).

Due to my complete lack of familiarity with PHP and the fact that I haven't explicitly done any of that sort of stuff for a while, I couldn't really offer any concrete suggestions on how to achieve that. But google might help (now you know keywords such as "pid file" & "trapping errors", etc). Armed with that, I'd be surprised if you couldn't find plenty of good info, plus keep in mind that TurnKey is based on Debian (v14.x = Debian Jessie, upcoming v15.x will be based on Stretch).

Personally, I'd be inclined to write a bash wrapper script for your PHP script and let that manage the error trapping and PID file creation/deletion etc. But OTOH that may not necessarily be the best way, it's just what I'd do. And in fairness, that's because I'm very familiar with bash! :)

Regardless, once you have a "proper" pid file plus error trapping sorted out, then a cron job is certainly a good option for this task (a cron job is essentially like a Windows "task" - triggering a task on a timed schedule). That's where the importance of a PID file & error trapping comes in. So each time the cron job is triggered, it can ensure your script isn't already running. Otherwise, you'll encounter the same issues you had before, with multiple instances of your script running.

I'd also recommend some form of logging, so if it does fail, you'll have a relatively easy way of working out what went wrong. A little extra initial development work, will likely save you a lot of headaches down the track. If you do decide to do logging, I highly recommend that you look into leveraging 'logrotate'. That can be configured to compress and/or delete old logs to avoid them growing uncontrollably. Again google will be your friend! :)

option 2: a systemd service file

Assuming that you are using TurnKey v14.x (anywhere other than an Proxmox/LXC container; earlier versions of TurnKey & v14.x LXC used an older init system called sysv-init) then another pretty incredible option IMO would be to turn this into a SystemD service (aka daemon).

You could essentially keep your looping script as is (perhaps include a sleep timer if you don't need/want it running all the time) and write a SystemD service file to trigger it. You could then enable it to auto start on boot ('systemctl enable php-script') and could easily check the state of it anytime you liked ('systemctl status php-script'), as well as manually stop ('systemctl stop php-script') and start it ('systemctl start php-script'). You could even make it auto stop under certain conditions, e.g. if MySQL is stopped). Whilst the learning curve may be a little steeper initially, you could then avoid needing to worry about the PID file and would get (albeit rudimentary) logging for free! SystemD is pretty damn awesome IMO!

A PID file is not necessary, as SystemD will keep track of your process. Note that if you still use a bash wrapper, then ensure that you launch your PHP script with the 'exec' command! Otherwise SystemD will only watch the bash script, not the PHP one too. If you can be bothered, I'd still be inclined to set up some logging (even if it's disabled by default). That will make debugging any future issues significantly easier!

getting further help

No matter which way you go, I highly recommend Server Fault (server related questions), 'nix Stack Exchange (Linux specific questions) and Stack Overflow (programming related questions) (and Stack Exchange sites in general) for any specific questions which google doesn't answer for you. As there is a bit of overlap between these sites, I suggest that you consult the help (append /help on the end of each URL) to avoid the risk of your questions being closed as off topic. I also suggested trying to keep your questions as tightly targeted as possible. I suggest writing a few separate short questions, rather than try to get everything answered in one.

Please don't take that last paragraph to be me trying to discourage you from asking here though. I'm more than happy to help as much as I can (I quite enjoy it). It's just you'll likely get a quicker (and perhaps even better) answer to your more generic questions there, than you will here. My knowledge tends to be quite broad, but not very deep (beyond TurnKey specifics). Plus you won't have to wade through my rambling posts...! :)

So please feel free to post back if you get stuck. Also it'd be awesome if you share your solution once you have something you're happy with. I have no doubt that other TurnKey users would get some value out of it!

Good luck!

Dan's picture

Just wanted to say thank you AGAIN so much Jeremy... I have not had time to attempt any of your possible solutions...  but my opening socket and keeping it open appears to be working in production now. :)

We have so many systems to create and support that sometimes the bird in the hand is really better than two in the bush.   Its a careful balance of creating stable systems, vs ones that get you 80% of the way there and then you deal with issues potentially.

I love Turnkey's platform keep churning out awesome tools for devs like me!

Thanks!

 

Add new comment