Ronan0's picture

Serializing MySQL database to /TKLBAM/myfs
------------------------------------------

Traceback (most recent call last):
  File "/usr/bin/tklbam-backup", line 510, in <module>
    main()
  File "/usr/bin/tklbam-backup", line 443, in main
    opt_resume, True, dump_path if dump_path else "/")
  File "/usr/lib/tklbam/backup.py", line 237, in __init__
    self._create_extras(extras_paths, profile_paths, backup_conf)
  File "/usr/lib/tklbam/backup.py", line 183, in _create_extras
    limits=conf.overrides.mydb, callback=mysql.cb_print()) if self.verbose else None
  File "/usr/lib/tklbam/mysql.py", line 553, in backup
    mna = MysqlNoAuth()
  File "/usr/lib/tklbam/mysql.py", line 698, in __init__
    self.orig_varrun_mode = stat.S_IMODE(os.stat(self.PATH_VARRUN).st_mode)
    
OSError: [Errno 2] No such file or directory: '/var/run/mysqld'

Exception AttributeError: "MysqlNoAuth instance has no attribute 'orig_varrun_mode'" in <bound method MysqlNoAuth.__del__ of <mysql.MysqlNoAuth instance at 0x7fa2f396e560>> ignored

 

I have mysqld in the following location - /usr/sbin/mysqld. 

Forum: 
Tags: 
Ronan0's picture

Additionally, after trying to run this backup, it has done something to my set up so that my systems cannot connect:

 

java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource DEFAULT_transactional_DS with 0 connection(s) (0 still available)

 

om.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

 

java.net.ConnectException: Connection refused
Ronan0's picture

A server restart, everything is fine. Does TKLBAM try to stop mysqld or something? 

(This server runs applications off Tomcat btw. )

Ronan0's picture

Any ideas on this? Stopping Tomcat before the backup does not help either. Only other way this server differs from my others is that it is from OVH. Thanks.

Ronan0's picture

Bump: OSError: [Errno 2] No such file or directory: '/var/run/mysqld'

rest of error as per OP.

Can't work out why I can't get this server to backup on TKLBAM.

Jeremy Davis's picture

As per the subject line, my guess is that when TKLBAM dumps the DB your system is running out of RAM which is causing MySQL to crash. Once it crashes, TKLBAM can't get any more DB data (so it too crashes). When you try to access the server, the website isn't working (because MySQL isn't running).

When you reboot, everything seems fine again (MySQL is running again) until next time you run out of RAM...

Obviously I'm only guessing and I can't be 100% sure, but going from experience, I think it's likely I'm right...

Ronan0's picture

This is a 6GB memory machine. The highest memory has ever peaked is 55%. There's a fairly small database on it.

Looking at the monitor from yesterday when I was trying this backup again, memory peaked at 28.75%.

So, you appear to understandfrom the error message that mySQL is crashing. - But there has to be something else causing it aside from a memory issue. 

Jeremy Davis's picture

Perhaps try checking the logs?

As a general rule most logs (especially system ones and most Debian packages) can be found in /var/log . Probably the one that will be most helpful here is the MySQL log. IIRC it's something like /var/log/mysql.log (or similar).

Hopefully that may give some clues.

Ronan0's picture

Thanks for that.

It seems to be denying access. Do I need to somehow put the root password into TKLBAM? 

2017-05-30T08:24:15.435239Z 23723 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2017-05-30T08:24:15.441292Z 23724 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2017-05-30T08:24:15.452058Z 23725 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2017-05-30T08:24:15.457254Z 23726 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2017-05-30T08:24:15.458883Z 0 [Note] Giving 0 client threads a chance to die gracefully
2017-05-30T08:24:15.458917Z 0 [Note] Shutting down slave threads
2017-05-30T08:24:15.458931Z 0 [Note] Forcefully disconnecting 0 remaining clients
2017-05-30T08:24:15.458955Z 0 [Note] Event Scheduler: Purging the queue. 0 events
2017-05-30T08:24:15.459033Z 0 [Note] Binlog end
2017-05-30T08:24:15.501800Z 0 [Note] Shutting down plugin...............................................

Jeremy Davis's picture

By default TKLBAM should be using the hidden "system maintenance" account (special root account only available locally which can access MySQL using a stored password).

But it certainly looks like something is trying to log in to MySQL using the root account (but no password)!

Also perhaps there is a separate MySQL error log? Actually, after the service has crashed, but before you restart it, if you run 'service mysql status' that should give you the last few lines of log for the MySQL daemon. Even if that doesn't give you a ton of clues, it should also give you a hint on what command to run to get further daemon log entries. You could also manually check the daemon log, but using SystemD is probably easier as it will only give you the MySQL related stuff.

Ronan0's picture

Thanks. I will look further into the logs. 

The above was copied from the mySQL error log.

Could TKLBAM trigger some other process to try and log in to mySQL?

The only other user account in this installation of mySQL is mysql.sys@localhost - is that the system maintenance account you mean that TKLBAM uses? 

Jeremy Davis's picture

TBH I thought it was something more like debian-sys-maint but perhaps that's changed?

Sorry I thought that looked more like a general log rather than an error log. It certainly seems like it was shutting down though. So perhaps look higher up/further back in that log (if there is higher up/further back). It'd be interesting to see what else was happening at "08:24:15", or just before...

If that is all there is, then perhaps have a look for a log file with the same name but with a number appended on the end. There may also be files with a .gz on the end. The files ending in /gz have been compressed so you'll need to use zcat to view them (rather than cat or tail).

Also IIRC MySQL has 2 logs, an error.log and a more general log. So I just checked on a server that I have running here and I actually see 3 log files (all in /var/log): mysql.log, mysql.err and mysql/error.log. On my system both mysql.log, mysql.err are empty but mysql/error.log has a few entires, so I'm guessing that's the one you're looking at. It may be worth double checking those other 2 log files just in case.

Also there are other logs which may give additional relevant info. daemon.log (log of all services), syslog (general system log), dmesg (kernel messages - maybe there is something funky going on with hardware?). I'd be looking for time stamps that roughly match what is in your MySQL error log (i.e. around 08:24 2017-05-30). Note that some of the logs may have a slightly different timestamp format. dmesg actually uses a completely different time scale (time in milliseconds since boot).

Once you think you may have found something of interest, to check, reboot your server (so everything is running nicely). Then log into your server using multiple separate SSH sessions (one for each log you wish to watch, plus one extra to run the backup). In all but one of the sessions, tee up the log files to display in real time. Do that like this:

 tail -f /var/log/<name-of-log-file>

That should display the last few lines of the log file, then sit there waiting. When new log entries are written, they'll display here instantly. To exit, press <Ctrl&gt-;<C>.

Once you have all that set up, manually trigger a TKLBAM backup in the remaining session ('tklbam-backup'). Hopefully that will give you some clearer idea of what is actually happening and when...

If you find anything that seems relevant, please post (probably better to err on the side of sharing too much, than not enough).

Actually, one other thing that occurs to me is that perhaps you are running out of HDD space? That would cause similar behaviour (although TBH seems unlikely as there'd probably also be other warnings). To double check, try this:

df -h

Look for the volume mounted on '/' (should be the first entry under the headings). Ideally that shouldn't be too much more that about 70% (to leave headroom for TKLBAM backups and restores) although it probably shouldn't cause immediate issues so long there is enough free room to create your backup.

Ronan0's picture

I still have a problem with this.

I never resolved the above, which meant I could not back up the server. I just left it, and backup manually.

I have the exact same issue now trying to restore a server.

They are different servers. The commonality is they are both running Apache Tomcat. So, this may underlie the problem.

Again, the main error returned in the console is:

File "/usr/bin/tklbam-restore", line 553, in <module>
    main()
  File "/usr/bin/tklbam-restore", line 529, in main
    restore.database()
  File "/usr/lib/tklbam/restore.py", line 68, in database
    self.rollback.save_database()
  File "/usr/lib/tklbam/rollback.py", line 186, in save_database
    mysql.backup(self.paths.myfs, self.paths.etc.mysql)
  File "/usr/lib/tklbam/mysql.py", line 553, in backup
    mna = MysqlNoAuth()
  File "/usr/lib/tklbam/mysql.py", line 698, in __init__
    self.orig_varrun_mode = stat.S_IMODE(os.stat(self.PATH_VARRUN).st_mode)
OSError: [Errno 2] No such file or directory: '/var/run/mysqld'
Exception AttributeError: "MysqlNoAuth instance has no attribute 'orig_varrun_mode'" in <bound method MysqlNoAuth.__del__ of <mysql.MysqlNoAuth instance at 0x7f7b9a143e60>> ignored

 

And the mySQL log returns:

2018-01-26T17:05:28.463670Z 44 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2018-01-26T17:05:28.475391Z 45 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2018-01-26T17:05:28.480323Z 46 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2018-01-26T17:05:28.481767Z 0 [Note] Giving 5 client threads a chance to die gracefully
2018-01-26T17:05:28.481851Z 0 [Note] Shutting down slave threads
2018-01-26T17:05:30.482112Z 0 [Note] Forcefully disconnecting 5 remaining clients

 

So, you said above that "... By default TKLBAM should be using the hidden "system maintenance" account (special root account only available locally which can access MySQL using a stored password)..."

It's obviously not doing this. Perhaps I would have switched this account off to harden the server in the past? - I don't recall doing so though. 

How can I check that account referred to is available to TKLBAM? Exactly?

Or perhaps Liraz would have some ideas on this?

This is on Debian version 8.9

MySQL version 5.7.20

Tomcat 7

 

Thanks.

Ronan0's picture

Ok, I seem to have disabled the debian-sys-maint account when hardening. That might explain it.

I need to look into how to reenable it, and you mentioned a 'stored password'.

Hopefully I get it resolved. I will update this if so.

Any tips welcome. Thanks.

Ronan0's picture

I don't recall removing that account manually though. I wonder how it came about. Any advice on reinstating it welcomed. There does not seem to be a /etc/mysql/debian.cnf file there either.

Jeremy Davis's picture

Restoring the system user account is pretty straight forward, you just need to reconfigure the package (basically resets it to default config).

TBH, I'm a bit surprised that anything works after removing that. You most likely wouldn't be able to start/stop MySQL service either (as the system wouldn't have permission).

This shouldn't be destructive to any data stored in your DB (or anything else), but as per always when you're doing system level changes, make sure you have a backup of everything important. Then simply reconfigure the MySQL package:

dpkg-reconfigure mysql-server-5.5

You should be good to go now...

Ronan0's picture

I'm using MySQL version 5.7.20.

Also, I have always used apt-get as my package manager.

Do you still recommend I use the above command modified to

dpkg-reconfigure mysql-server-5.7

?

Thanks.

 

Ronan0's picture

That command doesn't work. I am using 5.7.21 MySQL Community Server.

Ronan0's picture

The three users I have are:

mysql.session

mysql.sys

root 

Everything works fine with it except for TKLBAM.

Ronan0's picture

I would definitely not have removed that account and the debian.cnf file myself.

Jeremy Davis's picture

I'm assuming that you've installed that from a third party repo? If it's not a Debian package then it wouldn't surprise me if it doesn't have that file even by default. If it's a third party install then perhaps check with the third party? (Wherever you installed from).

You could try reinstalling it. I.e. 'apt-get install --reinstall <package-name>'. Not sure if that will affect your DB or not (shouldn't do, but I can't be sure).

Also we've never tested TKLBAM thoroughly with MySQL of a later version than 5.5, so perhaps it's incompatible? We've done some preliminary testing with MariaDB (10.1, what TurnKey v15.0 will have - roughly the same as MySQL 5.6) and it appears to work fine so far. TBH, I'd expect it to "just work" but there are significant changes between 5.5 and 5.6, let alone 5.7, so there are way to many unknowns to be sure.

In retrospect, first up I probably should have clarified the version of TurnKey you were running on and what customizations you had done... Hindsight...

Ronan0's picture

I ran 

dpkg-reconfigure mysql-community-server

but still those threee users.

Ronan0's picture

Yes. I had originally installed mySQL 5.7 from a downloaded package on these servers (I needed 5.7 rather than 5.5 for certain reasons.)

I just saw this:

"Nice to know your feedback on our APT repo packages. Unfortunately, you have been hit by an unsupported case where we do not provide both debian-sys-maint user debian.cnf file through our packages. They are provided only by the native repo packages." - https://bugs.mysql.com/bug.php?id=73711

So, to try resolve this, I put in a new debian.cnf file, at /etc/mysql/, copied from elsewhere.

And I ran command in mysql - GRANT ALL PRIVILEGES on *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY 'password-in-debian.conf' WITH GRANT OPTION; FLUSH PRIVILEGES;

It seems to be working now. At least I don't get that error, and it appears to restore ok.

So just to repeat, if anyone installs MySQL 5.6 or 5.7 from a downloaded package rather than the native mySQL package, they will be hit with this problem.

Any insights on this case, or experience in it, appreciated. Thanks.

Jeremy Davis's picture

Great work on that. That should do the trick. Glad to hear that that works.

Like I mentioned, I don't have any experience with MySQL-5.7, so can't offer any guidance, but now you mention it, it makes a fair bit of sense that it wouldn't work as it was. But your workaround should provide everything that TKLBAM requires.

One thing that it probably worth mention though is that you may hit this (or somewhat similar) issue if/when you use TKLBAM to migrate your data to a new server. TKLBAM will reinstall packages, but only packages from default repos (i.e. Debian and TurnKey).

So that will mean you'll probably want to split the restore into multiple operations. First download your backup to the new server. Then make the required tweaks, including adding the 3rd party repo and the new config required for the new version of MySQL. Then restore all the data etc. If you do it that way, it should install the new version of MySQL and everything should just work. Having said that, YMMV, but please do not hesitate to post with any questions or concerns and I'll do my best to help out.

Ronan0's picture

Thanks for your help, Jeremy. You pointed out the problem for me.

Just towards feedback, I only use TKLBAM to back up the databases - using overrides, and the skip packages and skip files tags on restore. - I use whole server snapshots to back up and restore the basic Debian set up, and the two server applications set up on it, and git to manage backup and restore of my main application files.

Also, I know from another server, that one shouldn't have any trouble if one upgraded from the native mySQL 5.5 to 5.6 or 5.7. If you do it that way, the debian-sys-maint user and debian.cnf file are kept. I only encountered this when installing straight from the non-native package.

All the best.

Jeremy Davis's picture

Glad you got it working and you're all systems go again.

Whilst snapshots are a nice easy way of backing up the file system, unless you stop the server when the snapshot is taken, there is a small risk of data corruption. I'm guessing that you are aware of that though, hence why you use TKLBAM for the DB and git for your app. But I thought it was worth saying anyway, just in case! :)

Good luck with it all moving forward.

Add new comment