Pat's picture

We have a Redmin appliance that has been running for a while.  Linux version 10.04.01 and webmin version 1.520.  We use it to share architectural bid documents with the contractors bidding on our projects so our usage level varies from week to week but is never extremely high.  

I back it up weekly.  March 19 (about 3 weeks ago) ago I did the backup (it was a full back up since my last incremental was too old.) then ran updates.   I had not run updates for a very long while and figured it would be good to get all the latest security patches etc.  When the update was done, the client site was not working.  Since I was running out of time and needed this to be available I restored from the backup I had done just before running the updates.  The restore worked fine and everything was working again.  However after the restore the webmin reported that the amount of space used just about doubled which has just about filled the hard drive (only 4.96 GB free).  

Two weeks ago I tried to run backup again.  It got most of the way through the full backup but it seemed like it hung.  I just left it running over the weekend but it never finished cleanly.  When I tried to run the backup again, it reported that there was a backup in progress and would not run.  So this last weekend I rebooted the machine and tried running the backup again.  Same problem with the backup getting very close to done then hanging and erroring out.  Again it is saying that I can't run a backup since there is a previous backup in progress.  We have bids underway wtith additional documents loaded and several additional users registered since the last successfull backup.  So I really want to get a good backup before I start digging into what doubled the used stoage space and/or migrating to a clean new install.  

Where in the logs locally could I find the specific error that is blocking the backup from finishing?  Unfortunately I did not write down the specifics before closing out the browser.   I'm assuming that is the right place to start troubleshooting.  Or is there a safe way to clear up some space on the hard drive without a verified current backup?  

Any ideas or suggestions on what is wrong and how I can get it fixed are really appreciated.    I'm not extremely proficient with Linux so the more detail the better.  Thanks for any help.   Pat

Forum: 
Tags: 
Pat's picture

Still have the problem - whenever I try to run a backup, I get "back up in progress" error.  I'll try rebooting the server this weekend when no users are on it and will try running a backup again to see if it will finish this time.  

If anyone has any suggestions or adjustments that I should make before restarting the backup, please let me know.  I appreciate any help.  Thanks   Pat

Pat

Pat's picture

I rebooted the machine and reran backup today.  It errored out before finishing again.  It looks like it stopped about the same place as last time.  This time I printed out the screen before closing it.  Here is the end with everything before it cut out.  Hope someone can look at this and tell me how to fixi it.  

Any help is appreciated.   Thanks Pat

 

A etc/.git/objects/d2/a2f39c11f7e163616def25bcc7903b96f42310
Uploading s3://s3.amazonaws.com/tklbam‐fngje3zvzp4dhgmp/duplicity‐full.20160409T143437Z.vol378.difftar.gpg to
STANDARD Storage
Processed volume 378
A etc/.git/objects/d2/a32eba69902baa6351491c8f453046f8b5e9f1
A etc/.git/objects/d2/a48e45acff0e1135dac8ee29cd1443f17e41cb
A etc/.git/objects/d2/a9334a7af4b7c76a25d25b5c4a36d8aa95cabc
A etc/.git/objects/d2/a99bcfbdb35ab47ac89f717716a5938fd26571
A etc/.git/objects/d2/ab3b0bd268a61dc9d61f0491143b95a6c8ad23
A etc/.git/objects/d2/ab947273c62a6c3e5d2f8c7b10a37930797293
A etc/.git/objects/d2/aba39b8ae0f2c6827ec8247cdb872f70f8c413
A etc/.git/objects/d2/abaf22e86e16f5d7fa7192a56eda4326f26dc3
A etc/.git/objects/d2/acc32672eea3346817faf20fe0c3c496d46c87
A etc/.git/objects/d2/ad4107b2de8213e58d36d37aac67e1b6db4650
A etc/.git/objects/d2/ae0f2c4857db0b6976957576865eb6fab29d52
A etc/.git/objects/d2/ae6cd4eddbe894e76156e41a94528f34fef429
A etc/.git/objects/d2/affb8a3dbfcda863928ce7028f1c7f04cd0629
A etc/.git/objects/d2/b117fba26a83a45788ea7670c37335e5e00216
A etc/.git/objects/d2/b25839df0006c23d1473e410dea5833c599ae9
A etc/.git/objects/d2/b30cbf0c76ba9926e729add7ebaf7e21135ae3
A etc/.git/objects/d2/b6bf180fd831faf3f385dd192fcf421ac34f57
A etc/.git/objects/d2/b746df83ee09b3da0e7ead2df0beff6de4c47d
A etc/.git/objects/d2/ba6be1e7a2f185d2263b943ce4c0a0b7599eca
A etc/.git/objects/d2/ba774b2f1789a8d999709b47664a6a719aed54
A etc/.git/objects/d2/ba95ce24fdc4dec92ec4cd04775719356bb0aa
A etc/.git/objects/d2/bad9ad15a6a09c750df387a5e276c53a4f15cd
A etc/.git/objects/d2/bb72df1269dc673d8f427369eea36b52538f45
A etc/.git/objects/d2/bd9d0dcba1ea6fcfd563a0d929cbb323ba8d43
A etc/.git/objects/d2/bdd296f20e81ee720bc0686b860123d7036df0
A etc/.git/objects/d2/be2a7d1dde8bad3f5953b4c08031df4a32e228
A etc/.git/objects/d2/bf00f9e7328fa17d6a1289806cf4426f638164
A etc/.git/objects/d2/bf689b89ed62b7dd52a5a86ce45d4a70ff5e82
A etc/.git/objects/d2/bf905e3b4408a1001ca42fe25925b8ccff88ac
A etc/.git/objects/d2/c0716513f7bc9b4fef608ac6a5be25c85d67cb
A etc/.git/objects/d2/c2608c98497849e79f4f7150ab2d83b0a094dc
A etc/.git/objects/d2/c2f84beff5a75ff6dc379a18df4730452b6414
A etc/.git/objects/d2/c55dc3deeedb942d8ecf3c938cec8dcb8dcaa1
A etc/.git/objects/d2/c675692df316be7b453f37d976e4d54f677a70
A etc/.git/objects/d2/c83793edf6755c69a16b79e654a8e902cbfac2
A etc/.git/objects/d2/c8675d09743b7c28662a971e54c1d1ed41fefc
A etc/.git/objects/d2/c921354db50b4d188c81126e1fcc2744e72d91
A etc/.git/objects/d2/c988fe24b1141aac103dffb39044c9d7063e04
A etc/.git/objects/d2/cbfe5eefe9a36e3c799f97f3c35d461276f6a5
A etc/.git/objects/d2/cc40cd3723bbae27cf340895789d2dee443035
A etc/.git/objects/d2/cdc6bb6bf6dcd78a2ee3e4972457574dc7c09e
A etc/.git/objects/d2/ce5020df479d377eb1aac138e0a1fc8f8ab7cf
A etc/.git/objects/d2/d1de650c861fb680ba61e64b005748d2e05d6a
A etc/.git/objects/d2/d205c9f6b30f35abda7e92d0c1bd6ccada1ccb
A etc/.git/objects/d2/d52f080ec85ffee55a8e77220b817bedd3c40d
A etc/.git/objects/d2/d6635575abdbedee0222a027c77928ede9bd23
A etc/.git/objects/d2/d93b212d4cadcc0cef3b2c47304c7150f23a16
A etc/.git/objects/d2/dcc767e146d3feb1c9a9b1987af8e5580cce46
A etc/.git/objects/d2/dec9c3cae2c32e1e6e83374ed1afa94f9e1f02
A etc/.git/objects/d2/e0ba25d3391a76edcff11b37398316e291392d
A etc/.git/objects/d2/e38d77f0df87ca8804a61f599d5cb099105f4b
A etc/.git/objects/d2/e3b46d8dbe73911b087322f2c02d3973fb917e
A etc/.git/objects/d2/e98df316b938f71c8fcf891a4993ebf0441dae
A etc/.git/objects/d2/eacbcbd62f20715497b60f221815af2cd9080d
A etc/.git/objects/d2/ead5d67b2f6f17ed6ade1cb667915f9590f2cc
A etc/.git/objects/d2/ed3a1cdf0b930c9130fc8cfc0cac90b32fba1d
A etc/.git/objects/d2/f03b0f964a98c433c0bb4ada8fca41abd7603f
A etc/.git/objects/d2/f062c786fe8ce19d6a3e636bdbde530a087229
A etc/.git/objects/d2/f1c4ca343ffb8605769906340da848e27dde14
A etc/.git/objects/d2/f2cb33f94b375aacd581c8fa4ac4e18fe1a186
A etc/.git/objects/d2/f2d5558b688494d06dc43c0bd1300c0374b33d
A etc/.git/objects/d2/f4a3dad5f6b78ac2b243fb0238434c3f94fdac
A etc/.git/objects/d2/f749c14abd0beca3f552f74ff4b958297834bf
A etc/.git/objects/d2/fa8b431e96e8990cd913f7c216678f47f6644f
A etc/.git/objects/d2/faee647de1d845dd98b9664eda72aa5c24fa61
A etc/.git/objects/d2/fbe9f98f8224341757251a6fdb2702e1db37cb
A etc/.git/objects/d2/fc3597453eea48ebbdc75954c5ebd3280ad573
A etc/.git/objects/d2/fe92539aa18cf742acc78ab2e0760b3042b1ca
A etc/.git/objects/d2/ff6a8530a96ae603fd92ddbc9867b8ee12458b
A etc/.git/objects/d3
A etc/.git/objects/d3/01eddcc32680eefe9f294f008adaa4eacc1e12
A etc/.git/objects/d3/043cbc028f4e906921609a7909350f7e069033
A etc/.git/objects/d3/0532f6e068c1c63141952d937e6f7eebae4dca
A etc/.git/objects/d3/056cd13bead220658ec6402408f44cbc00ec28
A etc/.git/objects/d3/05c8b6c670f67ae086f45988341c6dfa8054a2
A etc/.git/objects/d3/0645f1204e33756dcd6f53dfada295480b488b
A etc/.git/objects/d3/0a737e93f530a810e93933a130161251a6bff4
A etc/.git/objects/d3/0cd90f3e47eb1c4bcd65e69be5e1781824ea50
A etc/.git/objects/d3/0db2a499817918bfc6c35937f9e057a7bacd01
A etc/.git/objects/d3/0fa47fb53ee1c5781c9a0b6f2a6a75062651e7
A etc/.git/objects/d3/13275de8d352be7da9e3b9d127c4b9b9be4f97
A etc/.git/objects/d3/135b025a1eb4a7938b0e8b354c082968a87711
A etc/.git/objects/d3/13c7e123aa3ee40f59b6f34d76f0485512bdd6
A etc/.git/objects/d3/15668bc83fbc0a0af08c07b71ba3ed0826d6d4
A etc/.git/objects/d3/163cd3410bf9fdd06e9f93023735c4b539ba03
A etc/.git/objects/d3/16a144fc01b7889da5c8093a8c34b5d4bb5f2f
A etc/.git/objects/d3/19390007ef9bce8e249bc6fe05798cf983f3e6
A etc/.git/objects/d3/1dd760415982ea7613c882e56417f6cf0dd99f
A etc/.git/objects/d3/1f1fb5f2663984ba4fe6c17d27458acea9df51
A etc/.git/objects/d3/2208c59fb6b83c49e5dd5d3204852be65d502e
A etc/.git/objects/d3/2485d798f5623518058fe4aeaaa1dfdf9b8266
Uploading s3://s3.amazonaws.com/tklbam‐fngje3zvzp4dhgmp/duplicity‐full.20160409T143437Z.vol379.difftar.gpg to
STANDARD Storage
Processed volume 379
A etc/.git/objects/d3/24d027f520297dc284032bc6ce861378bf3be7
A etc/.git/objects/d3/26cca0483de81359e22e9b57bd9e07a9dcb0c9
A etc/.git/objects/d3/28cb009dc21877598f26ffbd60278bc279d476
A etc/.git/objects/d3/28f3a2bce05731f28f439849a06da2a99a0f9d
A etc/.git/objects/d3/2a052a9bb6cc9edfadee6205ddbf1d47354012
A etc/.git/objects/d3/2eb72eb6950c033cba52bc2ff27d1c3d85cd93
A etc/.git/objects/d3/2f1ce1e7f85870d368f90e654cc6242e40ef98
A etc/.git/objects/d3/33d73bcc2fb552009c98a8c9deeb7a27806012
A etc/.git/objects/d3/34766d3600bcde78ede410378097e4d71d6279
A etc/.git/objects/d3/34a3c3fce68fce50ed3bb552fbbf7c97a1a4db
A etc/.git/objects/d3/38a1e240df4df50bdafad49119245a25ba5407
A etc/.git/objects/d3/3d32e1e0020016a87dc07581edad26a339d669
A etc/.git/objects/d3/3da50881137899aed2760a37b2ed0587aa7af6
A etc/.git/objects/d3/41d047f1ae8c4093b61217fc785baccbf2b1cb
A etc/.git/objects/d3/41ddfaa3b676a7da86706f39d97434d411ba24
A etc/.git/objects/d3/41f9eb33a0630b03dbbc5fcafa1030809d71f9
A etc/.git/objects/d3/4383726c06cf10b3cb67a6fc4b2a9fbf2a9548
A etc/.git/objects/d3/44ae84f733b776bbac7b54808daef8948ebc49
A etc/.git/objects/d3/45d914a72fa264c250a73f590eb1234ea7c8af
A etc/.git/objects/d3/497d48b0a615cb99da19ab06c1a38ac0913104
A etc/.git/objects/d3/49997d94cba1eb97e343e16505edac7ffd855f
A etc/.git/objects/d3/4c5c4be9750060718661eaf83d1a55a8f8142a
A etc/.git/objects/d3/4cf65596a9064329283267afa13a6ebe6b466c
A etc/.git/objects/d3/4ef94ca68ba9a9b67d7dca20cd918fa6748cd2
A etc/.git/objects/d3/4f32658f2a7a294ab16c6f4ee5aaba98948f13
A etc/.git/objects/d3/5272e07201a7747b5556f30a95fe093099b055
A etc/.git/objects/d3/52ae6d1c04837964d684bf449a9f26dfd2a0fb
A etc/.git/objects/d3/537a027ed1dc563531216a8e53aca8b8720c19
A etc/.git/objects/d3/565d7df26b01aebfcb40a2215833b09adec14e
A etc/.git/objects/d3/56cd1d2b5a47000222f39ae9618d31e93e5a2c
A etc/.git/objects/d3/57efad656a5b1582b5b7b47210072d6af7ed34
A etc/.git/objects/d3/57f0c852273f6db8837e75025f45522aab559e
A etc/.git/objects/d3/587cf882677b020dda6efb0c3549cb7b0f6b5b
A etc/.git/objects/d3/5bb9bd58b3131359e3644df4d2b8e277ae756e
A etc/.git/objects/d3/5d9c501764b8e87105a6575c32f29afb991573
A etc/.git/objects/d3/5dafce79417dd09b10c1ea25fa496f79bb8b70
A etc/.git/objects/d3/5f61d27429d8cc20883952ed86b1acf75686c1
A etc/.git/objects/d3/5fae1deebe6d122e84069f59db2cdc2fcf5c8b
A etc/.git/objects/d3/604bfb47acb7cbf37953ca258d5b56e5b584d1
A etc/.git/objects/d3/66fce4976a9d4452addaf0f16c9b833f035d87
A etc/.git/objects/d3/67ad99413fa976502976d0e3c6b795ff87abd3
A etc/.git/objects/d3/699679a8256ed7a0cc5ed08fd91fb30cb310a6
A etc/.git/objects/d3/6be177a9195d346da9186fa7cf236d1879cc2a
A etc/.git/objects/d3/6c65c8d670d3569f33a60abe73b978029326ea
A etc/.git/objects/d3/769da2afc818f5c62ea4643b925f615385fd4b
A etc/.git/objects/d3/79126a47fb1ed7fc0371d53bf219c8dfa981dd
A etc/.git/objects/d3/7a9ecb4876703457648c6141817036ed3cab6b
A etc/.git/objects/d3/7b35bf5d8423ce023d80b90a10e55d456db36e
A etc/.git/objects/d3/7dd32ff49f3119b4a75d7c2c987f310093cefb
A etc/.git/objects/d3/81dbf4dbb5de8a44b99754782bd27421f030c0
A etc/.git/objects/d3/81f80bfeb4870e73534b801aea02c6b9e854e9
A etc/.git/objects/d3/8322baf0eed275abcbea41e2c07bbd0b272195
A etc/.git/objects/d3/848e521369d6f69e43d50d635ce5f6e3d9e4b4
A etc/.git/objects/d3/8613c50b6c2854c98909f3f30ec59368b3245f
A etc/.git/objects/d3/86991ab4a6c248262187d99ebbc3a6f199c31e
A etc/.git/objects/d3/8721f04d9d9aaf1e8a25c2c44272f2ef6d8609
A etc/.git/objects/d3/8a09bb208f6ba834ce7f96b1e8aaea10e9f741
A etc/.git/objects/d3/8a27ebc96584570e50b75f23d06447095a8a81
A etc/.git/objects/d3/8a35f5983956bf6af2084b001081b4e26aecd9
A etc/.git/objects/d3/8b68f79af74ac27366d9a8a842bc18a853f12a
A etc/.git/objects/d3/8cea7610102bb95c8125c8f61ebc176892968c
A etc/.git/objects/d3/9052c746b9e61389531cbb5ceb3e941bb9d487
A etc/.git/objects/d3/9108762963d8d4a381f36058613eaa97b45a3f
A etc/.git/objects/d3/9169eee7b9a277acfc5513de57a190e2f975e6
A etc/.git/objects/d3/955eb1809023f0ea853777fa91259dd7d6add7
A etc/.git/objects/d3/98816f9ada1deaa12ca098aae23da0a22fa66a
Traceback (most recent call last):
File "/usr/lib/tklbam/deps/bin/duplicity", line 1252, in <module>
with_tempdir(main)
File "/usr/lib/tklbam/deps/bin/duplicity", line 1245, in with_tempdir
fn()
File "/usr/lib/tklbam/deps/bin/duplicity", line 1218, in main
full_backup(col_stats)
File "/usr/lib/tklbam/deps/bin/duplicity", line 416, in full_backup
globals.backend)
File "/usr/lib/tklbam/deps/bin/duplicity", line 294, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/lib/tklbam/deps/lib/python2.6/site‐packages/duplicity/gpg.py", line 279, in GPGWriteFile
data = block_iter.next(min(block_size, bytes_to_go)).data
File "/usr/lib/tklbam/deps/lib/python2.6/site‐packages/duplicity/diffdir.py", line 505, in next
result = self.process(self.input_iter.next(), size)
File "/usr/lib/tklbam/deps/lib/python2.6/site‐packages/duplicity/diffdir.py", line 631, in process
data, last_block = self.get_data_block(fp, size ‐ 512)
File "/usr/lib/tklbam/deps/lib/python2.6/site‐packages/duplicity/diffdir.py", line 658, in get_data_block
buf = fp.read(read_size)
File "/usr/lib/tklbam/deps/lib/python2.6/site‐packages/duplicity/diffdir.py", line 415, in read
buf = self.infile.read(length)
File "/usr/lib/tklbam/deps/lib/python2.6/site‐packages/duplicity/diffdir.py", line 384, in read
buf = self.infile.read(length)
IOError: [Errno 5] Input/output error

 

 

Pat

Jeremy Davis's picture

I'm really sorry that I missed your post. Thanks so much for your patience. TBH I actually thought that I had responded previously, but obviously not...

FWIW I've been flat out trying to get v14.1 out the door!

When you say it's been running for a while; you're not joking! 4-5 years I'd guess! :) v10.04.01 is actually the Ubuntu version that it was based on. It would be a TurnKey v11.x appliance. We've since moved to a Debian base (in v12.0 - Debian Squeeze). If this server is internet facing then I strongly suggest that you plan to migrate your data to a newer server as Ubuntu stopped providing security updates for v10.04 almost a year ago...

WRT your issue, do you have plenty of free HDD space? It appears to be an I/O error and the most common cause for that is a full HDD. Also IIRC the earlier releases of v11.x had a couple of bugs. The first was that Webmin was saving stuff in /etc that shouldn't really be there (some of that you probably don't want in your backups like bulk logs etc). That issue was compounded by etckeeper (which is producing all those etc/.git/objects entries) not running it's garbage collection so the git repo that saves all your config changes (and all the cruft from Webmin) grows exponentially over time. IIRC I had a very basic LAMP server which over the space of a year ended up with over 500MB of rubbish. Assuming that you server is one that was affected by this bug I'd hate to imagine what level of cruft it has built up. Out of interest you can see how big your /etc dir is with this:

du -ch /etc | grep total
Or to just see how big the git repo that etckeeper stores everything is:
du -ch /etc/.git | grep total

If free space isn't your issue then I'm not really sure TBH. Technically we only support the previous major version (i.e. we're on v14.1 now so support only goes back to v13.0) but we hate to see customers stranded so will do what we can to help you out..

Pat's picture

I do really appreciate you taking the time to respond and for confirming my growing suspicions that the lack of free space on the drive might be the problem.  Also thanks for all the work you do providing Turnkey Linux to us. 

The disk is pretty full.  When I did the restore from a back up the disk space that was used just about doubled.  When I ran the commands you listed above I did find the /etc/.git was were large - 24G out of a total drive size of 63G.   I'm trying to get a good current back up then plan to wipe out the old version and start over with the most current Turnkey Redmine appliance.  

The hope is that I can import my data/users/and the like from the old Redmine databases.  I've Redmine migration notes so know that I may have a couple of manual adjustments but think I can make it work OK.  Since I plan to load the new Redmine appliance on a different machine I'll have the opportunity to work on it without impacting the current use.   

Is there a best practice way to clean out the /etc/,git/objects?  Shall I just go in an manually remove?  Thanks again for the assistance.  

 

 

Pat

Jeremy Davis's picture

Try this as a short term measure to clean up:
cd /etc
git gc
Then see how big your /etc dir is; that should bring it down significantly.

Otherwise you can reinitialise it completely. You will lose the history of all your config changes but assuming you haven't changed anything lately and everything is working ok it should be fine. Do that like this:

etckeeper uninit -f
etckeeper init
Without manually removing files from /etc (which IMO is generally a really bad idea...) that's probably as good as you'll get...

You write you already tried to do a restore, and it doubled disk usage. I think tklbam-restore downloads and unpacks the data on local disk. It will be cached somewhere. Look up documentation (or wait for Jeremy) to find out how to clean up cached data.

Jeremy Davis's picture

It needs to download the full archived backup; then extract it to a cache. Even with a relatively small backup it will still take a bit of space. If the whole server is taking up almost 60GB then your backup will be pretty big...

TBH I don't recall where TKLBAM stored the cache back in v11.x as I was just an enthusiastic volunteer back then! :) But if you 're still having issue I can find out.

Pat's picture

When I ran "git gc" it found 939 objects but errored out saying "bus errorobjects 939".  Since I have not updated or changed anything recently and the server is working OK, I went ahead and ran the uninit / init commands.  That cleared out a lot - etc/.git went from using 24G down to using 33M.   I'll run a full back up now and will post back if it finishes OK.  

I could track down where the restore is cached but if I can get a clean new back up, then I'll just move forward with starting over with the newest Redmine appliance.  

Thanks much for both of your help. 

Pat

Pat's picture

Just to update.  The back up ran and completed OK.  Now I'll move on to starting up a clean up to date Redmine appliance.   Thanks again for your assitance with this.  

Pat

Add new comment