#dev | Logs for 2018-08-19

« return
[13:56:59] -!- SemperOss [SemperOss!~torben@Soylent/Staff/Sysop/SemperOSS] has joined #dev
[13:57:39] <Bytram> SemperOss: Hi!
[13:57:47] <Bytram> can you kinit on boron?
[13:57:52] <SemperOss> Bytram: Hi back
[13:58:00] <Bytram> front and center, too!
[13:58:38] <SemperOss> No I can't as TMB has grabbed my password, changed to something HE only knows! :-(
[13:59:07] <Bytram> I've got privs for most things, but I'm more of a userland kind of guy. I've learned a few things over the years here, but I try to leave most of the sysops stuff to the other staff
[13:59:19] <Bytram> yes, sad face indeed
[13:59:37] <Bytram> okay... lemme see if I can retrieve a command from chat logs
[14:01:00] <Bytram> I'm going to try to do: /home/bob/bin/bounce on fluorine and hydrogen
[14:01:30] <Bytram> fluorine... done
[14:01:48] <SemperOss> Sounds like a good idea. It's probably not the load balancer due to the 502/503 errors
[14:01:57] <Bytram> hydrogen... done
[14:02:18] <Bytram> got same output on both servers:
[14:02:19] <Bytram> * Stopping HTTP accelerator varnishd [ OK ]
[14:02:19] <Bytram> * Starting HTTP accelerator varnishd
[14:02:38] <Bytram> so, it looks like that was successful.
[14:02:40] <SemperOss> Stiil retrying a couple of times to download the RSS file
[14:02:45] <Bytram> now, whether or not that was sufficient...
[14:02:55] <Bytram> my you're quick!
[14:03:25] <SemperOss> I wish I had access to the web servers now
[14:03:29] <Bytram> SemperOss: what command are you using?
[14:03:39] <Bytram> to download the rss file
[14:03:45] <SemperOss> I used wget
[14:03:49] <Bytram> k
[14:04:00] * Bytram is gonna try it on boron
[14:04:02] <Bytram> brb
[14:04:52] <SemperOss> K
[14:05:00] <Bytram> worked 1st try
[14:05:15] <SemperOss> I've tried it on boron and now it whizzes through
[14:05:26] <Bytram> yep, that's where I was, too.
[14:05:35] <Bytram> were you on boron, before?
[14:06:15] <Bytram> I'm getting the same errors from my home box
[14:06:36] <SemperOss> I was
[14:06:47] <Bytram> k
[14:07:04] <Bytram> any luck trying wget from your local machine?
[14:07:14] <SemperOss> I've tried from one of my overseas servers too. Same error
[14:07:21] <Bytram> 2018-08-19 10:07:06 (740 KB/s) - Connection closed at byte 31628. Retrying.
[14:07:24] <Bytram> nod nod
[14:07:35] <Bytram> hmmm
[14:08:05] <SemperOss> I'll try from my Singapore Linode server
[14:08:13] <Bytram> noice!
[14:08:57] <SemperOss> Same, lame result! :-(
[14:09:08] <Bytram> If you'd like to see an interesting Rom-Com movie, I really enjoyed Crazy Rich Asians (a large part of 3which was set in Singapore)
[14:09:13] <Bytram> okay
[14:09:31] <Bytram> so one can reasonable assume that Linode has good connectivity within Linode, right?
[14:09:33] <SemperOss> I think you'll have to give the load balancer a knock on th e h
[14:09:40] <SemperOss> the head
[14:09:55] <Bytram> an how am I gonna do that?
[14:09:57] <SemperOss> Yes, I'd assume that
[14:10:01] <Bytram> hold on
[14:10:23] <Bytram> so, that would be magnesium and sodium, IIRC
[14:10:40] <chromas> Can you pull the rss file directly from each of the servers?
[14:11:17] <SemperOss> Bytram: See, I don't really know as I haven't had access to the machines yet
[14:11:50] <SemperOss> I can tell you a lot about boron, though
[14:11:54] <Bytram> would one of you please look on the Wiki for the descriptipons of those servers... info may be out of date, but I recently discovered there is a page for at least a couple of them
[14:11:58] <Bytram> LOL
[14:12:46] <SemperOss> load-balancer Sodium, Magnesium
[14:13:00] <Bytram> nod nod
[14:13:27] <chromas> Those aren't even in the wiki
[14:13:30] <Bytram> I'll try them in this order:
[14:13:34] <Bytram> chromas: kthx
[14:13:35] <Bytram> lithium
[14:13:35] <Bytram> magnesium
[14:13:35] <Bytram> sodium
[14:13:35] <Bytram> fluorine
[14:13:35] <Bytram> helium
[14:13:36] <Bytram> hydrogen
[14:13:38] <Bytram> neon
[14:13:40] <Bytram> beryllium
[14:13:42] <Bytram> boron
[14:13:44] <Bytram> oxygen
[14:13:58] <chromas> The wiki is sad and unloved, like /. beta
[14:14:38] * Bytram *chuckles*
[14:14:43] <Bytram> well, that was... interesting
[14:14:44] <SemperOss> I'll send a warm thought in the Wiki direction and then ignore it again
[14:15:29] <SemperOss> Can you get root access to either of the two?
[14:15:48] <Bytram> I don't know. Probably.
[14:16:03] <Bytram> on lithium, I got: 2018-08-19 14:14:21 (300 MB/s) - Connection closed at byte 48023. Retrying.
[14:16:17] <Bytram> on the first retry it was successfull
[14:17:09] <Bytram> second try was identical
[14:17:26] <Bytram> third attempt required two retries
[14:18:19] <Bytram> one more time, single retry
[14:18:54] <Bytram> single retry, but at yet another offset: 64429.
[14:19:44] <Bytram> hmm, I bounced fluorine and hydrogen... think I'll try one of those, next, instead
[14:20:38] <Bytram> on fluorine, first attempt succeeded on the first try
[14:21:09] <Bytram> second attempt: Connection closed at byte 97199. Retrying.
[14:21:09] <Bytram> and then success
[14:21:13] <Bytram> :(
[14:21:44] <chromas> Must be time to reinstall Windows
[14:22:14] <Bytram> then 3 successes in a row on first try, and then another one where a retry was needed
[14:24:13] <SemperOss> I think you should try to restart the load balancers
[14:26:58] <Bytram> tried again using my browser, activated logging and noticed that there are attemts to load http: AND https:
[14:27:00] <Bytram> for #request# GET http://add.my.yahoo.com
[14:27:02] <Bytram> and for
[14:27:10] <Bytram> #request# GET http://soylentnews.org
[14:28:07] * Bytram is not sure why we are trying to load a yahoo favicon, unless there is a URL lacking a scheme or maybe a domain name or something like that?
[14:28:27] <Bytram> anyway, could load both the rss and the atom feeds just fine in Pale Moon
[14:28:49] <chromas> Does Pale Moon support ipv6?
[14:29:10] <Bytram> could be, but I'm (AFAIK) stricly IPv4
[14:31:12] <Bytram> SemperOss: I suppose I could do that, but before I do, and while TMB is away for the rest of the week and thus unable to clean up if I bork things... I gotta ask
[14:31:32] <SemperOss> Understandable
[14:31:43] <Bytram> what problems are we having besides the rss/atom feeds borking?
[14:31:58] <Bytram> an occasion 5xx that, upon retry, eventually succeeds?
[14:33:44] <Bytram> hmmm, I have no idea where they would be, but shouldn't there be some kind of logs somewhere?
[14:34:39] <SemperOss> I would look at the load balancer first and see why it gets the 50x errors. They normally mean that it cannot get in contact with the web servers
[14:34:49] <Bytram> sodium and mags
[14:34:51] <SemperOss> It beign nginx
[14:34:54] <Bytram> I'm on sodium atm
[14:35:08] <SemperOss> what does /var/log/syslog say
[14:35:53] <Bytram> permission denied
[14:35:55] <Bytram> hold on
[14:36:43] <SemperOss> Next one to look at is /var/log/nginx/error.log
[14:38:17] <Bytram> I don't see anything that strikes me as being an error or warning in /var/log/syslog
[14:38:58] <Bytram> oh? Now *that* is more interesting.
[14:39:27] <Bytram> root@sodium:~# cat /var/log/nginx/error.log
[14:39:28] <Bytram> 2018/08/19 09:21:21 [error] 821#0: OCSP response not successful (6: unauthorized) while requesting certificate status, responder: ocsp.usertrust.com
[14:39:28] <Bytram> 2018/08/19 12:40:23 [error] 822#0: OCSP response not successful (6: unauthorized) while requesting certificate status, responder: ocsp.usertrust.com
[14:39:28] <Bytram> 2018/08/19 12:55:40 [error] 822#0: OCSP response not successful (6: unauthorized) while requesting certificate status, responder: ocsp.usertrust.com
[14:39:29] <Bytram> 2018/08/19 14:00:59 [error] 822#0: OCSP response not successful (6: unauthorized) while requesting certificate status, responder: ocsp.usertrust.com
[14:39:32] <Bytram> root@sodium:~#
[14:41:16] <Bytram> SemperOss: do you use let's encrypt at all?
[14:41:21] <SemperOss> Why would it do that occasionally only? And it does not seem to explain the results we are seeing
[14:41:28] <SemperOss> I do
[14:41:42] -!- SemperOss [SemperOss!~torben@Soylent/Staff/Sysop/SemperOSS] has parted #dev
[14:42:05] -!- SemperOss [SemperOss!~torben@Soylent/Staff/Sysop/SemperOSS] has joined #dev
[14:42:19] <Bytram> SemperOss: (1) no idea, (2) agreed, and (3) but it probably isn't helping any, either
[14:42:36] <SemperOss> Hit the wrong key and lost the thread :-(
[14:42:45] <Bytram> ouchy
[14:42:47] <Bytram> https://crt.sh
[14:42:53] <Bytram> https://crt.sh
[14:43:15] <Bytram> as far as I can tell, that should have us covered for the site.
[14:43:48] <Bytram> and this should handle the rest of things:
[14:43:49] <Bytram> https://crt.sh
[14:43:53] <Bytram> https://crt.sh
[14:44:05] <Bytram> .op
[14:44:06] -!- mode/#dev [+o Bytram] by Hera
[14:44:30] <Bytram> .deop
[14:44:30] -!- mode/#dev [-o Bytram] by Hera
[14:44:53] <SemperOss> Oh, you're using the new wildcard certificates. Cool!
[14:45:23] <Bytram> =)
[14:45:46] <Bytram> yep, blame TMB for that one, IIRC
[14:46:14] <Bytram> okay, while I am on sodium, is there anyplace else you would like me to look?
[14:46:37] <SemperOss> I had to set up a truckload of names for mine ... wildcards are sooooo much easier, so good!
[14:46:44] <Bytram> yuppers
[14:46:59] <SemperOss> But, but, but ... This is NOT a certificate problem
[14:47:28] <Bytram> understood, that is why I asked if there was anyplace else you would like me to look on sodium?
[14:48:26] <SemperOss> Are the no other error messages in the nginx log file?
[14:48:33] <SemperOss> s/the/there/
[14:48:45] <Bytram> that was the entire file
[14:49:44] <SemperOss> That is very strange. Could you plz try error.log.1 too
[14:49:54] <Bytram> same path?
[14:49:58] <SemperOss> Yes
[14:51:00] <Bytram> just more OCSP stuff
[14:51:32] <SemperOss> Then it is either 1) not nginx or 2) not the server involved
[14:51:38] <Bytram> k
[14:51:46] <Bytram> off to magnesium, then?
[14:51:54] <SemperOss> I tend to believe 2) as the 50x errors normally come from nginx
[14:52:11] <SemperOss> So, yes
[14:53:35] <Bytram> root@magnesium log # tail -40 /var/log/syslog
[14:53:35] <Bytram> tail: cannot open '/var/log/syslog' for reading: No such file or directory
[14:53:35] <Bytram> root@magnesium log #
[14:53:47] <Bytram> nature calls... back in a few minutes
[14:54:09] <SemperOss> Is that a Gentoo machine?
[14:54:32] <SemperOss> If, you might have to try /var/log/messages instead
[14:57:53] <Bytram> back
[14:58:05] <Bytram> root@magnesium log # uname -a
[14:58:05] <Bytram> Linux magnesium.li694-22 4.9.7-x86_64-linode80 #2 SMP Thu Feb 2 15:43:55 EST 2017 x86_64 Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz GenuineIntel GNU/Linux
[14:58:05] <Bytram> root@magnesium log #
[14:59:20] <chromas> Is there an /etc/os-release or something similar?
[15:00:00] <Bytram> gentoo
[15:00:03] <SemperOss> On Debian/Ubuntu it often says it in /etc/issue (i.e. cat /etc/issue)
[15:00:19] <Bytram> root@magnesium log # cat /etc/os-release
[15:00:19] <Bytram> NAME=Gentoo
[15:00:19] <Bytram> ID=gentoo
[15:00:19] <Bytram> PRETTY_NAME="Gentoo/Linux"
[15:00:19] <Bytram> ANSI_COLOR="1;32"
[15:00:20] <Bytram> HOME_URL="https://www.gentoo.org/"
[15:00:22] <Bytram> SUPPORT_URL="https://www.gentoo.org/support/"
[15:00:24] <Bytram> BUG_REPORT_URL="https://bugs.gentoo.org/"
[15:00:28] <Bytram> root@magnesium log #
[15:00:47] <SemperOss> Then it is /var/log/messages, I think (haven't really done Gentoo for a while)
[15:01:05] <Bytram> yep, there is such a file, but it contains 0 bytes
[15:01:13] <SemperOss> And /var/log/nginx/error.log too
[15:01:21] <Bytram> -rw-r--r-- 1 root adm 106448 Aug 15 18:13 messages-20180819
[15:01:22] <SemperOss> And OH!
[15:02:21] <SemperOss> Then try messages-20180819 too
[15:02:48] <SemperOss> Nice wild goose chase we're on here
[15:02:55] <chromas> systemd would solve this issue :D
[15:02:58] <Bytram> wait a sec... just got to nginx dir
[15:03:00] <Bytram> root@magnesium nginx # ls -ltr | cat
[15:03:02] <Bytram> total 7934492
[15:03:04] <Bytram> -rw-r--r-- 1 root root 3262 Aug 14 2017 error_log-20170815
[15:03:06] <Bytram> -rw-r--r-- 1 nginx root 227 Oct 23 2017 error_log-20171023
[15:03:08] <Bytram> -rw-r--r-- 1 nginx root 508 Jul 1 01:27 error_log-20180701
[15:03:10] <Bytram> -rw-r--r-- 1 nginx root 0 Jul 1 03:10 error_log
[15:03:12] <Bytram> -rw-r--r-- 1 nginx root 253952 Aug 19 15:02 error.log
[15:03:14] <Bytram> -rw-r--r-- 1 nginx root 8124551168 Aug 19 15:02 access.log
[15:03:16] <Bytram> root@magnesium nginx #
[15:03:22] <Bytram> an 8GB access.log file?
[15:03:29] <Bytram> did I read that right?
[15:03:57] <SemperOss> Yup
[15:04:08] <SemperOss> Soylent News is popular
[15:04:17] <Bytram> apparently so!
[15:05:04] <Bytram> ahhh, /dev/root is full !!?!
[15:05:16] <Bytram> root@magnesium nginx # df -k
[15:05:16] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[15:05:16] <Bytram> devtmpfs 10240 0 10240 0% /dev
[15:05:16] <Bytram> tmpfs 204492 384 204108 1% /run
[15:05:17] <Bytram> shm 1022456 0 1022456 0% /dev/shm
[15:05:17] <SemperOss> Gulp!
[15:05:19] <Bytram> cgroup_root 10240 0 10240 0% /sys/fs/cgroup
[15:05:21] <Bytram> root@magnesium nginx # du -k
[15:05:23] <Bytram> 7934496 .
[15:05:25] <Bytram> root@magnesium nginx #
[15:05:46] <Bytram> I think that just might be factor in this case.
[15:05:48] <SemperOss> Where's / ?
[15:06:04] <Bytram> umm, at the top?
[15:06:06] <Bytram> ;)
[15:06:19] <SemperOss> In your df
[15:06:24] <SemperOss> ;-)
[15:06:46] <Bytram> funny, looks like it did not paste?
[15:06:49] * Bytram tries again
[15:07:20] <Bytram> root@magnesium nginx # df -k
[15:07:20] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[15:07:20] <Bytram> devtmpfs 10240 0 10240 0% /dev
[15:07:20] <Bytram> tmpfs 204492 384 204108 1% /run
[15:07:21] <Bytram> shm 1022456 0 1022456 0% /dev/shm
[15:07:23] <Bytram> cgroup_root 10240 0 10240 0% /sys/fs/cgroup
[15:07:25] <Bytram> root@magnesium nginx # du -k
[15:07:29] <Bytram> 7934496 .
[15:07:31] <SemperOss> No /
[15:07:31] <Bytram> root@magnesium nginx #
[15:07:53] <Bytram> no, not true... looks like the '/' is being treated as an escape char
[15:08:15] <SemperOss> Like in /join
[15:08:39] <Bytram> I swapped / with \
[15:08:40] <Bytram> root@magnesium nginx # df -k
[15:08:40] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[15:08:40] <Bytram> \dev\root 24189340 23927000 0 100% \
[15:08:40] <Bytram> devtmpfs 10240 0 10240 0% \dev
[15:08:40] <Bytram> tmpfs 204492 384 204108 1% \run
[15:08:42] <Bytram> shm 1022456 0 1022456 0% \dev\shm
[15:08:44] <Bytram> cgroup_root 10240 0 10240 0% \sys\fs\cgroup
[15:08:46] <Bytram> root@magnesium nginx # du -k
[15:08:48] <Bytram> 7934496 .
[15:08:50] <Bytram> root@magnesium nginx #
[15:08:52] <Bytram> that's better
[15:08:57] <chromas> You can escape the first / with another /
[15:09:27] <Bytram> global replace in an emacs buffer was easier
[15:09:37] <chromas> It's probably full of tmb's pr0n stash
[15:09:44] <Bytram> lol
[15:09:49] <chromas> not to be confused with a pr0n stache
[15:09:55] <SemperOss> Emacs ... SemperOSS runs away screaming
[15:09:59] <Bytram> or a must ash?
[15:10:08] <Bytram> ROFLMAO!
[15:10:13] * Bytram was waiting for that
[15:10:46] <Bytram> so.... any suggestions on a good place to look for a file that I can trim / remove
[15:10:47] <Bytram> ?
[15:10:59] <chromas> access.log
[15:11:15] <SemperOss> OK, we've got 24GB disk and 8GB access log ... We have to move that somewhere safe
[15:11:37] <chromas> I thought that got nuked by a cron job
[15:11:38] <Bytram> yes, nuke it is good, but should save it somewhere first, agreed.
[15:11:42] <chromas> or was supposed to
[15:12:18] <Bytram> fwiw:
[15:12:19] <Bytram> root@magnesium nginx # head access.log
[15:12:19] <Bytram> 164.52.7.132 - - [20/Jun/2017:08:00:52 +0000] "\x16\x03\x01\x01\x22\x01\x00\x01\x1E\x03\x03)\xAB\xC3\xAEa\xD3\x06E{\xE1i\xA1\xE5\x97\xB4a\x02\xEC\xB6\xD1\x83\xBE_\xBE\xF1\xD9|\x99" 400 173 "-" "-"
[15:12:19] <Bytram> 164.52.7.132 - - [20/Jun/2017:08:00:58 +0000] "USER test +iw test :Test Wuz Here" 400 173 "-" "-"
[15:12:19] <Bytram> 164.52.7.132 - - [20/Jun/2017:08:00:59 +0000] "GET / HTTP/1.1" 301 185 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
[15:12:21] <Bytram> 74.82.47.4 - - [20/Jun/2017:08:32:04 +0000] "GET / HTTP/1.1" 200 76594 "-" "-"
[15:12:51] <Bytram> so, looks like there's about 2 months' worth of logs in there.
[15:13:07] <chromas> 2017
[15:13:21] <Bytram> huh? Oh? Oh! OMG!
[15:13:34] <chromas> Not so popular after all :(
[15:13:48] <SemperOss> Methinks that Gentoo does not automatically perform log rotation, eh?
[15:13:58] * Bytram starts furiously hitting reload on the homepage in his browser
[15:14:06] <Bytram> sure looks like that!
[15:14:07] <SemperOss> LOL
[15:14:24] <chromas> Is there a reverse trunc?
[15:14:34] <Bytram> or, when they set it up, forgot to set up a cron job or something to make it happen
[15:14:42] <Bytram> ask an elephant
[15:15:23] <SemperOss> With this amount of disk space we can't even compress the file
[15:15:41] <chromas> I guess it's truncate. Missed opportunity for command abbrev
[15:15:44] <Bytram> yep
[15:16:07] <Bytram> I have an idea... hold on
[15:16:09] <chromas> move to another box?
[15:16:21] <SemperOss> That would be my suggestion
[15:16:21] <chromas> you can rsync and compress at the same time
[15:16:40] <Bytram> yep, but which one is the question... I'm gonna load the Linode dashboard and see which box has the largest drive
[15:16:43] <Bytram> *drives
[15:16:50] <chromas> Store it on the cloud
[15:16:57] <SemperOss> boron has got 17G free
[15:17:15] <Bytram> that should do it...
[15:17:31] <Bytram> now, how to copy it? sftp?
[15:17:40] <SemperOss> rsync
[15:18:06] <Bytram> understand the concept, have not used it personally
[15:18:24] <Bytram> goona need an explicit command line, and I would like to practice with a smaller file, first.
[15:18:52] <chromas> rsync file user@host:/path/to/file
[15:18:59] <SemperOss> rsync -Av /var/log/nginx/access.log boron:/tmp
[15:19:46] <Bytram> and Where am I when I issue that command? (1) server (2) path (3) and I assume as user root?
[15:20:38] <SemperOss> You probably need to make the file readable by anyone, then do it as yourself on the server
[15:20:56] <chromas> /tmp isn't a ramdisk on boron is it?
[15:21:08] <Bytram> no eye deer
[15:21:09] <SemperOss> But please hold the keys, just checking an rsync parameter for restart
[15:21:15] <Bytram> nod nod
[15:21:24] <SemperOss> And no, it's the main disk ... Have checked
[15:21:25] <Bytram> let's practice with a small[er] file, first
[15:22:16] <SemperOss> rsync -Av --partial /var/log/nginx/access.log boron:/tmp
[15:22:31] <SemperOss> Good idea with smaller file
[15:23:03] <SemperOss> And we need to stop it from being updated while we transfer, so the following commands:
[15:23:21] <SemperOss> cd /var/log/nginx # Just to save some typing
[15:23:24] <Bytram> ooooh, good catch!
[15:23:29] <Bytram> k
[15:23:56] <SemperOss> mv access.log access.log.old; touch access.log # As root
[15:24:23] <SemperOss> # Just guessing but ...
[15:24:27] <Bytram> and, to confirm, I do that on magnesium. correct?
[15:24:32] <SemperOss> /etc/init.d/nginx restart
[15:24:44] <SemperOss> Oh yes, magnesium
[15:24:49] <Bytram> k
[15:25:09] <SemperOss> Could you plz check that /etc/init.d/nginx exists
[15:25:19] <SemperOss> ls will do
[15:25:41] <Bytram> mv access.log access.20180819a.log ; touch access.log
[15:26:17] <Bytram> ^^^ is the command I will do on magnesium
[15:26:18] <Bytram> root@magnesium nginx # ls -l /etc/init.d/nginx
[15:26:18] <Bytram> -rwxr-xr-x 1 root root 2192 Apr 17 23:22 /etc/init.d/nginx
[15:26:18] <Bytram> root@magnesium nginx #
[15:26:30] <SemperOss> Looks very right
[15:26:41] <SemperOss> Proceed
[15:27:39] <Bytram> so, that command is with the honking big file...
[15:28:02] <Bytram> is there a smaller, practice file you might recommend?
[15:28:19] <SemperOss> Before the mv command?
[15:28:55] <Bytram> umm,yes... gimme a minute
[15:30:11] <SemperOss> No, the mv command is actually safe AS IT IS WRITTEN BY YOU ABOVE!
[15:30:18] <SemperOss> Sorry for the shouting
[15:31:53] <Bytram> np
[15:31:56] <Bytram> .
[15:31:56] <Bytram> .
[15:31:58] <Bytram> # On magnesium:
[15:32:00] <Bytram> sudo -i
[15:32:02] <Bytram> cd /var/log/nginx
[15:32:04] <Bytram> mv access.log access.20180819a.log
[15:32:06] <Bytram> touch access.log
[15:32:08] <Bytram> sync -Av --partial /var/log/nginx/access.20180819a.log boron:/tmp
[15:32:10] <Bytram> .
[15:32:12] <Bytram> .
[15:32:20] <SemperOss> Not sync, rsync
[15:32:25] <Bytram> should that cover it for the real file?
[15:32:27] <Bytram> oops...
[15:32:38] <Bytram> copy pasta error
[15:32:49] <SemperOss> Lasagne?
[15:32:53] <Bytram> good catch!
[15:33:02] * Bytram *chuckles*
[15:33:09] <Bytram> you'll fit in here just fine!
[15:33:17] <SemperOss> Thx
[15:33:46] <Bytram> so that ^^^^ will move the big file, just do it, or should I try with a small[er] file, first?
[15:34:17] <SemperOss> No, that will stop te file from being written to and then it will be copied ... and THEN we'll nuke it
[15:34:25] <Bytram> k
[15:34:39] <Bytram> squeak now or forever hold your grease!
[15:35:10] <Bytram> 3
[15:35:11] <Bytram> 2
[15:35:11] <SemperOss> It should be as safe as it can, provided there's no spleeing errors
[15:35:15] <Bytram> 1
[15:35:56] <Bytram> and... no joy.
[15:35:58] <Bytram> root@magnesium nginx # cd /var/log/nginx
[15:35:58] <Bytram> root@magnesium nginx # mv access.log access.20180819a.log
[15:36:00] <Bytram> root@magnesium nginx # touch access.log
[15:36:02] <Bytram> root@magnesium nginx # rsync -Av --partial /var/log/nginx/access.20180819a.log boron:/tmp
[15:36:04] <Bytram> rsync: ACLs are not supported on this client
[15:36:06] <Bytram> rsync error: syntax or usage error (code 1) at main.c(1578) [client=3.1.3]
[15:36:08] <Bytram> root@magnesium nginx #
[15:37:14] <SemperOss> OK, you'll have to move the file somewhere else within the same file system, like /tmp (try "df /tmp /var/log/nginx" to ensure they are on the same file system)
[15:37:56] <Bytram> k
[15:38:54] <Bytram> no luck on those... here's the skinny:
[15:39:01] <Bytram> root@magnesium nginx # df -a
[15:39:03] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[15:39:04] <SemperOss> Then you'll have to give yourself or everyone access to the file (yourself: chown bytram.bytram var/log/nginx/access.log boron:/tmp, everyone: chmod 666 var/log/nginx/access.log boron:/tmp)
[15:39:05] <Bytram> devtmpfs 10240 0 10240 0% /dev
[15:39:07] <Bytram> proc 0 0 0 - /proc
[15:39:09] <Bytram> tmpfs 204492 384 204108 1% /run
[15:39:11] <Bytram> mqueue 0 0 0 - /dev/mqueue
[15:39:13] <Bytram> devpts 0 0 0 - /dev/pts
[15:39:15] <Bytram> shm 1022456 0 1022456 0% /dev/shm
[15:39:17] <Bytram> sysfs 0 0 0 - /sys
[15:39:19] <Bytram> debugfs 0 0 0 - /sys/kernel/debug
[15:39:21] <Bytram> configfs 0 0 0 - /sys/kernel/config
[15:39:23] <Bytram> fusectl 0 0 0 - /sys/fs/fuse/connections
[15:39:25] <Bytram> cgroup_root 10240 0 10240 0% /sys/fs/cgroup
[15:39:29] <Bytram> openrc 0 0 0 - /sys/fs/cgroup/openrc
[15:39:31] <Bytram> none 0 0 0 - /sys/fs/cgroup/unified
[15:39:33] <Bytram> cpuset 0 0 0 - /sys/fs/cgroup/cpuset
[15:39:35] <Bytram> cpu 0 0 0 - /sys/fs/cgroup/cpu
[15:39:37] <Bytram> cpuacct 0 0 0 - /sys/fs/cgroup/cpuacct
[15:39:39] <Bytram> blkio 0 0 0 - /sys/fs/cgroup/blkio
[15:39:41] <Bytram> memory 0 0 0 - /sys/fs/cgroup/memory
[15:39:43] <Bytram> devices 0 0 0 - /sys/fs/cgroup/devices
[15:39:45] <Bytram> freezer 0 0 0 - /sys/fs/cgroup/freezer
[15:39:47] <Bytram> net_cls 0 0 0 - /sys/fs/cgroup/net_cls
[15:39:49] <Bytram> perf_event 0 0 0 - /sys/fs/cgroup/perf_event
[15:39:51] <Bytram> net_prio 0 0 0 - /sys/fs/cgroup/net_prio
[15:39:53] <Bytram> pids 0 0 0 - /sys/fs/cgroup/pids
[15:39:55] <Bytram> debug 0 0 0 - /sys/fs/cgroup/debug
[15:39:59] <Bytram> binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
[15:40:01] <Bytram> root@magnesium nginx #
[15:40:29] <SemperOss> Holy Spaghetti Monster. You sure that /tmp is not on the same filesystem as /var/log/nginx?
[15:40:39] <Bytram> no idea
[15:41:13] <SemperOss> df /tmp /var/log/nginx
[15:41:20] <SemperOss> Should give you an idea
[15:41:29] <Bytram> root@magnesium nginx # df /tmp /var/log/nginx
[15:41:30] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[15:41:30] <Bytram> root@magnesium nginx #
[15:41:36] <Bytram> ugh
[15:41:55] <SemperOss> You need to escape again, methinks
[15:41:55] <Bytram> root@magnesium nginx # df \tmp \var\log\nginx
[15:41:55] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[15:41:55] <Bytram> \dev\root 24189340 23927000 0 100% \
[15:41:55] <Bytram> \dev\root 24189340 23927000 0 100% \
[15:41:55] <Bytram> root@magnesium nginx #
[15:42:07] <Bytram> nod nod
[15:42:19] <SemperOss> They are! Brilliant!
[15:42:44] <Bytram> I'm glad YOU are excited!
[15:42:47] <SemperOss> mv /var/log/nging/access.20180819a.log /tmp
[15:43:10] <SemperOss> cd /tmp
[15:43:18] <SemperOss> chmod 666 access.20180819a.log
[15:43:42] <SemperOss> Now you can retrieve it from somewhere else like boron
[15:43:49] <Bytram> hold on pardner
[15:44:09] <SemperOss> Hoooooooooooollllllllllllllllllllllllllldddddddddddddddddddiiiiiiiiiiiiiiiiiiiiinnnnnnnnnnnnnnnnnnggggggggggggggggggg
[15:44:10] <Bytram> oh, there was a typo
[15:44:21] <SemperOss> Yes, I see it
[15:44:24] <SemperOss> nginx
[15:44:28] <Bytram> yuppp
[15:44:49] <Bytram> okay, *that* worked!
[15:44:52] <SemperOss> I'll get my next commands peer reviewed
[15:45:04] <Bytram> mv /var/log/nginx/access.20180819a.log /tmp
[15:45:19] <SemperOss> I'll be back in a couple of months then ...
[15:45:19] <Bytram> ^^ is what worked so I can refer back to it in this log in the future.
[15:45:34] <Bytram> cd /tmp
[15:45:46] <Bytram> chmod 666 access.20180819a.log
[15:45:59] <Bytram> Yays! that worked!
[15:46:14] <SemperOss> Right then, back to boron
[15:46:14] <Bytram> and, now for the rsync?
[15:46:21] <SemperOss> From boron
[15:46:41] <Bytram> oh, wait... I should go back to boron and leave magnesium?
[15:46:45] <Bytram> k
[15:47:00] <SemperOss> Yup, that's the idea
[15:47:07] <Bytram> okay... back to boron
[15:47:13] <SemperOss> rsync -Av magnesium:/tmp/access.20180819a.log /tmp
[15:47:26] <SemperOss> Now you're pulling it instead of pushing it
[15:47:35] <Bytram> now that we 666'd it, I can do this as a regular user... correct?
[15:47:39] <Bytram> k
[15:47:46] <SemperOss> Yes
[15:47:49] <Bytram> k
[15:48:09] <Bytram> martyb@boron:~/tmp$ cd /tmp
[15:48:10] <Bytram> martyb@boron:/tmp$ df -k .
[15:48:10] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[15:48:10] <Bytram> martyb@boron:/tmp$
[15:48:15] <Bytram> ugh, again
[15:48:43] <SemperOss> IRC is not made for Linux/UNIX administration
[15:48:55] <Bytram> orly?
[15:48:56] <Bytram> martyb@boron:~\tmp$ cd \tmp
[15:48:56] <Bytram> martyb@boron:\tmp$ df -k .
[15:48:56] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[15:48:58] <Bytram> \dev\sda 49056608 29528348 17038204 64% \
[15:49:00] <Bytram> martyb@boron:\tmp$
[15:49:10] <Bytram> that looks MUCH better
[15:49:20] <SemperOss> A sight for sore eyes
[15:49:28] <Bytram> rsync -Av magnesium:/tmp/access.20180819a.log /tmp
[15:50:04] <Bytram> and yet another ugh.
[15:50:05] <Bytram> martyb@boron:/tmp$ rsync -Av magnesium:/tmp/access.20180819a.log /tmp
[15:50:05] <Bytram> rsync: ACLs are not supported on this server
[15:50:05] <Bytram> rsync error: syntax or usage error (code 1) at main.c(1578) [server=3.1.3]
[15:50:05] <Bytram> rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
[15:50:06] <Bytram> rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.0]
[15:50:08] <Bytram> martyb@boron:/tmp$
[15:50:49] <SemperOss> Boooohooooo
[15:50:55] <Bytram> yep
[15:51:08] <SemperOss> OK, scp then
[15:51:21] <Bytram> so ACLs are seperate and distinct from chomd / chown
[15:51:22] <Bytram> k
[15:52:12] <SemperOss> scp magnesium:/tmp/access.20180819a.log /tmp
[15:52:17] <Bytram> SemperOss: In case you hadn't noticed, my professional background is in software QA/test
[15:52:40] <Bytram> ~woop
[15:52:42] <SemperOss> Mine is in ... everything, sort of :-)
[15:52:43] <Bytram> !woop
[15:52:43] <Bender> woop woop woop (\/) (;,,;) (\/)
[15:53:02] <Bytram> 15% done already
[15:53:06] <Bytram> 20
[15:53:16] <SemperOss> Looks terribly good
[15:53:28] <Bytram> ETA: 01"30
[15:53:35] <Bytram> 40%
[15:53:43] <Bytram> =)
[15:53:49] <Bytram> 50%
[15:53:59] <Bytram> ETA: 01:00
[15:54:04] <Bytram> 60%
[15:54:14] <Bytram> 67%
[15:54:16] <SemperOss> Bytram: I'm currently working as an IT Architect but my background is engineering, software development, computer science and business administration
[15:54:37] <Bytram> VERY nice mix!
[15:54:48] * Bytram has done some software dev, too. btw
[15:55:01] <Bytram> 100%
[15:55:08] <Bytram> Yay!!!
[15:55:09] <SemperOss> Yeah!!!!
[15:55:26] <SemperOss> Time for an integrity check
[15:55:28] <Bytram> now to go back and to delete the fatberg? right?
[15:55:35] <Bytram> oh. Yees!
[15:55:45] <Bytram> md5 hash?
[15:55:46] <SemperOss> md5sum /tmp/access.20180819a.log
[15:55:50] <Bytram> =)
[15:55:52] <SemperOss> So, yes
[15:56:23] <SemperOss> And on magnesium too, of course
[15:56:43] <Bytram> yep
[15:56:56] <Bytram> martyb@boron:/tmp$ md5sum /tmp/access.20180819a.log
[15:56:56] <Bytram> 3d58c614050d4412568540a14ee1b939 /tmp/access.20180819a.log
[15:56:56] <Bytram> martyb@boron:/tmp$
[15:58:58] <Bytram> martyb@magnesium:/tmp$ md5sum /tmp/access.20180819a.log
[15:59:00] <Bytram> 3d58c614050d4412568540a14ee1b939 /tmp/access.20180819a.log
[15:59:02] <Bytram> martyb@magnesium:/tmp$
[15:59:09] <Bytram> looks like success to me.
[15:59:11] <SemperOss> OK, the same
[15:59:14] <Bytram> nod nod
[15:59:41] <SemperOss> Now to the unpleasant rm command ... do NOT get it worng ;-)
[15:59:57] <Bytram> Yeah, I'm pretty careful about issuing nuclear commands
[16:00:07] <SemperOss> rm /tmp/access.20180819a.log ##### On MAGNESIUM
[16:00:30] <Bytram> martyb@magnesium:/tmp$ pwd
[16:00:31] <Bytram> martyb@magnesium:/tmp$
[16:00:53] <SemperOss> And, and, and ...
[16:01:25] <Bytram> martyb@magnesium:/tmp$ rm /tmp/access.20180819a.log
[16:01:25] <Bytram> rm: cannot remove '/tmp/access.20180819a.log': Operation not permitted
[16:01:25] <Bytram> martyb@magnesium:/tmp$
[16:01:32] * Bytram needs to sudo, I think
[16:01:48] <Bytram> martyb@magnesium:/tmp$ ls -ltr
[16:01:48] <Bytram> total 7934232
[16:01:48] <Bytram> -rw------- 1 bob firefighters 2438 Jun 30 19:13 tmpijtyd8lf
[16:01:48] <Bytram> -rw-rw-rw- 1 nginx root 8124551168 Aug 19 16:00 access.20180819a.log
[16:01:48] <Bytram> martyb@magnesium:/tmp$
[16:01:55] <Bytram> yep, owner is root
[16:01:59] <Bytram> right?
[16:02:06] <SemperOss> Sounds right
[16:02:08] <Bytram> k
[16:02:31] <SemperOss> You may still have a problem if the ACL is set to something unpleasant, but try it
[16:02:52] <Bytram> !woop
[16:02:52] <Bender> woop woop woop (\/) (;,,;) (\/)
[16:03:08] <Bytram> martyb@magnesium:/tmp$ sudo -i
[16:03:09] <Bytram> root@magnesium ~ # cd /tmp
[16:03:09] <Bytram> root@magnesium tmp # ls -ltr /tmp/access.20180819a.log
[16:03:09] <Bytram> -rw-rw-rw- 1 nginx root 8124551168 Aug 19 16:02 /tmp/access.20180819a.log
[16:03:09] <Bytram> root@magnesium tmp # rm access.20180819a.log
[16:03:10] <Bytram> root@magnesium tmp # ls-ltr
[16:03:10] <SemperOss> G-R-E-A-T-!-!-!-!
[16:03:12] <Bytram> -bash: ls-ltr: command not found
[16:03:14] <Bytram> root@magnesium tmp # ls -ltr
[16:03:16] <Bytram> total 4
[16:03:18] <Bytram> -rw------- 1 bob firefighters 2438 Jun 30 19:13 tmpijtyd8lf
[16:03:20] <Bytram> root@magnesium tmp #
[16:03:22] <Bytram> SemperOss++
[16:03:22] <Bender> karma - sempeross: 1
[16:03:29] <Bytram> root@magnesium tmp #
[16:03:32] <Bytram> ugh
[16:03:36] <Bytram> SemperOss++
[16:03:36] <Bender> karma - sempeross: 2
[16:03:38] <Bytram> SemperOss++
[16:03:38] <Bender> karma - sempeross: 3
[16:03:38] <Bytram> SemperOss++
[16:03:38] <Bender> karma - sempeross: 4
[16:03:44] <SemperOss> Thx!
[16:03:52] <Bytram> chromas++ thanks for your help, too!
[16:03:52] <Bender> karma - chromas: 8
[16:03:54] <Bytram> chromas++ thanks for your help, too!
[16:03:54] <Bender> karma - chromas: 9
[16:03:55] <SemperOss> What does df say now?
[16:03:55] <Bytram> chromas++ thanks for your help, too!
[16:03:55] <Bender> karma - chromas: 10
[16:04:12] <SemperOss> Bytram++
[16:04:12] <Bender> karma - bytram: 1
[16:04:13] <Bytram> oh magnesium?
[16:04:17] <SemperOss> Yes
[16:04:27] <SemperOss> Bytram++
[16:04:27] <Bender> karma - bytram: 2
[16:04:28] <SemperOss> Bytram++
[16:04:28] <Bender> karma - bytram: 3
[16:04:30] <SemperOss> Bytram++
[16:04:30] <Bender> karma - bytram: 4
[16:04:31] <SemperOss> Bytram++
[16:04:31] <Bender> karma - bytram: 5
[16:04:34] <Bytram> urk?
[16:04:36] <SemperOss> You did all the hard work
[16:04:49] <Bytram> prepare for a paste
[16:04:50] <Bytram> root@magnesium tmp # df -a
[16:04:50] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[16:04:50] <Bytram> devtmpfs 10240 0 10240 0% /dev
[16:04:50] <Bytram> proc 0 0 0 - /proc
[16:04:52] <Bytram> tmpfs 204492 384 204108 1% /run
[16:04:54] <Bytram> mqueue 0 0 0 - /dev/mqueue
[16:04:58] <Bytram> devpts 0 0 0 - /dev/pts
[16:05:00] <Bytram> shm 1022456 0 1022456 0% /dev/shm
[16:05:02] <Bytram> sysfs 0 0 0 - /sys
[16:05:04] <Bytram> debugfs 0 0 0 - /sys/kernel/debug
[16:05:06] <Bytram> configfs 0 0 0 - /sys/kernel/config
[16:05:08] <Bytram> fusectl 0 0 0 - /sys/fs/fuse/connections
[16:05:09] <SemperOss> Escape clauses to be inserted here!
[16:05:10] <Bytram> cgroup_root 10240 0 10240 0% /sys/fs/cgroup
[16:05:12] <Bytram> openrc 0 0 0 - /sys/fs/cgroup/openrc
[16:05:14] <Bytram> none 0 0 0 - /sys/fs/cgroup/unified
[16:05:16] <Bytram> cpuset 0 0 0 - /sys/fs/cgroup/cpuset
[16:05:18] <Bytram> cpu 0 0 0 - /sys/fs/cgroup/cpu
[16:05:20] <Bytram> cpuacct 0 0 0 - /sys/fs/cgroup/cpuacct
[16:05:22] <Bytram> blkio 0 0 0 - /sys/fs/cgroup/blkio
[16:05:24] <Bytram> memory 0 0 0 - /sys/fs/cgroup/memory
[16:05:28] <Bytram> devices 0 0 0 - /sys/fs/cgroup/devices
[16:05:30] <Bytram> freezer 0 0 0 - /sys/fs/cgroup/freezer
[16:05:32] <Bytram> net_cls 0 0 0 - /sys/fs/cgroup/net_cls
[16:05:34] <Bytram> perf_event 0 0 0 - /sys/fs/cgroup/perf_event
[16:05:36] <Bytram> net_prio 0 0 0 - /sys/fs/cgroup/net_prio
[16:05:38] <Bytram> pids 0 0 0 - /sys/fs/cgroup/pids
[16:05:40] <Bytram> debug 0 0 0 - /sys/fs/cgroup/debug
[16:05:42] <Bytram> binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
[16:05:44] <Bytram> root@magnesium tmp #
[16:05:46] <Bytram> .
[16:05:48] <Bytram> ugh... not AGAIN!
[16:05:50] <Bytram> let me out of here!
[16:05:54] <Bytram> here it comes....
[16:05:58] <Bytram> root@magnesium tmp # df -a
[16:06:00] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[16:06:02] <Bytram> \dev\root 24189340 23927008 0 100% \
[16:06:04] <Bytram> devtmpfs 10240 0 10240 0% \dev
[16:06:06] <Bytram> proc 0 0 0 - \proc
[16:06:08] <Bytram> tmpfs 204492 384 204108 1% \run
[16:06:10] <Bytram> mqueue 0 0 0 - \dev\mqueue
[16:06:12] <Bytram> devpts 0 0 0 - \dev\pts
[16:06:14] <Bytram> shm 1022456 0 1022456 0% \dev\shm
[16:06:16] <Bytram> sysfs 0 0 0 - \sys
[16:06:18] <Bytram> debugfs 0 0 0 - \sys\kernel\debug
[16:06:20] <Bytram> configfs 0 0 0 - \sys\kernel\config
[16:06:22] <Bytram> fusectl 0 0 0 - \sys\fs\fuse\connections
[16:06:24] <Bytram> cgroup_root 10240 0 10240 0% \sys\fs\cgroup
[16:06:28] <Bytram> openrc 0 0 0 - \sys\fs\cgroup\openrc
[16:06:30] <Bytram> none 0 0 0 - \sys\fs\cgroup\unified
[16:06:32] <Bytram> cpuset 0 0 0 - \sys\fs\cgroup\cpuset
[16:06:34] <Bytram> cpu 0 0 0 - \sys\fs\cgroup\cpu
[16:06:36] <Bytram> cpuacct 0 0 0 - \sys\fs\cgroup\cpuacct
[16:06:38] <SemperOss> And the 8GB are ... where?????
[16:06:38] <Bytram> blkio 0 0 0 - \sys\fs\cgroup\blkio
[16:06:40] <Bytram> memory 0 0 0 - \sys\fs\cgroup\memory
[16:06:42] <Bytram> devices 0 0 0 - \sys\fs\cgroup\devices
[16:06:44] <Bytram> freezer 0 0 0 - \sys\fs\cgroup\freezer
[16:06:46] <Bytram> net_cls 0 0 0 - \sys\fs\cgroup\net_cls
[16:06:48] <Bytram> perf_event 0 0 0 - \sys\fs\cgroup\perf_event
[16:06:50] <Bytram> net_prio 0 0 0 - \sys\fs\cgroup\net_prio
[16:06:52] <Bytram> pids 0 0 0 - \sys\fs\cgroup\pids
[16:06:54] <Bytram> debug 0 0 0 - \sys\fs\cgroup\debug
[16:06:58] <Bytram> binfmt_misc 0 0 0 - \proc\sys\fs\binfmt_misc
[16:07:00] <Bytram> root@magnesium tmp #
[16:07:02] <Bytram> finis
[16:07:04] <Bytram> I dunno!!!
[16:08:05] <SemperOss> And it is definitely gone?
[16:08:48] <Bytram> yes.
[16:08:50] <SemperOss> Did you restart nginx
[16:08:52] <SemperOss> ?
[16:09:13] <Bytram> no
[16:09:32] <SemperOss> Ahh, so it has it claws in the file still, deleted or not.
[16:09:40] <SemperOss> /etc/init.d/nginx restart
[16:09:43] <Bytram> inodes
[16:09:44] <SemperOss> as root
[16:09:47] <Bytram> k
[16:09:51] <SemperOss> Yes, inodes
[16:10:24] <Bytram> root@magnesium tmp # /etc/init.d/nginx restart
[16:10:24] <Bytram> * Checking nginx' configuration ... [ ok ]
[16:10:24] <Bytram> * Stopping nginx ... [ ok ]
[16:10:24] <Bytram> * Starting nginx ... [ ok ]
[16:10:28] <Bytram> root@magnesium tmp #
[16:10:35] <SemperOss> And the df?
[16:11:04] <Bytram> root@magnesium tmp # df -k .
[16:11:05] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[16:11:05] <Bytram> \dev\root 24189340 15992788 7934200 67% \
[16:11:06] <Bytram> root@magnesium tmp #
[16:11:17] <SemperOss> I like that MUCH better!
[16:11:17] <Bytram> =)
[16:11:23] <Bytram> #metoo!
[16:12:01] <SemperOss> Hm, that reference seems to be quite outside my nerdy area of expertise ;-)
[16:12:12] * Bytram *chcuckles*
[16:12:23] <Bytram> ok, need a quick break... back in 5 minuites or so
[16:12:31] <SemperOss> Right
[16:20:40] <Bytram> teamwork++
[16:20:40] <Bender> karma - teamwork: 13
[16:20:43] <Bytram> teamwork++
[16:20:43] <Bender> karma - teamwork: 14
[16:22:52] <SemperOss> Bytram: Well done. It's nice to cooperate in a positive environment without tempers flaring
[16:23:00] <Bytram> chromas: now that we have freed up space, can you please check and see if you can reproduce your the problems you experienced?
[16:23:10] <Bytram> WHAT!!!?!
[16:23:12] <Bytram> =)
[16:23:25] <Bytram> agreed!
[16:24:20] <SemperOss> What about double-checking sodium?
[16:24:25] <Bytram> Also helps that I have documented so many test cases that I know the importance of *clearly* identifying the command to be issued and documenting the expected and actual results
[16:24:28] <Bytram> k
[16:24:42] * Bytram leaves magnesium
[16:24:51] <Bytram> ssh sodium
[16:25:44] <SemperOss> Bytram: I am actually working on automatising test for the Danish tax people
[16:26:05] <Bytram> .
[16:26:06] <Bytram> martyb@sodium:~$ df -k
[16:26:06] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[16:26:06] <Bytram> udev 1020704 4 1020700 1% \dev
[16:26:06] <Bytram> tmpfs 204988 352 204636 1% \run
[16:26:06] <Bytram> \dev\sda 48378736 1488668 46382404 4% \
[16:26:08] <Bytram> none 4 0 4 0% \sys\fs\cgroup
[16:26:10] <Bytram> none 5120 0 5120 0% \run\lock
[16:26:12] <Bytram> none 1024936 0 1024936 0% \run\shm
[16:26:14] <Bytram> none 102400 0 102400 0% \run\user
[16:26:16] <Bytram> martyb@sodium:~$
[16:26:36] <SemperOss> Doesn't seem to have any problems
[16:26:38] <Bytram> how much extra do they think they can charge for their pastries?
[16:26:41] <Bytram> ;)
[16:26:43] <SemperOss> ... with space
[16:26:44] <Bytram> agreed.
[16:26:54] <Bytram> ...the final frontier
[16:27:34] <SemperOss> ... to boldly go where nothing is ... er
[16:27:41] <Bytram> Individual or corporate tax? or both?
[16:28:17] <SemperOss> It is *the* tax authority ... they grab any- and everything
[16:28:41] * Bytram shudders to think of how many ancient code paths have become ossified over the years
[16:29:05] <Bytram> don't even *think* of trying to fix *that* -- here be dragons.
[16:29:11] <Bytram> amirite?
[16:29:15] <SemperOss> You have NO ideas.
[16:29:27] <Bytram> umm, I worked at IBM.
[16:29:45] <Bytram> in the 80s
[16:30:05] <Bytram> I tested code that came around with CP67
[16:30:12] <SemperOss> They have a piece of COBOL code that interfaces with a huge IBM system through some Java code and then the IBM system talks to a gargantuan SAP system
[16:30:28] <Bytram> *groan*
[16:30:33] * Bytram is not surprised.
[16:30:47] <Bytram> put a wrapper around it and treat it as a black box
[16:31:11] <SemperOss> They have decide to kill all the systems bar the SAP system
[16:31:16] <SemperOss> decided
[16:31:31] <Bytram> are you looking to streamline that, or... huh? OMG.
[16:31:47] <SemperOss> Yup
[16:31:54] <Bytram> I suspect some consultants are becoming quite wealthy
[16:32:07] <SemperOss> I guess so too
[16:32:36] <Bytram> overstating things a bit, but from I've heard SAP is inceedibly flexible, but you need a load of consultants to do the tweaks to make it happen.
[16:32:41] <SemperOss> I'm contracted in through a consultancy that supplies me to Deloitte that have the contract with the tax authority
[16:32:45] <Bytram> *incredibly
[16:32:59] <Bytram> have head of 'em
[16:33:02] <Bytram> *heard
[16:33:23] <SemperOss> SAP is a MONSTROSITY! (sorry, shouting again)
[16:33:39] <Bytram> turtles all the way down... in the code... and in the staffing.
[16:33:41] <SemperOss> SAP is fine as long you don't fiddle to much with it
[16:33:56] <Bytram> but then it doesn't quite do what you need, right?
[16:34:06] <Bytram> just one more thing... and therein lies madness.
[16:34:10] <SemperOss> They have 80+% changed codebase
[16:34:22] <Bytram> zounds!
[16:34:47] <Bytram> I *hope* they plan to run things in parallel so they can compare results of new vs old?
[16:35:07] <SemperOss> Just to keep the monster alive is impossible ... and they are about 6 or so updates behind as they haven't been able to find people to do it at a reasonable price
[16:35:29] <Bytram> "reasonable" price?
[16:36:01] <Bytram> from what you just wrote, they are already paying through the nose...
[16:36:08] <Bytram> that must be *insane*
[16:36:39] <SemperOss> Yup, multi million project and SAP is now coming after them, saying they will not allow them to keep running on this o-l-d version
[16:37:13] * Bytram notices his eye twitching
[16:37:38] <Bytram> #g the pink panther movie wikipedia
[16:37:39] <MrPlow> https://en.wikipedia.org(2006_film) - "The Pink Panther is a 2006 American comedy film and a reboot of The Pink Panther franchise, marking the tenth installment in the series. In this film, Inspector ..."
[16:37:59] <Bytram> nope, I meant the original one.
[16:38:30] <Bytram> https://en.wikipedia.org(1963_film)
[16:38:39] <SemperOss> Ah, Inspector Clousseau's boss
[16:38:44] <Bytram> bingo!
[16:38:47] <SemperOss> Twitching eye
[16:38:54] <Bytram> 'zactly!
[16:40:16] <SemperOss> SKAT (that's the Danish tax authority) are contemplating the new system as the old system is a merry mish-mash and not the least transaction oriented
[16:40:45] <Bytram> not transactional == no rollback?
[16:40:53] <Bytram> or...
[16:41:11] <SemperOss> Oh yes .................. manually
[16:41:19] <Bytram> hardcoded calls to various and sundry modules that have been spaghetti-ed together
[16:41:30] <Bytram> no. really?
[16:41:43] <SemperOss> Really!
[16:41:56] <Bytram> more eye twitches ensue!
[16:42:14] <SemperOss> I should think now is a good time to hyperventilate
[16:42:24] <SemperOss> Brown bag provided
[16:43:28] <Bytram> nod nod
[16:43:32] <SemperOss> Haven't got a clue as to get the *** Responses like your "***Bytram notices his eye twitching". IRC is sort of a late patch for me
[16:43:34] <Bytram> appreciated!
[16:44:07] <Bytram> type this: /me wrote this
[16:44:25] * SemperOss provides brown bag
[16:44:30] <SemperOss> Ah, thx
[16:44:34] <Fnord666> /Bytram wrote this
[16:44:46] <Bytram> Fnord666++ /me chuckles
[16:44:46] <Bender> karma - fnord666: 1
[16:45:11] <Bytram> SemperOss: this is the movie I ws thinking of: https://en.wikipedia.org
[16:45:26] <SemperOss> Living and learning ... hi Fnord666
[16:45:27] <Fnord666> howdy all. Didn't want to interrupt the debugging
[16:45:34] <Fnord666> howdy SemperOss
[16:45:45] <Bytram> Fnord666: np... You been following along?
[16:45:50] <Fnord666> I have
[16:46:00] <Fnord666> a little in backscroll then in RT
[16:46:09] <Bytram> one never knows when one has an audience
[16:46:13] <Bytram> nodnod
[16:46:40] <Fnord666> Well if it's logged, you always have an audience, even if its just in reruns
[16:47:13] * SemperOss puts on clothes again and looks surreptitiously around
[16:47:15] <Fnord666> Didn't have anything constructive to add so I just listened and nodded
[16:47:24] <Bytram> cut to the commercial
[16:47:53] <Bytram> Fnord666: how'd we do?
[16:48:15] <Fnord666> Looked good to me.
[16:48:33] <Bytram> mare see!
[16:48:35] <Fnord666> stuffed file systems are quite the bain of unixkind
[16:48:59] <Fnord666> pity the access file wasn't set up to logrotate
[16:49:02] <SemperOss> Yeah, and often difficult to solve
[16:49:13] <Fnord666> yup
[16:49:16] <Bytram> that reminds me...
[16:49:26] <Bytram> we have fixed ONE instance of a problem...
[16:49:34] <Fnord666> might want to have a look at all of the servers
[16:49:47] <Bytram> are there others of this ilk waiting in the wings, and what can/should bwe do to prevent a repeat?
[16:49:47] <SemperOss> When TMB comes back I'll get my credentials and get it set up
[16:49:52] <Fnord666> Lol. Ninja's . I was just typing that
[16:50:01] <Fnord666> Ninja'd
[16:50:01] <Bytram> teamwork++
[16:50:01] <Bender> karma - teamwork: 15
[16:50:09] <Bytram> (and stop looking over my shoulder!)
[16:50:13] <Bytram> =)
[16:50:14] <Fnord666> :)
[16:50:17] <SemperOss> teamwork++ agreed
[16:50:17] <Bender> karma - teamwork: 16
[16:51:23] <SemperOss> I think we have to concoct some more monitoring ... Bytram, you get something already but that's more like usage statistics and such, right?
[16:51:33] <Fnord666> SemperOss good catch about the files being on the same FS
[16:51:43] <Bytram> agreed!
[16:51:46] <SemperOss> Thx
[16:52:32] <SemperOss> Which reminds me, we need to move it out of /tmp on boron as a reboot could kill it now
[16:52:48] <Bytram> huh?
[16:52:50] <Fnord666> god point also
[16:52:51] <SemperOss> Bytram: could you move it to your homedir?
[16:52:57] <Fnord666> s/god/good/
[16:53:20] <Bytram> wait a bit... I *just* closed my terminal window.
[16:53:33] <SemperOss> Ubuntu likes to do a good cleanup of /tmp on reboots
[16:53:48] <SemperOss> Bytram: Sorry
[16:53:52] <SemperOss> Bytram++
[16:53:53] <Bender> karma - bytram: 6
[16:54:08] <Fnord666> SemperOss Are you a vi type of person? I noticed a slight spitting on the sidewalk at the mention of Emacs
[16:54:47] <SemperOss> I most certainly am. My first runin with emacs had me kill it from another terminal window ... Never looked back
[16:54:59] <SemperOss> :-D
[16:55:16] * Bytram has a long password, that is not written down anywhere, and needs to be keyed in twice... once to login, and once to kinit
[16:55:34] * Bytram is on boron
[16:55:38] <SemperOss> Keepas to the rescue
[16:55:45] <SemperOss> Keepass, that is
[16:55:50] <Fnord666> exactly
[16:56:04] <Fnord666> and yes, I'm firmly in the vi camp myself
[16:56:05] * Bytram don't mess with no dongles
[16:56:21] * Bytram has lost objects, but so far, not his mind.
[16:56:28] <SemperOss> No dongles necessary, just a good master password
[16:56:51] <SemperOss> I keep it open and locks it when I leave the screen
[16:56:52] <Bytram> okay... I'm here and am your hands... what do i gotta do?
[16:58:01] <SemperOss> If you haven't moved anywhere out of your homedir: mv /tmp/access.20180819a.log .
[16:58:16] <Bytram> hold on
[16:58:18] <Bytram> cd
[16:58:23] <SemperOss> The last full stop is not to end the sentence ;-)
[16:58:24] <Bytram> worng window :(
[16:58:32] <Bytram> nod nod
[16:59:03] <Bytram> don't I have to sudo, first?
[16:59:20] <SemperOss> No, it should be owned by you
[16:59:26] <Bytram> okaaay
[16:59:52] <Bytram> got i.
[16:59:55] <Bytram> got it.
[17:00:03] <SemperOss> Bytram++
[17:00:03] <Bender> karma - bytram: 7
[17:00:45] <Bytram> martyb@boron:~/tmp$ ls -l /home/martyb/tmp/access.20180819a.log
[17:00:45] <Bytram> -rw-r--r-- 1 martyb firefighters 8124551168 Aug 19 15:54 /home/martyb/tmp/access.20180819a.log
[17:00:45] <Bytram> martyb@boron:~/tmp$
[17:00:55] <SemperOss> I think that ends today's drama ... and now to a description by a man watching paint dry
[17:01:04] <Bytram> SemperOss: hold on a sec?
[17:01:20] <Bytram> what did we just accomplish?
[17:02:01] <Bytram> moved it to a different file system?
[17:02:06] <SemperOss> We moved it away from /tmp, which is cleaned up by Ubuntu on a reboot, meaning the file would have just ... vanished if anything untowards happened
[17:02:18] <Bytram> ahh, got it. thanks!
[17:02:39] <SemperOss> Un-poof!
[17:02:48] * Bytram does not have AC in his apartment and it is slowly getting warmer and warmer... making things a bit harder to concentrate on
[17:02:51] <Fnord666> stepping away for a bit. Back alter. Nice work gentlemen
[17:03:02] <Bytram> the magic un-dragon?
[17:03:09] <Fnord666> back later. Not a chiropractic visit.
[17:03:10] <SemperOss> Fnord666: Thank you
[17:03:24] <Bytram> yes, is time for me to be going about my day, as well.
[17:03:32] <SemperOss> Bytram: Absolutely!
[17:03:47] <Bytram> SemperOss: Truly enjoyed working with you... thanks for the painstaking efforts to guide me through the process!
[17:04:59] <SemperOss> Bytram: Thank YOU! It's been a pleasure not being sworn at for trying different possibilities and such. Nice!
[17:05:23] <SemperOss> And the family dinner has been brewing here, so I'll leave for now too
[17:05:40] <Bytram> okay, while I have a momentary second wind... oh, for sure!
[17:06:05] <SemperOss> See y'all laters
[17:06:09] * Bytram was going to suggest taking a look at the other server to see if its access log was about to... *crunch*
[17:06:31] <SemperOss> In half an hour, maybe?
[17:06:51] <Bytram> umm, k... which server and I can go rumagin around ahead of time
[17:07:35] <Bytram> looks like sodium is the other frontend proxy besides magnesium
[17:08:25] <Bytram> and, it looks like we are okay for the time being:
[17:08:26] <Bytram> martyb@sodium:~$ df -k
[17:08:26] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[17:08:26] <Bytram> udev 1020704 4 1020700 1% \dev
[17:08:26] <Bytram> tmpfs 204988 352 204636 1% \run
[17:08:28] <Bytram> \dev\sda 48378736 1488672 46382400 4% \
[17:08:30] <Bytram> none 4 0 4 0% \sys\fs\cgroup
[17:08:32] <Bytram> none 5120 0 5120 0% \run\lock
[17:08:33] <SemperOss> Why don't you talk a nice walk around them all and see what's there ("df") and what /var/log is up to "find /var/log/ -type f -exec ls -l '{}' \;"
[17:08:34] <Bytram> none 1024936 0 1024936 0% \run\shm
[17:08:36] <Bytram> none 102400 0 102400 0% \run\user
[17:08:38] <Bytram> martyb@sodium:~$
[17:09:50] <Bytram> Well, now that I know sodium is okay, I need to step away and cool off for a bit... is 86F atm.
[17:10:08] <Bytram> will later set up a script to walk the servers and check on those things...
[17:10:16] <Bytram> thanks MUCH for the commands to use!
[17:10:41] <Bytram> let's just call it good for now, and enjoy the rest of the day away from here.
[17:11:03] <Bytram> if I come back and find anything, I'll put out a ping here.
[17:55:14] <SemperOss> Bender seems not to be on the bend at the mo ... collateral improvement?
[17:56:25] <Bytram> !uid
[17:56:25] <Bender> The current maximum UID is 7078, owned by rakslice
[19:11:56] <Bytram> .
[19:13:25] <Bytram> I apologize in advance, but I know no other way to ensure this information is located contemporaneously with the foregoing work... paste bomb coming.
[19:13:28] <Bytram> martyb@boron:~/log$ cat disk_check.20180819_180228.log
[19:13:28] <Bytram> 20180819_180228: hydrogen: Filesystem 1K-blocks Used Available Use% Mounted on
[19:13:28] <Bytram> 20180819_180228: hydrogen: udev 4084092 4 4084088 1% /dev
[19:13:28] <Bytram> 20180819_180228: hydrogen: tmpfs 817668 364 817304 1% /run
[19:13:29] <Bytram> 20180819_180228: hydrogen: /dev/sda 98502268 14618508 82867792 15% /
[19:13:31] <Bytram> 20180819_180228: hydrogen: none 4 0 4 0% /sys/fs/cgroup
[19:13:33] <Bytram> 20180819_180228: hydrogen: none 5120 0 5120 0% /run/lock
[19:13:35] <Bytram> 20180819_180228: hydrogen: none 4088320 0 4088320 0% /run/shm
[19:13:37] <Bytram> 20180819_180228: hydrogen: none 102400 0 102400 0% /run/user
[19:13:39] <Bytram> 20180819_180228: helium: Filesystem 1K-blocks Used Available Use% Mounted on
[19:13:41] <Bytram> 20180819_180228: helium: udev 4084160 4 4084156 1% /dev
[19:13:43] <Bytram> 20180819_180228: helium: tmpfs 817668 368 817300 1% /run
[19:13:45] <Bytram> 20180819_180228: helium: /dev/sda 98646156 34756708 58883020 38% /
[19:13:47] <Bytram> 20180819_180228: helium: none 4 0 4 0% /sys/fs/cgroup
[19:13:49] <Bytram> 20180819_180228: helium: none 5120 0 5120 0% /run/lock
[19:13:51] <Bytram> 20180819_180228: helium: none 4088320 0 4088320 0% /run/shm
[19:13:53] <Bytram> 20180819_180228: helium: none 102400 0 102400 0% /run/user
[19:13:55] <Bytram> 20180819_180229: lithium: Filesystem 1K-blocks Used Available Use% Mounted on
[19:13:59] <Bytram> 20180819_180229: lithium: /dev/root 49056608 35619816 10946736 77% /
[19:14:01] <Bytram> 20180819_180229: lithium: devtmpfs 10240 0 10240 0% /dev
[19:14:03] <Bytram> 20180819_180229: lithium: tmpfs 404320 424 403896 1% /run
[19:14:05] <Bytram> 20180819_180229: lithium: shm 2021596 0 2021596 0% /dev/shm
[19:14:07] <Bytram> 20180819_180229: lithium: cgroup_root 10240 0 10240 0% /sys/fs/cgroup
[19:14:09] <Bytram> 20180819_180229: beryllium: Filesystem 1K-blocks Used Available Use% Mounted on
[19:14:11] <Bytram> 20180819_180229: beryllium: /dev/sda 49218368 40031160 8687056 83% /
[19:14:13] <Bytram> 20180819_180229: beryllium: tmpfs 2014048 128 2013920 1% /dev/shm
[19:14:15] <Bytram> 20180819_180229: boron: Filesystem 1K-blocks Used Available Use% Mounted on
[19:14:17] <Bytram> 20180819_180229: boron: udev 2019884 4 2019880 1% /dev
[19:14:19] <Bytram> 20180819_180229: boron: tmpfs 404812 432 404380 1% /run
[19:14:21] <Bytram> 20180819_180229: boron: /dev/sda 49056608 37470936 9095616 81% /
[19:14:23] <Bytram> 20180819_180229: boron: none 4 0 4 0% /sys/fs/cgroup
[19:14:27] <Bytram> 20180819_180229: boron: none 5120 0 5120 0% /run/lock
[19:14:29] <Bytram> 20180819_180229: boron: none 2024044 0 2024044 0% /run/shm
[19:14:31] <Bytram> 20180819_180229: boron: none 102400 0 102400 0% /run/user
[19:14:33] <Bytram> 20180819_180230: oxygen: Filesystem 1K-blocks Used Available Use% Mounted on
[19:14:35] <Bytram> 20180819_180230: oxygen: /dev/simfs 209715200 207315156 2400044 99% /
[19:14:37] <Bytram> 20180819_180230: oxygen: none 524288 4 524284 1% /dev
[19:14:39] <Bytram> 20180819_180230: oxygen: none 4 0 4 0% /sys/fs/cgroup
[19:14:41] <Bytram> 20180819_180230: oxygen: none 104860 1032 103828 1% /run
[19:14:43] <Bytram> 20180819_180230: oxygen: none 5120 0 5120 0% /run/lock
[19:14:45] <Bytram> 20180819_180230: oxygen: none 524288 0 524288 0% /run/shm
[19:14:47] <Bytram> 20180819_180230: oxygen: none 102400 0 102400 0% /run/user
[19:14:49] <Bytram> 20180819_180231: fluorine: Filesystem 1K-blocks Used Available Use% Mounted on
[19:14:51] <Bytram> 20180819_180231: fluorine: udev 4084160 4 4084156 1% /dev
[19:14:53] <Bytram> 20180819_180231: fluorine: tmpfs 817668 396 817272 1% /run
[19:14:55] <Bytram> 20180819_180231: fluorine: /dev/sda 98646156 61537236 32102596 66% /
[19:14:59] <Bytram> 20180819_180231: fluorine: none 4 0 4 0% /sys/fs/cgroup
[19:15:01] <Bytram> 20180819_180231: fluorine: none 5120 0 5120 0% /run/lock
[19:15:03] <Bytram> 20180819_180231: fluorine: none 4088320 0 4088320 0% /run/shm
[19:15:05] <Bytram> 20180819_180231: fluorine: none 102400 0 102400 0% /run/user
[19:15:07] <Bytram> 20180819_180232: neon: Filesystem 1K-blocks Used Available Use% Mounted on
[19:15:09] <Bytram> 20180819_180232: neon: udev 4084160 4 4084156 1% /dev
[19:15:11] <Bytram> 20180819_180232: neon: tmpfs 817668 348 817320 1% /run
[19:15:13] <Bytram> 20180819_180232: neon: /dev/sda 98646156 58968216 34671616 63% /
[19:15:15] <Bytram> 20180819_180232: neon: none 4 0 4 0% /sys/fs/cgroup
[19:15:17] <Bytram> 20180819_180232: neon: none 5120 0 5120 0% /run/lock
[19:15:19] <Bytram> 20180819_180232: neon: none 4088320 0 4088320 0% /run/shm
[19:15:21] <Bytram> 20180819_180232: neon: none 102400 0 102400 0% /run/user
[19:15:23] <Bytram> 20180819_180232: sodium: Filesystem 1K-blocks Used Available Use% Mounted on
[19:15:25] <Bytram> 20180819_180232: sodium: udev 1020704 4 1020700 1% /dev
[19:15:29] <Bytram> 20180819_180232: sodium: tmpfs 204988 352 204636 1% /run
[19:15:31] <Bytram> 20180819_180232: sodium: /dev/sda 48378736 1488676 46382396 4% /
[19:15:33] <Bytram> 20180819_180232: sodium: none 4 0 4 0% /sys/fs/cgroup
[19:15:35] <Bytram> 20180819_180232: sodium: none 5120 0 5120 0% /run/lock
[19:15:37] <Bytram> 20180819_180232: sodium: none 1024936 0 1024936 0% /run/shm
[19:15:39] <Bytram> 20180819_180232: sodium: none 102400 0 102400 0% /run/user
[19:15:41] <Bytram> 20180819_180233: magnesium: Filesystem 1K-blocks Used Available Use% Mounted on
[19:15:43] <Bytram> 20180819_180233: magnesium: /dev/root 24189340 15995588 7931400 67% /
[19:15:45] <Bytram> 20180819_180233: magnesium: devtmpfs 10240 0 10240 0% /dev
[19:15:47] <Bytram> 20180819_180233: magnesium: tmpfs 204492 384 204108 1% /run
[19:15:49] <Bytram> 20180819_180233: magnesium: shm 1022456 0 1022456 0% /dev/shm
[19:15:51] <Bytram> 20180819_180233: magnesium: cgroup_root 10240 0 10240 0% /sys/fs/cgroup
[19:15:53] <Bytram> martyb@boron:~/log$
[19:20:57] <SemperOss> From this I can see that oxygen is the one most needing some fresh air ... albeit not urgently
[19:21:54] <SemperOss> Bytram: Good work
[19:21:56] * Bytram chuckles, once again
[19:22:16] <Bytram> btw, oxygen has had storage issues in the past... it is our backup server
[19:23:05] <Bytram> usually, TMB goes off, locates the offending files (manually) and rm's them as needed,
[19:23:43] <Bytram> rarely, though, some user's home dir has some huge file in it and then copies of it consume too much space on that server.
[19:24:23] <Bytram> it's an... unusual... machine and, IIRC, for safety sake, is located somewhere outside of our usual data center (Dallas??)
[19:24:58] <Bytram> yep, oxygen runs OpenVZ
[19:25:25] <Bytram> beryllium is centos, and the rest are Ubuntu, or have been converted over to gentoo
[19:25:51] <SemperOss> Ah, I see. 200+GB space to back up ~300GB ... Hmmm!
[19:26:11] <Bytram> so, if oxygen runs out of gas, then we just lose the ability to make backups... not a problem, right?
[19:26:43] <Bytram> they squeeze the bits as they transfer them over the pipe
[19:26:46] <Bytram> ;)
[19:26:47] <SemperOss> No, of course not and Linode bombs for the first time and ... everything's gone?
[19:27:12] <SemperOss> :-D
[19:27:35] <SemperOss> I know oxygen is not Linode, right?
[19:27:37] <Bytram> that's why oxy is over ------> there
[19:27:46] <Bytram> umm, I think it is
[19:27:57] <Bytram> give me a minute, k?
[19:28:17] <SemperOss> But in a *completely* different location, right?
[19:29:08] <Bytram> huh! Oxygen is *NOT* a linode.
[19:29:18] <Bytram> at least, I do not see it listed on the dashboard
[19:32:02] <SemperOss> Good news. My setup is local machines that back up everything to a local server and essential files via NextCloud to a server abroad (at a friend's house) where it is backed up to another server hourly and then backed up to my brother's system nightly. Hopefully that's enough redundancy
[19:32:35] <Bytram> only if you backup your brother's server on your system, right?
[19:33:16] <SemperOss> Oh yeah, I forgot that ... I'd better set that up immediately ... oh wait ...
[19:33:52] <SemperOss> We actually backs up his files to another partition at my friend's
[19:34:11] <SemperOss> To avoid looping the data around
[19:34:17] <Bytram> heh. reminds me of... racetrack memory
[19:34:23] <Bytram> #g racetrack memory
[19:34:24] <MrPlow> https://en.wikipedia.org - "Racetrack memory or domain-wall memory (DWM) is an experimental non-volatile memory device under development at IBM's Almaden Research Center by a ..."
[19:34:39] <Bytram> umm, this was decades ago.
[19:34:45] <SemperOss> Those good old TI, I think, magnetic bubble thingies
[19:34:54] <SemperOss> I've actually had a few in my hand
[19:35:20] <Bytram> basically, bits would be sent moving down a conductive substance and picked back up when you waited long enough for them to make it 'round again.
[19:35:25] <Bytram> neat!
[19:35:59] <SemperOss> Not the mag bubbles? Drat!
[19:36:01] <Bytram> substance, maybe mercury? wire made of a special material? I don't recall for sure... only heard about it.
[19:37:52] <SemperOss> https://en.wikipedia.org is the one I was thinking about
[19:38:15] <Bytram> okay... since things seem to be under control here, think I'm gonna head off and take care of a few things.
[19:38:36] <Bytram> yep, I remember when it was nnounced, right along with tesseract and holographic memory
[19:39:03] <SemperOss> Tesseract is still alive as OCR
[19:39:11] <Bytram> nod nod
[19:39:33] <Bytram> haven't *used* it, but have heard of it... threw me off the first time I saw the name!
[19:39:48] <SemperOss> OK, I've had my dinner and just finished strawberries with cream, so I'll sit back and do some of the things I have happily ignored for the rest of the evening
[19:40:22] <Bytram> good for you!
[19:40:30] <Bytram> enjoy your evening..
[19:40:45] <SemperOss> Yeah, me too. Tesseract is, unfortunately, not very good. Microsoft's OneNote is *better*. Sigh!
[19:40:50] <Bytram> until we meet again... and the case is solved
[19:40:54] <SemperOss> Thank you and you the rest of your day.
[19:41:08] * Bytram dons / doffs a peter sellers voiced
[19:41:16] <SemperOss> ¡Hasta la vista!
[19:41:25] <Bytram> baby!
[19:41:45] * SemperOss saunters off into the set sun
[19:41:56] <SemperOss> Aw, hoit hot hot hot!
[19:42:04] <SemperOss> hoit => hot