#dev | Logs for 2019-01-10

« return
[05:45:07] -!- drussell [drussell!~cde90fbb@sktf-157-135-05-777.broadband.nucleus.com] has joined #dev
[05:46:12] <drussell> 500 error?
[08:17:14] -!- fyngyrz_ [fyngyrz_!~fyngyrz@66.171.jo.vll] has joined #dev
[08:17:14] -!- fyngyrz has quit [Read error: Connection reset by peer]
[08:31:49] <Xyem> Hm, I don't think I have the access to look into/fix it..
[08:32:28] <Xyem> ah, found the issue I think
[08:33:07] <Xyem> Looks like the database cluster is down
[08:38:26] <FatPhil> I don't think I ever set up a SSH key on any of the machines :/
[08:39:07] <Xyem> To get onto boron/lithium?
[09:29:35] <SemperOSS> Seems to be a database problem on lithium
[09:30:49] <SemperOSS> Got error 4009 'Cluster Failure' from NDBCLUSTER -- SELECT name, value FROM vars WHERE name IN ('memcached','memcached_debug','memcached_keyprefix','memcached_servers','sitename')
[09:31:18] <SemperOSS> Any MySQL memcached experts in the panel?
[09:36:37] <FatPhil> I doubt memcached is the problem, I suspect it's the victim.
[09:37:11] <SemperOSS> The error message when restarting rehash seems to indicate a MySQL clustering problem
[09:37:32] <SemperOSS> I have tried to restart MySQL to no avail
[09:38:54] <FatPhil> "
[09:38:54] <FatPhil> When you issue the show command on the management node what does it say about the status of your cluster, and the various nodes? Perhaps you could post the output here.
[09:39:43] <FatPhil> I've never clustered, so I know no more than google tells me when I ask it dumb questions.
[09:40:23] <SemperOSS> That is currently slightly more than me then ... Google and I are trying to look into it ;-)
[09:50:26] <FatPhil> Looks like there's a fair bit of sniffing possible: https://dev.mysql.com
[09:51:32] <FatPhil> anything in the clusterlog? https://dev.mysql.com
[09:52:20] <FatPhil> Anyone know when it was last seen working? It was working for me 9 hours ago, as I couldn't sleep.
[09:53:30] <SemperOSS> No, and I have been out of the loop generally due to an injury ... I only got involved now as I realised SN didn't work
[09:54:10] <SemperOSS> and the show command gives the following result:
[09:54:19] <SemperOSS> ndb_mgm> show
[09:54:20] <SemperOSS> Connected to Management Server at: ipv4.helium.li694-22:1186
[09:54:20] <SemperOSS> Cluster Configuration
[09:54:20] <SemperOSS> ---------------------
[09:54:20] <SemperOSS> [ndbd(NDB)] 2 node(s)
[09:54:21] <SemperOSS> id=2 (not connected, accepting connect from ipv4.helium.li694-22)
[09:54:22] <SemperOSS> id=3 (not connected, accepting connect from ipv4.neon.li694-22)
[09:54:23] <SemperOSS> [ndb_mgmd(MGM)] 2 node(s)
[09:54:24] <SemperOSS> id=101 @192.168.174.17 (mysql-5.7.17 ndb-7.5.5)
[09:54:25] <SemperOSS> id=102 (not connected, accepting connect from mysql-ndb-mgm-2.li694-22)
[09:54:26] <SemperOSS> [mysqld(API)] 4 node(s)
[09:54:27] <SemperOSS> id=11 (not connected, accepting connect from ipv4.helium.li694-22)
[09:54:28] <SemperOSS> id=12 (not connected, accepting connect from ipv4.lithium.li694-22)
[09:54:29] <SemperOSS> id=13 (not connected, accepting connect from ipv4.hydrogen.li694-22)
[09:54:30] <SemperOSS> id=14 (not connected, accepting connect from ipv4.fluorine.li694-22)
[09:55:01] <SemperOSS> And neon seems to be inaccessible
[09:55:09] <SemperOSS> ... to me
[09:59:34] <SemperOSS>
[10:00:17] <SemperOSS> Tried the following:
[10:00:26] <SemperOSS> ndb_mgm> 101 restart
[10:00:27] <SemperOSS> Shutting down MGM node 101 for restart
[10:00:27] <SemperOSS> Node 101 is being restarted
[10:00:27] <SemperOSS> ndb_mgm> 101 status
[10:00:27] <SemperOSS> Warning, event thread startup failed, degraded printouts as result, errno=115
[10:00:28] <SemperOSS> Connected to Management Server at: ipv4.helium.li694-22:1186
[10:00:29] <SemperOSS> Connected to Management Server at: ipv4.helium.li694-22:1186
[10:00:30] <SemperOSS> Node 101: connected (Version 7.5.5)
[10:00:31] <SemperOSS> ndb_mgm> 102 status
[10:00:32] <SemperOSS> Node 102: not connected
[10:00:33] <SemperOSS> ndb_mgm> 102 restart
[10:00:34] <SemperOSS> Shutting down MGM node 102 for restart
[10:00:35] <SemperOSS> Restart failed.
[10:00:36] <SemperOSS> * 2003: Restart failed
[10:00:37] <SemperOSS> * 5005-Send to process or receive failed.: Permanent error: Application error
[10:00:38] <SemperOSS>
[10:02:55] <FatPhil> 8-o
[10:07:33] <FatPhil> maybe something in /var/log on the nodes themselves? resource issues - memory or HD space?
[10:09:34] <SemperOSS> I'll look at what I can ...
[10:12:48] <SemperOSS> helium is the ndb management server but I have no root access to it#
[10:13:33] <SemperOSS> There is 20+ GB available on all servers bar beryllium with 6.8 GB, so space should not be an issue
[10:26:05] <FatPhil> well, you've done what you could, so thanks for trying. I guess we need to just implant visions of catfish invading the server room in TMB's head...
[10:26:34] <SemperOSS> Something along those lines, yes
[10:27:44] <SemperOSS> It's frustrating that I cannot find an appropriate log to stare at with glazed eyes
[10:29:06] <FatPhil> I hate it when software invents their own logging locations/mechanisms/formats. Plain text file under /var/log is the best option.
[10:57:05] <FatPhil> anyone have admin rights enough to put up a "sorry, we know about the problem and are working on it" page rather than the default 500?
[11:20:33] -!- SoyCow4338 [SoyCow4338!~c3e48be3@zslzb7u69ud5.fixip.t-online.hu] has joined #dev
[11:34:01] <SemperOSS> FatPhil: I know where to change but have no rights to do it
[11:59:11] <Bytram> SemperOSS: I prolly have the privs... what do you need me to do?
[12:00:08] <SemperOSS> I was considering FatPhil's suggestion of putting up a page to tell people that SN is down but we're working on it
[12:00:26] <SemperOSS> Need to set up a static page in nginx on fluorine, I think?
[12:01:21] <Bytram> no idea
[12:01:40] <Bytram> I *just* txted TMB that the site is throwing 500s
[12:03:00] <Bytram> yowza
[12:03:35] <Bytram> neon went to 400% cpu at 0415 and i/o basically stopped at same time
[12:03:41] <SemperOSS> That might be a help ... I think we'll have to consider a contingency plan for plastering a site down message on the front page ... or maybe take a page out of slashdot's system and have a static copy of the latest front page lying around with any links working to server when down
[12:04:08] <Bytram> SemperOSS: FatPhil: I can reboot neon from the linode manager page
[12:04:34] <Bytram> ~time x
[12:04:37] <SemperOSS> That is worth a try and then we could do a post mortem later
[12:04:52] <Bytram> so, to confirm: reboot neon?
[12:05:04] <SemperOSS> I'd vote yes
[12:05:15] <SemperOSS> Can't be worse than it is now
[12:05:18] <Bytram> any opposed? any potential downside?
[12:05:45] <SemperOSS> The only thing could be data missing for a post mortem, depending on what is going on
[12:06:03] <SemperOSS> But the potential upside is greater I think
[12:06:54] <Bytram> any logging of the even should have happened by now... it's like 6 or 8 hours after cpu getting pegged.
[12:07:14] <Bytram> s/even/event/
[12:07:36] <SemperOSS> Agreed, but temp files are gone, which shouldn't be a problem
[12:07:48] <SemperOSS> Pull the trigger, I'd suggest
[12:08:03] <Bytram> cross your fingers
[12:08:15] <Bytram> reboot clicked
[12:08:21] <SemperOSS> ... and toes and legs and arms
[12:08:42] <Bytram> ... and eyes
[12:08:50] <SemperOSS> Can you see when it's back up again?
[12:08:55] <Bytram> nod nod
[12:09:29] <SemperOSS> It responds to pings now
[12:10:26] <SemperOSS> but SN still throws Internal Server Errors
[12:10:48] <SemperOSS> I'll try to restart rehash on lithium
[12:11:35] <SemperOSS> Nah, same error
[12:11:40] <Bytram> hold on...
[12:11:49] <Bytram> pls, let's try one thing at a time?
[12:12:21] <Bytram> okay, neon is now responding to a ping
[12:13:25] <Bytram> and... now I can ssh into neon
[12:13:50] <Bytram> still getting a 500 on main page
[12:14:18] <cosurgi> check free disk space, just to be sure it's not this kind of problem. See if maybe /var/log is full
[12:14:26] <SemperOSS> The cluster daemon (ndbd) is running on neon
[12:14:39] <SemperOSS> I've checked free space on all but neon
[12:15:01] <SemperOSS> Inoen's got 33GB free
[12:15:02] <Bytram> martyb@neon:~$ df .
[12:15:02] <Bytram> Filesystem 1K-blocks Used Available Use% Mounted on
[12:15:02] <Bytram> martyb@neon:~$
[12:15:02] * FatPhil stands to the side as a cheerleader
[12:15:25] * Bytram gets pom poms and pleated skirt for FatPhil
[12:15:50] <SemperOSS> FatPhil: Any suggestions, please don't hold tem back
[12:16:01] <SemperOSS> s/tem/them/
[12:16:44] <cosurgi> /etc/init.d/* stop / start ?
[12:16:44] <Bytram> bounce the servers to refresh caches? ( i.e. do: /home/bob/bin/bounce on both fluorine and hydrogen )
[12:17:26] <cosurgi> can you post the logfile somewhere where I could see it? Last 24 hours? Maybe I will spot siomething
[12:17:35] <Bytram> path?
[12:17:40] <SemperOSS> Currently I think we can do little damage and much good by doinmg, basically, ANYTHING!
[12:17:51] <cosurgi> /var/og syslog & dmesg
[12:17:58] <cosurgi> /var/log/syslog & dmesg
[12:19:24] <FatPhil> what distro does neon run? is there a record of package upgrades - did anything change?
[12:19:31] <SemperOSS> Bytram: I have no access to /v ar/log/apache2 on neon, where there may be a nugget or two too
[12:19:31] <Bytram> cosurgi: you gots a place I can put 'em?
[12:19:48] <FatPhil> I know debian-alikes have an apt log
[12:19:54] <SemperOSS> Ubuntu
[12:19:59] <Bytram> ^^^
[12:20:09] <Bytram> System Boot - Ubuntu 14.04 LTS
[12:20:52] <cosurgi> Bytram: damn, like and ftp server or something? no :( maybe use pastebin?
[12:21:12] <Bytram> would rather not make it that public...
[12:21:20] <cosurgi> ah, right.
[12:21:45] <Bytram> email addies it is for the time being: cosurgi SemperOSS FatPhil
[12:21:59] <Bytram> please provide ur email addy
[12:22:18] <cosurgi> Bytram: janek@kozicki.pl
[12:22:23] <Bytram> k
[12:22:43] * SemperOSS sempeross@ziberex.dk for me
[12:23:06] <Bytram> I don't have an easy way to copy from neon to send so did a cat of each and caputered to my putty log... now to extricate to separate files
[12:23:15] <Bytram> make take a minute... brb
[12:23:34] <SemperOSS> I'll make a cuppa while you're gone
[12:23:57] <SoyCow4338> Something changed, it's not 500 anymore
[12:24:08] <cosurgi> Bytram: if putty, then you are on windows? Then you could use winscp, because it uses ssh for sending files.
[12:25:57] <Bytram> appreciate the suggestion -- prolly try later (when things have calmed down) -- the longest distance between two points is a short cut. ;)
[12:26:01] <Bytram> gots files
[12:27:08] <cosurgi> wow, yeah. Something changed. I even managed to log in with my site pwd. The only thing missing is the stylesheet. Website looks like in a text terminal browser, like lynx.
[12:27:45] <SoyCow4338> Content is missing, looks like app error now
[12:28:03] <SoyCow4338> Article pages load empty with only HTML comments
[12:28:05] <SoyCow4338>    
[12:28:10] <SoyCow4338> that's all
[12:28:37] <cosurgi> Oh, right. I see it too.
[12:29:08] <SoyCow4338> My best guess is DB problem
[12:30:25] <Bytram> so syslog and dmesg... any other?
[12:31:18] <Bytram> well, I'll send those fornow
[12:31:42] <Bytram> SemperOSS: cosurgi: check your mails
[12:32:30] <Bytram> oh, site is kinda back... looks like main page is loading but css is 'wonky'
[12:32:54] <Bytram> ^^^ from https://soylentnews.org
[12:34:38] <Bytram> SoyCow4338: cosurgi: ugh... gotta step away for a couple mnutes
[12:34:46] <SoyCow4338> If I had access I'd check shell command history, who did what
[12:36:04] <SemperOSS> Bytram: We may need syslog.log.[1-7]* too for history lessons
[12:36:06] <chromas> Check out the page source. Seems to be missing stuff. Templates not getting filled?
[12:36:11] <chromas> oh derp
[12:36:12] <cosurgi> damn.
[12:36:17] <cosurgi> I didn't get this email :(
[12:36:23] <cosurgi> I'm refreshing like crazy
[12:36:44] <cosurgi> will check spam and others...
[12:36:50] <SemperOSS> cosurgi: I'll try forwarding the one I got
[12:37:51] <Bytram> back
[12:38:00] <SemperOSS> cosurgi: Done and no errors on my side
[12:38:29] <Bytram> SemperOSS: brilliant!
[12:39:40] <Bytram> I expect that there may be some miscommunications between servers at this point... propose bouncing fluorine and hydrogen to get apache refreshed
[12:40:07] <SoyCow4338> Dunno the setup but in most cases downtime is caused by a config or software change. If disk space is ok and no something like crontab auto-install-latest-software was in place, then the only thing I can think of is counter overflow in some auto-inrement DB column.
[12:40:27] <SoyCow4338> I mean the most probable cause is somebody changed something.
[12:40:48] <SemperOSS> Bytram: Is there a way to whack them ALL on the head and then resuscitate them?
[12:40:56] <Bytram> well, linode did a reboot of neon 3 days ago... could be the site was kinda working, but things weren't really well
[12:41:13] <Bytram> SemperOSS: not advised... race condition as to which servers boot when
[12:41:25] <Bytram> lemme restart apache, etc. on fluorine and hydrogen
[12:41:26] <SemperOSS> Is there a preferred order to do it in? (Like DBs up first, then web servers)
[12:41:28] <Bytram> ( i.e. do: /home/bob/bin/bounce on both fluorine and hydrogen )
[12:41:34] * Bytram is on fluorine
[12:41:42] <cosurgi> SemperOSS: screw this, try my work email: jkozicki@pg.edu.pl , I will ask later my email provider what is wrong with my janek@kozicki.pl address.
[12:41:56] <cosurgi> (it still didn't arrive there)
[12:42:04] <Bytram> martyb@fluorine:~$ /home/bob/bin/bounce
[12:42:04] <Bytram> * Stopping HTTP accelerator varnishd [ OK ]
[12:42:04] <Bytram> * Starting HTTP accelerator varnishd [ OK ]
[12:42:04] <Bytram> martyb@fluorine:~$
[12:42:21] <Bytram> now to hydrogen
[12:42:41] <Bytram> ditto
[12:42:44] <Bytram> try it now
[12:42:50] <SemperOSS> cosurgi: Done
[12:43:01] <cosurgi> SemperOSS: now I got it!
[12:43:01] <SoyCow4338> Back in business
[12:43:03] <Bytram> woo hoo! ??
[12:43:13] <SemperOSS> Bytram: WORKING!!!!!!
[12:43:45] <Bytram> yee haw!!
[12:44:40] <SemperOSS> Y'all: What to be learnt from this? If it doesn't work, turn it off and on again, i.e. not much. :-(
[12:44:45] <SoyCow4338> Whatever server you restarted last, it might be a good idea to copy all logs before they get rotated or smth
[12:44:46] <cosurgi> SemperOSS: lol, now it arrived at my both addresses :)
[12:45:05] <Bytram> imp[ortant question: when did the site first go sideways? date/time UTC?
[12:45:18] * Bytram is writing up a quick meta story to let the community know.
[12:45:23] <SemperOSS> cosurgi: That's computers for you
[12:45:23] <SoyCow4338> @cosurgi Are you using Internet Explorer? :P
[12:46:10] <SoyCow4338> I've noticed it down aroung 5h ago
[12:47:05] <cosurgi> lol :) no. I use claws-mail
[12:47:07] <Bytram> looking thru the channels here, I see a report of a 500 @ 00:12 today. anyone see anything earlier?
[12:47:25] <cosurgi> These logs are surprisingly short, even for a freshly restarted system
[12:47:56] <cosurgi> Only this looks suspicious: Jan 10 12:13:34 neon sudo: pam_ldap: ldap_starttls_s: Can't contact LDAP server
[12:48:04] <cosurgi> and this: Jan 10 12:11:24 neon kernel: [ 3.662545] init: failsafe main process (609) killed by TERM signal
[12:48:13] <SoyCow4338> I'd avoid logging in until the root cause is found
[12:49:03] <cosurgi> btw, why is there a print server? :) Jan 10 12:11:23 neon kernel: [ 2.464045] lp: driver loaded but no devices found
[12:49:44] <Bytram> 'cause we got a great deal on virtual paper, of course! We needed a virtual printer to make use of it. =)
[12:50:23] <SoyCow4338> Can you sudo lastlog on neon and check who logged around that time?
[12:50:58] <cosurgi> also, the command `last`
[12:51:49] <SemperOSS> lastlog gets truncated at boot and it is NOT logrotated, apparently
[12:52:31] <SemperOSS> I'm wrong, it doesn't
[12:52:47] <SemperOSS> That was wtmp
[12:59:44] <cosurgi> but last shouldn't be truncated I think?
[13:00:11] <SoyCow4338> Peace
[13:00:11] -!- SoyCow4338 [SoyCow4338!~c3e48be3@zslzb7u69ud5.fixip.t-online.hu] has parted #dev
[13:00:36] <cosurgi> ok. As far as I ccan tell, there is no critical error in these logs.
[13:00:54] <SemperOSS> I agree with cosurgi on that
[13:01:23] <cosurgi> I would need to examine more from /var/log/* maybe some service specific logs have some errors. And was spitting warnings for some time. Eventually it failed. And restart after some struggling brought it back up.
[13:01:52] <cosurgi> cd /var/log ; grep -i error . -r
[13:02:03] <cosurgi> this however would be too generic. There wil be thousandes of these :(
[13:02:26] <FatPhil> 100% CPU is a bit weird, any clues what was causing that
[13:02:36] <cosurgi> use htop to find out.
[13:02:40] <FatPhil> At least the time window seems to be narrowed down a bit now
[13:02:46] <FatPhil> htop --timetravel
[13:03:17] <cosurgi> ahh. How do you know it was 100% in the past?
[13:04:04] <FatPhil> cosurgi: 14:07 <@Bytram> neon went to 400% cpu at 0415 and i/o basically stopped at same time
[13:04:27] <cosurgi> if you send me all files from /var/log/* younger than 24h I *might* be able to spot something....
[13:05:17] <Bytram> Best estimates at this point is that the site was down: 20190110 00:15-07:45 UTC
[13:05:29] <SemperOSS> I can see that syslog.2.gz (from 8 Jan) is substantially larger than syslog.[3-7].gz
[13:05:42] * Bytram is highly under-caffeinated -- break time
[13:05:44] <cosurgi> that is interesting.
[13:05:47] <Bytram> biab
[13:06:10] <cosurgi> zcat /var/log/syslog.2.gz | less
[13:06:28] <SemperOSS> -rw-r----- 1 syslog adm 73546 Jan 10 12:55 syslog
[13:06:29] <SemperOSS> -rw-r----- 1 syslog adm 27532 Jan 9 06:45 syslog.1
[13:06:29] <SemperOSS> -rw-r----- 1 syslog adm 14832 Jan 8 06:35 syslog.2.gz
[13:06:29] <SemperOSS> -rw-r----- 1 syslog adm 1827 Jan 6 06:25 syslog.3.gz
[13:06:29] <SemperOSS> -rw-r----- 1 syslog adm 1863 Jan 5 06:45 syslog.4.gz
[13:06:30] <SemperOSS> -rw-r----- 1 syslog adm 1972 Jan 4 06:45 syslog.5.gz
[13:06:31] <SemperOSS> -rw-r----- 1 syslog adm 1892 Jan 3 06:45 syslog.6.gz
[13:06:32] <SemperOSS> -rw-r----- 1 syslog adm 1868 Jan 2 06:25 syslog.7.gz
[13:06:40] <SemperOSS> I have no read access to the files
[13:07:17] <FatPhil> woh, why is .2 so much bigger than .3...
[13:07:35] <SemperOSS> Good question
[13:07:50] <FatPhil> makes sense at least to pop you in the adm group, no?
[13:08:22] <SemperOSS> Would be nice, I think ... We'll have to wait for TMB to wake up
[13:08:51] <SemperOSS> I must say I have been less of a help than perhaps would be nice in the last months due to an injury
[13:09:28] <Bytram> but you were here when we needed you, today.
[13:09:32] <Bytram> SemperOSS++ yays
[13:09:32] <Bender> karma - sempeross: 5
[13:09:37] <Bytram> =)
[13:10:31] <SemperOSS> Yeah, only because I was waking up with a nagging feeling something was wrong, it wasn't my broken wrist so I checked SN ;-)
[13:11:06] <FatPhil> SemperOSS++
[13:11:06] <Bender> karma - sempeross: 6
[13:11:50] <cosurgi> I only wanted to look at news in the morning, and I could not. That mobilised me. But in fact I haven't done anything useful.
[13:12:16] <cosurgi> So it means that without TMB we cannot look at syslog.2.gz ?
[13:12:59] <cosurgi> looks like we lack some admin power.
[13:13:18] <SemperOSS> Bytram can copy it (scp, Bytram, scp)
[13:13:58] * Bytram heard the ping
[13:14:06] <Bytram> exact syntax, pls?
[13:14:19] <cosurgi> or, Bytram can `cp /var/log/syslog.2.gz /tmp/ ; chmod a+r /tmp/syslog.2.gz` and then SemperOSS would have read access.
[13:14:37] <Bytram> on neon, right
[13:14:38] <Bytram> ?
[13:14:44] <cosurgi> yes
[13:14:47] <Bytram> k
[13:15:00] <cosurgi> SemperOSS: now it's your turn ;)
[13:15:10] <SemperOSS> Do that for all of them on neon: 'cp /var/log/syslog.*.gz /tmp/ ; chmod a+r /tmp/syslog.*.gz'
[13:15:31] <Bytram> now you tell me! just did the first one
[13:15:45] <SemperOSS> Just to have something to compare with
[13:15:49] <Bytram> nod nod
[13:15:53] <Bytram> should be good
[13:16:52] <SemperOSS> I ain't seein' nuthin' in /tmp on neon
[13:17:09] <SemperOSS> ... apart from some kerberos files
[13:17:37] <Bytram> coffee++
[13:17:37] <Bender> karma - coffee: 19
[13:17:39] <Bytram> baib
[13:18:11] <cosurgi> hmm. the command looks good for me. Of course you do it wihout the quotes '
[13:18:40] <Bytram> could one of you please keep an eye on the meta story announcing the site's outage to see if there are any reports of continuing/other problems? https://soylentnews.org
[13:18:42] <SemperOSS> w on neon shows Bytram is doing 'ssh hydrogen'
[13:19:01] <Bytram> oh... yup
[13:19:03] <cosurgi> ahh. maybe Bytram you are doing this on hydrogen by accident?
[13:19:08] <Bytram> yes
[13:19:13] * Bytram make sad face
[13:19:27] <Bytram> u wants me on neon?
[13:19:46] <SemperOSS> Ah well, only a slight opps, the egg wears off quickly ;-)
[13:19:53] <cosurgi> no worries :)
[13:19:56] <SemperOSS> s/opps/oops/
[13:20:05] <Bytram> case in point, eh?
[13:20:07] <Bytram> ;)
[13:20:11] <SemperOSS> Yup!
[13:20:16] <SemperOSS> :D
[13:20:18] <Bytram> issued cp cmd on neon... try it now
[13:20:33] <SemperOSS> Mucho better
[13:20:44] <Bytram> yay yay yay
[13:20:45] <SemperOSS> cosurgi, do you want to share the honours?
[13:20:52] <Bytram> ok, this time for sure... break time
[13:20:57] <cosurgi> SemperOSS: uh, how? :)
[13:21:14] <Bytram> is only my 2nd day off from work since christmas and am very much not awake yet.
[13:21:16] <SemperOSS> The waterlogged log files scrutiny
[13:21:32] <cosurgi> ah, yeas of course. I want to see it
[13:21:37] <cosurgi> send it to me.
[13:21:51] <SemperOSS> the .gz or unpacked files?
[13:21:55] <cosurgi> .gz
[13:22:04] <SemperOSS> 'K just a mo
[13:26:33] <cosurgi> just a small hint: I like using midnight commander `mc` menu->left->Shell link for copying files around. it is ofter faster (because ncurses-GUI, you know) than scp or rsync.
[13:28:25] <cosurgi> I have it!
[13:28:30] <cosurgi> ok, examining now
[13:28:32] <fyngyrz_> agree: midnight commander is the bomb
[13:29:19] <SemperOSS> the .2 file is longer because the server was rebooted
[13:29:19] -!- fyngyrz_ has quit [Quit: Leaving...]
[13:29:28] -!- fyngyrz [fyngyrz!~fyngyrz@Soylent/Staff/Editor/fyngyrz] has joined #dev
[13:29:42] <SemperOSS> There seems to be problems with the perf monitor taking longer and longer, so maybe another issue
[13:29:59] <cosurgi> in mc's source code there is even a small hunk of code which I patched ;) Just a dozen of lines.
[13:30:31] <Bytram> back
[13:30:47] <Bytram> fwiw, I get the feeling this might be connected to a cron job?
[13:31:25] <Bytram> ISTR the site sends out e-mails and does some other stuff in slashd(?) at about 00:15
[13:32:52] <SemperOSS> Jan 7 07:11:51 neon kernel: [ 28.842737] perf samples too long (2769 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
[13:32:53] <SemperOSS> Jan 7 07:13:28 neon kernel: [ 126.778793] perf samples too long (5143 > 5000), lowering kernel.perf_event_max_sample_rate to 25000
[13:32:53] <SemperOSS> Jan 7 07:14:59 neon kernel: [ 217.883160] perf samples too long (10042 > 10000), lowering kernel.perf_event_max_sample_rate to 12500
[13:32:53] <SemperOSS> Jan 7 07:16:13 neon kernel: [ 292.583156] perf samples too long (20015 > 20000), lowering kernel.perf_event_max_sample_rate to 6250
[13:32:53] <SemperOSS> Jan 7 23:28:36 neon kernel: [58634.756489] perf samples too long (52870 > 40000), lowering kernel.perf_event_max_sample_rate to 3250
[13:32:54] <SemperOSS> Jan 7 23:28:36 neon kernel: [58634.756498] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 2.880 msecs
[13:34:39] <cosurgi> quick google tells me that this indicates too high load on the system.
[13:34:57] <cosurgi> that's consistent with this 400% usage reported earlier.
[13:35:35] <SemperOSS> No perf messages in the other files, but then again they may have been and gone before the oldest logfile retained
[13:36:08] <cosurgi> who did this reboot on Jan 7 07:11:51 ?
[13:36:22] <fyngyrz> I think that was probably TMB
[13:36:34] <SemperOSS> I concur. There's something strange going on after the reboot done by Linode. Bytram, why? Do you know?
[13:37:04] <Bytram> know eyed dear
[13:37:35] <SemperOSS> Arrrrrrrrrrrgh ... I mean: Ah, well
[13:37:56] <cosurgi> Bytram: ls -la /var/spool/cron ?
[13:38:02] <cosurgi> let's examine crontab...
[13:38:19] <fyngyrz> after he found the problem with the "GROUP BY" in the query that sped the site up, the site's CSS sort of... went away. He fixed it, I asked what it was, and he said "gremlins" which sort of made me wonder, but, it was TMB :)
[13:38:32] <fyngyrz> that may have been the reboot
[13:38:32] <Bytram> root@neon:~# ls -la /var/spool/cron
[13:38:32] <Bytram> total 20
[13:38:32] <Bytram> drwxr-xr-x 5 root root 4096 Apr 26 2012 .
[13:38:32] <Bytram> drwxr-xr-x 5 root root 4096 Apr 26 2012 ..
[13:38:32] <Bytram> drwxrwx--T 2 daemon daemon 4096 Apr 26 2012 atjobs
[13:38:33] <Bytram> drwxrwx--T 2 daemon daemon 4096 Oct 25 2011 atspool
[13:38:35] <Bytram> drwx-wx--T 2 root crontab 4096 Sep 14 2016 crontabs
[13:38:37] <Bytram> root@neon:~#
[13:39:00] <cosurgi> Bytram: ls -laR /var/spool/cron/ ?
[13:39:01] <cosurgi> sorry.
[13:39:04] <Bytram> ugh, story queue is empty
[13:39:05] <cosurgi> try with -R
[13:39:09] <Bytram> k
[13:39:37] <Bytram> root@neon:~# ls -laR /var/spool/cron/
[13:39:37] <Bytram> total 20
[13:39:37] <Bytram> drwxr-xr-x 5 root root 4096 Apr 26 2012 .
[13:39:37] <Bytram> drwxr-xr-x 5 root root 4096 Apr 26 2012 ..
[13:39:43] <Bytram> drwxrwx--T 2 daemon daemon 4096 Apr 26 2012 atjobs
[13:39:46] <Bytram> drwxrwx--T 2 daemon daemon 4096 Oct 25 2011 atspool
[13:39:50] <Bytram> drwx-wx--T 2 root crontab 4096 Sep 14 2016 crontabs
[13:39:52] <Bytram> total 12
[13:39:54] <Bytram> drwxrwx--T 2 daemon daemon 4096 Apr 26 2012 .
[13:39:56] <Bytram> drwxr-xr-x 5 root root 4096 Apr 26 2012 ..
[13:39:58] <Bytram> -rw------- 1 daemon daemon 2 Apr 26 2012 .SEQ
[13:40:00] <Bytram> total 8
[13:40:02] <Bytram> drwxrwx--T 2 daemon daemon 4096 Oct 25 2011 .
[13:40:04] <Bytram> drwxr-xr-x 5 root root 4096 Apr 26 2012 ..
[13:40:06] <Bytram> total 20
[13:40:08] <Bytram> drwx-wx--T 2 root crontab 4096 Sep 14 2016 .
[13:40:10] <Bytram> drwxr-xr-x 5 root root 4096 Apr 26 2012 ..
[13:40:12] <Bytram> -rw------- 1 mysql crontab 1388 Sep 14 2016 mysql
[13:40:14] <Bytram> -rw------- 1 root crontab 1089 Mar 22 2015 root
[13:40:16] <Bytram> -rw------- 1 slash crontab 1187 Jun 4 2015 slash
[13:40:20] <Bytram> root@neon:~#
[13:41:25] <cosurgi> cat /var/spool/cron/crontabs/mysql | grep -v '#' | uniq
[13:41:30] <Bytram> did u get it ok?
[13:41:35] <Bytram> ahh, on it
[13:41:44] <cosurgi> yes. Now I would like to see the command invoked by each of them
[13:42:00] <Bytram> no output from: cat /var/spool/cron/crontabs/mysql | grep -v '#' | uniq
[13:42:08] <cosurgi> oh?
[13:42:13] <Bytram> yup
[13:42:43] <cosurgi> try cat /var/spool/cron/crontabs/mysql just to be sure, but noe need to paste if its long. It should be all commented out
[13:42:58] <Bytram> just did that =)
[13:43:06] <cosurgi> ok. so let's see the next one.
[13:43:07] <Bytram> just comments with a couple blank lines at end
[13:43:15] <cosurgi> cat /var/spool/cron/crontabs/root | grep -v '#' | uniq
[13:43:29] <Bytram> same; nothin
[13:43:34] <cosurgi> cat /var/spool/cron/crontabs/sash | grep -v '#' | uniq
[13:43:41] <cosurgi> cat /var/spool/cron/crontabs/slash | grep -v '#' | uniq
[13:44:01] <SemperOSS> Try 'crontab -l' as root
[13:44:14] <Bytram> root@neon:~# cat /var/spool/cron/crontabs/slash | grep -v '#' | uniq
[13:44:14] <Bytram> @reboot /srv/soylentnews.org/startup.sh
[13:44:14] <Bytram> root@neon:~#
[13:44:31] <Bytram> .
[13:44:41] <Bytram> crontab -l shows only comments
[13:44:43] <cosurgi> ah. So we have just one command in crontab. and it happens at reboot.
[13:44:56] <cosurgi> but site is cleaning up at 00:15 right?
[13:45:07] <Bytram> that's what I vaguely recall
[13:45:18] <cosurgi> ls -laR /etc/cron.d
[13:45:32] <Bytram> slashd is a daemon that sets off a bunch of tallying, emailing, etc.
[13:45:40] <Bytram> (IIRC)
[13:45:58] <cosurgi> check if TMB is using /etc/cron.d/ dir. for something.
[13:46:24] <SemperOSS> startup.sh contains the following three lines:
[13:46:28] <SemperOSS> #!/bin/sh
[13:46:29] <SemperOSS> cd /srv/soylentnews.org/
[13:46:29] <SemperOSS> /opt/mysql-5.6.17-linux-glibc2.5-x86_64/support-files/mysql.server start
[13:46:33] <SemperOSS>
[13:46:50] <cosurgi> Bytram: `ls -laR /etc/cron.d` please? :)
[13:47:19] <Bytram> root@neon:/etc/cron.d# ls -laR /etc/cron.d
[13:47:19] <Bytram> total 28
[13:47:19] <Bytram> drwxr-xr-x 2 root root 4096 Sep 1 2017 .
[13:47:19] <Bytram> drwxr-xr-x 99 root root 12288 Jan 10 12:11 ..
[13:47:20] <Bytram> -rw-r--r-- 1 root root 510 Apr 17 2015 php5
[13:47:22] <Bytram> -rw-r--r-- 1 root root 102 Apr 2 2012 .placeholder
[13:47:24] <Bytram> -rw-r--r-- 1 root root 396 Dec 16 2011 sysstat
[13:47:26] <Bytram> root@neon:/etc/cron.d#
[13:47:41] <cosurgi> oh. I saw php5 in syslog. Is that rellated?
[13:47:46] <SemperOSS> May I suggest using -lAR so we don't get . and ..?
[13:48:00] <Bytram> you may suggest =)
[13:48:03] <cosurgi> oh, right. sorry. musce memory habit/
[13:48:07] <cosurgi> *muscle
[13:48:25] <cosurgi> cat /etc/cron.d/php5 | grep -v '#' | uniq
[13:48:26] <Bytram> root@neon:/etc/cron.d# ls -IAR /etc/cron.d
[13:48:26] <Bytram> php5 sysstat
[13:48:26] <Bytram> root@neon:/etc/cron.d#
[13:48:27] <FatPhil> my muscle memory is to type -lart
[13:48:28] <SemperOSS> I know
[13:48:59] <FatPhil> but I'm a BOFH fan
[13:49:47] <cosurgi> Bytram: SemperOSS suggested lAR no IAR, but never mind. ;)
[13:49:57] <cosurgi> Bytram: please check cat /etc/cron.d/php5 | grep -v '#' | uniq
[13:50:24] <SemperOSS> Why have a font where I and l cannot be distinguished?
[13:50:38] <Bytram> beats me
[13:50:47] <cosurgi> that's putty IIRC. no worries ;)
[13:50:48] <Bytram> courier++
[13:50:48] <Bender> karma - courier: 1
[13:52:34] <cosurgi> we could also check /etc/cron.d/sysstat | grep -v '#' | uniq
[13:52:58] <cosurgi> but I must adming that poking around the system using Bytram as terminal is a bit tiresome for all of us. So sorry about that.
[13:53:07] <cosurgi> *admit
[13:53:38] <FatPhil> cron's actions would be logged in the logs anyway
[13:53:41] <Bytram> agreed, but at least I *have* the privs so I can help out, however awkwardly.
[13:53:54] <Bytram> should really see about you guys getting updated privs/access
[13:54:33] <cosurgi> I didn't expect it would come to this. Consider giving me some read-only access, at least to the system. The I could help out more.
[13:54:38] <cosurgi> *Then
[13:54:57] <Bytram> that's... strange.
[13:54:58] <Bytram> root@neon:/etc/cron.d# /etc/cron.d/sysstat | grep -v '#' | uniq
[13:54:58] <Bytram> -bash: /etc/cron.d/sysstat: Permission denied
[13:54:58] <Bytram> root@neon:/etc/cron.d#
[13:55:24] <cosurgi> In fact NCommander gave me access 2 or 3 years ago. But it got lost in one of the upgrades, and I didn't ask to get it back.
[13:55:27] <chromas> ACLs again?
[13:55:37] <FatPhil> Bytram: you just tried to execute it, not read it
[13:55:40] <Bytram> umm, think you wanted a 'cat' at the beginning of that
[13:55:53] <Bytram> root@neon:/etc/cron.d# cat /etc/cron.d/sysstat | grep -v '#' | uniq
[13:55:53] <Bytram> PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin
[13:55:55] <Bytram> 5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1
[13:55:57] <Bytram> 59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2
[13:55:59] <Bytram> root@neon:/etc/cron.d#
[13:56:50] <cosurgi> no command at 00:15
[13:57:03] <cosurgi> hm. yes. There is
[13:57:12] <Bytram> i may be off a bit... what about 04:15 ??
[13:57:17] <cosurgi> 5-55/10 is starting from 5th minute every 10 minutes, right?
[13:57:27] <cosurgi> so 15-th minute is there too.
[13:57:28] <FatPhil> every 10 minutes, at 05, 15, 25, ...
[13:57:48] <cosurgi> Just to be sure, lets see the other file:
[13:57:57] <cosurgi> cat /etc/cron.d/php5 | grep -v '#' | uniq
[13:57:58] <cosurgi> ?
[13:58:16] <Bytram> root@neon:/etc/cron.d# cat /etc/cron.d/php5 | grep -v '#' | uniq
[13:58:16] <Bytram> 09,39 * * * * root [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime)
[13:58:16] <Bytram> root@neon:/etc/cron.d#
[13:59:01] <cosurgi> ok. I would suppose that php5 is innocent.
[13:59:03] <Bytram> FatPhil: you need any privs while TMB is at it?
[13:59:11] <Bytram> IIRC, that is used by the exec bot
[13:59:26] <Bytram> anyone else need privs?
[13:59:41] <cosurgi> I suppose that there is a collision between 59 23 and 5-55/10
[13:59:55] <cosurgi> the one at 23:59 didn't finish yet, but the one at 00:05 is starting.
[14:00:33] <FatPhil> Bytram: I don't think I even handed over a public key, so can't even access the machines
[14:00:49] <FatPhil> If we have sysstat - did it log anything interesting this morning?
[14:01:29] <Bytram> FatPhil: pm?
[14:01:55] <FatPhil> # grep 'Jan 10.*CRON' /var/log/syslog
[14:03:33] <Bytram> is 65 lines... paste bomb coming
[14:04:26] <Bytram> nah... that's nasty
[14:04:47] <chromas> Need a secret staff pastebin
[14:05:57] -!- SoyCow4338 [SoyCow4338!~c3e48be3@zslzb7u69ud5.fixip.t-online.hu] has joined #dev
[14:05:58] <Bytram> agreed!
[14:06:18] <Bytram> it's still about 30-40 lines, but here's the most recent / significant stuff:
[14:06:19] <Bytram> Jan 10 04:09:01 neon CRON[10681]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
[14:06:19] <Bytram> Jan 10 04:15:01 neon CRON[10709]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:06:19] <Bytram> Jan 10 04:17:01 neon CRON[10822]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
[14:06:19] <Bytram> Jan 10 04:25:01 neon CRON[10827]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:06:20] <Bytram> Jan 10 04:35:01 neon CRON[10834]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:06:22] <Bytram> Jan 10 04:39:01 neon CRON[10844]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
[14:06:25] <Bytram> Jan 10 04:45:01 neon CRON[10858]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:06:27] <Bytram> Jan 10 12:11:24 neon cron[767]: (CRON) INFO (pidfile fd = 3)
[14:06:29] <Bytram> Jan 10 12:11:24 neon cron[790]: (CRON) STARTUP (fork ok)
[14:06:31] <Bytram> Jan 10 12:11:24 neon cron[790]: (CRON) INFO (Running @reboot jobs)
[14:06:33] <Bytram> Jan 10 12:11:24 neon CRON[817]: (slash) CMD (/srv/soylentnews.org/startup.sh)
[14:06:35] <Bytram> Jan 10 12:11:25 neon CRON[801]: (CRON) info (No MTA installed, discarding output)
[14:06:37] <Bytram> Jan 10 12:15:01 neon CRON[1346]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:06:39] <Bytram> Jan 10 12:17:02 neon CRON[1349]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
[14:06:41] <Bytram> Jan 10 12:25:01 neon CRON[1404]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:06:43] <Bytram> Jan 10 12:35:01 neon CRON[1411]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:06:45] <Bytram> Jan 10 12:39:01 neon CRON[1415]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
[14:06:50] <Bytram> Jan 10 12:45:01 neon CRON[1429]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:06:52] <Bytram> Jan 10 12:55:01 neon CRON[1473]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:06:54] <Bytram> Jan 10 13:05:01 neon CRON[1477]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:06:56] <Bytram> Jan 10 13:09:01 neon CRON[1485]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
[14:06:59] <Bytram> Jan 10 13:15:01 neon CRON[1502]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:07:01] <Bytram> Jan 10 13:17:01 neon CRON[1508]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
[14:07:03] <Bytram> Jan 10 13:25:01 neon CRON[1639]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:07:05] <Bytram> Jan 10 13:35:01 neon CRON[1642]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:07:07] <Bytram> Jan 10 13:39:01 neon CRON[1646]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
[14:07:10] <Bytram> Jan 10 13:45:01 neon CRON[1672]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:07:12] <Bytram> Jan 10 13:55:01 neon CRON[1777]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:07:14] <Bytram> .
[14:07:49] <SemperOSS> Try the same only for syslog.1, please
[14:08:44] <Bytram> root@neon:/etc/cron.d# grep 'Jan 10.*CRON' /var/log/syslog.1 | wc
[14:08:44] <Bytram> 0 0 0
[14:08:44] <Bytram> root@neon:/etc/cron.d#
[14:09:20] <SemperOSS> OK, so no data from midnight until 4 AM, hmmmm
[14:09:37] <cosurgi> 4:45
[14:09:55] <Bytram> to confirm that:
[14:09:56] <Bytram> root@neon:/etc/cron.d# tail /var/log/syslog.1
[14:09:56] <Bytram> Jan 9 05:55:01 neon CRON[8059]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:09:56] <Bytram> Jan 9 06:05:01 neon CRON[8066]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:09:56] <Bytram> Jan 9 06:09:01 neon CRON[8069]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
[14:09:57] <Bytram> Jan 9 06:15:01 neon CRON[8081]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:09:59] <Bytram> Jan 9 06:17:01 neon CRON[8088]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
[14:10:01] <Bytram> Jan 9 06:25:01 neon CRON[8098]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
[14:10:04] <Bytram> Jan 9 06:25:01 neon CRON[8099]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:10:06] <Bytram> Jan 9 06:35:01 neon CRON[8142]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:10:08] <Bytram> Jan 9 06:39:01 neon CRON[8149]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
[14:10:11] <Bytram> Jan 9 06:45:01 neon CRON[8163]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
[14:10:13] <Bytram> root@neon:/etc/cron.d#
[14:10:44] <Bytram> ugh; afk, brb
[14:10:59] <Bytram> oh, and expecting a phone call too... may be a bit
[14:11:00] <Bytram> sorry!
[14:11:18] * chromas dials M for Marty
[14:11:20] <SemperOSS> Bytram: Laters then
[14:14:07] <SemperOSS> Where have the debian-sa1 commands gone between Jan 9 06:45 to Jan 10 04:09? And again missing between Jan 10 04:45 to 12:15?
[14:20:38] -!- SoyCow4338 has quit [Quit: Web client closed]
[15:46:36] <cosurgi> IMHO it means that the computer was down/not working/locked up
[16:08:54] <SemperOSS>
[16:26:07] <FatPhil> OK, ssh in from outside world works, kinit works, ssh to other hosts work, and sudo works - I'm hot to trot
[16:29:54] <FatPhil> neon seems to have gone kaput some time after 04:45
[16:30:45] <FatPhil> oddly, nothing's in /var/log/sysstat - there's little point in having the package if you're not going to use it
[16:31:04] <FatPhil> HAving said that, it was a regular heartbeat in the syslog, so did have some use
[16:33:10] <FatPhil> nothing interesting in kern.log, so probably not a device failing to respond.
[18:21:09] -!- fyngyrz_ [fyngyrz_!~fyngyrz@66.171.jo.vll] has joined #dev
[18:21:09] -!- fyngyrz has quit [Read error: Connection reset by peer]
[18:28:19] -!- drussell has quit [Ping timeout: 248 seconds]
[18:49:51] -!- fyngyrz_ has quit [Quit: Leaving...]
[18:49:54] -!- fyngyrz [fyngyrz!~fyngyrz@Soylent/Staff/Editor/fyngyrz] has joined #dev
[20:05:50] -!- MrPlow has quit [Remote host closed the connection]
[20:06:01] -!- MrPlow [MrPlow!MrPlow@Soylent/BotArmy] has joined #dev
[20:06:01] -!- MrPlow has quit [Changing host]
[20:06:01] -!- MrPlow [MrPlow!MrPlow@nsa.gov] has joined #dev
[20:06:30] -!- MrPlow has quit [Remote host closed the connection]
[20:07:01] -!- MrPlow [MrPlow!MrPlow@Soylent/BotArmy] has joined #dev
[20:07:01] -!- MrPlow has quit [Changing host]
[20:07:01] -!- MrPlow [MrPlow!MrPlow@nsa.gov] has joined #dev
[20:07:43] -!- MrPlow has quit [Remote host closed the connection]
[20:08:01] -!- MrPlow [MrPlow!MrPlow@Soylent/BotArmy] has joined #dev
[20:08:01] -!- MrPlow has quit [Changing host]
[20:08:01] -!- MrPlow [MrPlow!MrPlow@nsa.gov] has joined #dev
[20:08:30] -!- MrPlow has quit [Remote host closed the connection]
[20:09:01] -!- MrPlow [MrPlow!MrPlow@Soylent/BotArmy] has joined #dev
[20:09:01] -!- MrPlow has quit [Changing host]
[20:09:01] -!- MrPlow [MrPlow!MrPlow@nsa.gov] has joined #dev
[20:22:12] -!- MrPlow has quit [Remote host closed the connection]
[20:23:01] -!- MrPlow [MrPlow!MrPlow@Soylent/BotArmy] has joined #dev
[20:23:01] -!- MrPlow has quit [Changing host]
[20:23:01] -!- MrPlow [MrPlow!MrPlow@nsa.gov] has joined #dev
[20:25:43] -!- MrPlow has quit [Remote host closed the connection]
[20:26:01] -!- MrPlow [MrPlow!MrPlow@Soylent/BotArmy] has joined #dev
[20:26:01] -!- MrPlow has quit [Changing host]
[20:26:01] -!- MrPlow [MrPlow!MrPlow@nsa.gov] has joined #dev