#dev | Logs for 2014-07-21
« return
[00:08:01] juggs|afk is now known as juggs
[00:12:33] TheMightyBuzzard is now known as TheMightyBuzzard|Fishing
[01:07:55] -!- Tachyon [Tachyon!~Tachyon@hollhb.kolej.mff.cuni.cz] has joined #dev
[01:08:03] -!- Tachyon has quit [Remote host closed the connection]
[01:32:04] -!- Tachyon [Tachyon!~Tachyon@hollhb.kolej.mff.cuni.cz] has joined #dev
[07:13:07] juggs is now known as juggs|afk
[09:55:52] -!- Bytram|away [Bytram|away!~pc@Soylent/Staff/Developer/martyb] has joined #dev
[09:55:52] -!- mode/#dev [+v Bytram|away] by SkyNet
[09:56:23] Bytram|away is now known as Bytram
[09:57:43] <Bytram> paulej72: ping
[09:57:46] <Bytram> NCommander: ping
[09:57:59] <NCommander> Bytram, pong
[09:58:00] <Bytram> am having trouble logging into the site...
[09:58:04] <NCommander> Bytram, the site is kinda flakely at the moment
[09:58:07] <NCommander> Gluster shat itself hard
[09:58:11] <Bytram> ouch
[09:58:13] <NCommander> Yeah
[09:58:19] <NCommander> I only caught it because the backup went boom
[09:58:22] <Bytram> just wanted to make sure that someone who could do something about new about it, too.
[09:58:24] <NCommander> Due to access issues
[09:58:30] <Bytram> ouch ouch.
[09:58:46] <Bytram> anything I can do to help (besides shutup atm) ?
[09:59:21] <NCommander> Bytram, I'm scanning for more corruption issues, then going to flush everything and restart slash fully
[09:59:26] <NCommander> I'm not sure what happened
[09:59:36] <NCommander> At some point, it looks like the self-healing daemon on boron shat itself
[09:59:45] <NCommander> And then boron and the rest of the cluster got out of sync and well
[09:59:46] <NCommander> boom
[09:59:47] <Bytram> can you put up the static page that says the site is unavailable - cuts down on confusion
[09:59:54] <NCommander> It shouldn't be down entirely
[09:59:55] <NCommander> Ugh
[10:00:08] <Bytram> I'm getting a 503 when trying to login
[10:00:12] <Bytram> let me try again
[10:00:38] <Bytram> strange... now I'm on.
[10:00:53] <NCommander> Bytram, fluorine is flaking out
[10:01:03] <NCommander> The load-balancer isn't properly detecting if its active or not
[10:01:08] <NCommander> I need to fix the check script
[10:01:22] <NCommander> Ok, I manually failed fluorine
[10:01:29] <NCommander> Hydrogen should pick up the slack and the site should be "mostly" up
[10:02:33] <NCommander> Technical description:
[10:02:33] <NCommander> 502 Bad Gateway - Response Error, a bad response was received from another proxy server or the destination origin server.
[10:02:34] <NCommander> Er
[10:02:34] <NCommander> shit
[10:03:00] <Bytram> I'm putting up a brief blurb about the system being unstable.
[10:05:33] <NCommander> Bytram, yeah, its really unhappy
[10:05:37] <NCommander> Looking to see what happened
[10:05:38] <Bytram> nod nod
[10:05:42] <NCommander> The system itself *is* up
[10:05:48] <NCommander> I can access it on loopback from hydrogen
[10:05:55] <NCommander> The loadbalancer broke its brain
[10:06:36] <NCommander> or I might have just broken my own setup
[10:06:37] <NCommander> 198.58.127.45 soylentnews.org
[10:06:41] <NCommander> Bytram, ugh
[10:07:02] <Bytram> As of 6:00AM EDT, SoylentNews is experiencing some technically difficulties; service may be unstable.
[10:07:02] <Bytram> It is being worked on and we hope to have things running smoothly again in a short while.
[10:07:02] <Bytram> We apologize for any inconvenience.
[10:07:06] <NCommander> Bytram, we have a site news box for this
[10:07:29] <NCommander> Bytram, don't run articles on this unless its exceptional
[10:07:33] <NCommander> I think I have the errors cleared though
[10:07:46] <Bytram> ok, where is it? And do I have the privs to update it?
[10:07:47] <NCommander> The file system appears to be consistent again, so I'm firing up fluorine and returning us back to normal
[10:07:54] <Bytram> nod nod
[10:07:55] <NCommander> Bytram, its buried in the templates, and I'm not sure
[10:08:22] <Bytram> this was a quick-n-dirty way to get the msg out, for now.
[10:08:40] <Bytram> wil do it better when we have more time for my training
[10:09:47] <NCommander> argh bahahiehfoiwehfoiwehoifwhe
[10:09:50] * NCommander swears more
[10:09:56] <NCommander> Ok, a few files got corrupted
[10:09:59] <NCommander> Argh
[10:10:07] <NCommander> I think we have a backup of the keytab which is the most critical one
[10:10:22] <Bytram> on oxygen?
[10:10:37] * Bytram needs to get a copy of IP addresses to add to his hosts file
[10:12:22] <NCommander> Bytram, gah, looks like we backed up a bad one
[10:12:31] * NCommander regenerated the keytab by hand which is annoying but so be it
[10:12:39] <Bytram> that would explain why I can't get to boron using putty?
[10:12:45] <NCommander> I don't have full incremental backups setup
[10:12:49] <NCommander> Bytram, boron should be up just file
[10:12:50] <NCommander> *fine
[10:13:00] <Bytram> hrrm
[10:13:43] <Bytram> ok, I'm on. /me needs a not-so-ridiculously long pwd
[10:14:05] <NCommander> Port 80 http source table http 2 up, 0 down
[10:14:09] <NCommander> Ok
[10:14:12] <NCommander> We're back in business
[10:14:41] <Bytram> y!
[10:14:44] <Bytram> yay!
[10:14:59] <NCommander> Wonder when the heck that happened :-/
[10:15:11] <NCommander> !todo finish rebuilding helium ASAP
[10:15:11] <Bender> todo item 6 added
[10:16:46] <Bytram> NCommander: at about 0530 EDT, I tried to get on the site... it ignored my cookie and each time I tried to logon, it just kept coming back to the story screen I was on, with empty fields for my nick and password. I assumed it was just a cache issue,
[10:17:23] <NCommander> Bytram, that's what the site does if the backend has shat itself
[10:17:35] <NCommander> Usually because we're bouncing slash and its expected to return in a minute or so
[10:17:39] <Bytram> that was on my phone. I then powered up my laptop around 0555 EDT, to ask someone to kick the cache server and that's when I bumped into you here.
[10:17:51] <Bytram> nod nod
[10:18:41] <Bytram> when I found the same symptoms on my laptop -- no longer logged in -- and trying to login in gave me a 503, I figured it was a bit more "interesting" problem.
[10:18:55] <Bytram> so, are we good? and how can I tell?
[10:21:23] <NCommander> Bytram, I think we're good
[10:21:27] <NCommander> I'm not seeing any errors here
[10:21:38] <NCommander> I'm re-running the backup which is how I found the corruption issue in the first place
[10:22:01] <Bytram> saving errors for later re-use? How *creative* !! :/
[10:22:18] <NCommander> Bytram, we don't have a 100% full backup of the site since I hadn't setup the crontabs, that was on my TODO for the night
[10:22:26] <Bytram> LOL
[10:22:51] <NCommander> Anyway, that will be fixed as soon as I confirm that this passes
[10:22:56] <Bytram> nod nod
[10:23:31] <Bytram> seems like icinga still isn't working... so I can't help you out much on this end.
[10:25:09] <NCommander> Bytram, that is xlefay's baby, he can fix it :-)
[10:25:20] <Bytram> I just went to pull the "system issues" story, and it looks like you did it already?
[10:25:53] <Bytram> NCommander: yeah, i know it's xlefay's... from over here, my toolkit is a bit, ummm, limited.
[10:27:20] <NCommander> Bytram, yeah, I did
[10:28:07] <Bytram> looks strange to have a story that went live and had absolutely zero hits. My editor skills are definitely on the wane. :(
[10:28:18] <Bytram> coffee++
[10:28:18] <Bender> karma - coffee: 1
[11:18:03] TheMightyBuzzard|Fishing is now known as TheMightyBuzzard|
[11:18:10] TheMightyBuzzard| is now known as TheMightyBuzzard
[11:46:45] -!- cosurgi [cosurgi!~cosurgi@ocee.bl.pg.gda.pl] has joined #dev
[11:47:45] cosurgi is now known as SoyGuest71713
[11:57:01] <paulej72> NCommander: was this a Kerberos issue. if so I was trying to set up juggs on Saturday with a Kerberos account and what ever I did he could not kinit.
[12:03:10] <NCommander> paulej72, I don't know what happened; I rewrote the keytab file
[12:05:06] <paulej72> NCommander: can you look into juggs kinit issue when you have a few
[12:06:18] <paulej72> NCommander: also I am not sure how to specify a new cap'n module other than in a readme
[12:07:10] <NCommander> paulej72, we probably need to create something similiar to Bundle::Slash
[12:09:09] <TheMightyBuzzard> mornin folks
[12:11:01] <paulej72> TheMightyBuzzard: so how much stuff do you have left on UTF8 before we release it
[12:11:36] <TheMightyBuzzard> paulej72, none except the inevitable bug fixes
[12:12:14] <TheMightyBuzzard> oh wait, the readme.utf8
[12:12:32] <TheMightyBuzzard> would help if the person upgrading the db had instructions
[12:14:50] <TheMightyBuzzard> already have them written, hang on and i'll pull request them up
[12:14:59] <paulej72> we need to get some full testing done on it as well
[12:15:56] <paulej72> NCommander: what is the staus of the wildcard cert
[12:18:06] <NCommander> paulej72, not sure
[12:18:07] <NCommander> Argh
[12:18:17] <TheMightyBuzzard> paulej72, hence its time on dev, yep
[12:18:19] * NCommander is suffering from "overload" today
[12:25:56] Bytram is now known as Bytram|afk
[12:28:50] -!- Tachyon has quit [Quit: De omnibus dubitandum est.]
[12:31:34] <TheMightyBuzzard> paulej72, new README.utf8 in.
[12:33:42] <TheMightyBuzzard> this concludes the unicode work until someone finds me a bug or i have time to do up the 4byte unicode work.
[12:38:00] <TheMightyBuzzard> ima go head and merge the new README.utf8. it's instructions rather than code and we aren't going to lose the old one.
[12:42:24] Bytram|afk is now known as Bytram
[12:48:01] <TheMightyBuzzard> paulej72, nother pull request in. non-unicode-related, you may merge or reject as you like. https://github.com
[12:58:34] <TheMightyBuzzard> Think I'll work on https://github.com unless anyone has anything more pressing.
[13:00:15] <paulej72> TheMightyBuzzard: i do not think there is more pressing issues, but we do need to start buttoning up what we got so we can do a release on 14.08.01 or there abouts
[13:01:56] <TheMightyBuzzard> nod nod. i think i can get 204 cleared in a couple days tops if you want it in for 14.08.01. haven't looked yet though.
[13:20:10] <TheMightyBuzzard> hrm. way i read it, you already CAN set archive_delay=0 and have stories never archive. likely other logic that doesn't do things as sanely as the daily_archive.pl script though.
[13:26:25] <paulej72> TheMightyBuzzard: i was thinking that we would need a second var like comment_timeout and have the comment sytem stop after so many days and another one for the mod system. You could set all three independently.
[13:28:15] <TheMightyBuzzard> nod nod, mod already has its own variable but adding a comment one would be a good idea.
[13:28:31] <TheMightyBuzzard> archive_delay_mod
[13:30:04] <paulej72> TheMightyBuzzard: we sort of need this as we have spammers hitting older stories with viagra comments. would like to turn off comments and mods on older stories to combat this
[13:30:38] <TheMightyBuzzard> yep, we could probably set it as low as 30 and not hurt anyone's feelings.
[13:32:14] <paulej72> We could probably set it as low as 15 and not many would notice.
[13:33:45] <TheMightyBuzzard> likely. okay, i'll get on adding the var and moving comments to use it instead of archive_delay then. after that's done i'll go through and make sure all the archive_delay uses play nice with zero.
[15:19:14] Bytram is now known as Bytram|afk
[15:56:42] -!- Alberto [Alberto!~Alberto@187.143.gwk.gr] has joined #dev
[16:00:11] Bytram|afk is now known as Bytram|away
[16:01:32] -!- Bytram|away has quit [Quit: Leaving]
[16:17:28] -!- Alberto_ [Alberto_!~Alberto@187.135.wls.nog] has joined #dev
[16:18:15] -!- Alberto has quit [Killed (NickServ (GHOST command used by Alberto_!~Alberto@187.135.wls.nog))]
[16:18:24] Alberto_ is now known as Alberto
[17:03:43] -!- Alberto_ [Alberto_!~Alberto@187.214.mnu.li] has joined #dev
[17:07:35] -!- Alberto has quit [Ping timeout: 244 seconds]
[17:35:51] Alberto_ is now known as Alberto
[17:36:28] Alberto is now known as ElKommander
[17:36:51] ElKommander is now known as TheMightyBrazzers
[18:40:28] TheMightyBrazzers is now known as TheMightyBrazzersBorg
[19:21:53] <xlefay> yo
[20:03:58] -!- Tachyon [Tachyon!Tachyon@hollhb.kolej.mff.cuni.cz] has joined #dev
[20:33:20] -!- Bytram|away [Bytram|away!~pc@Soylent/Staff/Developer/martyb] has joined #dev
[20:33:20] -!- mode/#dev [+v Bytram|away] by SkyNet
[20:35:13] Bytram|away is now known as Bytram
[21:58:02] Bytram is now known as Bytram|away
[21:58:32] -!- Bytram|away has quit [Quit: Leaving]
[22:09:50] -!- bspar has quit [Ping timeout: 244 seconds]
[22:11:21] TheMightyBrazzersBorg is now known as Alberto
[22:14:05] -!- bspar [bspar!~bspar@waol.bspar.org] has joined #dev
[22:18:16] -!- Alberto has quit [Changing host]
[22:18:16] -!- Alberto [Alberto!~Alberto@187.214.103.67] has joined #dev
[22:42:23] -!- bspar has quit [Ping timeout: 244 seconds]
[22:44:17] -!- bspar [bspar!~bspar@waol.bspar.org] has joined #dev
[23:01:44] juggs|afk is now known as juggs