#staff | Logs for 2014-03-21

« return
[00:09:27] <paulej72> NCommander: are you around?
[00:18:47] -!- mrcoolbp [mrcoolbp!~mrcoolbp@Soylent/Staff/mrcoolbp] has joined #staff
[00:18:47] -!- mode/#staff [+v mrcoolbp] by SkyNet
[00:20:51] <paulej72> OK I am still at work, so I am going home now that I have caught up on my irc logs. <voice+"terminator">I'll be back.</voice>
[00:21:45] paulej72 is now known as paulej72_away
[00:23:34] -!- pbnjoe has quit [Read error: Connection reset by peer]
[00:24:28] -!- pbnjoe [pbnjoe!~pbnjoe@Soylent/Users/313/pbnjoe] has joined #staff
[00:41:54] * NCommander is here
[00:43:06] <xlefay> wb :)
[00:44:42] <xlefay> :o
[00:44:51] <xlefay> I haven't smoked in at least 8 hours or so :o
[00:44:52] <Landon> ooh ooh
[00:44:53] <Landon> m etoo
[00:44:54] * Landon is here
[00:45:02] <xlefay> wb!
[00:47:52] <xlefay> Landon: are there any files in particular with Bender, I shouldn't put on Git?
[00:49:01] <Landon> hm
[00:49:23] <Landon> anything in .jsb you're going to need to be careful about
[00:49:42] <Landon> actually, the only things in .jsb that should even consider going to git are in myplugs directory
[00:49:52] <xlefay> Ok, but .jsb is auto created or not?
[00:50:00] <Landon> it is for the most part
[00:50:04] <Landon> it _does_ have custom plugins
[00:50:12] <Landon> like the soylent rss plugin
[00:50:16] <Landon> that uses our sylnt shortener
[00:51:11] <Landon> suggestions plugin has a hardcoded email
[00:51:27] <Landon> grep around for atheme
[00:51:34] <xlefay> Does the suggestion plugin still work? Does it have anti flood/spam measures?
[00:51:37] <Landon> I _might_ have bender's nickserv pass in there
[00:51:45] <Landon> I turned it off temporarily for whatever reason
[00:51:56] <xlefay> aah ok
[00:52:12] <Landon> oh right it got bork
[00:52:19] <xlefay> bork?
[00:52:19] <Landon> I needed to add more headers
[00:52:25] <xlefay> ooh the suggestion plugin?
[00:52:27] <Landon> because the SN mail server got a lot more picky
[00:52:28] <Landon> yeah
[00:52:34] <xlefay> Yeah it has some more checks rofl
[00:52:40] <Landon> the one it complained about was missing a Date header, but I'm sure there's one or two more I might not have
[00:53:25] <NCommander> so
[00:53:28] <NCommander> looking at devotee
[00:53:30] <Landon> twitter authentication is in a config file, so that's not a worry
[00:53:34] <NCommander> it wants GPG signed emails
[00:53:39] * NCommander thinks that may not work
[00:53:39] <xlefay> Good, good. :)
[00:53:50] <Landon> NCommander: oh lord, we only have one person on the mailing list even signing emails
[00:53:59] <Landon> and we're an uber techy group
[00:54:07] * NCommander only got that resetup today
[00:54:18] <Landon> I should note, my muscle reflex for mutt when sending emails is p-c-y
[00:54:18] <xlefay> And we can't even confirm he's authentic, least that's why my mail client says!
[00:54:19] <Landon> :(
[00:54:23] <xlefay> It has red flags and all that
[00:54:29] <Landon> p-c-y == PGP settings -> clear -> send
[00:55:49] * Landon goes to play with the unreal engine
[00:56:11] <xlefay> Thanks Landon for the infos :)
[00:56:46] <mrcoolbp> so suggestions email piping is still borked?
[00:57:02] <Landon> yep
[00:57:41] <xlefay> Adding that header shouldn't be to difficult, I'm more concerned about people flooding that command though
[00:58:01] <Landon> yeah, but
[00:58:14] <Landon> I'm of the opinion that it's easy enough to fix that we shouldn't do anything about it until flooding happens
[00:58:28] <Landon> no one's going to send 400TB of suggestions emails a second ;)
[00:58:36] <xlefay> !grab Landon
[00:58:36] <Bender> Added quote 75
[00:59:07] <mrcoolbp> check this out https://intertwinkles.org
[00:59:10] <xlefay> hah, that's funny
[00:59:19] <mrcoolbp> it might work for staff decision making
[01:00:40] <Landon> https://github.com ah! cool :)
[01:00:43] * xlefay likes the effect of hovering over 'twinkles'
[01:01:01] <Landon> xlefay: do you read HN?
[01:01:23] <xlefay> Not often, anything interesting?
[01:01:28] <Landon> http://elrumordelaluz.github.io
[01:01:30] <Landon> *evil grin*
[01:01:59] <xlefay> ooooh
[01:02:01] <xlefay> *bookmarks*
[01:02:02] <Landon> NCommander: april 1 should have the whole site a'shakin
[01:02:15] <xlefay> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[01:02:39] <NCommander> Landon, migration to HURD :-P
[01:03:06] <NCommander> Landon, you can't validaate my GPG key?'
[01:03:12] * xlefay wonders why SN doesn't have a humans.txt!
[01:03:19] <Landon> NCommander: nope, I can't validate it either
[01:03:22] <Landon> little s
[01:03:24] <Landon> :(
[01:03:31] <NCommander> Landon, probably need to import my public key
[01:04:05] <NCommander> Landon, gpg --recv-key 37F011893BAE3611C45B 8E15733E1A42D2247639
[01:04:16] <NCommander> Landon, gpg --recv-key 37F011893BAE3611C45B8E15733E1A42D2247639
[01:04:23] <mrcoolbp> looks like intertwinkles depends on javascript
[01:04:34] <xlefay> Keyserver?
[01:04:55] <Landon> hm
[01:04:59] <Landon> it's taking a long time to request the key
[01:05:05] <Landon> I'm using subkeys.pgp.net
[01:06:50] <xlefay> same here
[01:07:01] <NCommander> Landon, gpg --keyserver keyserver.ubuntu.com:1137 --recv-key 37F011893BAE3611C45B8E15733E1A42D2247639
[01:07:11] <NCommander> er
[01:07:16] <NCommander> Landon, gpg --keyserver keyserver.ubuntu.com --recv-key 37F011893BAE3611C45B8E15733E1A42D2247639
[01:07:23] <xlefay> I was just about to say that rofl
[01:07:27] <xlefay> that one works fine
[01:07:39] <Landon> yay, capital S
[01:07:40] <Landon> :)
[01:07:57] <Landon> for some reason, mutt isn't autodetecting that you signed the email
[01:08:10] <NCommander> Might be due to the fact that its on a mailing list
[01:08:30] <xlefay> it detects it here though ;-)
[01:08:33] <NCommander> I've had issues with mutt and PGP emails
[01:08:34] <Landon> perhaps, but I can tell it "Check key" and it'll find it
[01:08:38] <xlefay> must be mutt
[01:08:43] <NCommander> xlefay, yours probably says verified but not trusted
[01:08:48] <NCommander> Unless you have a chain of trust to my key
[01:09:00] <xlefay> NCommander: just imported your key, now it's no longer yellowish
[01:09:13] <NCommander> xlefay, with Enigmamail, it will be blue if its trusted
[01:09:36] <Landon> mmhmm
[01:10:39] <xlefay> "X-Ham-Report: Spam detection software, running on the system "ns1.web-refinery.com", has identified this incoming email as possible spam. " lol
[01:11:42] <xlefay> don't see it as being yellow but don't see anything indicated it's trusted either... weird kmail *opens the docs*
[01:16:50] <mrcoolbp> is the svc linode overloaded??
[01:16:54] <mrcoolbp> I'm having issues
[01:17:05] <mrcoolbp> Sorry! This site is experiencing technical difficulties.
[01:17:05] <mrcoolbp> Try waiting a few minutes and reloading.
[01:18:46] -!- combatserver has quit [Quit: zzz]
[01:19:55] -!- combatserver [combatserver!~chatzilla@k-21-827-19-772.hsd5.wa.comcast.net] has joined #staff
[01:19:55] <xlefay> mrcoolbp: which site? The wiki?
[01:20:10] <mrcoolbp> yeah and I'm having trouble with my mailbox as well
[01:20:46] <xlefay> There's definitely something with the wiki, the httpd logs show some fatal errors concerning functions *and why would the wiki be executing a system command*?!
[01:20:48] <xlefay> FunPika: ^
[01:22:13] <xlefay> Also, in the future saying the DB is unavailable for the wiki might be more useful
[01:22:33] <xlefay> FunPika: just logged into SSH, guessing he's on it
[01:24:19] <FunPika> all those fatals look like php functions
[01:24:38] * stderr is reading the backlog before bed...
[01:24:55] <xlefay> FunPika: correct
[01:25:08] <stderr> NCommander: Devotee can do without GPG signed emails. Just skip to the dvt-parse part...
[01:25:10] * xlefay starts to wonder why MediaWiki wouldn't just use PDO, those insensitive clods!
[01:25:11] <Landon> little bit of light reading, stderr
[01:25:21] paulej72_away is now known as paulej72
[01:25:31] <mrcoolbp> I cannot get into my mailbox via the webclient either that's why I was suggesting the svc linode might be unhappy
[01:26:03] <xlefay> ooh
[01:26:04] <xlefay> I see the issue
[01:26:07] <xlefay> someone killed the mysql
[01:26:14] <FunPika> Oh look at all the "unable to allocate memory" and similar
[01:26:25] <xlefay> that means, the mysql was killed because of lack of memory
[01:26:35] <mrcoolbp> didn't NCommander say that linode was overloaded?
[01:26:36] <FunPika> 30k free...fuuuuuuuuuu
[01:26:41] <FunPika> *m
[01:26:45] <xlefay> NCommander: I'd say we should upgrade the svc box or get another one, really soon ;)
[01:26:47] <FunPika> *30M
[01:27:08] <xlefay> FunPika: don't check now
[01:27:20] <FunPika> 9900k free, i still have top open! :(
[01:27:33] <xlefay> Aah don't we all love to read how the kernel is killing things?
[01:27:36] <stderr> Landon: Yes. :-)
[01:27:50] <xlefay> spamd is down now
[01:27:52] <paulej72> NCommander: are you here now?
[01:28:28] <xlefay> I thought all SVC db crap was done on the DB server.. because it being the DB server, ya'know
[01:28:34] <paulej72> mrcoolbp: you mentioned to NCommander earlier about problems with the slashboxes what was the issue?
[01:28:36] <stderr> I'm actually working a little bit on making a simple web frontend for devotee to see how easy it might be to replace the current voting booth.
[01:29:09] <mrcoolbp> paulej72: only that one of them (lower right) was still linking to FF which I believe he nuked already ?
[01:29:17] <xlefay> 'FF'?
[01:29:22] <xlefay> Fusion Forge?
[01:29:36] <stderr> But I should go to bed now... And I probably won't have time to do any coding tomorrow at work. The department is moving to another build Monday, so most of the day tomorrow I'll be packing down all my computers and stuff...
[01:29:59] <mrcoolbp> Fusion forge yes
[01:30:23] <paulej72> I have that fix in my stuff that has not been merge red or has been merged and not put on productn yet. We need to get out testing server finished.
[01:30:55] <xlefay> I'm just going to point out that there's a good chance no-one will be receiving mail anytime soon
[01:31:02] <mrcoolbp> at all
[01:31:06] <paulej72> mrcoolbp: see if it is fixed on slashcot and if it is then I have a fix in the pipeline
[01:31:06] <xlefay> *starts spamd again*
[01:31:12] <xlefay> but it's probably going to get killed again anyway
[01:32:01] <xlefay> PHP is running excessively, FunPika does MediaWiki know of the concept called 'cache'? If so, if it's not enabled, could we do that?
[01:32:31] <FunPika> yes it does...not 100% sure if its configured properly at the moment
[01:32:43] <xlefay> I'm calling this kernel evil, it kills spamd before it finishes starting up ;-)
[01:33:04] <xlefay> Could you look into tha?
[01:33:05] <xlefay> that*
[01:33:09] <mrcoolbp> paulej72 I will look now
[01:33:28] <mrcoolbp> paulej72 are you able to check out the svc linode? it's shitting the bed right now
[01:33:59] <mrcoolbp> email sent to staff is returning undelivered right now
[01:34:01] <paulej72> sorry I don't think I have access to that yet.
[01:34:03] <mrcoolbp> wiki is doqwn
[01:34:05] <mrcoolbp> down
[01:34:08] <xlefay> mrcoolbp: I'm on it ;-)
[01:34:18] <mrcoolbp> WTF? didn't we restore access to servers the other day?
[01:34:23] <mrcoolbp> xlefay: great
[01:34:32] <xlefay> not to worry, it'll probably break down again anyway
[01:34:34] <mrcoolbp> xlefay: give us a status update when you can
[01:34:59] <paulej72> what is the fqdn and ip address of that box
[01:35:05] <xlefay> Whatever happens, it'll just break down till either the frontend server gets up or we increase it's power or offload heavy duty services to another machine
[01:35:19] <xlefay> paulej72: just use wiki.soylentnews.org
[01:35:45] <mrcoolbp> xlefay: are you the only one with access to that box?
[01:35:54] <xlefay> wait..
[01:35:56] <mrcoolbp> s/box/linode/
[01:36:19] <xlefay> paulej72: check notice
[01:36:21] <xlefay> mrcoolbp: nope
[01:36:51] <mrcoolbp> well we can't send out anything on the mailing list...
[01:36:55] <xlefay> eeeh FunPika
[01:37:05] <xlefay> there are a dozen php wiki task running?
[01:37:21] <xlefay> apache 21126 0.0 0.3 307076 3108 ? S 00:31 0:00 /usr/bin/php /var/www/vhosts/wiki/w/maintenance/runJobs.php --maxjobs 1
[01:37:22] <xlefay> ^ 251
[01:37:30] <FunPika> what the hell? O_o
[01:37:37] <xlefay> Think that may be the culprit?
[01:38:07] <xlefay> total used free shared buffers cached
[01:38:08] <xlefay> Mem: 988 220 768 0 1 29
[01:38:12] <xlefay> fairly sure the kernel killed more crap
[01:38:26] <xlefay> and whatcha know? Spamd starts again!
[01:39:03] <FunPika> holy crap 700k free
[01:39:06] <FunPika> *M
[01:39:11] <FunPika> why do I keep saying k <_<
[01:39:14] <mrcoolbp> wiki seems to be up
[01:39:37] <mrcoolbp> mailbox is back up, good work xlefay
[01:39:37] <xlefay> OK, httpd, mysqld & spamd (all those killed are back up).
[01:40:04] <xlefay> FunPika: please look into why the wiki would run 251 tasks at once
[01:40:48] <xlefay> ehm
[01:40:55] <xlefay> it's starting all those processes again ;-) ;-)
[01:41:22] <stderr> On http://soylentnews.org there's a box for "Public Key". Is that shown anywhere?
[01:41:57] <xlefay> FunPika: I've renamed the 'runJobs.php' to 'meh.php' for now
[01:42:26] <stderr> ... and how tired was I when I filled out that form earlier? I pasted my ssh public key. :-)
[01:42:28] <FunPika> I'll BRB then I'll look at it some more
[01:42:30] <paulej72> stderr: probably not
[01:42:44] <xlefay> rofl
[01:42:52] <xlefay> think it is shown though
[01:43:03] <xlefay> or someone has a key in their sig, nvm
[01:43:39] <stderr> I don't see it in my own posts.
[01:44:10] <stderr> Don't see it on http://soylentnews.org either.
[01:44:31] <xlefay> All's good then
[01:45:12] <stderr> Oh, well... If an admin wants my ssh public key, they know where to find it... Hidden deep down in a database table somewhere...
[01:45:45] <paulej72> stderr: at least it was the public key not the private key
[01:50:16] <mrcoolbp> xlefay: should we document this "incident" ?
[01:50:39] <xlefay> It's not over yet.
[01:51:02] <xlefay> _something_ is still starting up those processes, even though they aren't really doing anything atm (since I renamed the file they started)
[01:51:10] <xlefay> also, we haven't even confirmed at this point it was those processes
[01:51:28] <mrcoolbp> I gotta run off, I'll catch ya'll tomorrow
[01:51:29] <xlefay> and I need to take a shower;-)
[01:51:34] <xlefay> alright, take care mate
[01:51:39] <mrcoolbp> later sir
[01:51:45] <xlefay> ;-)
[01:51:52] -!- mrcoolbp has quit []
[01:53:29] -!- robind has quit [Ping timeout: 246 seconds]
[01:58:41] <xlefay> FunPika: I'm thinking this may be a side effect of those missing functions, just a theory though, in which case it'd be pretty crappy code, any theories on your side?
[02:03:12] <FunPika> still not sure why its doing it
[02:03:50] <xlefay> I moved the file to 'meh.php' and recreated it with the contents '<?php die; ?>'
[02:04:05] <xlefay> it should exit directly _instead_ they stay open, doesn't make much sense
[02:04:46] <xlefay> FunPika: I'm guessing the wiki gets accessed and the jobs try to run?
[02:05:06] <FunPika> http://wiki.soylentnews.org There don't even appear to be jobs in the queue right now waiting to run.
[02:05:20] <FunPika> jobs="0" on the far right
[02:05:25] <xlefay> I notice
[02:06:08] <xlefay> It seems we've got a frontend server @ wiki.soylentnews.org now?
[02:06:43] <xlefay> and our httpd doesn't grab the x-forward-for, if any.
[02:08:14] <xlefay> Ok, so a x-forwarded-for is present but the SVC's httpd isn't picking it up, it's apache 2.2 so we'll need to get mod_rpaf in there
[02:09:29] <xlefay> apache(48)/48 <-- shouldn't that be the wiki user, since mod_ruid2 (.. and suphp, since that's all in the vhost.)?
[02:09:32] <FunPika> I'm not seeing any more missing function errors in the logs, so those were probably all side effects of no memory
[02:09:48] <xlefay> although, we're not using suphp, simply mod_php
[02:09:57] -!- mechanicjay [mechanicjay!~jhowe@Soylent/Staff/Developer/mechanicjay] has joined #staff
[02:09:57] -!- mode/#staff [+v mechanicjay] by SkyNet
[02:10:07] <xlefay> mechanicjay: yoo ;-)
[02:10:11] <xlefay> just the person we needed :P
[02:10:16] <mechanicjay> heya
[02:10:24] <mechanicjay> what's up?
[02:10:25] <xlefay> How are ya?
[02:10:33] <mechanicjay> Just got a linode alert for disk io on the services box
[02:10:39] <xlefay> aah ;-)
[02:10:47] <mechanicjay> ...what did you do?
[02:10:49] <mechanicjay> :)
[02:10:58] <xlefay> The service box was overloading and spinning up a php wiki maintenance script
[02:11:16] <xlefay> funny enough, after killing all those, only 400 MB of mem was in use but.. the problem remains
[02:11:34] <xlefay> there were 251 php processes that were eating but not doing a whole lot ;-)
[02:11:40] <mechanicjay> lovely
[02:12:03] <xlefay> and even now, php processes are still being sprung up and memory increases
[02:12:13] * FunPika just killed about 200MB worth of them :/
[02:12:21] <xlefay> Do we have mod_ruid, or suphp? (I'm asking because of the vhost config..)
[02:12:28] <mechanicjay> we do not
[02:12:44] <xlefay> auto generated configs?
[02:13:25] <mechanicjay> The wiki and forms were configs I ripped from bluehost. others are by hand. It's all just running as the system apache user
[02:13:57] <xlefay> FunPika: I've just established that the script does get run when someone hit the URL
[02:14:04] <mechanicjay> what incantation are you using to view the php procs
[02:14:06] <xlefay> mechanicjay: and we've got a frontend server for the wiki and such?
[02:14:17] <xlefay> ps aux | grep 'php' and we used htop
[02:15:11] <mechanicjay> xlefay: can you rephrase that question, i dont' get your meaning.
[02:15:28] <mechanicjay> ...ew, why is wiki.old doing anything
[02:15:36] <mechanicjay> lets nuke that.
[02:15:47] <xlefay> wait!
[02:15:52] <xlefay> mechanicjay: wiki.soylentnews.org points to another IP and it forwards x-forwarded-for but it doesn't seem we are using mod_rpaf
[02:16:07] <xlefay> thus our httpd logs show the server in front of the svc box
[02:16:31] <mechanicjay> Non-authoritative answer:
[02:16:31] <mechanicjay> Name: wiki.soylentnews.org
[02:16:31] <mechanicjay> Address: 72.14.184.41
[02:16:35] <xlefay> mechanicjay: I think FunPika recently upgraded the wiki, but I'm getting the feeling this issue has been going around for a while
[02:16:40] <mechanicjay> that's this box.
[02:16:43] <xlefay> [03/21/14 02:16:41] [DNS] Resolved wiki.soylentnews.org to: 198.58.118.73
[02:16:46] <xlefay> aah old dns cache prolly
[02:17:00] <FunPika> yeah i remember you complaining about the box being a bit sluggish before i did anything to the wiki
[02:17:01] <xlefay> you set it up to proxy I guess
[02:17:01] <mechanicjay> yes, that's the old linode box.
[02:17:13] <mechanicjay> I did to handle stale dns entries ;)
[02:17:24] <xlefay> FunPika: correct. and after I killed all those php processes, our memory usage went to 350 - 400
[02:17:35] <xlefay> mechanicjay: hehe that's good ;-)
[02:17:49] <xlefay> (in total, instead of maxing the memory out)
[02:18:02] <xlefay> > htop -u apache
[02:18:30] <xlefay> "headable top"
[02:18:58] <mechanicjay> I discovered htop about 2 months ago -- I'm starting to get a feel for it -- I think I really like it
[02:19:14] <xlefay> I've been using it for years :P
[02:19:18] <xlefay> It's something actually readable!
[02:19:22] <mechanicjay> indeed!~
[02:19:41] <xlefay> anyway what doesn't make sense is that I moved the file and put '<?php die; ?>' in the file that's getting hit
[02:19:57] <xlefay> meaning, the process should cancel immediately, instead they don't.
[02:20:04] <xlefay> Can we somehow trace that back?
[02:20:23] <xlefay> strace -p 22731
[02:20:28] <xlefay> ... it's just waiting, like wtf
[02:20:33] <FunPika> https://www.mediawiki.org Only quickly skimmed through this so far...but this sounds like a bug in MediaWiki causing problems.
[02:21:14] <mechanicjay> hang on, have we looked at the apache error logs -- there s aphp lib being called that doesn't exist by something.
[02:21:41] <xlefay> "Simply set $wgPhpCli = false; until the related bugs are fixed"
[02:21:47] -!- drussell has quit [Read error: Connection timed out]
[02:21:59] -!- bytram [bytram!~pc@Soylent/Staff/Developer/martyb] has joined #staff
[02:21:59] -!- mode/#staff [+v bytram] by SkyNet
[02:21:59] <xlefay> mechanicjay: I've notice those yes
[02:22:15] <FunPika> all right, I'll try setting that in LocalSettings.php
[02:22:32] -!- drussell [drussell!~drussell@205.233.pt.umv] has joined #staff
[02:22:55] <xlefay> So I'm suggesting we install mod_ruid2 and start using cgroups to prevent this crap from downing the server(s) in the future ;-)
[02:23:03] <xlefay> s/\'m//
[02:23:58] <xlefay> That would however mean, we'd have to create users per vhost, but I don't think that's a big issue
[02:24:01] <mechanicjay> http://www.mediawiki.org
[02:24:40] <xlefay> mechanicjay: the fact there are so many missing extensions is just silly
[02:24:55] <xlefay> oh wait
[02:25:07] <FunPika> looks like that fixed it for now at least
[02:25:12] <FunPika> not seeing those processes popping up now
[02:25:15] <xlefay> nvm, misread... /me notes they should definitely make those log excerpts more readable
[02:25:21] <mechanicjay> yeah, theyr'e not missing media wiki is running out some sort of memory
[02:25:23] <xlefay> FunPika: then I should rename 'meh.php' back ;-)
[02:25:32] <xlefay> mechanicjay: the entire server was running out of memory
[02:25:44] <mechanicjay> xlefay: indeed
[02:25:46] <xlefay> the kernel killed MySQL, spamd & httpd iirc
[02:25:52] <FunPika> down to less than 10M at once point
[02:25:56] <FunPika> *one
[02:26:00] <mechanicjay> :(
[02:26:19] <mechanicjay> it's been in a steady state for a couple week though, what changed tonight?
[02:26:58] <xlefay> Update of mediawiki I'm guessing
[02:27:08] <xlefay> I do suspect it was already drowning the server earlier though
[02:27:28] <xlefay> 310 mb mem in use
[02:27:51] <xlefay> 63 swap, I'd say life is good at this point
[02:28:04] <mechanicjay> indeed
[02:28:52] <xlefay> Anyway, I should send NC my public key for linode access, can you add me to those alert messages?
[02:29:07] <xlefay> (I'm also wondering whether we set up service alerts at linode yet, NC mentioned those)
[02:29:21] <xlefay> .. when we were discussing monitoring options the other day
[02:30:13] <mechanicjay> let me dig around and see if I can
[02:30:28] <xlefay> Awesome ;-)
[02:31:18] <xlefay> anyway, what do you think of the mod_ruid2 and cgroups?
[02:31:20] <mechanicjay> actually, do you have access to the linode account?
[02:31:34] <xlefay> nope, I don't have access to any linode account atm
[02:32:15] <xlefay> only just got my head nailed in since maybe 12 hours or so :P
[02:32:36] <xlefay> (for those missing the reference, nail guns were involved)
[02:33:03] <mechanicjay> I seem to just get the alerts by virtue of having a login on the account -- don't see a place to explicitly add anyone for alerts
[02:33:12] <xlefay> ah ok
[02:34:04] <xlefay> so cgroups, ever worked with them?
[02:34:10] <mechanicjay> Given the resource constrains of the box at least cgroups makes a lot of sense
[02:34:14] <mechanicjay> no, reading up on them now
[02:34:24] <xlefay> I haven't used them either, but I'm fairly sure we can make it work ;)
[02:34:35] <xlefay> mod_ruid2 would force stuff to run under their own users
[02:34:55] <xlefay> e.g. wiki gets an user 'wiki', everytime the wiki gets accessed stuff gets executed under the wiki account instead of apache
[02:34:55] <mechanicjay> for fucks sake.. Did you catch this function in the runJobs.php script?
[02:35:02] <mechanicjay> public function memoryLimit() {
[02:35:13] <mechanicjay> // Don't eat all memory on the machine if we get a bad job.
[02:35:22] <mechanicjay> derp, nice sanity checking guys.
[02:35:25] <xlefay> LOL
[02:35:51] <xlefay> imo, they should set a pid and check if the pid exists, if so, don't start a fucking process
[02:36:32] <xlefay> silly thing was mechanicjay, there were no jobs to be done
[02:38:13] -!- drussell has quit [Read error: Connection timed out]
[02:38:17] <mechanicjay> It just struck me as funny
[02:38:55] <mechanicjay> memory usage seems stable
[02:38:57] <mechanicjay> nice
[02:39:11] <mechanicjay> now..how much more crap can we cram on this box?
[02:39:16] <mechanicjay> ;)
[02:39:32] -!- drussell [drussell!~drussell@205.233.pt.umv] has joined #staff
[02:39:39] <xlefay> if we implement cgroups alone, quite a bit
[02:39:43] <xlefay> if it's wise, is another question
[02:40:08] <xlefay> only using 300 MB mem, 700 MB free, we should be able to cram quite a bit more onto it
[02:40:17] <mechanicjay> anyway, process separation under different user accounts is usually a good idea, though I usually only do that for security reasons. Like if one site gets owned, it protects the other kind of thing
[02:40:22] * xlefay notes he fucking hate bash
[02:40:47] <bytram> Hi there! What's up?
[02:40:49] <mechanicjay> do we need to use both modules in conjunction?
[02:40:55] <paulej72> xlefay: what i wrong with bash?
[02:41:32] <xlefay> paulej72: I'm using to fish.. meaning, I type on thing once, it detects the pattern if I'm writing it again, and I can just press right arrow ;-)
[02:41:45] <xlefay> also typing a command, 'ls' I can press up and get everything matching that history. I'm lazy ;-)
[02:41:53] <xlefay> mechanicjay: cgroups is a kernel thing afaik
[02:42:04] <xlefay> as for mod_ruid2, it just forces setuid per vhost
[02:42:35] <xlefay> we could use suphp or whatever, but that'll only work for PHP, but afaik mod_ruid2 works for pretty much everything
[02:43:18] <xlefay> Really, if you get the chance, try fish... you'll love it
[02:44:05] <mechanicjay> xlefay: yeah that makes sense. Def willing to test it out -- especially interested in cgroups.
[02:44:26] <mechanicjay> So, if we only we could get a copy of the linode vm to f- with...
[02:44:30] <xlefay> cgroups will save us a whole lot of trouble like this in the future
[02:44:36] <paulej72> I'll give it a try sometime. I'll need to see if my standard repp has fish
[02:44:38] <xlefay> exactly
[02:45:14] <xlefay> paulej72: 100% honest, I thought it was lame at first and bash was absolutely the best but tried it anyway, never look back ;-)
[02:45:41] <xlefay> the user accounts have one more bonus! cd ~wiki ;=)
[02:45:41] <mechanicjay> I hate my ERP servers -- the default shell is ksh
[02:45:48] <xlefay> ugh
[02:45:55] <xlefay> I feel for you
[02:45:57] <paulej72> ugh is right
[02:46:02] <mechanicjay> pretty much the first thing I do is /bin/bash when I login
[02:46:41] <xlefay> The only thing I dislike about fish at this point is that I have a habit of writing statements and stuff in Bash and it doesn't accept that ;)
[02:46:46] <mechanicjay> I'm hesitant to change roots default shell because alot of stuff needs to be run as root, and the scripts are make for ksh
[02:47:01] <xlefay> ooh.. don't they just have #!/bin/bash?
[02:47:16] <mechanicjay> I don't trust that all do
[02:47:28] <mechanicjay> And I'm like tertiary admin them, so it's really not my place
[02:47:37] <xlefay> aah
[02:47:47] <xlefay> same reason why I didn't install fish :P
[02:47:59] <xlefay> besides it not being in the standard centos repos *swears*
[02:48:40] <paulej72> that answers my question probably not in the springdale repos either.
[02:49:06] <xlefay> it's silly how much is omitted in the default centos repos
[02:49:12] <xlefay> paulej72: yum search fish | grep shell
[02:49:22] <mechanicjay> okay, well. Let's figure out how get mod_ruid2 and cgroups working and see if a migration plan is feasable.
[02:49:27] <mechanicjay> xlefay: that was EPEL is for
[02:49:50] <paulej72> xlefay: too lazy to ssh into one of my boxes to see
[02:49:58] <xlefay> "fish.x86_64 : A friendly interactive shell"
[02:50:01] <mechanicjay> no fish in the suse standard repos.
[02:50:12] <xlefay> It's in the korora/fedora repos, fortunately
[02:50:36] <xlefay> nope not in epel either
[02:50:44] <xlefay> iirc I compiled it myself on my server
[02:51:01] <xlefay> http://fishshell.com
[02:51:03] <FunPika> http://packages.ubuntu.com Ubuntu has it
[02:51:13] <xlefay> nope, downloaded that rpm from fishshell.com ;)
[02:51:27] <paulej72> cool it is in the unsupported repo for springdale linux (checked the website springdale.princeton.edu)
[02:51:32] <xlefay> FunPika: it's become more and more apparent that ubuntu pretty much has everything
[02:51:35] <mechanicjay> ooh, they have a suse binary built!
[02:51:44] <xlefay> hehe that's nice paulej72, try it out :P
[02:51:49] <xlefay> mechanicjay: try it out
[02:51:52] <mechanicjay> ..installing
[02:52:09] * xlefay 's master plan: having other sysadmins test it out so they'll love it and it'll get installed :P
[02:52:09] <FunPika> https://packages.debian.org as does Debian stable
[02:52:41] <xlefay> It's absolutely ridiculous how much debian/ubuntu have and how little CentOS has
[02:53:31] <mechanicjay> they're suse rpm is broken
[02:53:40] <xlefay> :<
[02:53:41] <mechanicjay> error: Failed dependencies:
[02:53:41] <mechanicjay> which is needed by fish-2.1.0-2.1.x86_64
[02:53:45] <xlefay> ooooooooooooh
[02:53:51] <xlefay> I've got a postgresql update
[02:54:02] <xlefay> hmm
[02:54:03] <mechanicjay> it would be fucking helpful if the dependecies it needed were listed :p
[02:54:24] <xlefay> isn't it listed?
[02:54:35] <xlefay> iirc, yum does list it "yum install ./...rpm"
[02:54:37] <paulej72> which is the failed dependency
[02:54:41] <xlefay> guessing you're rpm -Uvh?
[02:54:46] <paulej72> yum install which
[02:54:56] <xlefay> which only shows location of a package doesn't it?
[02:55:23] <mechanicjay> ideally there is a line between the ":" and the which, that lists the needed packages
[02:55:30] <paulej72> I am only half joking here.
[02:55:33] <xlefay> fish requires a curses implementation, such as ncurses, to run.
[02:55:34] <xlefay> fish requires a number of utilities to operate, which should be present on any Unix, GNU/Linux or OS X system. These include (but are not limited to) hostname, grep, awk, sed, which, and getopt. fish also requires the bc program.
[02:56:22] <xlefay> mechanicjay: https://github.com you could try a nightly build although I generally don't advocate nightly's...
[02:57:10] <mechanicjay> it's okay, someone put a package up on software.opensuse.org
[02:57:23] <mechanicjay> ..installing
[02:57:24] <xlefay> http://fishshell.com <-- is a nice way to see what you'll get
[02:58:03] <mechanicjay> it runs
[02:58:06] -!- FunPika has quit [Quit: Leaving]
[02:58:14] <mechanicjay> historic command completion -- neat
[02:58:27] <xlefay> it is!
[02:58:33] <xlefay> it should also pick up your bash history iirc
[02:58:37] <mechanicjay> it does
[02:58:49] <xlefay> be sure to just press right arrow, instead of typing things fully
[02:59:13] <xlefay> also try: 'ls' and arrow up
[02:59:14] <mechanicjay> yeah, I was trying to hit tab to complete,
[02:59:22] <xlefay> only historical 'ls' commands
[02:59:28] <xlefay> tab works if it's not in the history
[02:59:37] <mechanicjay> interesting
[02:59:59] <xlefay> be sure to: fish_update_completions
[03:00:16] <xlefay> then try, 'ps' [tab]
[03:00:21] <mechanicjay> okay, I have to drop off. I'll give you fish report tomorrow ;)
[03:00:26] <xlefay> (without a space after 'ps')
[03:00:29] <xlefay> haha alright ;-)
[03:00:33] <xlefay> Have fun!
[03:00:47] <xlefay> And welcome to the fishy side (in advance :P)!
[03:00:57] <mechanicjay> let's play around with cgroups and ruid2 independantly and compare notes in a couple days
[03:01:03] <xlefay> Will do :)
[03:01:15] <xlefay> Got a shitload of other stuff to do also, so I'll try to get some time :)
[03:02:35] <mechanicjay> me too, but's it interesting enough, and may have some applications elsewhere on other stuff I"m working on, so I'll try and raise it's priority
[03:02:48] <mechanicjay> in my personal queue :)
[03:02:49] <xlefay> same here
[03:02:55] <xlefay> but it's in my SN queue :p
[03:03:09] <mechanicjay> maybe that's my problem, I don't have good queue separation.
[03:03:14] <mechanicjay> :)
[03:03:16] <xlefay> my advantage and disadvantage is, is that I don't have a job atm (looking)
[03:03:25] <xlefay> so I've got a bit more time, but still a lot of stuff to do ;-)
[03:03:54] <mechanicjay> understood, I'm afraid if I become unemployed, I'll be too busy to find a new job
[03:04:10] <mechanicjay> ..darn these people who demand money every month
[03:04:13] <xlefay> I'm looking pretty much all the time, but one needs a distraction from that after a while tho
[03:04:15] <mechanicjay> ..and this need to eat
[03:04:43] <xlefay> Spending so much time finding a job is just shitty, especially when everywhere there are a few dozen other people who want it
[03:04:49] <xlefay> haha exactly
[03:05:21] <mechanicjay> I'm sorry to hear that.
[03:05:38] <mechanicjay> Anyway, take care, I'll catch you later!
[03:05:45] <xlefay> No worries, take care, talk later ;-)
[03:06:00] -!- mechanicjay has quit [Quit: Leaving.]
[03:15:51] -!- drussell has quit [Read error: Connection timed out]
[03:17:24] -!- drussell [drussell!~drussell@205.233.pt.umv] has joined #staff
[03:23:05] -!- Popeidol has quit [Ping timeout: 246 seconds]
[03:25:02] -!- Popeidol [Popeidol!~matt@526-545-648-109.dyn.iinet.net.au] has joined #staff
[03:33:47] -!- drussell has quit [Read error: Connection timed out]
[03:37:52] -!- FunPika [FunPika!~FunPika@Soylent/Staff/Wiki/FunPika] has joined #staff
[03:37:52] -!- mode/#staff [+v FunPika] by SkyNet
[03:44:12] -!- paulej72 [paulej72!~paulej72@Soylent/Staff/Developer/paulej72] has parted #staff
[03:44:47] -!- FunPika has quit [Ping timeout: 246 seconds]
[03:45:08] -!- bytram has quit [Ping timeout: 246 seconds]
[04:17:18] audioguy is now known as audioguyafk
[04:34:34] -!- mrcoolbp [mrcoolbp!~mrcoolbp@Soylent/Staff/mrcoolbp] has joined #staff
[04:34:34] -!- mode/#staff [+v mrcoolbp] by SkyNet
[04:43:09] -!- mrcoolbp has quit []
[04:52:03] -!- mrcoolbp [mrcoolbp!~mrcoolbp@Soylent/Staff/mrcoolbp] has joined #staff
[04:52:03] -!- mode/#staff [+v mrcoolbp] by SkyNet
[05:22:44] <mrcoolbp> anyone awake
[05:51:45] audioguyafk is now known as audioguy
[06:17:15] -!- combatserver_ [combatserver_!~chatzilla@k-21-827-19-772.hsd5.wa.comcast.net] has joined #staff
[06:18:39] -!- pbnjoe has quit [Quit: Leaving]
[06:18:47] -!- combatserver has quit [Ping timeout: 246 seconds]
[06:18:56] combatserver_ is now known as combatserver
[06:23:01] -!- mrcoolbp has quit []
[06:24:56] FoobarBazbot|afk is now known as FoobarBazbot
[06:43:59] -!- combatserver has quit [Quit: zzz]
[07:24:22] MrBluze|afk is now known as MrBluze
[07:49:29] MrBluze is now known as MrBluze|afk
[08:11:45] -!- pbnjoe [pbnjoe!~pbnjoe@Soylent/Users/313/pbnjoe] has joined #staff
[08:14:43] -!- audioguy has quit [Ping timeout: 256 seconds]
[08:20:58] MrBluze|afk is now known as MrBluze
[08:48:13] -!- audioguy [audioguy!~freenode@Soylent/Staff/Developer/audioguy] has joined #staff
[08:48:13] -!- mode/#staff [+v audioguy] by SkyNet
[09:22:39] -!- mattiep [mattiep!~mattie_p@Soylent/Staff/Editor/mattiep] has joined #staff
[09:22:39] -!- mode/#staff [+v mattiep] by SkyNet
[09:26:23] -!- mattie_p has quit [Ping timeout: 246 seconds]
[09:47:35] -!- drussell [drussell!~drussell@205.233.pt.umv] has joined #staff
[10:03:10] -!- stdhell [stdhell!~pohol@GetOffMyLawn/stderr] has joined #staff
[10:03:10] -!- mode/#staff [+v stdhell] by SkyNet
[11:50:52] <MrBluze> NCommander: poke
[12:11:11] -!- FunPika [FunPika!~FunPika@Soylent/Staff/Wiki/FunPika] has joined #staff
[12:11:11] -!- mode/#staff [+v FunPika] by SkyNet
[12:59:21] -!- mrcoolbp [mrcoolbp!~48226104@Soylent/Staff/mrcoolbp] has joined #staff
[12:59:21] -!- mode/#staff [+v mrcoolbp] by SkyNet
[13:02:05] <mrcoolbp> Landon: the link to "Who'sWho" in #soylent is incorrect
[13:02:35] <mrcoolbp> .op
[13:02:35] -!- mode/#staff [+o mrcoolbp] by SkyNet
[13:05:35] <NCommander> MrBluze, *wobbles*
[13:06:49] <mrcoolbp> Whoa there boy, y'all be careful now.
[13:21:46] * NCommander feels mull
[13:27:31] <mrcoolbp> Sorry to hear that, I'll hopefully have a plan out to staff later today about the name change for review
[13:36:58] <NCommander> mrcoolbp, I'm going to try and finish the dev server today
[13:36:59] -!- bytram [bytram!~pc@Soylent/Staff/Developer/martyb] has joined #staff
[13:36:59] -!- mode/#staff [+v bytram] by SkyNet
[13:37:20] <NCommander> mrcoolbp, I think our best bet for voting is a reworked pollbooth
[13:37:37] <mrcoolbp> NCommander, copy that
[13:38:35] <mrcoolbp> abandoning the devotee option?
[13:38:53] <NCommander> mrcoolbp, asking people to fight with GPG is probably a bad idea
[13:39:07] * NCommander is currently dumping the sql database
[13:39:15] <NCommander> and importing it on the dev box
[13:39:41] <bytram> Hi all! what's happenin'?
[13:39:50] * NCommander feels like garbage
[13:40:09] <bytram> NCommander++ # hope you feel better soon!
[13:40:09] <Bender> karma - ncommander: 4
[13:44:11] -!- prospectacle [prospectacle!~3a6b4c99@g98-792-85-862.mit871.act.optusnet.com.au] has joined #staff
[13:55:58] <NCommander> .voice prospectacle
[13:55:58] -!- mode/#staff [+v prospectacle] by SkyNet
[13:56:03] <NCommander> Welcome prospectacle
[13:56:29] <prospectacle> thanks
[13:56:45] * NCommander notes its kinda amusing checking various peoples MUA
[13:56:51] <NCommander> prospectacle, no one on staff uses the same MUA :-)
[13:57:04] <NCommander> (well, supposedly, there's a second thunderbird user, but I haven't seen it)
[13:57:10] <NCommander> Cool
[13:57:12] <NCommander> import complete
[13:57:35] <prospectacle> lol
[13:58:34] <prospectacle> the web is 25 years old and there's still no good general-purpose messaging program for it
[13:59:29] <NCommander> prospectacle, it won't be hard to add user-to-user PMing
[13:59:36] <NCommander> There's already a decent infrastructure for it already in slash
[13:59:40] <prospectacle> nice
[14:00:32] <prospectacle> Seems like slash has a lot that slashdot doesn't show
[14:00:47] <NCommander> INSERT INTO vars (name, value, description) VALUES ('feature_story_enabled','0','Simple Boolean to determine if homepage prints feature story');
[14:00:48] <prospectacle> I mean voldemort
[14:00:48] <NCommander> Hrm
[14:00:54] * NCommander wonders what that is
[14:01:10] <NCommander> mcasadevall@tranquility:~/src/slashcode$ grep feature_story_enabled -r *
[14:01:11] <NCommander> sql/mysql/defaults.sql:INSERT INTO vars (name, value, description) VALUES ('feature_story_enabled','0','Simple Boolean to determine if homepage prints feature story');
[14:01:11] <NCommander> mcasadevall@tranquility:~/src/slashcode$
[14:01:13] <NCommander> ABSOLUTELY NOTHING
[14:02:35] <NCommander> Think this is what I want
[14:02:36] <NCommander> no_prerendered_stories
[14:02:45] * NCommander is trying to remember the var to disable static HTML rendering
[14:03:07] -!- matt_ [matt_!~4c76b3cf@t-59-991-957-035.hsd4.ma.comcast.net] has joined #staff
[14:03:10] -!- mode/#staff [+v matt_] by SkyNet
[14:03:16] <NCommander> n
[14:03:18] <NCommander> *no
[14:03:22] * NCommander beats matt_ with Outlook
[14:03:37] * matt_ was hoping nobody would notice :)
[14:04:05] <prospectacle> hi matt_
[14:04:05] <matt_> NCommander, do you think that you will be ready to post your vision article today? if so, I can post the journal entry and send you a link.
[14:04:15] <matt_> prospectacle, hi. how are you?
[14:04:19] <NCommander> matt_, no, right now I need to do some infrastructure and hardering work
[14:04:27] <NCommander> matt_, I'm trying to apparmor apache so we can't be hacked as easily
[14:05:04] <prospectacle> matt_, good thanks. how are things in the deep end?
[14:05:08] <matt_> NCommander, ok. do you want to set a deadline (say tomorrow, Sunday or Monday) to keep up our momentum?
[14:06:31] <matt_> prospectacle, pretty good.
[14:08:16] <NCommander> matt_, I don't know
[14:08:41] <NCommander> Found it
[14:08:43] <NCommander> sql/mysql/upgrades:INSERT INTO vars (name, value, description) VALUES ("article_link_story_dynamic", "0", "Change default dynamic status for story linking");
[14:08:48] <matt_> NCommander, actually, I think that the last half of your last long-ish email sent to the list would work quite well. perhaps it just needs a bit of polishing.
[14:08:50] <NCommander> WHY ISN'T THAT IN THE DATABASE SCHEMA
[14:09:02] <NCommander> matt_, probably, do you want to edit it for me a bit? :-)
[14:09:28] <NCommander> Ok, static rendering disabled
[14:09:31] <NCommander> (on dev)
[14:09:34] <matt_> NCommander, i thought you'd never ask! :) I'll send you an edited version later today.
[14:09:53] <prospectacle> beautiful
[14:10:05] <prospectacle> If nothing else, you can write the book on slashcode when this is all set up.
[14:10:12] <NCommander> prospectacle, someone alrady did
[14:10:30] <NCommander> prospectacle, http://shop.oreilly.com
[14:10:35] <prospectacle> oh. well i searched for some of the things you've been grappling with, and the only results were from soylentnews git repository
[14:10:49] <prospectacle> So you can write the book that people actually use
[14:10:57] <matt_> gtg, see you later!
[14:11:17] <mrcoolbp> Later matt_
[14:11:34] -!- matt_ has quit [Quit: Web client closed]
[14:11:38] <NCommander> gah
[14:11:39] <NCommander> oops
[14:11:49] * NCommander deleting the wrong folder on dev
[14:11:50] <NCommander> piss
[14:11:58] <bytram> ouch.
[14:12:05] <prospectacle> Also that book has 0 reviews. I predict your book would have at least 3
[14:12:26] <bytram> prospectacle, lol!
[14:12:57] <NCommander> Crap
[14:13:03] <NCommander> We're getting OOMs on the web server
[14:13:38] <bytram> NCommander, I pulled a bunch of pages yesterday... wonder if it's related?
[14:13:46] <NCommander> bytram, possibly.
[14:13:47] <bytram> !current-uid
[14:13:47] <Bender> The current maximum UID is 3919, owned by Aenima
[14:14:49] <NCommander> -V display version information and exit
[14:14:49] <NCommander> mcasadevall@soylent-www:~$ free -m
[14:14:49] <NCommander> total used free shared buffers cached
[14:14:49] <NCommander> Mem: 1992 1360 632 0 8 276
[14:14:49] <NCommander> -/+ buffers/cache: 1074 918
[14:14:49] <NCommander> Swap: 511 300 211
[14:14:51] <NCommander> Its OK for now
[14:14:56] <NCommander> But ...
[14:15:38] <mrcoolbp> NCommander, did you see the issues on the svc linode yesterday?
[14:16:10] <NCommander> mrcoolbp, no, I don't touch that box
[14:16:34] <mrcoolbp> Xlefay posted a log to the staff mailing list....
[14:16:38] <bytram> I just checked and it was only 1K web pages; and I spaced them out every 5-10 seconds.
[14:19:53] <NCommander> http://dev.soylentnews.org - w00t, guru meditation errors!
[14:20:37] <prospectacle> at last!
[14:21:48] <NCommander> mcasadevall@soylent-dev:/srv/soylentnews.org/src/slashcode$ sudo -u slash git checkout master
[14:21:49] <NCommander> Switched to branch 'master'
[14:21:49] <NCommander> Your branch and 'origin/master' have diverged,
[14:21:49] <NCommander> and have 11 and 12 different commits each, respectively.
[14:21:50] <NCommander> Shit
[14:24:29] <stdhell> NCommander: When you were using the nail gun to make xlefay a proper member of the SysAdmin team, I asked if you could use it on me too, but I never got a real answer?
[14:24:47] <NCommander> stdhell, oh, I thought I nailed you too
[14:24:49] * NCommander nails stdhell to a wall
[14:25:11] <mrcoolbp> Nail a terminal beside him
[14:26:31] <bytram> NCommander, FYI: I'm still looking at pollbooth...
[14:26:32] <stdhell> Yeah, unless you redirect me to /dev/null or a file, you should probably give me a terminal I can use.
[14:27:12] * NCommander gives him an actual vt120
[14:27:12] <bytram> stderr, I often end my regexps with "$"... does that help?
[14:27:24] <bytram> stdhell, I often end my regexps with "$"... does that help?
[14:27:35] * bytram made rror
[14:27:52] <stdhell> bytram: Our department is in the process of moving to another building, so I can't really work on anything today, but I'm looking at making a simple web frontend for devotee to see if it will be easy to include it in Slashcode.
[14:28:20] <NCommander> Ugh
[14:28:22] <stdhell> bytram: stderr is me at home. std-hell is me at work...
[14:28:24] <NCommander> I think I broke it
[14:28:24] <NCommander> damn it
[14:28:34] <NCommander> stdhell, how do you handle the fact debvotee required GPG signed votes?
[14:29:02] <stdhell> As I say yesterday, skip that part and jump directly to dvt-parse.
[14:29:11] <NCommander> Did github go down?
[14:29:23] * NCommander can't access it
[14:29:27] <NCommander> from here or the nodes
[14:29:49] <bytram> NCommander, I'm checking from my end... hold on
[14:30:46] <bytram> NCommander, not looking good at all. I'm getting just a few *bytes*/sec down
[14:30:56] <NCommander> That's exceedingly annoying
[14:31:15] * bytram notes NCommander's knack for understatement
[14:31:37] <bytram> I suspect we may not be the only ones inconvenienced...
[14:31:56] <NCommander> bytram, bytram,
[14:31:57] <NCommander> 13:13 UTC We are currently working to mitigate an incoming DDoS attack. We'll provide additional information as it becomes available.
[14:32:02] <NCommander> https://status.github.com
[14:32:16] <mrcoolbp> Wow
[14:32:28] <mrcoolbp> Probably khyber
[14:32:37] <mrcoolbp> = )
[14:32:46] * NCommander groans
[14:32:54] <bytram> that's not nice... very not nice.
[14:33:05] * NCommander clones from his laptop
[14:33:07] <NCommander> <3 IPv5
[14:33:10] <NCommander> *IPv6
[14:34:07] <NCommander> Huh
[14:34:11] <NCommander> Git doesn't like IPv6 addresses
[14:35:40] * NCommander sticks it in /etc/hosts
[14:35:41] <bytram> github DDOS status msg was posted 30 minutes ago
[14:36:32] <bytram> it's the afternoon in europe, and things are just getting going in the US; *nasty* timing!
[14:36:57] <stdhell> Mar 20 22:39:35 <stderr> mrcoolbp (and others?): As far as I can tell, to include Devotee.pm in Slashcode, we need to rewrite parse_messages() from dvt-parse (since our votes ain't in emails), create_tally() from dvt-tally is probably ok as it is, create_list() (...of voters) from dvt-voters is probably ok too (if we even want to make a list of voters) and finally, the real meat, winner() from dvt-rslt seems ok too.
[14:37:15] <NCommander> I *love* IPv6
[14:37:15] <NCommander> slash@soylent-dev:~$ git clone ssh://mcasadevall@tranquility
[14:37:16] <NCommander> Cloning into 'slashcode'...
[14:37:16] <NCommander> The authenticity of host 'tranquility (2001:0:53aa:64c:2ce1:1d13:bb51:4dc3)' can't be established.
[14:37:16] <NCommander> ECDSA key fingerprint is 08:a9:e0:1c:27:c9:19:c9:32:2b:28:7c:ab:ee:a0:e4.
[14:37:18] <NCommander> Are you sure you want to continue connecting (yes/no)? yes
[14:37:51] <stdhell> What?! No VerifyHostKeyDNS?
[14:38:07] <stdhell> Oh, wait, ECDSA... Nevermind...
[14:38:40] <mrcoolbp> NCommander: stdhell thinks we can implement devotee using parse_messages() ^^^^
[14:39:00] <NCommander> stdhell, I only put the IPv6 address in hosts because git has issues w/ raw IPv6 addresses
[14:39:10] <stdhell> mrcoolbp: I'm going to test that when I get home.
[14:39:20] <mrcoolbp> Cool, let me know please
[14:39:21] <NCommander> stdhell, I kinda liked the concept of using emails for the vote but ...
[14:39:29] <prospectacle> stdhell, how do you make ui for indicating preferences? Is it as simple as changing radio buttons to text (or check) boxes?
[14:40:36] <stdhell> prospectacle: I was thinking a drop down menu?
[14:41:03] -!- dentonj [dentonj!~dentonj@217.33.iwh.wjw] has joined #staff
[14:41:08] -!- dentonj [dentonj!~dentonj@217.33.iwh.wjw] has parted #staff
[14:41:10] <stdhell> Maybe someone can later do some fancy javascript crap^Wstuff with drag and drop and what not...
[14:41:16] <bytram> what if we go for vote ranking (or w/e it's called)
[14:41:44] <prospectacle> Drop down # next to each option? That would work.
[14:41:48] <stdhell> bytram: That's what the preferences is... As far as I understood the question.
[14:41:53] <stdhell> prospectacle: Yes.
[14:41:58] <prospectacle> Devotee lets you give multiple options same rank if you want. So you can't really mess it up.
[14:42:01] <NCommander> stdhell, I think this is going to take quite awhile
[14:42:14] <NCommander> stdhell, so I think we should implement debvotee as a plugin, and go with pollbooth in the short future
[14:42:46] <prospectacle> stdhell. That would work, as long as the drop downs say 1st, 2nd, 3rd" instead of 1,2,3, which might make people think they're assigning points.
[14:43:07] <stdhell> NCommander: I can tell you when I get home and actual do the coding... :-)
[14:43:11] <mrcoolbp> Stdhell: people are strongly opposed to JS
[14:43:21] <bytram> agreed
[14:44:08] <stdhell> mrcoolbp: Even from the webserver itself? I can understand opposing 3rd party stuff.
[14:44:40] <stdhell> ... and of course you should ALWAYS whitelist tzdata-javascript.org. :-)
[14:44:53] <prospectacle> If pollbooth needed to be rewritten to be viable (even for short term use for anything serious), it might be less (code)work just to make a hidden-comments article, and do a comment-scrape, which would only be half a dozen lines of perl/php/whatever
[14:45:02] <bytram> On my mobile, I lack granularity. Either ALL sites can run JS, or NO sites can... I set it to no sites.
[14:45:24] <mrcoolbp> Stdhell: not sure
[14:45:39] <stdhell> bytram: Ok... s/^Wstuff// Better?
[14:45:51] <mrcoolbp> But generally people want to avoid JS at all costs
[14:46:20] <bytram> prospectacle, ^^^^
[14:46:22] <stdhell> Ok... So drop down menu or simple text boxes...
[14:46:27] <bytram> right.
[14:46:43] <NCommander> stdhell, http://www.howtogeek.com - I'm going to run this on Apache and slashd
[14:46:59] <NCommander> stdhell, if kyberd really has an evil exploit, this might stop it in its trracks
[14:47:15] * stdhell thinks NCommander is talking to the wrong person...
[14:47:19] <stdhell> NCommander: Ok... :-)
[14:47:42] <bytram> hey. back in about 10minutes
[14:47:43] <prospectacle> bytram, I agree re: keeping JS unecessary
[14:47:56] <bytram> glad to hear it!
[14:48:16] * bytram just loves doing laundry :/
[14:48:25] <prospectacle> bytram: drop downs would still work, as would comment-scraping from a hidden-comments article, without JS.
[14:48:30] <stdhell> bytram: Wanna do mine too?
[14:48:36] <bytram> no
[14:49:06] <bytram> prospectacle, what about validation? accidentally vote two different things as #1, for example?
[14:49:14] <bytram> brb
[14:49:28] bytram is now known as bytram|afk
[14:49:35] <stdhell> bytram: I think you're allowed to preferer two options equally.
[14:49:51] <prospectacle> bytram: you can use counting systems that allow this. Debian uses such a system and publishes their method.
[14:52:30] <prospectacle> bytram: they base everything on what # of people prefer a to b, what # prefer a to c, etc. So if someone ranks a and b equally then they just contribute nothing to the "a vs b" count, but they might contribute something to the "b vs c" count.
[14:54:05] <stdhell> http://www.debian.org
[14:54:57] <stdhell> Anyway... I'm out for a while to see if I can find my boss...
[14:54:57] <prospectacle> stdhell: Yes, also http://www.debian.org (find "a.6" vote counting for basic explanation)
[14:58:34] -!- weeds [weeds!~4118a13c@cwz-29-45-637-17.columbus.res.rr.com] has joined #staff
[14:59:04] <bytram|afk> stdhell, hmmmm, maybe it would be best (for the time being) to go with something that has already been "firetested" rather than roll our own.
[14:59:53] <NCommander> BOOM
[14:59:54] <NCommander> http://dev.soylentnews.org
[14:59:59] <prospectacle> bytram: if there's an easy way to collect the polling data (e.g. modified poll-booth gui, or comment-scraping. Then any number of counting systems could be applied after the fact, as it doesn't have to be live/integrated etc)
[15:00:13] <prospectacle> hey
[15:00:18] <prospectacle> ncommander++
[15:00:19] <Bender> karma - ncommander: 5
[15:00:55] <prospectacle> it even works!
[15:01:40] bytram|afk is now known as Bytram
[15:02:16] <Bytram> NCommander, I can has dev / admin access plz?
[15:03:14] <NCommander> Bytram, what's your account
[15:03:19] <NCommander> Bytram, I'll make you a suadmin
[15:03:25] <Bytram> NCommander, martyb
[15:04:11] <NCommander> Bytram, boom
[15:04:27] <Bytram> yummy! thanks!
[15:04:46] <NCommander> Anyone else want admin bits?
[15:04:55] <mrcoolbp> I do
[15:05:15] <Bytram> NCommander, I assume al who had it before, still do?
[15:05:20] <Bytram> s/al/al/
[15:05:25] <Bytram> s/al/all/
[15:05:29] * Bytram needs new kd
[15:05:33] * Bytram needs new kbd
[15:05:37] <prospectacle> well it accepts comments
[15:05:37] <Bytram> =)
[15:05:50] <NCommander> prospectacle, nice
[15:05:53] <NCommander> Bytram, yeah
[15:06:02] <NCommander> Give me a moment, I need to turn it off and start running apparmor profiler on it
[15:06:16] <Bytram> k.
[15:07:22] -!- mattiep has quit [Quit: Leaving]
[15:07:34] <NCommander> actually ...
[15:07:37] * NCommander just writes one from scratch
[15:07:39] <NCommander> Less work
[15:08:14] <Bytram> NCommander, do we still want b to have suadmin
[15:08:16] <mrcoolbp> NCommander /me would take admin bits, want to play around a bit
[15:08:35] <Bytram> mrcoolbp, I see you already do.
[15:08:57] <NCommander> mrcoolbp, ENJOY
[15:08:59] * NCommander posts an article
[15:09:16] <NCommander> fuck
[15:09:20] <NCommander> the javascript is still borked
[15:09:26] <NCommander> Crap, this happened before
[15:09:26] <mrcoolbp> Nice thx
[15:10:37] -!- weeds [weeds!~4118a13c@cwz-29-45-637-17.columbus.res.rr.com] has parted #staff
[15:11:02] <NCommander> ugh
[15:11:04] <NCommander> I hate this bug
[15:11:12] <NCommander> I don't know why, but the jquery stuff never installs properly
[15:11:32] <NCommander> "joy"
[15:11:32] <NCommander> [Fri Mar 21 14:09:43 2014] [error] /admin.pl:Slash::DB::Utility:/srv/soylentnews.org/local/lib/perl5/site_perl/5.10.1/x86_64-linux/Slash/DB/Utility.pm:748:virtuser='slash' -- hostinfo='Localhost via UNIX socket' -- You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ') AND stories.stoid IN (534,533,532,531,530,529,528,527,524,525,526,523,522,521,' at lin
[15:11:32] <NCommander> e 1 -- SELECT DISTINCT stories.stoid FROM stories, story_topics_rendered AS str WHERE stories.stoid=str.stoid AND str.tid IN () AND stories.stoid IN (534,533,532,531,530,529,528,527,524,525,526,523,522,521,520,519,518,512,516,515,514,513,517,511,510,509,508,507,506,505,504,503,499,500,502,501,498,497,496,495) ;; Which was called by:Slash::DB::MySQL:/srv/soylentnews.org/local/lib/perl5/site_perl/5.10.1/x86_64-linux/Slash/DB/MyS
[15:11:34] <NCommander> QL.pm:5936
[15:11:41] <Bytram> NCommander, while u are at it, use the minified version!
[15:12:51] <Bytram> NCommander, getting link... hold on.
[15:13:22] * NCommander grumbles
[15:13:27] <Bytram> NCommander, http://code.jquery.com
[15:14:10] <NCommander> OH
[15:14:12] <NCommander> ugh
[15:14:33] <Bytram> it's 56KB instead of 118KB
[15:15:38] <NCommander> fuck it
[15:16:02] -!- janrinok [janrinok!~janrinok@Soylent/Staff/Editor/janrinok] has joined #staff
[15:16:02] -!- mode/#staff [+v janrinok] by SkyNet
[15:16:13] <janrinok> hi guys
[15:16:26] <NCommander> janrinok, we haz dev server
[15:16:42] <janrinok> Great - how long did that take?
[15:16:47] <Bytram> NCommander++ # you da best!
[15:16:47] <Bender> karma - ncommander: 6
[15:17:11] <Bytram> !grab Bytram
[15:17:11] <Bender> Added quote 76
[15:17:16] <Bytram> !quote 76
[15:17:16] <Bender> Quote 76 - <Bytram> !grab Bytram
[15:17:35] * Bytram urk!
[15:17:43] <janrinok> lol - not quite what you expected, eh?
[15:17:49] <NCommander> janrinok, not too long, required a lot of duct tape but
[15:17:52] <prospectacle> re: mysql, not allowed an empty "in ()"
[15:17:55] <NCommander> janrinok, http://dev.soylentnews.org
[15:18:17] <janrinok> thx - I'll look now
[15:18:38] <janrinok> Brilliant
[15:19:06] * NCommander needs to add a message somewhere that this is dev server
[15:19:08] <NCommander> Not real server
[15:19:12] <Bytram> NCommander, just a heads up that it'll be important to keep track of ALL one-off changes made to dev so we know what's different whe3n we roll it over to prod.
[15:19:21] -!- mattie_p [mattie_p!~mattie_p@Soylent/Staff/Editor/mattiep] has joined #staff
[15:19:21] -!- mode/#staff [+v mattie_p] by SkyNet
[15:19:32] <Bytram> mattie_p, hiya!
[15:19:39] <janrinok> hi mattie_p just in time to celebrate our new dev.server
[15:19:40] <mattie_p> hey, bytram
[15:19:51] <mattie_p> oh really? I just downloaded the vm last night
[15:20:04] <mattie_p> needed to restart laptop to enable VTx
[15:20:15] <NCommander> mattie_p, dev.soylentnews.org
[15:20:32] * NCommander thinks he'll change the logo back to the slashcode logo
[15:20:37] <NCommander> That should make it clear that this is a dev box
[15:21:06] <mattie_p> so this has the latest code from git before it goes to production?
[15:21:10] <NCommander> This is also our experiment to see if we can disable static HTML
[15:21:17] <NCommander> mattie_p, yeah, against a copy of the production database
[15:21:34] <mattie_p> ok, so a copy, not live against the production db, that was my next question
[15:22:08] <NCommander> mattie_p, the live one will come when we ahve a way to have multiple themes without killing each other
[15:22:16] <mattie_p> sorry I wasn't on yesterday, had a lot to do and the break from the site ended up doing me good anyway
[15:22:31] <mattie_p> did I miss much?
[15:22:34] <NCommander> mattie_p, I wish I could take one of those
[15:22:36] <NCommander> mattie_p, nope.
[15:22:48] <mattie_p> NCommander, you've been traveling, while not a "break
[15:22:54] <mattie_p> it is at least offline
[15:23:00] <mattie_p> I needed a mostly unplugged day
[15:23:02] <NCommander> mattie_p, "true"
[15:23:11] <NCommander> But 20 hours on a plane isn't relaxing
[15:24:04] <mattie_p> I'm well aware of that
[15:24:07] <NCommander> mattie_p, I'm working out apparmor profiles now to try and lock down the server
[15:24:12] <NCommander> !todo
[15:24:12] <Bender> todo for ncommander: 1) make sure install-slashsite installs proper schema 2) quit smoking 3) look at rewiring pollbooth for SERIOUS votes 4) write up YAFAP for nethack 5) find volunteer who may be willing to work on mod_perl rework effort 6) poof development server into existence 7) clean production database of unused vars/tables from Tags/FIrehose/Achievements 8) make - 1 more
[15:24:16] <NCommander> !todo-done 6
[15:24:16] <Bender> 1 item deleted
[15:24:21] <NCommander> (that took long enough)
[15:24:24] <NCommander> !todo
[15:24:24] <Bender> todo for ncommander: 1) make sure install-slashsite installs proper schema 2) quit smoking 3) look at rewiring pollbooth for SERIOUS votes 4) write up YAFAP for nethack 5) find volunteer who may be willing to work on mod_perl rework effort 6) clean production database of unused vars/tables from Tags/FIrehose/Achievements 7) make LDAP a thing yesterday 8) fix slashboxs
[15:24:25] <mattie_p> did you find something about that alleged threat?
[15:24:34] <NCommander> mattie_p, no, but I rather get us well locked down
[15:24:40] <mattie_p> true
[15:24:44] <NCommander> mattie_p, apparmor will go a long way in preventing exploits from being workable
[15:24:44] <mattie_p> might as well get it done now
[15:24:56] <NCommander> I need to LDAP the setup
[15:24:58] <NCommander> That's my next TODO
[15:25:00] <NCommander> ugh
[15:26:16] <NCommander> and document the dev box
[15:26:35] <mrcoolbp> Mattie_p: we are revisiting the name change domain gathering, I'll be online around 5pm EST if you want to catch up
[15:26:36] <mattie_p> I need to check on everyone's documentation, we should be making some serious progress by now
[15:26:45] <mrcoolbp> I gotta get back to work
[15:26:46] <mattie_p> mrcoolbp ok, sounds good
[15:26:54] <mrcoolbp> Great
[15:27:12] <NCommander> mattie_p, perfect
[15:27:20] * NCommander has some things to add on that, and has to work on his dayjob soonish
[15:27:40] <Bytram> NCommander, FYI; GitHub is still under attack, but seems to be back up.
[15:27:45] <mattie_p> dayjobs are the worst, except for not having one
[15:27:58] <Bytram> mrcoolbp, cya! have a great day!
[15:28:49] <prospectacle> You can't fund "organisations" on kickstarter, but you can fund projects. Maybe you could get paid to do a "better OSS poll booth" or an "Easy App Armor profile creator"
[15:29:08] <prospectacle> it would just be a coincidence that this is exactly what the organisation needs
[15:29:10] <NCommander> prospectacle, the later already exists :-)
[15:29:22] <NCommander> Its just not great for Apache
[15:29:27] * NCommander brbs
[15:29:51] <prospectacle> "easy app armor profile creator for apache"
[15:30:09] <NCommander> prospectacle, :-)
[15:30:16] * NCommander goes to have a cigarette and early luck
[15:30:38] <prospectacle> Anyway just a suggestion, since it seems like a lot of free work you're doing for us all. I would pledge if I got a sweet sticker out of it.
[15:31:27] <NCommander> prospectacle, heh :-)
[15:31:29] <NCommander> thanks
[15:31:32] <mattie_p> ok, vm is up and running, I'm logged in now
[15:31:38] <NCommander> prospectacle, you could always buy me a beer if you're in New York
[15:31:41] <mattie_p> where is the instruction page?
[15:31:49] <NCommander> mattie_p, what instructions :-)
[15:31:52] <mattie_p> NCommander I'll be there around 12 April, I think we're doing a meetup
[15:31:58] <prospectacle> well like you were saying abotu 20 hr plane rides (I'm in aus)
[15:32:01] <NCommander> mattie_p, *nods*
[15:32:03] <mattie_p> NCommander, at least start slash running
[15:32:10] <mattie_p> is it on the wiki?
[15:32:12] <NCommander> mattie_p, slash:slash for login
[15:32:16] <NCommander> apachectl start
[15:32:19] <mattie_p> yeah, I remember that much
[15:32:24] <mattie_p> k
[15:32:50] <NCommander> ah
[15:32:50] <NCommander> shit
[15:32:55] * NCommander knew he forgot something
[15:33:04] <mattie_p> what now?
[15:33:20] <NCommander> Oh, server timezone is UTC
[15:33:25] <NCommander> mattie_p, Slash wants the database to be in GMT
[15:33:34] <NCommander> We're on UTC which is "good enough", I don't want to muck more crap up
[15:33:35] <mattie_p> oh
[15:40:12] -!- mrcoolbp has quit [Quit: Web client closed]
[15:46:23] -!- mechanicjay [mechanicjay!~jhowe@Soylent/Staff/Developer/mechanicjay] has joined #staff
[15:46:23] -!- mode/#staff [+v mechanicjay] by SkyNet
[15:47:29] -!- robind [robind!~robind@Soylent/Staff/Sysop/robind] has joined #staff
[15:47:29] -!- mode/#staff [+v robind] by SkyNet
[15:47:30] <mechanicjay> woot! Karma level 50 achieved on the site!
[15:47:41] <xlefay> Congrats ;-)
[15:47:50] <prospectacle> mechanicjay++
[15:47:50] <Bender> karma - mechanicjay: 7
[15:47:53] <xlefay> & good morning, just woke up ;)
[15:48:10] <Bytram> mechanicjay++ # from a fellow former Atari 800 owner
[15:48:10] <Bender> karma - mechanicjay: 8
[15:48:34] -!- robinld [robinld!~robind@Soylent/Staff/Sysop/robind] has joined #staff
[15:48:35] -!- mode/#staff [+v robinld] by SkyNet
[15:48:36] <mechanicjay> good morning to you too! I've already fed my kid, sat in traffic for an hour and taught my class this morning. Now for the first cup of coffee...
[15:48:52] <xlefay> woa
[15:48:52] <mechanicjay> Bytram++ Atari FTW
[15:48:52] <Bender> karma - bytram: 1
[15:48:53] <mattie_p> what? you're only having your first cup now?
[15:49:01] <mechanicjay> mattie_p: I know, it's tragic
[15:49:18] <mattie_p> I'll email you a cup of mine. I grind my beans fresh every morning
[15:49:26] <Bytram> mechanicjay, I wasted way too many hours with Star Raiders!
[15:49:44] <xlefay> Well, time for dinner, bbs
[15:49:50] <Bytram> xlefay, cya!
[15:50:04] <mechanicjay> later xlefay!
[15:50:22] <xlefay> mechanicjay: btw, do we really need spamd? It uses approx 50% of the systems resources
[15:50:24] <mechanicjay> Bytram: Frogger was my downfall
[15:50:44] <xlefay> err, clamd*
[15:50:45] <mechanicjay> xlefay: depends on how much spam your in your SN mailbox?
[15:50:54] <Bytram> NCommander, re: UTC vs GMT; ISTM we should use UTC, though there's not much difference? http://aa.usno.navy.mil
[15:50:55] <xlefay> err, clamd* <-
[15:51:12] <mechanicjay> I'd say virus scanning is more critical
[15:51:44] -!- robind has quit [Ping timeout: 246 seconds]
[15:52:09] <Bytram> mechanicjay, never got frogger, but played Star Raiders at college on a 12-foot diagonal projection screen, with a 2 KW sound system... in 1981!!
[15:52:36] <mechanicjay> Bytram: 1981? I'm jealous now!
[15:52:48] <xlefay> clamav definitions suck and most people that are on Windows and such have their own virus scanner, I'm curious if it's really required (just out of curiosity, I checked my cPanel server for clamd, couldn't find it)
[15:53:50] <Bytram> mattie_p, LOL! at the time, it was one sweet box! Later got a C64 and had a lot of fun with that, too.
[15:54:00] <mattie_p> brb
[15:54:01] <Bytram> mechanicjay, LOL! at the time, it was one sweet box! Later got a C64 and had a lot of fun with that, too.
[15:54:16] * Bytram urk! too many M's!
[15:54:57] <mechanicjay> xlefay: IDK, just seems like a best practice. I agree it's pig, but...
[15:55:47] <xlefay> I'll look into if we can lower it's memory usage, 24/7 usage of 50% of the systems mem isn't awesome
[15:56:36] <xlefay> but dinner time, I'll let you know if I find something after I return ;-)
[15:56:40] <NCommander> Bytram, slashcode says it should be using GMT
[15:58:23] <Bytram> NCommander, I plead ignorance, but what's the practical difference?
[15:58:46] <Bytram> NCommander, 'cept that UTC is based on atomic clock.
[15:59:03] <NCommander> Bytram, GMT still has DST
[15:59:43] <Bytram> oh. what? really? I thought that would be BST/BDT? Blegh!!!
[16:00:36] -!- LaminatorX [LaminatorX!~18d900fb@Soylent/Staff/Editor/LaminatorX] has joined #staff
[16:00:36] -!- mode/#staff [+v LaminatorX] by SkyNet
[16:00:46] <janrinok> NCommander: GMT does not have DST, that is British Standard Time
[16:01:01] <janrinok> GMT is stuck at UTC or vice versa
[16:01:05] <Bytram> janrinok, that's what I thought.
[16:01:20] <stdhell> GMT becomes BST during DST...
[16:02:09] <janrinok> stdhell: I think that I agree with you.... BST and GMT are the same during winter, but 1 hour different in Summer (I think)
[16:02:27] <janrinok> GMT never changes though
[16:03:06] <Bytram> ^^^ it would make no sense to have a "standard time" that repeats an hour during the transition.
[16:03:09] <janrinok> That is why it is used by the military worldwide
[16:03:37] <Bytram> so what is wrong with saying "UTC == GMT" ??
[16:03:50] <stdhell> If you're talking about the timezone that doesn't have DST, please call it UTC...
[16:04:05] <janrinok> Nothing wrong - but one is based on an atomic clock, the other on some uncertain standard in the UK.
[16:04:08] <prospectacle> UTC replaced GMT when clocks and timekeeping became more accurante
[16:04:49] <prospectacle> s/accurante/accurate/
[16:05:10] <prospectacle> s/accurate/precise/
[16:05:23] <NCommander> http://dev.soylentnews.org - boom more
[16:05:42] <janrinok> stdhell: I agree, but both UTC and GMT are the same but based on different sources. BST is the same as UTC/GMT for 6 months of the year.
[16:06:11] <stdhell> janrinok: I'm sorry, but that's nonsense...
[16:06:37] <Bytram> from a *practical*, layman's perspective they're about the same; but GMT is historical, and UTC is the official, atomic-clock-based time
[16:07:23] * NCommander runs apparmor profile gen again httpd
[16:07:50] -!- robinld has quit [Ping timeout: 246 seconds]
[16:07:54] <janrinok> shtdhell: http:// en.wikipedia.org/wiki/Coordinated_Universal_Time
[16:07:55] * stdhell runs amok.
[16:08:05] <janrinok> stdhell: ^
[16:08:18] <stdhell> BST is WITH DST... IT's NEVER the same as UTC or GMT.
[16:08:31] <Bytram> ^^^
[16:08:40] <janrinok> stdhell: For 6 months of the year they are.
[16:09:01] <stdhell> And saying "bla bla bla UTC" vs "bla bla bla GMT" doesn't say ANYTHING about the source.
[16:09:11] <stdhell> janrinok: No! NEVER!
[16:09:14] <janrinok> read the wiki link I posted
[16:09:14] <NCommander> AppArmor does not appear to be started. Please enable AppArmor and try again.
[16:09:15] <NCommander> ARGH
[16:09:18] <NCommander> fuck linode
[16:09:22] * NCommander gets a proper kernel installed
[16:10:13] <NCommander> Fortunately, its easy to get linode to run with a distro kernel then their neutral one
[16:10:28] <prospectacle> janrirok: bst doesn't exist for part of the year, for the other part (when it does exist) it's an hour off GMT
[16:10:31] <stdhell> janrinok: BST is between (last sunday of march) and (last sunday of october). It is UTC+1.
[16:10:57] <Bytram> stdhell, YES! from the link: "For most purposes, UTC is synonymous with GMT, but GMT is no longer precisely defined by the scientific community."
[16:11:06] <stdhell> janrinok: I'm pretty sure I have done more work with timezones than you... Trust me on this one...
[16:11:17] <janrinok> BST == British _Summer_ Time, during winter it is the same as UTC and GMT
[16:11:32] <stdhell> _Summer_ time during winter?
[16:11:56] <janrinok> There is NO adjustment during winter - which is why it is the SAME as GMT/UTC
[16:11:57] <stdhell> (Actually they did that during 1969->1971, but...)
[16:12:16] <stdhell> But then it's not called B_S_T...
[16:12:25] <stdhell> For a very good reason...
[16:13:01] <prospectacle> janrinok, BST ceases to exist during winter. it is ontologically, unobtainable. It vanishes into the ether, biding its time
[16:13:07] <janrinok> So the UK uses GMT for 6 months'ish and BST for 6 months'ish
[16:13:34] <stdhell> If by 6 months-ish you mean "5" and "7". Yes.
[16:14:32] <janrinok> When the (misnamed) British Standard Time is used, it refers to whichever is currently in force: GMT or B Summer T, but it is not an official abbreviation.
[16:14:34] <stdhell> Which makes it even more stupid when people call it standard time... It's only used 5 out of 12 months...
[16:14:59] <Bytram> stdhell, ^^^^^^^^
[16:15:23] <janrinok> I agree - and being a Brit does not make me think it is a good idea - but that is what we chose to do..... Look at the problems it causes!
[16:17:14] <FunPika> it gets worse in the United States...we call our timezones stuff like "Eastern Standard Time" when it actually only lasts 4 months a year
[16:17:39] <janrinok> What timezone is it for the other 8 months?
[16:17:48] <FunPika> Eastern Daylight Time
[16:17:52] <janrinok> lolol
[16:17:55] <stdhell> If you want to see something "interesting" with timezones, try TZ=UTC date -d @0 (Epoch: Jan 1st 1970 00:00:00, no surprise there...), but see if you can guess the output of TZ=Europe/London date -d @0 before trying it...
[16:18:08] <janrinok> Did we catch it from you FunPika, or did you copy us?
[16:18:24] <FunPika> what?
[16:18:41] <janrinok> This idiocy regarding tz.
[16:18:45] <LaminatorX> In the US, we wrangle with Arizona Time, which is just they're special way of saying, "We don't do DST here." They stay on Mountain Standard year-round. Except that is for the Navajo areas, where they do switch to MDT. Makes traditional dawn prayers more convenient, as I understand it.
[16:19:08] -!- weeds [weeds!~4118a13c@cwz-29-45-637-17.columbus.res.rr.com] has joined #staff
[16:19:23] <LaminatorX> "their" sorry.
[16:19:28] <mechanicjay> My brother, who lived in Arizona for a bit said the attitude of folks who live in the Desert is, "WHAT? You want to give us MORE sun?"
[16:19:35] * stdhell would like to point you all to http://tzdata-javascript.org
[16:19:36] <NCommander> Ok
[16:19:38] <NCommander> we're in business
[16:19:44] <NCommander> We're now on a stock Ubuntu kernel on the dev server
[16:19:46] <NCommander> yay apparmor
[16:19:48] <janrinok> There are various excuses for our daylight saving: to help the farmers, schoolchildren, commuters, etc
[16:19:59] <Bytram> brb... afk
[16:20:01] <stdhell> janrinok: All of them are nonsense...
[16:20:10] <LaminatorX> In actuality, it's an assist for golf courses and retailers.
[16:20:11] Bytram is now known as Bytram|afk
[16:20:13] <janrinok> absolutely
[16:20:28] <mechanicjay> janrinok: I think an exercise in government compliance, but that sounds loony, so I try not to say it out loud to much.
[16:20:41] <janrinok> lol
[16:20:53] <mechanicjay> :)
[16:21:36] <LaminatorX> People really do shop more when there's more daylight after work. It's a measured phenomenon.
[16:21:53] <FunPika> https://en.wikipedia.org Looks like this started in Germany, and it was proposed by a guy from New Zealand.
[16:21:54] <janrinok> You say that as if its a good thing?
[16:22:05] <mechanicjay> janrinok++
[16:22:05] <Bender> karma - janrinok: 2
[16:22:14] <janrinok> Bloody Kiwis, we'll blame them for today
[16:22:26] <LaminatorX> It's a good thing for the stores, certainly. Me, I have no money, so it's merely of academic interest.
[16:23:12] <prospectacle> dst = going for a walk in the sunlight at 7:30pm
[16:23:18] <mechanicjay> My interest is that the week it takes my kids adjust to the time differnce sucks.
[16:23:34] <janrinok> brb - time for a cup of tea...
[16:24:24] <mechanicjay> I have a "Joy of Painting" episode playing in the office right now...zzzzzz
[16:24:50] <mechanicjay> Do folks outside the US know who Bob Ross is?
[16:25:06] <prospectacle> only from family guy
[16:25:24] <stdhell> mechanicjay: Sounds more interesting than my episode of "move this thing over here, then replace the two <br />'s with spaces"... :-(
[16:25:49] <mechanicjay> Oh, it's just background noise while think about pretending to be a graphic designer for the day.
[16:26:49] <mechanicjay> I've been waiting for artwork for the University Mobile app from our communications department for over a month. I need to get it tested and out the door, so gimp, here I come!
[16:27:00] <mechanicjay> Bob Ross is giving me inspriation
[16:27:04] <stdhell> At least I'm not packing moving boxes for other people... (Still need to pack the last one of my own.)
[16:27:35] <mechanicjay> please, we're supposedly moving offices in 2-6 weeks, that'll be fun.
[16:28:04] <stdhell> We moved here about a year ago... And now we're moving again.
[16:28:27] <mechanicjay> This area has been the "Computer Center" for over 30 years and they're moving us to the basement of the Library
[16:28:59] <stdhell> And we had to move from our old old place in a hurry, because others were going to use the building... A couple of weeks ago when I was in the old building it was still empty...
[16:29:01] <mechanicjay> I mean we're in the basement of one of the academic buildings, so really no difference -- I'm actually getting a major office upgrade out the deal.
[16:29:55] <mechanicjay> I currently share an office with another guy. Access to the Systems room is through our office. So, we're on the same A/C circuit as the Systems Room, and have to keep a path open from the hall door to the Systems room door
[16:30:25] <mechanicjay> We're basically confined to using the opposite corners of a square as the doors are of course in opposite corners of the room.
[16:30:29] <stdhell> We're moving to a bigger office, but officially we're 5 in that room, so we need 5 tables in there... Everybody knows that we're only going to be 2 people in that room for the next 6 months or so, but for some stupid reason that nobody can explain, we need 5 tables _NOW!!!_
[16:30:54] <stdhell> ... or a couch too, because "it would be sad, if we had to put it down in the storage room".
[16:30:57] <stdhell> WTF?!
[16:31:22] <mechanicjay> I think I to bring my classic 1960's Seelcase desk to the new office, which has about 3x more usable space, and is still only the two of us.
[16:31:42] <stdhell> So don't only do we have to fit 5 tables in a room designed for 4, we also need to make room for a BIG FUCKING COUCH...
[16:31:56] <mechanicjay> nice
[16:32:36] <mattie_p> does anyone know the su password for the dev vm?
[16:32:38] <LaminatorX> Ross was kind of a one-trick-pony, be ha sure mastered that trick. Plus, I feel my shoulder muscles relaxing just thinking about the sound of his voice. https://www.youtube.com
[16:33:04] <Bytram|afk> stdhell, can the tables be folded up and moved aside?
[16:33:26] <stdhell> And the stupid idiot who wanted us to have that couch in our room, had made a drawing where it should go in the room... A drawing where the 2.8x2.8 meter long couch is going to fit in a 1.6m corner...
[16:34:05] <mechanicjay> stdhell: room layout requires graph paper and to scale cutouts of furniture.
[16:34:07] <janrinok> mattie_p: No, and it doesn't look like anyone else does either ;-)
[16:34:08] <stdhell> Bytram|afk: Our plan is that as soon as somebody has seen that we got 5 tables, we're going to put two of them on top of eachother and never move them again...
[16:34:28] <stdhell> mechanicjay: Yes, that's why I measured the couch before making a drawing...
[16:34:33] <mattie_p> janrinok apparently not
[16:34:35] <Bytram|afk> stdhell, I like!
[16:34:54] * Bytram|afk hears laundry is done... brb
[16:35:04] <stdhell> On my drawing the couch is labeled as "BIG F***ING COUCH".
[16:35:09] <janrinok> lol
[16:35:15] <mattie_p> stdhell sounds like umm, Douglas Adams' book Long Dark Tea Time of the Soul
[16:35:16] <mechanicjay> HA!
[16:35:33] <mattie_p> it features a couch that gets stuck in a stairwell and cannot get in or out, somehow
[16:35:56] <mattie_p> owner of the flat needs to go under or over it everytime to get in and out
[16:36:11] <stdhell> I haven't read that one. Only THHGTTG.
[16:37:10] <mattie_p> its a decent read, much in the same style
[16:38:35] * NCommander is getting there
[16:42:12] <NCommander> huh
[16:42:12] <NCommander> Mar 21 15:41:57 soylent-dev kernel: [3245587.198283] type=1400 audit(1395416517.415:251): apparmor="DENIED" operation="open" parent=3556 profile="/srv/soylentnews.org/apache/bin/httpd" name="/usr/share/mysql/charsets/Index.xml" pid=3558 comm="httpd" requested_mask="r" denied_mask="r" fsuid=1001 ouid=0
[16:42:16] <NCommander> oh
[16:42:21] <NCommander> that's probably DBD::mysql
[16:46:15] janrinok is now known as janrinok|afk
[16:46:23] <NCommander> w00t
[16:46:24] <NCommander> Ok
[16:46:28] <NCommander> Got an initial apparmor profile made
[16:46:32] <NCommander> That was suprisingly easy
[16:46:52] <Bytram|afk> NCommander++
[16:46:52] <Bender> karma - ncommander: 7
[16:47:40] <NCommander> paste.ubuntu.com/7131216/
[16:47:45] * NCommander has it *seriously* locked down
[16:48:02] <Bytram|afk> NCommander, orly? against what?
[16:48:40] <NCommander> Bytram|afk, general attacks
[16:48:57] <NCommander> Basically, if an intruder can pwn Apache, they can only override the log files
[16:49:04] <NCommander> Or read the slashcode or perl modules directories
[16:49:23] <Bytram|afk> glad to hear it! that's great!
[16:49:29] <NCommander> apparmor is awesome
[16:49:38] * NCommander needs to do the same with varnish, but that gets security updates at least
[16:51:02] <NCommander> and slashd
[16:51:06] <NCommander> But thats a huge improvement
[16:51:09] <Bytram|afk> NCommander, pm?
[16:51:13] <NCommander> Bytram|afk, sure
[16:53:10] * mechanicjay notes when the ups at my desk self tests, the alpha workstation's power supply makes an unsettling buzzing noise.
[16:57:21] -!- robind [robind!~robind@Soylent/Staff/Sysop/robind] has joined #staff
[16:57:21] -!- mode/#staff [+v robind] by SkyNet
[17:01:44] <NCommander> mechanicjay, check staff email :-)
[17:01:45] <NCommander> robind, ^
[17:01:51] <NCommander> We've got a straightjacket for slashcode
[17:02:01] <robind> what does that mean
[17:02:25] <NCommander> robind, check the email to see how locked down Apache can be made
[17:02:30] <NCommander> <3 AppArmor
[17:02:59] -!- prospectacle has quit [Quit: Web client closed]
[17:03:38] <robind> gmail is down for me lol
[17:03:43] <NCommander> robind, see PM
[17:05:05] <xlefay> NCommander: hint, if we an get a frontend varnish up, switch the IPs from prod to the frontend server, you can set varnish up to do caching, and when you start the maintainance window, strip cookies and show everyone a static cached version
[17:05:12] <xlefay> s/hint/suggestion/
[17:05:22] <NCommander> xlefay, I need to physically shut the nodes done
[17:05:33] <NCommander> xlefay, and drop into bootloader
[17:06:41] <xlefay> aah you misunderstand, I meant a separate node - that also serves cache when the backend server is done
[17:06:58] <xlefay> down*
[17:07:01] <NCommander> xlefay, ah, yeah, in the future :-)
[17:08:04] LaminatorX is now known as LaminatorX|afk
[17:08:27] <xlefay> ;-)
[17:21:05] <Landon> editor alert: novelity->novelty
[17:21:12] <NCommander> Landon, thanks
[17:21:20] <NCommander> mechanicjay, BTW, I'm testing killing static pages in slash
[17:22:51] <NCommander> Huh
[17:22:52] <NCommander> Interesting
[17:22:53] <mattie_p> oh, so less memcache issues then?
[17:22:58] <NCommander> mattie_p, yeah
[17:23:06] * NCommander semi-reproduced the stupid issue with stories not showing up
[17:23:11] <mattie_p> I know that was cause for many complaints
[17:23:26] <NCommander> Yeah
[17:23:27] <mattie_p> polls, especially, were / are taking a long time to show up
[17:23:36] <NCommander> I'm tempted just to do this on production now
[17:24:45] * NCommander hrms
[17:25:52] <xlefay> Well you scheduled a window already, best to wait for that, but do we know the long term effect of this change?
[17:27:37] <NCommander> grumble
[17:27:44] <NCommander> xlefay, no, which is why I haven't flipped it on
[17:30:10] <NCommander> Ok
[17:30:19] <NCommander> It looks like I got most of the dyn-only hiccups fixed
[17:40:52] <NCommander> xlefay, so .... Brtram just hit the dev server with a index/30+ threads, and seems most of its ok
[17:40:59] <NCommander> I'm tempted to switch us over dynamic fully
[17:41:03] <NCommander> We can alway revert it
[17:41:12] <NCommander> ^- robind, mechanicjay - thoughts?
[17:43:15] <robind> idk sounds scary
[17:43:37] <xlefay> The real question is, is the current issue people have big enough to warrant a change that may or may not cause issues for more people? SN is still rather small; but I don't think we should just rush something like this through.
[17:43:40] Bytram|afk is now known as Bytram
[17:43:57] <xlefay> Better to be safe than sorry, I think
[17:44:10] <xlefay> s/safe/more sure/
[17:44:19] <Bytram> keep in mind that I have a *skinny* pipe, at best I can pull about 50KB/sec
[17:44:31] <xlefay> Did you guys run apache bench?
[17:44:52] <Bytram> xlefay, great idea.
[17:45:00] <xlefay> Bytram: how did you test it?
[17:45:05] <Bytram> I can't do it. NCommander?
[17:45:28] <xlefay> I'm fairly sure I can, but I'm more curious how you tested it before
[17:45:31] <Bytram> xlefay, a web link checking tool
[17:45:47] * xlefay remembers messing up linodes traffic graph ;-)
[17:46:23] <xlefay> NCommander: say that word; would an AB help determine if it'll work properly under extreme loads?
[17:46:28] <robind> i have a 30MB connection right now
[17:46:42] <xlefay> I'm fairly sure I can throw 200+ at it
[17:46:59] <robind> err I mean Mb whatever
[17:47:00] <xlefay> but an apache bench should suffice I'd say
[17:47:12] <NCommander> xlefay, point it at dev.soylentnews.org
[17:47:13] <Bytram> I'm getting quite a few timeouts and cancelled and connection aborted
[17:47:17] <xlefay> robind: so do I: http://www.speedtest.net
[17:47:17] <NCommander> xlefay, but let me disable the log files
[17:47:30] <xlefay> ok
[17:47:39] <robind> fancy
[17:47:48] <Bytram> pls give me 5 minutes or so to finish scanning.
[17:48:08] <NCommander> xlefay, hit it
[17:48:17] <robind> we bought like the cheapest fiber we could get
[17:48:27] <xlefay> ab -n 5000 -c 100 http://dev.soylentnews.org
[17:48:31] <xlefay> too much? too low?
[17:48:39] <NCommander> xlefay, use your disgression
[17:49:07] * xlefay notices he hasn't got apache bench installed .___.
[17:49:16] <robind> might be important idk
[17:49:42] * NCommander wants to see how far we can push it before it falls over
[17:50:24] <xlefay> running..
[17:50:57] <Bytram> round 2 completed... retrying broken links with round 3
[17:51:06] <xlefay> it's done.. that was a low test I think tho
[17:51:16] <NCommander> xlefay, there some links I expect to be broken
[17:51:24] <NCommander> DUe to the way I managed the DB
[17:51:27] <xlefay> Apache test doesn't follow links though eh
[17:51:33] <NCommander> xlefay, bandwdith is at 3 Mbits
[17:51:40] <xlefay> it just benchmarks how much apache can handle
[17:51:57] <xlefay> http://paste.ubuntu.com
[17:52:07] <xlefay> err
[17:52:07] <NCommander> xlefay, so seige is a better tool for this http://www.joedog.org
[17:52:08] <xlefay> http://paste.ubuntu.com
[17:52:09] <Bytram> heh. number of URLs jumped from 8925 to 9598
[17:52:28] <NCommander> Varnish needs a longer timeout
[17:52:30] <NCommander> Complete requests: 5000
[17:52:30] <NCommander> Failed requests: 3840
[17:52:30] <NCommander> (Connect: 0, Receive: 0, Length: 3840, Exceptions: 0)
[17:52:30] <NCommander> Total transferred: 253070966 bytes
[17:53:00] <xlefay> That's actually what AB does too, but don't think AB supports authentication and such *looks*
[17:53:32] <Bytram> oh... 9821 URLs, now.. that is the number of broken/failed links from prior run that it is trying to resolve.
[17:53:48] <NCommander> xlefay, so what's the Failed requests
[17:53:51] <NCommander> 404s or?
[17:54:02] <xlefay> I suspect that's when dev got overloaded
[17:54:41] <Bytram> hmmm, nope. Had it right the first time... total URLs (that it knows of so far: 9941). I'm 99% done
[17:54:52] <xlefay> https://httpd.apache.org
[17:55:01] <xlefay> Failed requests
[17:55:03] <xlefay> The number of requests that were considered a failure. If the number is greater than zero, another line will be printed showing the numer of requests that failed due to connecting, reading, incorrect content length, or exceptions.
[17:55:18] <NCommander> "incorrect length"
[17:55:19] <NCommander> wtf?
[17:55:39] <xlefay> Yes, I'm wondering the same now
[17:55:55] <xlefay> I'm guessing due to the amount of request, apache didn't finish every one?
[17:56:01] <NCommander> Might be slash being bitchy
[17:56:05] * NCommander is running it locally here
[17:56:13] <NCommander> My pipe is shit
[17:56:18] <xlefay> dev. points to you?
[17:56:19] <NCommander> so this will take longer
[17:56:28] * xlefay thought it was a linode
[17:56:31] <NCommander> xlefay, no, its a linode
[17:56:34] <xlefay> ooh
[17:56:34] <NCommander> I'm running ab from here
[17:56:41] <xlefay> aah
[17:56:42] <NCommander> I'm going to fire it off from within the linode datacentre in a bit
[17:56:48] <xlefay> which command line are you using?
[17:56:53] <NCommander> xlefay, same one you used
[17:57:00] <xlefay> Linode to Linode seems unfair, internal networks and such?
[17:57:21] <NCommander> xlefay, I want to toast Apache :-)
[17:57:29] <xlefay> ooh sec
[17:57:42] <NCommander> 3000 requests done
[17:57:49] <xlefay> Let me know if this works
[17:58:01] <NCommander> ?
[17:58:13] <xlefay> running some more ABs ;-)
[17:58:18] <xlefay> You want to toast Apache, right? :)
[17:58:27] * NCommander waits to see if smoke escapes
[17:58:37] <NCommander> xlefay, I'm honestly curious what it takes to bring us down
[17:58:38] <Bytram> LOL!
[17:58:53] <NCommander> Its lagging
[17:58:55] <xlefay> Only thing I'm afraid of is that AB is slowing us down
[17:58:56] <NCommander> But its still up
[17:58:56] <Bytram> I'm at 10152 or 11386 URLs (89%)
[17:59:02] <xlefay> apr_socket_recv: Connection timed out (110)
[17:59:04] <xlefay> Total of 38375 requests completed
[17:59:09] <xlefay> LOL
[17:59:23] <xlefay> I don't think dev. liked that, very much
[17:59:30] <NCommander> xlefay, increasing varnish's timeout helped
[17:59:34] <Bytram> Now i"m getting connection aborted and cancelled/timout
[17:59:45] <xlefay> Bytram: probably my fault..
[17:59:53] <Bytram> but some are still making it through
[18:00:04] <xlefay> NCommander: that's good, I got server "Apache" though?
[18:00:11] <xlefay> Shouldn't it say Varnishd?
[18:00:56] <Bytram> Oh! a new one! "temporarily overloaded"
[18:00:58] <xlefay> wow http://dev.soylentnews.org it's so slow :P
[18:01:12] <NCommander> xlefay hit us with a low orbital cannon
[18:01:23] <NCommander> xlefay, server "apache"?
[18:01:24] <NCommander> apr_socket_recv: Connection reset by peer (104)
[18:01:24] <NCommander> Total of 4412 requests completed
[18:01:24] <NCommander> Crap
[18:01:24] <NCommander> Funny enough
[18:01:24] <NCommander> I can still access it with my web browser
[18:01:24] <NCommander> So lagging fiercely
[18:01:25] <NCommander> But still usable
[18:01:25] <NCommander> (this is a linode 1024 with DB on the same box)
[18:01:26] <NCommander> Nope
[18:01:26] <NCommander> wait
[18:01:27] <NCommander> I think its dead Jim
[18:01:29] <xlefay> LOOOOOL
[18:01:40] <xlefay> Maaan, you were lagging like crap
[18:02:05] <xlefay> Error 503 Service Unavailable
[18:02:14] <Bytram> I'm getting "temporarily overloaded" left and right, but an occasional request makes it through
[18:02:43] <xlefay> .voice Popeidol
[18:02:43] -!- mode/#staff [+v Popeidol] by SkyNet
[18:02:57] <xlefay> I'm fairly sure, Apache is still trying to respond ;-)
[18:03:07] <Bytram> about 1 in 10 are making it through, now.
[18:04:05] * xlefay doesn't think NCommander is here anymore...
[18:04:14] <xlefay> prolly not
[18:04:40] <NCommander> xlefay, dude, did you hit me with AB?
[18:04:42] * NCommander got knocked offline on the ISP end
[18:04:45] <xlefay> an apache restart *should* fix most issues
[18:04:52] <xlefay> NCommander: nope, I hit dev.soylentnews.org
[18:05:08] <NCommander> xlefay, did Apache deadlock?
[18:05:22] <xlefay> but you may have maxed out your connection via AB
[18:05:35] <xlefay> not sure.. Apache doesn't seem to handle the load very well
[18:05:38] <NCommander> xlefay, System information disabled due to load higher than 8.0
[18:05:40] <NCommander> ...
[18:05:45] <xlefay> LOL
[18:05:49] <xlefay> That's @ dev?
[18:05:58] <NCommander> xlefay, yeah
[18:06:07] <xlefay> Kill apache
[18:06:07] * NCommander takes a look
[18:06:19] <xlefay> It probably started many many processes
[18:06:27] <NCommander> mcasadevall@soylent-dev:~$ ps ax | grep httpd | wc -l
[18:06:27] <NCommander> 100
[18:06:29] <Bytram> just finished round 3; 11618 URLs checked many timeouts; starting round 4
[18:06:29] <xlefay> Hey... least there was smoke
[18:06:30] <NCommander> dude
[18:06:39] <xlefay> Only 100?
[18:06:46] <xlefay> Varnish is definitely doing its job
[18:06:51] <xlefay> (also, AB is disabled for a while now...)
[18:06:59] <NCommander> xlefay, let me see if I can make things better
[18:07:04] * NCommander wishes we could use something beside prefork
[18:07:13] <xlefay> I haven't been testing for like 7 mins
[18:07:31] <Bytram> getting consistent: temporarily overloaded msgs now.
[18:07:35] <xlefay> Bytram: ;-)
[18:07:45] * xlefay still sees the smoke
[18:08:10] <Bytram> need to get ready for work... back in about 15-20 minutes.
[18:08:22] Bytram is now known as Bytram|afk
[18:08:26] <NCommander> xlefay, thoughts: http://httpd.apache.org
[18:08:28] <mattie_p> later Bytram!
[18:08:39] <Bytram|afk> I'll be back!
[18:09:13] <xlefay> ciao Bytram|afk
[18:10:17] <NCommander> xlefay, I'm open to thoughts on how to keep us up
[18:11:53] <xlefay> I think our current set up is pretty good UNLESS someone does an AB or worse; but we need better than that, so what do we get if we tune Apache further?
[18:12:00] <xlefay> It can handle, a little bit more?
[18:13:40] <NCommander> xlefay, well, despite what the system said, linode said disk i/o nor CPU went berserk
[18:13:50] <xlefay> Maybe a load balancing situation will allow us to handle even more? But it would get pricey. I would optimize Apache as much as I can (but wouldn't over do it) and seriously consider a 3rd party to do the frontend crap, I realize some here are against such a thing (e.g. cloudflare) but it would allow us to serve -so- much more without having to worry about our own infra
[18:13:59] <NCommander> oooh
[18:14:02] <NCommander> memcache wasn't installed
[18:14:15] <xlefay> memcache wouldn't help that much out either at these loads, would it?
[18:14:35] <NCommander> xlefay, SQL lookups
[18:14:47] <NCommander> Also, not using a tuned SQL database config
[18:14:50] * NCommander guesses that's our blocker
[18:15:25] <xlefay> As long as memcache is setup properly, I suppose I couldn't forcefully overload the systems memory
[18:15:28] <NCommander> xlefay, ah ...
[18:15:28] <NCommander> ahem
[18:15:33] <NCommander> the mysqld server crashed
[18:15:35] <NCommander> THATS why it broke
[18:15:39] <xlefay> We seriously need cgroups and crap
[18:15:45] <xlefay> NCommander: you sure it wasn't killed by the kernel?
[18:16:19] <NCommander> xlefay, checking
[18:16:37] <xlefay> Also... MySQL, what does one expect.
[18:16:58] <NCommander> Mar 21 17:07:00 soylent-dev kernel: [3250688.762825] Out of memory: Kill process 13872 (httpd) score 37 or sacrifice child
[18:16:58] <NCommander> Mar 21 17:07:00 soylent-dev kernel: [3250688.762837] Killed process 13872 (httpd) total-vm:243868kB, anon-rss:8348kB, file-rss:0kB
[18:16:58] <NCommander> Mar 21 17:07:00 soylent-dev kernel: [3250688.769025] httpd: page allocation failure: order:0, mode:0x200da
[18:16:58] <NCommander> Mar 21 17:07:00 soylent-dev kernel: [3250688.769033] Pid: 13872, comm: httpd Not tainted 3.2.0-60-virtual #91-Ubuntu
[18:17:00] <NCommander> Mar 21 17:07:00 soylent-dev kernel: [3250688.769036] Call Trace:
[18:17:02] <NCommander> Yeah, it started OOMing
[18:17:06] <NCommander> probably what happened to mysql
[18:17:11] <xlefay> NCommander: trust me, Memcache wouldn't help much either ;-)
[18:17:27] <xlefay> Unless you configure it up to the detail, it wouldn't be too hard to overload either
[18:17:49] <xlefay> Just have to hit hard on a few dozen different pages that are SQL intensive
[18:17:51] <NCommander> Hrm
[18:17:55] <NCommander> My guess if we're leaking memory
[18:17:58] <NCommander> somewhere in slash
[18:18:03] * NCommander sets the child to die after 100 requests
[18:18:15] <xlefay> Mind you, I was trowing 1000 per run
[18:18:26] <xlefay> I wonder how long it took to process
[18:18:44] <xlefay> and where was varnish?!
[18:18:49] <NCommander> xlefay, varnishing :-P
[18:18:54] <xlefay> I was hitting the frontpage without being signed in
[18:18:56] * NCommander sets a ulimit
[18:19:02] <NCommander> huh
[18:19:03] <NCommander> hrm
[18:19:03] <xlefay> Each time, I should've been presented with a static HTML page
[18:19:09] <NCommander> xlefay, I nuked the static HTML page
[18:19:12] <xlefay> ooh
[18:19:13] <NCommander> so now its just index.pl
[18:19:14] <xlefay> right
[18:19:32] <NCommander> Varnish should be able to cache that though
[18:20:01] <xlefay> yea I thought slashd created that cache..
[18:20:18] <NCommander> xlefay, yeah, I nuked that
[18:20:22] <xlefay> I'd just have varnish provide caching services imo, it's good at that
[18:20:22] <NCommander> On dev
[18:20:31] <NCommander> Yeah, varnish should be caching .pl
[18:20:45] <NCommander> Unless index.pl is setting a cookie
[18:20:58] <xlefay> Honestly, yesterday 251 php scripts nuked a lot @ services server
[18:21:04] <xlefay> I would hate to see what a perl script can do
[18:21:13] <NCommander> Its not
[18:21:35] <xlefay> It's not a contributing factor either?
[18:21:55] <xlefay> Say it takes 5 seconds to deliver those 1000 requests, it'd be 200 .pl runs per second?
[18:22:21] <xlefay> 200 db connections p/s, and then we're talking about Varnish & Apache still
[18:22:53] <NCommander> xlefay, well, varnish always goes down if apache goes down
[18:22:57] <NCommander> hrm
[18:22:58] <xlefay> 200 p/s > Varnish <-> Apache (<-> slashd) <-> index.pl ?
[18:23:01] <xlefay> ooh
[18:23:08] <NCommander> so it still hits the backend apache processes
[18:23:14] <xlefay> 200 p/s > Varnish <-> Apache (<-> slashd) <-> index.pl <-> [memcache <-> ]MySQL ?
[18:23:37] <NCommander> Bingo
[18:23:38] <xlefay> I really think varnish should keep serving static cache after apache goes down
[18:23:46] <NCommander> I'm trying to figure out how to do that
[18:23:53] <NCommander> And have the cache die after five minutes
[18:23:54] <xlefay> ooh
[18:23:57] <xlefay> you should have said so already!
[18:24:01] <NCommander> so an AC never sees content more than 5 minutes old
[18:24:05] <xlefay> sec, let me give you a link
[18:25:35] <xlefay> https://www.varnish-cache.org
[18:25:38] <xlefay> should work?
[18:25:47] <xlefay> I had a better one but can't find it :/
[18:25:50] <xlefay> *keeps searching*
[18:25:51] <NCommander> I think I need to set Cache-Control from Apache
[18:26:33] <xlefay> https://www.varnish-cache.org also informative
[18:26:54] <xlefay> Personally, I'd just say, in varnish if there isn't a cookie present, hard cache that shit
[18:27:11] <NCommander> xlefay, I'm tinkering with the varnish config now
[18:28:21] <NCommander> trying to figure out what this probe.cgi is
[18:28:25] <xlefay> Also, while the dev was terribly show, the CSS was loading terrible slow (who would've guessed?) and JUST before the dark, ugly red, there was a nice color for the "tops" of articles... wonder if anyone else seen that one?
[18:28:38] <NCommander> no idea
[18:28:45] * NCommander wonders if varnish is inflating slash hit counts
[18:28:59] <xlefay> probe.cgi isn't on github?
[18:29:04] <xlefay> @ soylentnews/slashcode
[18:29:33] <xlefay> If varnish is, that might even be enough to hit the DB a lot
[18:29:48] <xlefay> In which case, if memcache can cache those before updating, it'd be nice
[18:29:51] <NCommander> ah
[18:29:53] <NCommander> I know how to fix it
[18:29:54] <NCommander> Hold on
[18:30:11] <xlefay> but iirc, redis might be better suited for key->value but..... yet another service isn't awesome
[18:30:12] <xlefay> ok
[18:31:04] <NCommander> xlefay, hit it again
[18:31:27] <NCommander> If we go belly up, varnish SHOULD prevent us from going boom
[18:31:37] <NCommander> varnish is set to cache 30 seconds if the backend is healthy ATM
[18:31:39] <NCommander> 1h if its dead
[18:31:41] <mattie_p> so I think I broke the dev vm :)
[18:31:42] <xlefay> ~ $ /usr/local/apache/bin/ab -n 50000 -c 1000 http://dev.soylentnews.org
[18:31:42] <Bytram|afk> only got a minute; finished round 4; 13585 URLs found. Still have approx 60% timeout links.
[18:31:56] <xlefay> 10.000 req done
[18:31:57] <Bytram|afk> need to disconect and go to work.
[18:31:59] <Bytram|afk> have fun everybody!
[18:32:04] <mattie_p> later, Bytram|afk
[18:32:05] <xlefay> 15.000
[18:32:11] <xlefay> 20.000
[18:32:20] <xlefay> takes approx 5 seconds for 5000 req
[18:32:21] <mattie_p> does csh have a dependency on apache?
[18:32:35] <mattie_p> apacheclt start no longer works for me after I installed that
[18:32:41] <NCommander> mattie_p, no?
[18:32:48] <mattie_p> hrm
[18:32:49] <NCommander> mattie_p, oh, you need .profile
[18:32:54] <NCommander> You have to set the path and other vars correctly
[18:32:58] <xlefay> apr_socket_recv: Connection timed out (110)
[18:32:59] <xlefay> Total of 39937 requests completed
[18:33:05] <NCommander> xlefay, well, thats better
[18:33:06] <mattie_p> ahh, k
[18:33:12] <NCommander> xlefay, its still up
[18:33:17] <mattie_p> I sometimes forget the simple stuff
[18:33:17] <xlefay> Error 503 Service Unavailable
[18:33:29] <xlefay> but yeah it's up
[18:33:38] <NCommander> Er, it 503 hered
[18:33:43] <xlefay> here too
[18:33:44] <NCommander> Now its not
[18:33:46] <NCommander> what happened
[18:33:48] <xlefay> still is
[18:33:52] <Popeidol> 503 here also.
[18:34:11] <NCommander> Mar 21 17:22:20 soylent-dev varnishd[12967]: Manager got SIGINT
[18:34:11] <NCommander> Mar 21 17:22:20 soylent-dev varnishd[12967]: Stopping Child
[18:34:11] <NCommander> Mar 21 17:22:21 soylent-dev varnishd[17692]: Platform: Linux,3.2.0-60-virtual,x86_64,-smalloc,-smalloc,-hcritbit
[18:34:11] <NCommander> Mar 21 17:22:21 soylent-dev varnishd[17692]: child (17693) Started
[18:34:12] <NCommander> Hrm
[18:34:16] <NCommander> I think vaarnish got OOMed
[18:34:27] <xlefay> NCommander: I honestly don't think we can handle this load on our own. Not unless we actually have a load balancing situation which would be expensive.
[18:34:28] <NCommander> xlefay, Mar 21 17:31:32 soylent-dev kernel: [3252162.480921] TCP: Possible SYN flooding on port 80. Sending cookies. Check SNMP counters.
[18:34:29] <NCommander> :-P
[18:34:46] <xlefay> I was trowing approx 1000 r/q per second
[18:34:49] <NCommander> xlefay, we can use the linode load balancer at it
[18:34:54] <xlefay> would be expensive?
[18:34:59] <NCommander> 20 dollars a month
[18:35:09] <xlefay> hmm
[18:35:15] <NCommander> yeah
[18:35:22] <NCommander> We start OOMing and http instances start dropping
[18:35:23] <NCommander> Hrm
[18:35:39] <NCommander> Varnish isn't doing its job though
[18:35:54] <xlefay> Have you looked at cloudflare, the one I mentioned I couple of times? It would be a cheap way of saving us money for now. + protect us from ddos's and crap...
[18:36:14] -!- Bytram|afk has quit [Ping timeout: 246 seconds]
[18:36:24] <xlefay> cloudflare does exactly what we want, for free
[18:36:39] <xlefay> load balanced and all (all we need to do is let cloudflare handle the incoming connections)
[18:37:23] <mechanicjay> oof, that's quite the scroll back I just caught up on.
[18:38:08] <mechanicjay> What you want to do is to tune the number of apache servers and workers so that you don't let apache grow it's process count so high that it starves the box for memory.
[18:38:22] <mechanicjay> If the OOM killer kicks in, you're dead in the water anyway, better to drop connections at that point.
[18:38:34] <xlefay> and cgroups to control max memory stuff preventing other stuff from killing the entire server
[18:38:58] <NCommander> mechanicjay, yeah, I'm working on it
[18:39:11] <xlefay> NCommander: have you worked with cgroups before?
[18:39:21] <mechanicjay> I think that cgroups can be a safety net underneath it all, but if we have it tuned right, we shouldn't need the net.
[18:39:27] <xlefay> MJ and I are going to look into them, but if you already worked with them you may be able to tell us whether it's the right way to go
[18:39:35] <NCommander> xlefay, yeah
[18:39:43] <xlefay> mechanicjay: that really depends on how much is being throw at us
[18:39:52] <xlefay> if more is thrown at us than apache can handle, apache will still go down
[18:40:00] <xlefay> but..
[18:40:03] <xlefay> that's always the case lol
[18:40:04] <NCommander> xlefay, its lagging, but its up
[18:40:16] <NCommander> xlefay, hit it
[18:40:24] <NCommander> varnish should prevent Apache from exploding now
[18:40:57] <xlefay> apr_socket_recv: Connection reset by peer (104)
[18:40:58] <xlefay> Total of 16715 requests completed
[18:41:01] <NCommander> shit
[18:41:05] <NCommander> thats going the wrong way!
[18:41:17] <NCommander> xlefay, er, its still up
[18:41:18] <xlefay> it didn't een get to 20.000 this time! dammit
[18:41:34] <NCommander> we didn't OOM ...
[18:41:45] <xlefay> AB stops when stuff on the other end goes to far
[18:41:54] <xlefay> (hence why I chose AB, I don't want to kill the box entirely)
[18:41:58] <mechanicjay> xlefay: I'd rather have apache croak than kill mysql and risk the db.
[18:42:07] <NCommander> mechanicjay, mysql is on a seperate box
[18:42:13] <NCommander> (not true on the dev server)
[18:42:14] <xlefay> 25
[18:42:16] <mechanicjay> I thought we were talking about dev
[18:42:20] <NCommander> mechanicjay, ah, true
[18:42:22] <xlefay> 30
[18:42:32] <xlefay> 35
[18:42:34] <mechanicjay> what does varnishstat say on the dev box?
[18:42:36] <NCommander> dev server lagging fercily
[18:42:41] <xlefay> 39....
[18:42:45] <xlefay> and it crapped out
[18:42:49] <NCommander> Hitrate ratio: 1 1 1
[18:42:49] <NCommander> Hitrate avg: 0.9989 0.9989 0.9989
[18:43:17] <mechanicjay> right, run varnishstat for the entirety of the next load test
[18:43:23] <NCommander> xlefay, HIT IT
[18:43:32] <xlefay> hitting..
[18:43:38] <xlefay> err:
[18:43:40] <NCommander> mechanicjay, well I told varnish to cache for 30s
[18:43:43] * xlefay starts smakcing it around
[18:43:53] <xlefay> smacking*
[18:44:04] <xlefay> You might want to increase 30 to a more sensible number
[18:44:05] <NCommander> I can't copy and paste from varnishstat
[18:44:13] <xlefay> least, 1 or 2 minutes I'd say
[18:44:19] <NCommander> xlefay, well, I want to tune it for static assests and such
[18:44:22] <mechanicjay> just the ratio and avg are really interesting numbers
[18:44:28] <xlefay> 30
[18:44:29] <xlefay> 35
[18:44:30] <mechanicjay> they'll let you know how much varnish is actually caching
[18:44:39] <xlefay> 40
[18:44:48] <NCommander> So the backend connections are double digit
[18:44:49] <xlefay> 42.... crapped out
[18:44:57] <NCommander> varnish is reporting 40k connections inbound
[18:45:06] <NCommander> xlefay, I think varnish itself is farting
[18:45:09] <NCommander> And dropping connections
[18:45:15] <NCommander> We're not exploding under our own weight
[18:45:38] <xlefay> Well that's 50% good and 50% bad
[18:45:48] <NCommander> let me see if I can tweak varnish
[18:45:49] <xlefay> How many connections p/s on production?
[18:45:56] * NCommander runs varnishstat on production
[18:46:40] <mechanicjay> do I have to bounce to prod via the services box now?
[18:46:54] <NCommander> From web
[18:46:59] <NCommander> ugh
[18:47:23] <NCommander> mechanicjay, yeah, but you also need a SSH key installed
[18:47:24] <xlefay> Dev's holding out better now btw
[18:47:37] <NCommander> xlefay, we haven't caused it to collaspe
[18:47:39] <NCommander> Which is a win
[18:48:01] <NCommander> xlefay, can you get ab to keep going even if a connection drops?
[18:48:07] <xlefay> maybe not sure
[18:48:36] * NCommander tests the backend
[18:48:40] <NCommander> me unplugs Apache
[18:48:43] <NCommander> If I did this right
[18:48:52] <NCommander> varnish should keep caching what it knows
[18:48:53] <xlefay> hitting
[18:49:15] <NCommander> COOL
[18:49:17] <NCommander> Its still up
[18:49:19] <NCommander> httpd is stopped
[18:49:20] <mattie_p> ok, I've figured it out. I have a mostly clean compiling version of a MUD running on the VM :)
[18:49:31] <xlefay> 20
[18:49:38] <NCommander> slash@soylent-dev:~$ ps ax | grep httpd
[18:49:38] <NCommander> 22657 pts/5 S+ 0:00 grep --color=auto httpd
[18:49:45] <xlefay> hmm
[18:50:06] * NCommander should plug apache back in :-)
[18:50:22] <xlefay> So, what if I were to hit pages that weren't cached, it'd 503, but would it hit the filesystem every time?
[18:50:23] <NCommander> I need to make it do the following
[18:50:34] <xlefay> or does varnish keep track in memory what it cached or not?
[18:50:40] <NCommander> xlefay, http://dev.soylentnews.org - yup
[18:50:48] <NCommander> xlefay, it keeps track what it cached and isn't cached
[18:50:51] <xlefay> I know about the 503
[18:50:55] <NCommander> So if Apache is down, it will 503 stuff it won't know about
[18:51:04] <xlefay> but I was curious if it checked the filesystem or just had the data in memory
[18:51:10] <NCommander> in memory
[18:51:17] <NCommander> For stuff it doesn't know, it asks Apache
[18:51:17] <xlefay> in case it was the filesystem I could just hit arbitrary URLs
[18:51:22] <NCommander> varnish doesn't write anything to disk
[18:51:30] <xlefay> apr_pollset_poll: The timeout specified has expired (70007)
[18:51:32] <xlefay> Total of 49784 requests completed
[18:51:33] <xlefay> That's good
[18:51:40] <NCommander> So under very high load
[18:51:43] <NCommander> varnish drops a connection
[18:51:45] <NCommander> Honestly
[18:51:49] <NCommander> Not a big deal :-)
[18:51:59] <xlefay> Is that a fact or is that an assumption?
[18:52:11] <xlefay> If it's an assumption, I'm guessing it just doesn't even get the connection
[18:52:12] <NCommander> xlefay, fact. You've been hitting varnish without hitting backend apache right now
[18:52:15] <NCommander> oh
[18:52:19] <NCommander> That might be true
[18:52:36] <xlefay> Honestly, I should have been blacklisted already
[18:52:43] <NCommander> I plugged Apache back in
[18:52:53] <xlefay> We need to have something like that I think
[18:52:56] <NCommander> Yeah
[18:53:10] <xlefay> approx 1000 r/q p/s from a single IP should never constitute "good"
[18:53:36] <xlefay> unless it's localhost or a cluster environment.. but then it's still high
[18:53:51] <xlefay> way to high*
[18:54:26] * xlefay goes to check his bw graphs
[18:54:36] <NCommander> Ok
[18:54:43] <NCommander> So for AC
[18:54:47] <NCommander> Varnish caches for 5 minutes
[18:54:56] <NCommander> For logged in users
[18:55:03] <xlefay> How does Varnish play with IPv6?
[18:55:08] <NCommander> Plays great
[18:55:15] <NCommander> want to test it :-)?
[18:55:17] <xlefay> Good :)
[18:55:19] <xlefay> Sure
[18:55:48] <xlefay> ah man, linode can only take so little ;(
[18:56:08] <NCommander> xlefay, 2600:3c00::f03c:91ff:fe6e:d0a3
[18:56:10] <xlefay> Do you have a bin file I can download to test max speed?
[18:56:23] <mechanicjay> NCommander: when trying to ssh from services to www I get "No route to host"? Have my keys setup on services now, btw
[18:56:35] <xlefay> benchmarking
[18:56:38] <xlefay> 25
[18:56:43] <NCommander> xlefay, er, I just reset varnish
[18:56:46] <NCommander> xlefay, try it again
[18:56:46] * xlefay notes IPv6 goes somewhat quicker
[18:57:02] <NCommander> No, I was updating the config and killed varnishd :-)
[18:57:06] <xlefay> oh
[18:57:15] <NCommander> mechanicjay, give me a sec, I need to install your key on the www box
[18:57:19] <xlefay> but seriously, IPv6 looks to be going so much faster is that because of the config update
[18:57:20] <xlefay> ?
[18:57:21] * NCommander notes LDAP: priority +++
[18:57:25] <mechanicjay> NCommander: sure thing
[18:57:26] <xlefay> Time taken for tests: 12.879 seconds
[18:57:31] <xlefay> Concurrency Level: 1000
[18:57:34] <NCommander> xlefay, try it against dev again
[18:57:37] <xlefay> Complete requests: 50000
[18:57:39] <xlefay> Failed requests: 0
[18:57:44] <xlefay> Wasn't that the dev ipv6?
[18:57:56] <NCommander> xlefay, it was, do IPv4
[18:57:57] <NCommander> rofl
[18:57:57] <xlefay> ooh goes quicker now
[18:58:02] <NCommander> hitratio: 1
[18:58:07] <xlefay> Concurrency Level: 1000
[18:58:09] <xlefay> Time taken for tests: 12.708 seconds
[18:58:10] <xlefay> Complete requests: 50000
[18:58:12] <xlefay> Failed requests: 0
[18:58:13] <xlefay> Non-2xx responses: 50000
[18:58:16] <NCommander> Nice
[18:58:25] <NCommander> er
[18:58:26] <NCommander> ...
[18:58:28] * NCommander is getting 503s
[18:58:29] <NCommander> oops
[18:58:40] <NCommander> xlefay, so you got 20k 503s
[18:58:42] <xlefay> let me update my ssh key and ssh it over, then you can also set mine directly when you do MJ's. ;-)
[18:58:52] <xlefay> Non-2xx responses: 50000
[18:58:59] <xlefay> So I'm guessing 50k 503's
[18:59:07] <xlefay> That would explain why it went so fast
[18:59:14] <stdhell> Nobody wants my ssh pub key... :-(
[18:59:29] <NCommander> mechanicjay, you can get into web via the linode console
[19:00:49] <mechanicjay> NCommander: ...right!
[19:01:21] <mechanicjay> NCommander: I just see varnish stat running, is that you?
[19:01:24] <NCommander> xlefay, ok, its back up
[19:01:26] <NCommander> mechanicjay, yeah, kill it
[19:01:32] <NCommander> xlefay, HIT IT
[19:01:42] <xlefay> hitting ipv6
[19:01:47] <xlefay> after ipv4
[19:01:51] <xlefay> afterwards*
[19:01:59] <xlefay> Complete requests: 50000
[19:02:01] <xlefay> Failed requests: 49104
[19:02:02] <xlefay> (Connect: 0, Receive: 0, Length: 49104, Exceptions: 0)
[19:02:04] <xlefay> Non-2xx responses: 50000
[19:02:08] <NCommander> hrm
[19:02:16] <NCommander> Length ...
[19:02:18] <NCommander> I *wonder* if thats the gzip stuff in varnish
[19:02:49] <xlefay> It's either that or varnish is breaking under the load sending half ass responses
[19:03:01] <NCommander> xlefay, http://stackoverflow.com
[19:03:18] <xlefay> but the content isn't different, right?
[19:03:27] <xlefay> It's a static page that should be returned the same, all the time?
[19:03:37] <NCommander> The apache benchmarking tool (ab) assumes that length of response content will be the same during entire test. It stores the content length of the first response. If any of further responses have different content length, they result in "length failures".
[19:03:37] <NCommander> Following apache bug report seems to confirm that: ASF Bug 42040
[19:03:37] <NCommander> Summary: If you are serving any content of variable length, you should probably ignore this kind of ab request failures.
[19:03:48] <NCommander> xlefay, point it at dev.soylentnews.org/about.shtml
[19:03:50] <NCommander> That's a static page
[19:03:57] <NCommander> We shouldn't get length failures there
[19:04:04] <NCommander> to rule varnish goes nuts
[19:04:05] <xlefay> hitting
[19:04:22] <xlefay> Time taken for tests: 12.708 seconds
[19:04:23] <xlefay> Complete requests: 50000
[19:04:25] <xlefay> Failed requests: 0
[19:04:26] <xlefay> Non-2xx responses: 50000
[19:04:28] <NCommander> \o/
[19:04:30] <NCommander> wait
[19:04:31] <NCommander> fuck
[19:04:35] <xlefay> apache not up? :)
[19:04:51] <NCommander> Ah
[19:04:52] <NCommander> Hrm
[19:04:56] <NCommander> appears apache crapped itself
[19:05:13] <xlefay> FML
[19:05:14] * NCommander tinkers
[19:05:53] <NCommander> xlefay, try it now
[19:05:54] <xlefay> I'm apache breaker :'(
[19:06:00] * NCommander did something to apache that makes it unhappy
[19:06:12] <xlefay> I don't want any details :o
[19:06:19] <xlefay> hitting
[19:06:28] <NCommander> xlefay, I'm tinkering with it, but I want tos ee if varnish is going mad, on the about.shtml
[19:06:44] <xlefay> Failed requests: 49461
[19:06:45] <xlefay> (Connect: 0, Receive: 0, Length: 49461, Exceptions: 0)
[19:06:47] <xlefay> Non-2xx responses: 50000
[19:06:59] <NCommander> ...
[19:07:09] <NCommander> "Non-2xx responses"
[19:07:11] <NCommander> the hell
[19:07:15] <xlefay> ~ $ /usr/local/apache/bin/ab -n 50000 -c 1000 -r http://[2600:3c00::f03c:91ff:fe6e:d0a3]
[19:07:23] <NCommander> oh
[19:07:24] <mechanicjay> ...are they 302?
[19:07:28] <NCommander> probably
[19:07:31] <xlefay> --2014-03-21 19:07:22-- http://[2600:3c00::f03c:91ff:fe6e:d0a3]
[19:07:32] <NCommander> xlefay, IPv4 only
[19:07:33] <xlefay> Connecting to 2600:3c00::f03c:91ff:fe6e:d0a3:80... connected.
[19:07:34] <xlefay> HTTP request sent, awaiting response... 503 Service Unavailable
[19:07:36] <xlefay> 2014-03-21 19:07:22 ERROR 503: Service Unavailable.
[19:07:37] <NCommander> dev.soylentnews.org
[19:07:44] <NCommander> Varnish might be having issues with Ipv6
[19:07:53] <xlefay> ipv4
[19:08:03] <Popeidol> xlefay: add some -v in there
[19:08:09] <NCommander> Nope
[19:08:12] <NCommander> meditation errors
[19:08:13] <NCommander> Ugh
[19:08:14] <NCommander> Hold on
[19:08:21] <xlefay> actually
[19:08:23] <xlefay> it went fine now
[19:08:25] * NCommander is grimreaping
[19:08:30] <xlefay> Time taken for tests: 24.813 seconds
[19:08:31] <xlefay> Complete requests: 50000
[19:08:33] <xlefay> Failed requests: 0
[19:08:34] <xlefay> Total transferred: 654200000 bytes
[19:08:55] <xlefay> Popeidol: which -v do you suggest?
[19:09:57] <NCommander> xlefay, yeah, that never touched apache
[19:10:35] <xlefay> btw.. is the IPv6 directly to apache or to varnish?
[19:11:10] <NCommander> Ok
[19:11:13] <NCommander> xlefay, varnish
[19:11:19] <NCommander> xlefay, ok, so when I'm logged in
[19:11:34] <NCommander> It looks like I'm properly hitting the backend and getting fresh HTML
[19:12:00] * xlefay tinkers whether he should hit with a cookie
[19:12:07] <xlefay> I'm fairly sure that'd kill the server still
[19:12:07] <NCommander> xlefay, DO IT
[19:12:13] <NCommander> xlefay, you need to set one we don't filter :-)
[19:12:27] <xlefay> authenticated users bypass cache controls, right?
[19:12:33] <NCommander> xlefay, not completely
[19:12:43] <NCommander> They hit index.pl, but everything else is cached
[19:12:59] <NCommander> xlefay, set user=/seasonkey= to something
[19:13:13] <NCommander> xlefay, or log in manually and copy your cookies
[19:15:41] <NCommander> xlefay, ?
[19:15:45] <xlefay> sec
[19:16:07] <audioguy> wiki is down?
[19:16:25] <xlefay> works here
[19:16:30] <mechanicjay> works here too
[19:16:50] <audioguy> Must be cache here,
[19:17:07] <xlefay> seasonkey= correct?
[19:17:22] <xlefay> NCommander: ?
[19:17:26] <audioguy> Yup, dang browsers.
[19:17:33] <NCommander> xlefay, yeah.
[19:17:36] <xlefay> ok
[19:17:39] <xlefay> trying
[19:17:43] <xlefay> wait
[19:17:51] <xlefay> nvm
[19:17:53] <NCommander> xlefay, or users
[19:17:54] <NCommander> *user
[19:17:55] <xlefay> hitting about.shtml though
[19:18:01] <NCommander> xlefay, hrm
[19:18:05] <NCommander> xlefay, actually two things
[19:18:06] <xlefay> user=Test & seasonkey=youknowyouwanna
[19:18:11] <NCommander> xlefay, FIRST, hit index.pl with no cookies
[19:18:13] <NCommander> lets make sure that works
[19:18:13] <xlefay> and requests failed again
[19:19:01] <NCommander> xlefay, hrm .... but it didn't die
[19:19:09] <xlefay> wait
[19:19:14] <xlefay> hitting index.pl no cookies
[19:19:14] <NCommander> xlefay, which means my apache tuning did the trick
[19:19:26] <xlefay> hitting
[19:19:27] <xlefay> hopefully ;-)
[19:19:32] * NCommander notes apache should drop requests if varnish lets too many through
[19:19:46] <xlefay> it's going slowly
[19:19:49] <NCommander> But not dead?
[19:19:59] <xlefay> 15
[19:20:01] <xlefay> 20
[19:20:06] <xlefay> 25
[19:20:10] <NCommander> xlefay, its *not* passing to the backend
[19:20:15] * NCommander is watching on varnishstat
[19:20:21] <xlefay> 30
[19:20:23] <NCommander> so it should run all 50
[19:20:25] <xlefay> 35
[19:20:30] <xlefay> *should* yea
[19:20:32] <xlefay> 40
[19:20:39] <xlefay> 45
[19:20:48] <xlefay> 50
[19:21:13] <xlefay> Time taken for tests: 83.516 seconds
[19:21:15] <xlefay> Complete requests: 50000
[19:21:16] <xlefay> Failed requests: 195
[19:21:18] <xlefay> (Connect: 0, Receive: 61, Length: 73, Exceptions: 61)
[19:21:19] <xlefay> Total transferred: 2517386064 bytes
[19:21:21] <xlefay> HTML transferred: 2500209112 bytes
[19:21:27] <xlefay> Requests per second: 598.69 [#/sec] (mean)
[19:21:28] <xlefay> Time per request: 1670.325 [ms] (mean)
[19:21:30] <xlefay> Time per request: 1.670 [ms] (mean, across all concurrent requests)
[19:21:31] <xlefay> Transfer rate: 29436.00 [Kbytes/sec] received
[19:21:39] <NCommander> Not bad
[19:21:59] <NCommander> Looks like the varnish cache flushed once on timeout while you're hitting it
[19:22:02] <xlefay> now with cookie
[19:22:12] * NCommander excepts it to DC
[19:22:17] <xlefay> /usr/local/apache/bin/ab -n 50000 -v debug -c 1000 -r -C 'user=Test' -C 'seasonkey=youknowyouwanna' http://dev.soylentnews.org
[19:22:26] * NCommander braces
[19:22:36] <xlefay> 15
[19:22:51] <xlefay> 25
[19:22:54] <xlefay> 30
[19:22:55] <NCommander> xlefay, its not passing to the backend
[19:23:02] <xlefay> hmm
[19:23:10] <xlefay> maybe I'm missing a cookie then
[19:23:29] <xlefay> -C attribute Add cookie, eg. 'Apache=1234'. (repeatable)
[19:23:37] <NCommander> looking
[19:23:46] <NCommander> the only cookie I have from dev.soylentnews.org is user
[19:23:58] <xlefay> Failed requests: 45
[19:23:59] <xlefay> (Connect: 0, Receive: 10, Length: 25, Exceptions: 10)
[19:24:01] <xlefay> not bad
[19:24:09] <xlefay> NCommander: but.. if you go around there, does it hit apache or not?
[19:24:23] <xlefay> @ dev site that is
[19:24:25] <NCommander> Yeah
[19:24:27] <NCommander> It does
[19:24:32] <xlefay> hmm
[19:24:37] <xlefay> maybe I should try hitting /my/user
[19:24:47] <xlefay> http://dev.soylentnews.org 503 tho
[19:24:48] <NCommander> ack
[19:24:50] <NCommander> apache blew
[19:25:20] <NCommander> xlefay, what happens is apache deadlocks
[19:25:26] <xlefay> why did it?
[19:25:47] <xlefay> it didn't even got hit except maybe 3 times or so by varnish itself?
[19:26:50] <NCommander> Odd
[19:26:57] <xlefay> ?
[19:26:59] <NCommander> varnishstat doesn't seem to work properly for backend_conn
[19:27:08] <xlefay> Apache did get hit?
[19:27:09] <NCommander> I think I need to look at cache_miss
[19:27:12] <NCommander> yeah
[19:27:15] <NCommander> but backend_conn didn't go up
[19:27:15] <xlefay> cache_miss yea
[19:27:18] <NCommander> Must be using keep alive
[19:27:22] <xlefay> backend_conn = how much?
[19:27:23] <xlefay> 1?
[19:27:44] <xlefay> cache_miss = how much gets passed through to the backend server because there was no cache
[19:28:02] <xlefay> least iirc
[19:28:05] <NCommander> 2092
[19:28:08] <NCommander> on backend_conn
[19:28:16] <xlefay> That's not bad
[19:28:21] <NCommander> right
[19:28:22] <xlefay> but I have no clue what that means
[19:28:29] <NCommander> so if theres ever a cookie
[19:28:33] <NCommander> it seems to completely bypass the cache
[19:28:38] <NCommander> And cache misses go up
[19:29:09] <mechanicjay> Many times, varnish is configured to strip the cookie off static resources for just this reason'
[19:29:09] <xlefay> how much cache miss?
[19:29:11] <NCommander> Cause cachemiss goes up by 16 every time I trl-F5
[19:29:20] <NCommander> mechanicjay, yeah, we are supposed to be doing that
[19:29:24] <NCommander> Doesn't look like it works
[19:29:26] <xlefay> Can you reset cache_miss?
[19:29:38] <NCommander> xlefay, let me tinker with the config more
[19:29:55] <xlefay> I'm curious, 50.000 * 16?
[19:30:07] <xlefay> That would be how much my AB does with cookies enabled?
[19:30:17] <NCommander> xlefay, if its pulling things like images and such
[19:30:21] <NCommander> if (req.url ~ "\.(png|gif|jpg|swf|css|js)(\?.*|)$") {
[19:30:21] <NCommander> unset req.http.cookie;
[19:30:21] <NCommander> }
[19:30:29] * NCommander changes how this works
[19:30:51] <xlefay> I've heard people complain that @ Slashdot it wasn't easy to see if you were signed in or not
[19:31:08] <xlefay> Could it not be, that slashdot just cached' everything except certain URLs, e.g. /my/ etc?
[19:31:26] -!- paulej72 [paulej72!~paulej72@Soylent/Staff/Developer/paulej72] has joined #staff
[19:31:26] -!- mode/#staff [+v paulej72] by SkyNet
[19:31:39] <xlefay> paulej72: you do realize, you don't actually have to leave this channel when you disconnect, right?
[19:32:09] <NCommander> Ok, lets try that
[19:32:36] <paulej72> xlefay: was I gone?
[19:32:43] <xlefay> you /part channels when you leave
[19:33:10] <xlefay> ok, hitting it
[19:33:16] <paulej72> Looks like I don't have everything setup correctly on my clients
[19:33:22] <NCommander> varnish reset
[19:33:23] <NCommander> hrm
[19:33:28] <xlefay> 40
[19:33:29] <xlefay> woa
[19:33:31] <xlefay> this goes fast
[19:33:34] <xlefay> and all 503
[19:33:35] <NCommander> and it all cache missed
[19:33:37] <NCommander> :-)
[19:33:39] <NCommander> oops
[19:33:43] <xlefay> Non-2xx responses: 50000
[19:33:49] <NCommander> 50022 cache missed
[19:34:00] <paulej72> what are we working on?
[19:34:08] <NCommander> Who wrote the varnish config
[19:34:10] <NCommander> zford?
[19:34:35] <xlefay> I think that's actually an example in the standard varnish config
[19:35:16] <paulej72> NCommander: the fixes for the slash boxes that you put on your todo yesterday, should be in the code that I have in the current pull request
[19:35:53] <NCommander> if (req.url ~ "\.(png|gif|jpg|swf|css|js)(\?.*|)$") {
[19:35:53] <NCommander> unset req.http.cookie;
[19:35:53] <NCommander> }
[19:36:03] <NCommander> So that's the bit of magic we have
[19:36:13] <NCommander> and when I moved it to always be run vs. req.http.cookie ...
[19:38:01] <mechanicjay> yeah zford did the varnish config, I consulted on it.
[19:39:11] <NCommander> Huh
[19:39:18] <NCommander> its possible to get per-user caching
[19:40:02] <xlefay> hmm
[19:40:26] janrinok|afk is now known as janrinok
[19:40:29] <NCommander> Got it
[19:40:30] <NCommander> if (req.url ~ "\.(png|gif|jpg|swf|css|js)(\?.*|)$") {
[19:40:30] <NCommander> return (lookup);
[19:40:30] <NCommander> }
[19:40:47] -!- drussell has quit [Quit: Leaving]
[19:40:52] <NCommander> nope
[19:40:57] <NCommander> appears to broke varnish :-)
[19:41:14] <xlefay> hmm
[19:41:29] <NCommander> fixed properly
[19:41:41] <NCommander> The only downside to this is we'll have to purge varnish if we update the skin
[19:41:45] <NCommander> */not a big deal*
[19:41:49] <xlefay> That's not really a downside
[19:42:07] <NCommander> Ok
[19:42:09] <NCommander> its happy
[19:42:12] <xlefay> You can set a git action no?
[19:42:15] <NCommander> cache_miss holding at 50
[19:42:20] <NCommander> After several refreshes
[19:42:24] <xlefay> e.g. git pull, it runs a script and sends a purge?
[19:42:25] <NCommander> xlefay, hit index.pl, no cookies
[19:42:32] <NCommander> xlefay, we could do it in make install
[19:42:42] <xlefay> hitting
[19:42:50] <NCommander> cache_miss steady at 50
[19:42:57] <xlefay> no cookies :[
[19:43:02] * xlefay steal SkyNet's cookies
[19:43:05] <xlefay> 10
[19:43:06] <NCommander> yeah, but this is making sure I didn't do something stupid
[19:43:18] <xlefay> 20
[19:43:18] <NCommander> cache ratio 99%
[19:43:19] <NCommander> :-)
[19:43:22] <NCommander> hit rati :-)
[19:43:32] <xlefay> 30
[19:43:50] <xlefay> 40
[19:43:57] <NCommander> xlefay, theres a rate limiter plugin for varnish
[19:44:03] <NCommander> Next thing to setup
[19:44:08] <xlefay> 50
[19:44:18] <NCommander> You bypassed the cache 22 times out of 50k
[19:44:19] <NCommander> Not bad
[19:44:19] <xlefay> Failed requests: 66
[19:44:21] <xlefay> (Connect: 0, Receive: 22, Length: 22, Exceptions: 22)
[19:44:25] <xlefay> With cookie now?
[19:44:27] <NCommander> Yeah
[19:44:32] <NCommander> I expect Apache to die but
[19:44:41] <NCommander> huh
[19:44:44] <NCommander> cache_miss at 77
[19:44:46] <NCommander> Not going up
[19:44:47] <xlefay> 10k
[19:44:53] <xlefay> ooh it will ;-)
[19:45:03] <xlefay> 20
[19:45:07] <NCommander> apache fell over
[19:45:12] <NCommander> Why the fuck did apache fallover
[19:45:12] <xlefay> hah
[19:45:13] <NCommander> 77 hits
[19:45:17] <xlefay> hmm
[19:45:23] <xlefay> check kernel logs
[19:45:30] <xlefay> maybe it's a memory thing that killed apache?
[19:45:39] <NCommander> no, httpd is running
[19:45:54] <xlefay> Failed requests: 37981
[19:45:55] <xlefay> (Connect: 0, Receive: 1515, Length: 34951, Exceptions: 1515)
[19:45:57] <xlefay> Non-2xx responses: 15067
[19:46:07] <xlefay> honestly... are we sure it's caching at all?
[19:46:13] <NCommander> I'm checking
[19:46:14] <xlefay> different content-length suggests otherwise
[19:46:14] <NCommander> stand by
[19:46:24] <NCommander> The server locked up
[19:46:34] <NCommander> syslog is fucking huge
[19:46:39] <NCommander> 13534
[19:47:16] <NCommander> yeah
[19:47:17] <NCommander> we'lre OOMing
[19:47:18] <NCommander> shit
[19:47:28] <xlefay> yep
[19:47:30] <xlefay> thought so
[19:47:46] <paulej72> how much memory?
[19:47:49] <NCommander> 1 GiB
[19:47:51] <xlefay> Well least Khyber will be happy to know he doesn't even need 400TB ;-)
[19:47:59] * NCommander is going to setup rate limiting
[19:48:08] <NCommander> xlefay, well, its still a drastic improvement
[19:48:15] <NCommander> You have to be logged in or fake it to destory the box
[19:48:28] <xlefay> NCommander: I agree, let's see how we can improve further. :)
[19:48:36] <xlefay> Well; faking it isn't difficult at all.
[19:48:37] <NCommander> xlefay, TBH, I'm confortible rolling full dynamic out to the userbase
[19:48:46] <NCommander> xlefay, right, I'm fiddling with varnish rate limiting
[19:48:52] <NCommander> Trying to figure out how you install the plugin
[19:49:09] <xlefay> Preferably, we would be able to have varnish check against memcached if a cookie is valid
[19:49:23] <xlefay> but that would have to be implemented well otherwise it in itself is a point of failure
[19:49:30] <paulej72> are you working on the new dev server?
[19:49:40] <xlefay> paulej72: fortunately, yes
[19:50:09] <NCommander> https://github.com
[19:50:10] <NCommander> hrm
[19:50:12] <NCommander> Not packaged
[19:50:15] <paulej72> with the latest slash code?
[19:50:32] <xlefay> Wouldn't it be better to throttle http connections via iptables per IP?
[19:50:48] <xlefay> iptables work on the kernel level, correct? So it would be faster and more efficient than varnish?
[19:51:12] <xlefay> paulej72: think so
[19:51:22] <NCommander> libvmod-throttle has packaging
[19:51:23] <NCommander> awesome
[19:51:38] <xlefay> Guessing iptables aren't more efficient then?
[19:51:48] <NCommander> xlefay, hrm ...
[19:51:56] <NCommander> But we can't set a return code on that
[19:52:02] <xlefay> but we don't need to
[19:52:04] <NCommander> Varnish works fine for what your hitting as long as we don't hit apache
[19:52:11] <NCommander> We can send a 432 error, too much traffic
[19:52:19] <xlefay> if there's more than 15 connections from one IP per second, I'd say, that's too much
[19:52:35] <NCommander> xlefay, proxied workplaces
[19:52:39] <NCommander> proxied ISPs
[19:52:41] * NCommander has seen that
[19:52:43] <xlefay> hmm
[19:52:44] <janrinok> NCommander: xlefay I just came to tell you that the dev server is down - but you already know!
[19:52:45] <paulej72> xlefay: what a bout isp level nat and proxies
[19:52:54] <xlefay> I hadn't thought of that
[19:53:00] <xlefay> but per second? how likely is that?
[19:53:15] <xlefay> let's say, 500 connection/s
[19:53:35] * NCommander compiles
[19:54:02] -!- FunPika has quit [Quit: Leaving]
[19:54:05] <NCommander> argh
[19:54:12] <NCommander> I might need to cook varnish from source to build this in
[19:54:15] <xlefay> hmm
[19:55:38] <xlefay> Also, does varnish really not have that built in?
[19:55:44] <NCommander> xlefay, nope
[19:55:55] <NCommander> I think it expects you to use apache rate limiting
[19:55:56] <xlefay> I wonder why
[19:55:58] <NCommander> (introduced in apache 2.x)
[19:56:21] <xlefay> hmm
[19:57:29] <NCommander> Oh
[19:57:32] <NCommander> so it needs varnishs source
[19:57:35] <NCommander> But not compiled from source
[19:57:37] <NCommander> thats easy
[19:57:40] <NCommander> apt-get source varnish :-)
[19:58:01] <xlefay> I'm just curious what the benefit is besides being able to "reply" too too many requests
[19:58:34] <NCommander> No idea
[19:58:41] <NCommander> But we'll start 429 erroring if we get too many request
[19:58:45] <NCommander> 5 request per sec?
[19:58:58] <xlefay> If used incorrectly, this tool could let an attacker force Varnish to consume all available memory and crash. It would be too bad to be DoS'ed by a tool that prevents DoS! What you need to know is that this vmod will keep in memory the time of the revelant last requests for each key you provide. And this memory is outside of the memory you specify to Varnish for caching. (So if you specify 4G of RAM to varnish, this vmod memory will be on top of it.
[19:58:59] <xlefay> )
[19:59:28] <xlefay> Mind you, a regular varnish installation has at least that amount of memory last I checked.. we only have 1 GB or so
[19:59:42] <xlefay> and not even all of that is allocated to Varnish
[19:59:55] <NCommander> xlefay, I'm hope to suggestions
[20:00:17] <NCommander> (open
[20:00:31] <paulej72> add memory
[20:00:37] <xlefay> If we really want to do this right, we either need a dedicated Varnish box with enough memory or a load balancing situation
[20:01:13] <NCommander> xlefay, agreed, but this is a decent internim solution
[20:01:19] <NCommander> xlefay, its actually not that bad
[20:01:29] <NCommander> If we just rate limit to 3/s, then the keys expire 3 seconds later
[20:01:31] <NCommander> (keep reading)
[20:01:57] <paulej72> are we sure varnish is set to use a proper amount of the available ram?
[20:02:02] <NCommander> paulej72, no
[20:02:36] <paulej72> that may be the reason for the OOMs
[20:02:55] <xlefay> interesting
[20:02:58] <xlefay> NCommander: cgroups
[20:03:07] <xlefay> to limit it, but that'd be fataaaaaaaaaal
[20:03:16] <xlefay> I like how they explain it hmm
[20:03:33] <NCommander> ok, libvmod-throttle installed
[20:03:35] * NCommander configures it
[20:03:40] <xlefay> It seems like a reasonable suggestion, I'm just wondering what'll happen if a huge botnet were to attack the server
[20:03:52] <xlefay> 1,000 * 30 * 16 = 480 kbytes.
[20:03:56] <xlefay> needs to be extremely huge LOL
[20:04:17] <xlefay> I think we're safe.. unless some asshole really has that much capacity
[20:04:26] <NCommander> xlefay, should we rate limit cache misses, or in general
[20:04:40] * xlefay considers someone using a /48 IPv6..
[20:04:56] * NCommander notes theorically, we could be DOSed out of existence by someone eating all our bandwidth
[20:05:16] <xlefay> I'm trying so hard not to mention a certain name :P
[20:05:29] <xlefay> Rate limit cache misses initially, I'd say
[20:05:37] <paulej72> our list of verboten names is growing
[20:05:52] <NCommander> alright
[20:06:12] <xlefay> Well "theoretically" we could be saving a lot of bandwidth and varnish magic quite easily
[20:06:16] <paulej72> xlefay: agree with cache misses
[20:06:26] <xlefay> but NCommander I'd keep the config for general ready
[20:06:31] <xlefay> e.g. commented out
[20:06:35] <NCommander> error 429 "Too many requests; 400 TiB not accepted"
[20:06:36] -!- FunPika [FunPika!~FunPika@Soylent/Staff/Wiki/FunPika] has joined #staff
[20:06:36] -!- mode/#staff [+v FunPika] by SkyNet
[20:06:37] <NCommander> muhuhuhu
[20:06:44] <xlefay> No no, it was TB!
[20:06:50] <xlefay> He never mentioned TiB!
[20:07:08] <xlefay> Although, I agree, he should have
[20:07:40] <xlefay> You know, we could set an additional rule for exceptionally large requests...
[20:07:54] <NCommander> hrm
[20:07:55] <xlefay> maybe we should redirect that to an off side server which takes them on a redirect ride?
[20:08:05] <xlefay> unfortunately, many browsers nowadays have protection against that :/
[20:08:13] <NCommander> The only problem with setting this too low
[20:08:16] <NCommander> If the cache is empty
[20:08:19] <NCommander> it might start 429ing
[20:08:33] <xlefay> So, have a cronjob auto fetching it every cache interval
[20:08:44] <xlefay> or perhaps varnish can fetch it, itself after it expires?
[20:08:47] <xlefay> I think it should..
[20:08:57] <NCommander> AH
[20:09:00] <NCommander> I know how to fix that
[20:09:08] <paulej72> don't modern browsers send multiple requests when loading a page?
[20:09:11] <NCommander> Only rate limit on non-static assets
[20:09:24] <xlefay> but.. if you misconfigure it, varnish will be DoS'ing apache
[20:09:30] <xlefay> paulej72: depends
[20:09:37] <NCommander> Alright
[20:09:40] <NCommander> We should be in business
[20:09:46] <xlefay> NCommander: you know - we could consider getting maxcdn for static assets
[20:09:58] <xlefay> $8/m - and we can use their cdn platform
[20:10:20] <xlefay> Would save us bandwidth + it'll lower the load on the box(es)
[20:10:20] <NCommander> xlefay, perhaps
[20:10:28] <NCommander> xlefay, I don't want our IPs going anywhere but us though
[20:10:31] <NCommander> xlefay, Alright hit it
[20:10:38] <NCommander> You should start getting a LOT of non--200 errors
[20:11:16] <xlefay> Complete requests: 50000
[20:11:18] <xlefay> Failed requests: 39
[20:11:19] <xlefay> (Connect: 0, Receive: 0, Length: 39, Exceptions: 0)
[20:11:21] <xlefay> Non-2xx responses: 49962
[20:11:23] <NCommander> ...
[20:11:40] * NCommander looks
[20:11:41] * xlefay wonders what's up with the 39 length's
[20:13:04] <NCommander> OH
[20:13:07] <NCommander> I know what happened
[20:13:10] * NCommander reorders the config file
[20:13:11] <xlefay> How much bandwidth do we have anyway?
[20:13:18] <NCommander> 1.4 TiB I think?
[20:13:21] <xlefay> ;o
[20:13:24] <xlefay> 1.4?
[20:13:34] <xlefay> I suppose if we don't get attacked, that oughta suffice.
[20:13:37] <NCommander> xlefay, its a pool between all nodes
[20:13:45] <NCommander> xlefay, try it again
[20:13:55] <xlefay> pooled?
[20:14:01] <xlefay> that's tricky
[20:14:14] <NCommander> The rate limiting is a bit too low
[20:14:16] <NCommander> Oops
[20:15:37] <NCommander> xlefay, try it now
[20:16:19] <NCommander> looks like it did EXACTLY what it was supposed to
[20:16:27] <xlefay> Non-2xx responses: 49961
[20:16:31] <NCommander> ahaha
[20:16:32] <NCommander> Yup
[20:16:33] <xlefay> same as previously, except -1 request
[20:16:43] <xlefay> 20:11:16] <xlefay> Non-2xx responses: 49962
[20:16:45] <NCommander> What was your total time of execution
[20:16:49] <xlefay> 20:16:26] <xlefay> Non-2xx responses: 49961
[20:16:57] <xlefay> Time taken for tests: 13.880 seconds
[20:16:59] <NCommander> so
[20:17:06] <NCommander> 13*3
[20:17:11] <NCommander> 39
[20:17:33] <NCommander> 49963+39
[20:17:33] <NCommander> 50002
[20:17:35] <NCommander> Well
[20:17:37] <NCommander> It works
[20:17:53] <xlefay> 49963 ?
[20:17:55] <NCommander> pity you can't see if your getting 429
[20:18:01] <xlefay> 49961 you mean?
[20:18:03] <xlefay> oh I can ;-)
[20:18:08] <NCommander> xlefay, you're only allowed 3 request per second EXCLUDING static assets
[20:18:13] <xlefay> -v5 lot's of verbose
[20:18:16] <NCommander> (i.e, you can only hit .pl pages 3 times a second)
[20:18:24] <xlefay> <h1>Error 429 Too many requests; 400 TB not accepted</h1>
[20:18:26] <xlefay> <p>Too many requests; 400 TB not accepted</p>
[20:18:27] <xlefay> <h3>Guru Meditation:</h3>
[20:18:29] <xlefay> <p>XID: 1176268601</p>
[20:18:36] <NCommander> That way, if the cache is empty, it won't kill a user
[20:18:42] <xlefay> good ;)
[20:18:53] <NCommander> It means you could theorically DDOS us by hitting one of the js files repeatively
[20:19:04] <NCommander> But even then, its just eating bandwidth, NOT the backend
[20:19:16] <xlefay> True
[20:19:25] <NCommander> We can iptables those out if it happens
[20:19:33] <NCommander> This is just to prevent Apache from absolutely shitting itself
[20:19:40] <xlefay> Content-Length: 476 - I wonder if varnish were to reply with plaintext only ;-)
[20:19:40] * NCommander wonders if 3 reqs per second too low)
[20:19:56] <paulej72> I think it is
[20:20:01] <xlefay> Think it is though, an automated program will easily hit more, and a normal user *MIGHT* hit 3 reqs p/s
[20:20:06] <xlefay> especially if it also includes assets
[20:20:16] <NCommander> xlefay, assets bypass rate limiting
[20:20:20] <xlefay> oh
[20:20:28] <NCommander> They aren't counted against the user
[20:20:31] <xlefay> I forgot that for a sec
[20:20:57] <xlefay> I'd set it to 5 or 6 just in case
[20:21:02] <NCommander> xlefay, here's the config as it sits now
[20:22:37] <stdhell> See you!!!
[20:22:44] -!- stdhell has quit [Quit: Leaving]
[20:22:56] <NCommander> http://paste.ubuntu.com
[20:23:18] <xlefay> btw http://soylentnews.org
[20:24:05] <NCommander> ?
[20:24:13] <NCommander> xlefay, ugh
[20:24:18] * NCommander goes to take a shower
[20:24:36] <xlefay> ok ;-)
[20:24:57] <xlefay> you took out the "fry" and "bender" stuff?
[20:25:38] <mattie_p> managed to get a clean compile with some ugly hacks
[20:27:01] <mattie_p> ugg, not with -Wpedantic tho
[20:28:25] <mattie_p> back to work
[20:30:52] LaminatorX|afk is now known as LaminatorX
[20:30:58] <mattie_p> some of these warnings are simple, some ... not so much
[20:34:10] <NCommander> xlefay, Fry and Bender only shows up if the request goes to the backemd
[20:34:20] <xlefay> aah
[20:34:21] <xlefay> k
[20:34:37] <NCommander> xlefay, I dunno, I'm feeling confortible pushing this revised config on production
[20:34:45] <mattie_p> now I need to see if I can hook the MUD up to IRC
[20:34:47] <xlefay> hmm
[20:34:52] * NCommander notes with the apparmooring, I'm also much happy with us being on Apache 1.3
[20:35:38] <xlefay> I like the current config, and I'll second pushing it out -however- let's keep the old config handy just in case and be sure all sysops are aware of what to do in case the varnish config does mess up
[20:35:55] <paulej72> Is dev suspposed to be up? I get a 503 error?
[20:36:01] <NCommander> paulej72, yes?
[20:36:06] <xlefay> same here
[20:36:09] <NCommander> shit
[20:36:22] <NCommander> WTF
[20:36:28] <NCommander> http is eating 5.3% sysram
[20:36:30] <xlefay> I'm retracting that statement.
[20:36:45] <paulej72> !grab xlefay
[20:36:45] <Bender> Added quote 77
[20:36:45] <xlefay> regarding pushing it out :P let's first see what's going on with dev
[20:37:05] <NCommander> looking
[20:37:06] <xlefay> if it's entirely unrelated to varnish, I'll re-second it x'D
[20:37:07] <NCommander> Apache having issues
[20:37:10] <NCommander> I think its unrelated
[20:37:12] <mechanicjay> yeah, making sure the config is stable on dev for at least a couple days before pushing live would be good.
[20:37:25] <NCommander> mechanicjay, well I'm going to migrate production to an apparmor capable kernel
[20:37:25] <xlefay> I hope so too.
[20:37:29] <NCommander> (which is what we need the downtime for)
[20:37:39] <xlefay> It's really sad you can't just clone a VM :/
[20:37:48] <NCommander> xlefay, you can clone to a new linode
[20:37:51] <xlefay> oh
[20:38:02] <xlefay> then just re-clone it, set up the app armor, and rebind it's IP
[20:38:09] <xlefay> that's possible, right?
[20:38:16] <xlefay> Then, no downtime, whatsoever.
[20:38:18] <mattie_p> !quote shit
[20:38:18] <Bender> Quote 8 - <NCommander> shit
[20:38:22] <Bender> Also in quotes: 58, 66, 73
[20:38:24] <NCommander> xlefay, not quite that simple
[20:38:30] <xlefay> because?
[20:38:34] <xlefay> existing sessions, etc?
[20:38:37] <NCommander> xlefay, you have to change linode's configuration to use grub so you can boot a non-linode kernel
[20:38:53] <xlefay> Ooh you're talking about the kernel thing now
[20:38:58] <NCommander> yeah
[20:39:05] <NCommander> apache restarted
[20:39:06] <NCommander> hrm
[20:39:11] <paulej72> NCommander: I merged my pull that was sitting in the queue for the last week or so. Can you update dev with this.
[20:39:23] <NCommander> paulej72, k
[20:39:27] <xlefay> I thought you meant the process of cloning -> {doing what you need to} -> re-bind the IP
[20:39:32] <NCommander> xlefay, honestly, I think we're just starting too many httpd instances
[20:39:33] <NCommander> Nothing more
[20:39:40] <xlefay> NCommander: ^ that would work though, right? Which would also give you more time to set it all up
[20:39:52] <NCommander> xlefay, I made a lot of changes on dev to work with a cloned DB
[20:40:19] <xlefay> but in this case, you wouldn't need to work with a cloned DB. I'm merely talking as an upgrade situation
[20:40:39] <xlefay> e.g. "upgrading the kernel" - in this case, it'd be a fuss free and transparent upgrade
[20:41:08] <NCommander> xlefay, I don't want to clone dev -> prod
[20:41:12] <paulej72> When I just logged into dev I got an internal server error. Upon reloading the main page I was logged in.
[20:41:18] <NCommander> paulej72, I just restarted apache
[20:41:20] <xlefay> seems we're having a misunderstanding
[20:41:23] <NCommander> There's some weird shit in the syslog
[20:41:29] <NCommander> Mar 21 19:01:03 soylent-dev /tmp/vtc.18103.59f54d67/v1[31159]: Stopping Child
[20:41:29] <NCommander> Mar 21 19:01:03 soylent-dev /tmp/vtc.18103.59f54d67/v1[31159]: Child (31182) said Child dies
[20:41:29] <NCommander> Mar 21 19:01:03 soylent-dev /tmp/vtc.18103.59f54d67/v1[31159]: Child (31182) died status=1
[20:41:31] <NCommander> Mar 21 19:01:03 soylent-dev /tmp/vtc.18103.59f54d67/v1[31159]: Child cleanup complete
[20:41:33] <NCommander> WTF
[20:42:18] <xlefay> To clarify, this is what I meant, clone prod -> 'prod2' ... fix things on 'prod2' then re-bind prod's IP to prod2, and dump/archive 'prod'; if that works without downtime, that is
[20:42:26] <xlefay> Child said Child dies.... well, that's very useful
[20:42:42] <NCommander> xlefay, that works fine
[20:42:56] <NCommander> xlefay, for future downtime, but I need to get the clone-from thing setup
[20:43:06] * NCommander notes you CAN swap IPs with Linode ...
[20:43:36] <NCommander> hrmmmmmmmmmmm
[20:43:40] <xlefay> ^ that's what I meant with "re-bind IP"
[20:44:02] <NCommander> xlefay, yeah, I could do that
[20:44:08] <paulej72> xlefay: I like that method of updating if it doesn't mess with the db
[20:44:15] <NCommander> We need to mess w/ the DB though
[20:44:17] <xlefay> paulej72: the db isn't on prod
[20:44:18] <xlefay> right?
[20:44:21] <NCommander> That one also needs an app armor profile
[20:44:23] <NCommander> xlefay, no, its not
[20:44:29] <xlefay> so it wouldn't paulej72 ;-)
[20:44:39] <NCommander> Yeah
[20:44:44] <NCommander> Apacheis memory leaking
[20:44:45] <NCommander> WTF
[20:45:08] * NCommander has 3 httpd processes%MEM 5.3
[20:45:14] <NCommander> It wasn't doing this earlier, I must have broken the config
[20:45:16] <xlefay> holy shit
[20:45:37] <NCommander> Probably a side effect of gettign rid of dynamic page generation
[20:45:41] <NCommander> er
[20:45:42] <NCommander> static
[20:46:12] <xlefay> possibly... then it's slashd or however the linkage is called
[20:46:23] <NCommander> I'm looking through http config
[20:47:00] <NCommander> 8499 slash 20 0 241m 61m 2772 S 0 6.2 0:00.91 httpd
[20:47:00] <NCommander> 8501 slash 20 0 240m 60m 2812 S 0 6.1 0:00.80 httpd
[20:47:00] <NCommander> 8498 slash 20 0 239m 59m 2728 S 0 6.1 0:00.85 httpd
[20:47:00] <NCommander> 8512 slash 20 0 239m 59m 2768 S 0 6.0 0:00.76 httpd
[20:47:00] <NCommander> 8507 slash 20 0 239m 59m 2772 S 0 6.0 0:00.67 httpd
[20:47:01] <NCommander> 8500 slash 20 0 234m 54m 2752 S 0 5.5 0:00.46 httpd
[20:47:03] <NCommander> 8497 slash 20 0 233m 53m 2732 S 0 5.4 0:00.43 httpd
[20:47:05] <NCommander> 8508 slash 20 0 233m 52m 2644 S 0 5.3 0:00.30 httpd
[20:47:07] <NCommander> 8511 slash 20 0 232m 52m 2536 S 0 5.3 0:00.28 httpd
[20:47:09] <NCommander> 8504 slash 20 0 232m 52m 2500 S 0 5.3 0:00.31 httpd
[20:47:11] <NCommander> 4510 slash 20 0 232m 52m 2468 S 1 5.3 0:33.47 httpd
[20:47:13] <NCommander> 9622 slash 20 0 232m 52m 2380 S 0 5.3 0:00.01 httpd
[20:47:17] <NCommander> 9625 slash 20 0 232m 52m 2380 S 0 5.3 0:00.01 httpd
[20:47:19] <NCommander> 9626 slash 20 0 232m 52m 2380 S 0 5.3 0:00.01 httpd
[20:47:21] <NCommander> 9631 slash 20 0 232m 52m 2380 S 0 5.3 0:00.00 httpd
[20:47:23] <NCommander> 9634 slash 20 0 232m 52m 2380 S 0 5.3 0:00.01 httpd
[20:47:25] <NCommander> yeash
[20:47:27] <NCommander> 52 instances running
[20:47:29] <xlefay> woa
[20:47:31] <NCommander> right
[20:47:35] <NCommander> and now we're guru mediation
[20:47:38] <NCommander> WTF
[20:47:48] <NCommander> slash is deadlocking for some reason
[20:47:50] * NCommander debugs
[20:48:35] <paulej72> I have the homage, but no css loading
[20:48:40] <NCommander> paulej72, varnished
[20:49:00] <paulej72> OK makes sense
[20:49:07] <NCommander> Apache restarted with logging re-enabled
[20:49:50] <NCommander> oH
[20:49:51] <NCommander> fuck
[20:49:53] <NCommander> I know whats going on
[20:50:11] <NCommander> varnish polls for about.shtml every second
[20:50:19] <paulej72> I got the page when apache restarted, but was logged out. I got the internal server error again when logging in.
[20:50:36] <NCommander> paulej72, I logged in just fine
[20:50:48] <NCommander> paulej72, I'm waiting the error log
[20:50:52] <NCommander> if it 500s again, I'll know why
[20:51:03] <paulej72> why is varnish polling about.shtml?
[20:51:21] <NCommander> Gah
[20:51:25] <NCommander> paulej72, to see if Apache is up
[20:51:28] <NCommander> The old cookie bug is back
[20:51:31] * NCommander fixes THAT
[20:51:49] <paulej72> yes it is an I am currently logged out.
[20:52:10] <paulej72> clicked logged in and got he internal server error.
[20:52:27] <paulej72> reload home page and logged in.
[20:52:35] <paulej72> and I was logged in
[20:52:55] <NCommander> working on it
[20:53:01] * NCommander remembers what caused this bug
[20:53:03] <NCommander> stand by
[20:58:14] <NCommander> that's really irritating
[21:00:14] <NCommander> I can't figure out if this is varnish or slash misbehaving
[21:04:50] <paulej72> bugs from a month ago, already forgotten to the eather
[21:07:33] <paulej72> Got to go. Spending the evening with my nephew. BBL
[21:09:53] <NCommander> ah
[21:09:54] <NCommander> I get it
[21:11:04] -!- weeds [weeds!~4118a13c@cwz-29-45-637-17.columbus.res.rr.com] has parted #staff
[21:12:55] <NCommander> xlefay, so ... our old varnish config only worked by accept :-)
[21:12:58] <NCommander> *accident
[21:13:05] <xlefay> why is that?
[21:13:13] <NCommander> It didn't properly cache some pages
[21:13:23] <NCommander> The current login bug was due to login being cached
[21:13:29] <xlefay> for SSL?
[21:13:32] <NCommander> And not sending along the form back to Apache
[21:13:38] * NCommander is fiddling a bit more with it
[21:13:45] <NCommander> Making sure POST never gets varnished
[21:13:59] <xlefay> "login bug" only makes me think of SSL, but I'm fairly sure that forms gets back to apache
[21:16:17] <NCommander> xlefay, it wasn't
[21:16:19] <NCommander> Not consistently
[21:16:22] <xlefay> oh
[21:16:32] * NCommander just added a SAN check to varnish to pipe through POSTs
[21:16:38] <xlefay> Oh ok
[21:16:48] <xlefay> So you're thinking SSL auth will work after this fix?
[21:17:27] <NCommander> xlefay, probably
[21:17:36] <xlefay> Great :D
[21:17:36] <NCommander> xlefay, do you want to wham on it
[21:17:44] <xlefay> rofl
[21:17:45] <xlefay> sure
[21:17:46] <NCommander> But yeah, POST requests work properly now all the time
[21:17:47] <xlefay> which url?
[21:17:57] <NCommander> xlefay, dev.soylentnews.org
[21:18:02] <xlefay> just index.pl?
[21:18:08] <xlefay> cookies, yes/no?
[21:18:27] * xlefay just ate the last of the pancakes :/
[21:18:37] <NCommander> xlefay, well, ab isn't going to help, I meant using it and testing it as a user :-P
[21:18:42] <NCommander> xlefay, http://paste.ubuntu.com heres our rewritten varnish config
[21:18:45] <NCommander> Less stupid
[21:18:51] <NCommander> er
[21:18:52] <NCommander> actually
[21:18:53] <NCommander> Wait
[21:19:13] * NCommander adds AC gracing
[21:19:14] <NCommander> <NCommander> xlefay, dev.soylentnews.org
[21:19:16] <NCommander> er
[21:19:28] <NCommander> set req.grace = 5m;
[21:19:28] <NCommander> return(lookup);
[21:20:09] <NCommander> It looks happy
[21:21:23] * xlefay notes he isn't happy with these round corners
[21:21:35] <xlefay> those buttons fugly the layout :'(
[21:21:46] * NCommander is kinda eh on them
[21:22:15] * NCommander thinks he prefered the sharp corners
[21:22:19] <NCommander> But I'm not that picky
[21:23:47] <xlefay> They could work.. just not the way they are atm... but meh
[21:25:35] <xlefay> Also, why is there a link to irc-alternate, or am I missing the link to forum-alternate and wiki-alternate?
[21:26:53] <NCommander> xlefay, no idea. I'm not updating production slashcode with this downtime
[21:27:17] <xlefay> understandable
[21:27:24] <NCommander> xlefay, for this downtime, I'm going to switch us over to stock Ubuntu kernels
[21:27:29] <NCommander> and apparmor apache 1.3
[21:27:42] * NCommander notes all our pounding been on Apache and there's been no apparmor errors so I think the file is right
[21:27:56] <xlefay> but in the future just clone the machine and update it there, switching & swapping IPs is easy ;-)
[21:28:03] <xlefay> Oh I don't doubt the apparmor at all
[21:28:07] <NCommander> xlefay, right, this is our one time downtime :-)
[21:28:11] <NCommander> Because we need to switch to pv-grub
[21:28:27] <NCommander> (intentional downtime anyway I hope, unless we do another server migration)
[21:28:39] <NCommander> xlefay, we can look at setup up the load balancer tomorrow, and terminating SSL on it
[21:28:41] <xlefay> pv-grub? I've heard of grub, grub2, grub-legacy and all the other grub names, but pv-grub not yet
[21:28:48] <NCommander> xlefay, grub for xen
[21:28:50] <xlefay> oh
[21:28:55] <NCommander> paravirtualization-grub
[21:29:29] <xlefay> ah ok, so today apparmor, let's hope nothing breaks (you'll keep the current kernel, handy, correct?)
[21:29:51] <NCommander> xlefay, yeah, though obvious reboot is required
[21:29:57] <xlefay> e.g. in case something does break, it would be trivial for someone to reverse it
[21:29:58] <NCommander> xlefay, but we're going from linode custom -> stock ubuntu
[21:30:09] <NCommander> We shold expect LESS breakage with this setup
[21:30:12] <xlefay> Understood, I'm just thinking the worst cases ;-)
[21:30:15] <NCommander> Agreed
[21:30:31] <xlefay> Also, can you grant me access to that box or linode itself so I can set up my key on there?
[21:32:28] <NCommander> xlefay, I'm going to go smoke
[21:32:31] <NCommander> So when I come back
[21:32:38] <xlefay> Alright, thank you. :)
[21:33:00] <NCommander> xlefay, BTW, to get on production, you need something like this in your .ssh/config
[21:33:14] <NCommander> xlefay,
[21:33:21] <NCommander> Host soylent-www
[21:33:21] <NCommander> ProxyCommand ssh mcasadevall@dev.soylentnews.org nc -q0 192.168.129.146 22
[21:33:55] <xlefay> my own local .ssh/config or on the svc box?
[21:34:03] <NCommander> Your
[21:34:05] <NCommander> *yours
[21:34:10] <NCommander> Because you have to bounce off another node
[21:34:15] <xlefay> Where's the reference to the svc box in there?
[21:34:15] <NCommander> (replace mcasadevall@ with your own username
[21:34:30] <xlefay> e.g. you said we needed to bounce via the svc box, correct?
[21:34:39] <NCommander> well, its ANY linode in the Dallas datacentre
[21:34:43] <NCommander> I haven't seutp the svc box yet
[21:34:47] * NCommander doesn't want to touch it
[21:34:53] <NCommander> I'm going to setup a "shell" server tomorrow
[21:34:54] <xlefay> so, my local pc -> svc -> soylent-www ?
[21:34:55] <NCommander> With the LDAP
[21:35:02] <NCommander> xlefay, or pc -> dev - soylent-www
[21:35:09] <NCommander> then you can ssh soylent-www
[21:35:11] <xlefay> aah
[21:35:12] <NCommander> or scp soylent-www
[21:35:21] <NCommander> Also know as the right way to SSH from SSH
[21:35:31] <xlefay> so it doesn't matter, as long as I go through a server with ssh enabled on public interface
[21:35:32] <NCommander> (aka, when you ahve to SSH into a box, to SSH to a second box)
[21:35:35] <NCommander> xlefay, bingo
[21:35:40] <xlefay> (currently, I only have access to SVC so, that's why I was confused)
[21:35:46] <NCommander> Its security through obsecurity, but it works
[21:35:55] <NCommander> At least you have to know what our internal linode IPs are
[21:36:03] <NCommander> Which isn't documented anywhere but the linode manager
[21:36:15] <xlefay> Perhaps, it's useful to have an internal DNS for our boxes?
[21:36:26] <xlefay> so if an IP changes, we only have to change it in our zone file?
[21:36:50] <NCommander> xlefay, hosts-in-LDAP
[21:36:51] <xlefay> eg, 'www1.soylent' is easier to remember than '192........'
[21:36:53] <xlefay> aaaah
[21:36:55] <NCommander> Two steps ahead of you
[21:37:01] <xlefay> that's awesome, didn't know LDAP had that.
[21:37:07] <NCommander> LDAP can do everything NIS can
[21:37:08] * xlefay hasn't worked with LDAP a lot.
[21:37:10] <NCommander> Its just harder
[21:37:25] <NCommander> If I can't get it to work, I'll through an internal bind up somewhere
[21:37:48] <xlefay> yea
[21:38:19] <xlefay> imma smoke too
[21:54:19] <NCommander> 5 minutes to magic time
[21:55:20] <NCommander> xlefay, you know what? Fuck it, I'm going to update the varnish config on production
[21:55:28] <NCommander> If we break shit, we'll revert
[21:57:26] <xlefay> That'd be baad
[21:57:34] <xlefay> but ok, let's hope for the best then ;)
[21:57:52] * NCommander has to document this voodoo on the wiki
[21:57:53] <NCommander> bleck
[21:57:55] <NCommander> I hate wikis
[21:57:59] <xlefay> 45 karma!
[21:58:01] <xlefay> hell yes.
[21:58:28] <xlefay> rofl
[21:58:37] <NCommander> 2 minutes ...
[21:58:53] <xlefay> Just be sure to let people have those 2 minutes... I need moar karma
[21:58:59] <NCommander> lol
[21:59:04] <xlefay> *refreshes endlessly*
[21:59:33] <xlefay> NCommander: Good luck and don't muck it up!
[22:01:44] <NCommander> xlefay, I look forward to when we upgrade to 14.04 :-)
[22:01:50] <NCommander> xlefay, BTW, get to play w/ juju?
[22:02:18] <xlefay> Haven't yet, been laying in bed a lot, sick :'(
[22:02:41] <mechanicjay> Bad varnish config?
[22:05:14] <NCommander> mechanicjay, I rewrote most of the varnish config
[22:05:24] <NCommander> But right now, I'm doing updates and migrating to an apparmor capable kernel
[22:05:26] <NCommander> 77GB Used, 15018GB Remaining, 15095GB Quota
[22:05:29] * NCommander stands corrected
[22:05:33] <NCommander> we have 15 TiB of bandwidth
[22:05:51] <xlefay> I was already wondering how the hell you only had 1.5 if Linode gives 2TB per 1024 node
[22:05:58] <NCommander> oooh
[22:06:02] <NCommander> official Ubuntu kernel
[22:06:03] <NCommander> Linux soylent-www 3.2.0-60-virtual #91-Ubuntu SMP Wed Feb 19 04:13:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[22:06:04] <NCommander> SHINY
[22:06:08] <NCommander> one down, one to go
[22:06:28] * xlefay notes he had a global notice and all standing by to send.............
[22:06:43] <xlefay> then NC swoops in and simply changes the topic in #soylent, such a notice kill
[22:06:58] <NCommander> :-)
[22:07:05] <xlefay> http://soylentnews.org
[22:07:10] <xlefay> OH YES, GURU MEDITATION!
[22:07:19] <NCommander> ah fuck
[22:07:22] <NCommander> varnish came back on the reboot
[22:07:46] * NCommander fixes that
[22:09:38] <NCommander> added bonus
[22:09:43] <NCommander> mysql gets its apparmor profile applied!
[22:09:49] <NCommander> (ubuntu ships one for it out of the box)
[22:10:23] <NCommander> uh oh
[22:10:31] <NCommander> ...
[22:10:35] <NCommander> db lost its IP address
[22:10:55] <NCommander> correction
[22:10:57] <NCommander> db FAILED to boot
[22:10:58] <NCommander> crap
[22:11:00] * NCommander debugs
[22:11:43] <NCommander> Ok
[22:11:44] <NCommander> fixed
[22:11:46] <NCommander> that was easy enough
[22:11:58] <xlefay> Well if it all fails in the end, we still got an wiki to fall back on
[22:12:01] <xlefay> dammit there goes my master plan
[22:12:26] <xlefay> what was the issue?
[22:13:20] <NCommander> xlefay, typo in grub.lst
[22:13:27] <xlefay> aah
[22:14:51] <NCommander> root@soylent-db:~# uname -a
[22:14:51] <NCommander> Linux soylent-db 3.2.0-60-virtual #91-Ubuntu SMP Wed Feb 19 04:13:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[22:14:52] <NCommander> Sweet
[22:14:54] <NCommander> 2/2
[22:15:33] <xlefay> nice ;-)
[22:20:53] <NCommander> apache/bin/apachectl restart: httpd started
[22:20:54] <NCommander> boom
[22:20:55] <NCommander> almost done
[22:24:47] * MrBluze bought salientnews.net / salientnews.org
[22:24:55] <MrBluze> .com is owned
[22:27:32] <Landon> but
[22:27:36] <Landon> what is the shorturl for salient :(
[22:27:52] <xlefay> saws
[22:28:02] <xlefay> sals
[22:28:05] <xlefay> alient
[22:28:09] -!- LaminatorX has quit [Quit: Web client closed]
[22:28:22] <xlefay> tnews
[22:28:27] <xlefay> snews
[22:29:05] <xlefay> s-alien
[22:34:47] <MrBluze> slnt.net ? i can buy that
[22:35:43] -!- mrcoolbp [mrcoolbp!~mrcoolbp@Soylent/Staff/mrcoolbp] has joined #staff
[22:35:43] -!- mode/#staff [+v mrcoolbp] by SkyNet
[22:37:40] <mrcoolbp> what's going on xlefay?
[22:37:47] <xlefay> mrcoolbp: we broke it
[22:37:55] <xlefay> we absolutely messed it up beyond repair
[22:37:57] <mrcoolbp> nice work
[22:38:02] <xlefay> Nah, I'm kidding
[22:38:29] <xlefay> NCommander: scheduled a maintenance window earlier today - he's doing the maintenance now
[22:38:38] <mrcoolbp> is he installing the armor?
[22:38:49] * mrcoolbp just got home from work
[22:38:51] <xlefay> Among other things yes.
[22:39:02] * xlefay notes this means we can't use our pitchforks on the servers :-/
[22:39:16] <mrcoolbp> aw shucks
[22:39:22] <xlefay> !grab mrcoolbp
[22:39:22] <Bender> Added quote 78
[22:39:53] <MrBluze> mrcoolbp:
[22:39:58] <mrcoolbp> hey
[22:39:58] <MrBluze> found ya
[22:40:01] <mrcoolbp> right?
[22:40:06] <MrBluze> a bit late i suspect
[22:40:17] <mrcoolbp> we've been missing eachother, late for what?
[22:40:18] * xlefay notes the website works..
[22:40:26] <MrBluze> i dunno
[22:40:30] <mrcoolbp> yup, it's up
[22:40:31] <xlefay> not sure if this was intended already? :)
[22:40:32] <MrBluze> u were after me several days ago
[22:40:48] <xlefay> mrcoolbp: stalker!
[22:41:36] <xlefay> NCommander: doesn't seem like the SSL was fixed; the silly thing is, the cookie actually gets set (last I tried it anyway) - but it gets invalidated when you get forwarded to non-SSL or something like that
[22:41:56] <xlefay> Anyway, I'm guessing App Armor, new varnish config, etc.. all is up & working? ;-)
[22:42:22] <mrcoolbp> MrBluze: yeah I mostly just wanted to let you know that the shortlist was vetoed
[22:42:32] <MrBluze> yeah i know
[22:42:39] <mrcoolbp> well you do now
[22:42:46] <MrBluze> thats fair enough if people want to delay, deliberate and whatever other d words
[22:42:46] <mrcoolbp> not then though
[22:42:55] <mrcoolbp> hehe
[22:43:00] <MrBluze> just a tactic
[22:43:24] <MrBluze> but im not fussed - do things properly or not at all is a good approach
[22:43:29] <mrcoolbp> MrBluze: do you think if we make the name collections private (via email) we could not make people register a bunch of domains?
[22:43:37] <MrBluze> maybe
[22:43:41] <MrBluze> they didnt register them before
[22:43:54] <MrBluze> itcosts $10
[22:43:55] <MrBluze> thats too much
[22:44:10] <MrBluze> ive registered 12 domain names for this already
[22:44:15] <mrcoolbp> eeek
[22:44:27] <xlefay> mrcoolbp: to keep the list manageable and regular, we opted initially, that people had to register them before submitting
[22:44:30] <MrBluze> oh they are good domain names i have uses for them
[22:44:36] <xlefay> that would also indicate people are serious about the domain names.
[22:44:49] <xlefay> Honestly, 500+ domain name suggestions isn't going to do us any good either ;-)
[22:44:54] <MrBluze> also, people didnt even bother checking availability for domain names
[22:44:59] <MrBluze> it was totally lame
[22:45:04] <MrBluze> especiallly on the wiki
[22:45:06] <mrcoolbp> hmm...
[22:45:20] <NCommander> crap
[22:45:25] <NCommander> We're 500ing when you try to edit stories
[22:45:26] <NCommander> debugging
[22:45:39] <xlefay> Least that's on the private side of things..
[22:45:47] <NCommander> [Fri Mar 21 21:45:42 2014] [error] :Slash::Utility::Environment:/srv/soylentnews.org/local/lib/perl5/site_perl/5.10.1/x86_64-linux/Slash/Utility/Environment.pm:683:cannot getSkin for empty skid='' ;; Which was called by:Slash::Apache:/srv/soylentnews.org/local/lib/perl5/site_perl/5.10.1/x86_64-linux/Slash/Apache.pm:359
[22:45:51] <NCommander> Shit, that bug is back?
[22:45:52] <NCommander> ugh
[22:46:11] <NCommander> Oh
[22:46:17] <NCommander> apparmor profile too restrictive
[22:46:19] * NCommander lets it play in tmp
[22:46:53] <mrcoolbp> Xlefay: MrBluze: Audioguy: I think to come up with some good names we should make a story submission requesting domain names
[22:47:15] <mrcoolbp> question is should we still require them be registered?
[22:47:30] <NCommander> Looks like its for the stupid spellchecker
[22:47:35] * NCommander debates killing the spellchecker
[22:48:00] <mrcoolbp> the reason I'd like them *not* registered is that we could do it ourselves
[22:48:05] <MrBluze> i agree
[22:48:19] <MrBluze> we require them to be registered and freely available to us
[22:48:27] <MrBluze> if we are to vote on them
[22:48:29] <mrcoolbp> I don't want to have 10 different names we all like and then have trouble getting people to donate
[22:48:43] <mrcoolbp> maybe I'm overthinking it
[22:48:50] <MrBluze> maybe
[22:49:05] <xlefay> NCommander: any decent modern browser nowadays has a spell checker built in.
[22:49:08] <xlefay> Kill it, with fire!
[22:49:12] <MrBluze> but i think people are overthinking if they consider soylentnews to be somehow worth anything just because it cost $2k to buy it of an extortionist
[22:49:23] <xlefay> MrBluze: I think so yes...
[22:49:28] <NCommander> xlefay, I just added the apparmor stuff for now
[22:49:36] <MrBluze> its JUST a name
[22:49:41] <NCommander> xlefay, though I'd love to see an exploit which took advantage of a vunerability in ispell :-)
[22:50:01] <MrBluze> and B did say one thing right - our target community is in the millions, not the thousands
[22:50:04] <xlefay> NCommander: hmm... well we can always analyze it and see what we can come up with ;-)
[22:50:16] <MrBluze> we are nowhere on the way to that, so a name change now is harmless
[22:50:36] <xlefay> I disagree, it's far from harmless.
[22:50:51] <mrcoolbp> MrBluze: the vote is going to happen, I'm discussing the plan not whether or not to do it (you don't have to convince me)
[22:50:54] <xlefay> SoylentNews is established, surely it's easier to change now - but it may be harmful.
[22:51:16] <MrBluze> xlefay: if we keep the soylentnews domain, we can keep redirecting until the traffic on it dies
[22:51:28] <mrcoolbp> ^^^^^^
[22:51:53] <MrBluze> mrcoolbp: yes, of course - best way though is not to have people say "lets call ourselves news.com"
[22:52:02] <MrBluze> which is obviously taken
[22:52:08] <xlefay> MrBluze: correct, _but_ we still have the entire SN, Soylent, Soylentis, etc...
[22:52:27] <xlefay> we are established as it is and not everyone would like the move, that's what I mean with harm
[22:52:32] <MrBluze> xlefay: we do, unfortunately
[22:52:53] <mrcoolbp> I'm thinking It just seems wasteful to ask users to spend hundreds of dollars collectively when we only *really* need to register before the final vote
[22:53:20] <MrBluze> ok
[22:53:29] <NCommander> Hitrate ratio: 10 69 69
[22:53:29] <NCommander> Hitrate avg: 0.8367 0.8567 0.8567
[22:53:32] <NCommander> That's extremely sexy
[22:53:34] <NCommander> ^- mechanicjay
[22:53:39] <MrBluze> but i have noticed as soon as someone comes up with a good name, it gets a domain squatter within hours
[22:53:46] <mrcoolbp> xlefay: soylentnews will be one of the options, doesn't that cover your reservations?
[22:53:55] <xlefay> mrcoolbp: you misunderstand
[22:54:08] <xlefay> I don't mind the move, in fact, I would prefer we had tossed the name already
[22:54:08] <MrBluze> xlefay is right, the brand is now getting established
[22:54:13] <MrBluze> albeit amongst a small group only
[22:54:38] <mrcoolbp> I agree sure, but we (B) promised from the beginning there'd be a vote
[22:54:42] <xlefay> Anyway, NCommander congrats on a job well done ;-)
[22:54:46] <NCommander> We're giong to have to vote
[22:54:50] * NCommander has his thoughts to weigh on this
[22:54:54] <mrcoolbp> ^^^^
[22:55:05] * NCommander is right now dealing with getting the site secured and locked down
[22:55:15] <NCommander> I don't want to be worried about us getting DDoS'ed or rooted
[22:55:19] <NCommander> That's this weekend
[22:55:35] <NCommander> Then I can fully focus on the political aspects
[22:55:49] <MrBluze> fair enough NCommander
[22:55:52] <mrcoolbp> NCommander: we'll handle the name/vote thing, we'll propose a plan when it's solid
[22:55:56] * NCommander swears
[22:56:01] <NCommander> Fucking login bug came back
[22:56:04] <NCommander> I fixed this on dev
[22:56:04] <NCommander> argh
[22:56:11] <xlefay> yep
[22:56:47] <NCommander> No, looks like I just logged out
[22:56:49] <NCommander> Weird
[22:56:57] <MrBluze> just want to say, i dont really have my heart set on it because i do think the window of opportunity for change is waning - but once we know what the name is, it's easier to actually skin the site for it
[22:57:00] <xlefay> I've got a feeling slash is actually a raw version of SkyNet in disguise, slowly be surely introducing random "features"
[22:57:20] <NCommander> MrBluze, I know you're right
[22:57:25] <NCommander> MrBluze, this is partially my fault
[22:57:29] <NCommander> !todo
[22:57:29] <Bender> todo for ncommander: 1) make sure install-slashsite installs proper schema 2) quit smoking 3) look at rewiring pollbooth for SERIOUS votes 4) write up YAFAP for nethack 5) find volunteer who may be willing to work on mod_perl rework effort 6) clean production database of unused vars/tables from Tags/FIrehose/Achievements 7) make LDAP a thing yesterday 8) fix slashboxs
[22:57:30] <mrcoolbp> MrBluze hence my tendency to try to get this wrapped up
[22:57:36] <NCommander> Ugh
[22:57:41] <NCommander> I didn't even have a TODO on this one to strike off
[22:57:50] <MrBluze> its no drama, i am hopelessly behind on my general life todo list .. we are all busy
[22:57:50] <mrcoolbp> you were hoping eh?
[22:59:06] <MrBluze> however tbh soylentnews is really .. uninspiring .. its hard to come up with anything that isn't simply generic .. except we could go for the 1970's look or something to pitch to the era of the movie
[22:59:17] * xlefay disapproves
[22:59:42] * MrBluze isnt surprised
[23:00:21] <MrBluze> i logged in ok
[23:00:49] <xlefay> NCommander: you finally had that smoke? :)
[23:00:53] <NCommander> MrBluze, its making sure you logged in
[23:00:54] <NCommander> xlefay, I did
[23:02:03] <MrBluze> regarding: https / http login/logout issues, u need to either put a session id cookie into each https link, or simply force logins to direct to https form the beginning
[23:02:18] <xlefay> hehe cool; could I bother you to get the ssh stuff settled then? I would like to take a look around, get familiar with the systems in case something ever does go wrong so I don't find myself in to an unknown environment ;)
[23:02:35] <MrBluze> thats the main bug with https .. make the submit url https for login box
[23:02:41] <xlefay> heh session id cookie in links? NO way.
[23:02:48] <xlefay> MrBluze: it does
[23:02:50] <NCommander> MrBluze, we SHOULD have https properly fixed for good
[23:02:54] <xlefay> NCommander: nope
[23:02:58] <NCommander> fuck
[23:02:59] <MrBluze> when i log in, it directs to http
[23:03:01] <NCommander> OH
[23:03:03] <MrBluze> its just the url is broken
[23:03:06] <NCommander> That's a different problem
[23:03:10] <xlefay> [03/21/14 22:41:35] <xlefay> NCommander: doesn't seem like the SSL was fixed; the silly thing is, the cookie actually gets set (last I tried it anyway) - but it gets invalidated when you get forwarded to non-SSL or something like that
[23:03:12] <NCommander> There was an issue where the POST won't work properly
[23:03:19] <NCommander> Hold on
[23:03:21] <NCommander> I can fix that
[23:03:26] <NCommander> This is supposed to work
[23:03:27] <MrBluze> yeah its just one letter
[23:03:44] <xlefay> MrBluze: the action param in the form is correct
[23:03:54] <xlefay> it specifies //.... instead of http:// or https://
[23:04:11] <NCommander> https normally is reserved for subscribers
[23:04:13] <xlefay> If you use http://, then '//' will be just that. Same with 'https'
[23:04:29] <xlefay> So it's a settings that's disallowing SSL?
[23:04:54] <MrBluze> u must be using https://
[23:04:59] <MrBluze> dont use http:// its insulting
[23:05:01] <xlefay> Can we beat slash into submission
[23:05:02] <MrBluze> or //
[23:05:06] <xlefay> MrBluze: // = perfect
[23:05:09] <MrBluze> because the cookies are separated
[23:05:15] <xlefay> // != 'http' or 'https'
[23:05:20] <MrBluze> no, ive done this all before
[23:05:36] <MrBluze> and u have to make it https for the action for logins, everything else is not ok
[23:05:39] <xlefay> Me too. '//' = auto select based on how you're accessing, e.g. https or http
[23:05:56] <MrBluze> if u try to log in http ... u must submit to an https
[23:05:59] <MrBluze> its insecure otherwise
[23:06:09] <NCommander> Hold on
[23:06:13] <NCommander> One good whack and it will be fixed
[23:07:01] <xlefay> You don't seem to understand the '//' link in the 'action' param. When the browser sees an URL prepended with '//' only, it changes that to 'https://' or 'http://' depending on how you're accessing the site (e.g. if your already using https, it'll replace it with https)
[23:07:18] <NCommander> Try it now
[23:07:20] <xlefay> Also, MrBluze, if your on SSL and you submit a form to non-ssl on Firefox for instance it warns you, which it doesn't @ SN
[23:07:36] <MrBluze> xlefay: nobody should be submitting their login specs via anything other than https
[23:07:38] <NCommander> Bah
[23:07:43] <xlefay> MrBluze: I agree
[23:07:54] <MrBluze> that is determined in the action url, xlefay
[23:07:54] -!- mechanicjay has quit [Quit: Leaving.]
[23:08:00] <NCommander> Oh
[23:08:01] <xlefay> MrBluze: // ====== autodetect
[23:08:01] <NCommander> irony
[23:08:03] <NCommander> Hrm
[23:08:07] * NCommander is working on it
[23:08:09] <MrBluze> / .. no
[23:08:14] <xlefay> https://soylentnews.org
[23:08:44] <xlefay> two '/'s = auto detect, really. If you use https, it'll use https, if you use http, it'll use http.
[23:08:51] <xlefay> Check the source of the page, click on a // link, see what happens
[23:09:01] <MrBluze> xlefay: thing is, if u dont force the browser to use https, half the browsers will attempt to send POST via http
[23:09:13] <xlefay> MrBluze: nah. Check this: https://soylentnews.org <-- use invalid data on that page
[23:09:18] <MrBluze> and once logged in, u should ensure that the person is using https or default to log-out
[23:09:22] <xlefay> you'll be kept on https
[23:09:54] <MrBluze> no, i am on http after login
[23:10:05] * MrBluze knows he is right because he has done this before!!!
[23:10:13] * NCommander is working on it
[23:10:21] <xlefay> MrBluze: use INVALID data
[23:10:32] <xlefay> e.g. wrong username/password, see, you'll be kept on SSL
[23:10:47] <xlefay> when you submit that form, you'll be kept on SSL - it's the webserver not your browser which sends you to http
[23:11:39] <MrBluze> oh i se
[23:11:41] <MrBluze> +e
[23:11:57] <xlefay> You tried the invalid data? It keeps you on SSL
[23:12:10] <NCommander> xlefay, its the login script that bumps you back
[23:12:13] <xlefay> If you trace your way through the login process, you'll see - it'll keep you on SSL till the webserver redirects you
[23:12:17] <NCommander> I think its breaking because the subscription module is MIA
[23:12:18] <xlefay> NCommander: that's what I've been saying all along!
[23:12:25] <NCommander> xlefay, no, I mean I see where its doing it
[23:12:25] <xlefay> well not the subscription module
[23:13:06] <MrBluze> ok
[23:13:18] <MrBluze> i forget this thing is overly complex for what it does
[23:13:31] <MrBluze> by orders of magnitude tbh
[23:13:35] <xlefay> true
[23:14:25] <NCommander> stand by
[23:14:27] <NCommander> Give me a moment
[23:14:28] <xlefay> So essentially, you log in via SSL, then the subscription module (! the missing piece of information !) kicks you back out, which also explains why you get logged out if you go to https and were previously signed in via http
[23:14:35] <NCommander> xlefay, kinda, hold on
[23:14:38] <NCommander> I think I see the problem
[23:14:42] * NCommander hits it wth a wrench
[23:14:49] * xlefay hands the 4 by 4
[23:14:55] * xlefay ... to NC
[23:15:33] <NCommander> Ugh
[23:15:34] <NCommander> I hate slash
[23:15:39] <NCommander> And its overcomplicated engineering
[23:16:16] <NCommander> Ugh
[23:16:23] <xlefay> I'm guessing there's a hook for SSL or something?
[23:16:24] <NCommander> Ok
[23:16:26] <NCommander> So slash wants SSL and non-SSL to be different physical boxes
[23:16:36] * NCommander is debating if he just wants to comment out the redirect
[23:16:38] <xlefay> When I looked over the code it wasn't visible
[23:16:47] <xlefay> NCommander: tias ;-)
[23:16:57] <NCommander> Fuck
[23:16:57] <NCommander> No
[23:17:00] <NCommander> Its more complicated than that
[23:17:01] <xlefay> if it works and it's stable, good, if not... let's panic
[23:17:07] <NCommander> Its looking at Apache to see if the connection is SSL
[23:17:12] <NCommander> But we're not terminating the connection there
[23:17:25] <xlefay> wait, what file are you looking at?
[23:17:28] <NCommander> Users.pm
[23:17:33] <NCommander> Slash/Apache/User/User.pm
[23:17:38] <xlefay> I noticed there was a whitelist of sorts that allowed there servers to be SSL?
[23:17:42] <NCommander> (there's an SSL check in other places)
[23:17:54] <NCommander> xlefay, slash requires admins to login in SSL
[23:17:57] <NCommander> I commented out that check
[23:18:13] <xlefay> I thought there was an whitelist for servers but I may be wrong, lemme check if I wrote it down somewhere
[23:18:19] <NCommander> It won't be helpful
[23:18:22] <NCommander> Apache only sees varnish
[23:18:31] <NCommander> nginx -> varnish -> apache
[23:18:35] <NCommander> We'd have to bypass varnish
[23:18:46] <NCommander> Thats a bad thing
[23:18:51] <xlefay> NCommander: I think this was actually a smart idea, about passing through varnish, for caching
[23:18:52] <MrBluze> can we force http->https redirect on all page loads?
[23:18:59] <MrBluze> ie: at the DNS
[23:19:02] <NCommander> No
[23:19:02] <xlefay> we can have varnish pass nginx's x-forwarded for?
[23:19:10] <NCommander> Also, SSL does fucking huge amounts of CPU processing
[23:19:11] <xlefay> eh, you can't do http vs https in the dns
[23:19:18] <NCommander> I don't want to turn it on at the world level until I'm sure we can take it
[23:19:42] <NCommander> xlefay, hrm ... if we can pass a flag from nginx to varnish that tells its SSLed
[23:19:45] <NCommander> That woud work
[23:19:49] <NCommander> I can modify the check for SSL then
[23:19:49] <xlefay> nginx can do that
[23:19:53] <NCommander> And make those security features work
[23:19:53] <xlefay> just pass a header no?
[23:19:57] <NCommander> yeah
[23:20:01] <xlefay> That's simple!
[23:20:12] <NCommander> xlefay, I need to modify the SSL handler in Apache to handle this setup
[23:20:22] * NCommander understands why its this complicated, but ugh
[23:20:25] <xlefay> Well, I was only talking about the header being simple
[23:20:33] <NCommander> Yeah
[23:20:35] <NCommander> The rest is easy
[23:20:45] <NCommander> We need to make sure the header goes from nginx -> varnish
[23:20:49] <NCommander> Varnish needs to pass it to Apache
[23:21:03] <xlefay> well technically, you could also check in varnish if the connection comes from localhost
[23:21:18] <NCommander> Yeah, but we might ahve other things coming in from localhost
[23:21:31] <NCommander> It should only pass the SSL flag if it gets it on 127.0.0.1
[23:21:49] <xlefay> Or, run varnish on a secondary port, locally, check if that port is hit? If so, it's nginx?
[23:21:57] <xlefay> header works fine too tho
[23:21:58] * NCommander is mulling
[23:22:04] <NCommander> xlefay, what's your ssh pubkey
[23:22:08] <NCommander> I'll stick it on production now
[23:22:27] <xlefay> You don't happen to have an SSH connection open to SVC do you?
[23:23:37] <xlefay> check your SN e-mail
[23:24:10] <NCommander> xlefay, no
[23:24:15] <NCommander> let me see if I can even get into svc
[23:24:29] <NCommander> xlefay, ENOEMAIL
[23:25:00] <xlefay> yeah sorry, IPv6 tunnel prevented me from sending >.<
[23:25:08] <xlefay> apparently, HE.net gots port 25 blocked
[23:25:14] <NCommander> xlefay, remind me to push AAAA records for dev
[23:25:20] <NCommander> I want to see if Slash breaks horribly if we have that setup
[23:25:28] * NCommander notes thats the ONLY reason we haven't got IPv6 on production
[23:26:09] <NCommander> You know
[23:26:19] * NCommander is debating if he wants to kill this check
[23:26:23] <NCommander> I like forcing admins to be on SSL
[23:26:36] <xlefay> We're testing on dev
[23:26:37] <NCommander> (it actually nukes their admin privelleges if they log in on unsecure networks until they change their pass)
[23:26:38] <xlefay> ?
[23:26:51] <NCommander> xlefay, no, I'm looking at SSL on production, I don't have nginx setup on dev
[23:26:51] <xlefay> Heh, that's pretty good
[23:26:58] <NCommander> That's why its overly complicated
[23:27:02] <NCommander> And slash needs to know the SSL state
[23:28:01] <xlefay> Well... let's say, we created a second vhost for apache, for SSL - and set an environmental variable, could you use that to bypass the check?
[23:28:09] <xlefay> e.g. if (ENV['whatnot']) { ssl = true }
[23:28:30] * xlefay is probably simplifying this a ton
[23:28:34] <xlefay> e-mail is sent btw
[23:28:43] <NCommander> xlefay, the magic function you need to look at
[23:28:51] <NCommander> Slash/Apache/Apache.pm
[23:28:55] <NCommander> ConnectionIsSSL()
[23:29:14] <NCommander> OH!
[23:29:15] <NCommander> Look
[23:29:16] <NCommander> $x = $r->header_in('X-SSL-On');
[23:29:21] <NCommander> Slashcode actually does what we want
[23:29:23] <xlefay> Dammit
[23:29:29] <xlefay> I looked into that function beforfe
[23:29:32] <NCommander> xlefay, can you set that header in nginx?
[23:29:35] <xlefay> the other day actually >.<
[23:29:38] <xlefay> NCommander: yeah sure, ssh?
[23:29:52] <NCommander> yeah, what do you want your username to be?
[23:29:56] <xlefay> xlefay
[23:30:52] <NCommander> ...
[23:30:58] * NCommander adds it on production and not his laptop
[23:31:09] <xlefay> LOL
[23:31:18] <xlefay> haha, well laptop is fine too tho ;-)
[23:31:34] <xlefay> I'll just ssh via the laptop, onto the svc, onto production, what could go wrong, right? :)
[23:31:49] <NCommander> xlefay, you should be good to go
[23:31:56] <NCommander> xlefay, bounce from svc to production
[23:32:34] -!- janrinok has quit [Quit: leaving]
[23:32:53] * NCommander notes the Varnish configuration is getting to the point it can eat babies
[23:33:01] <xlefay> .146?
[23:33:29] <NCommander> yeah
[23:48:04] <mrcoolbp> xlefay: can I have your input on something?
[23:48:15] <xlefay> sec
[23:48:18] <mrcoolbp> sure
[23:50:03] <xlefay> mrcoolbp: is it a quick one?
[23:50:10] <mrcoolbp> yes
[23:50:11] <xlefay> question... that is, don't get funny now
[23:50:54] <mrcoolbp> The plan would be story submission to main site asking for suggestions for *available* domains (send to suggestions@soylentnews or namechange@SN), staff vote on a shortlist ( ~10-12 names ), check trademarks on the shortlist, register domains that survive, public vote
[23:51:02] <mrcoolbp> looks like this:
[23:51:03] <mrcoolbp> -Story Submission
[23:51:03] <mrcoolbp> -Long List (unregistered) domains
[23:51:03] <mrcoolbp> -staff vote
[23:51:03] <mrcoolbp> -short list
[23:51:03] <mrcoolbp> -check trademarks
[23:51:03] <mrcoolbp> -register
[23:51:03] <mrcoolbp> -final vote
[23:51:48] <mrcoolbp> we can use audioguys code for all this, he thinks he can use email tokens for SN users for the final vote
[23:52:02] <mrcoolbp> no need to rework slashcode and we can get moving
[23:52:04] <mrcoolbp> thoughts?
[23:52:25] <xlefay> What method is audioguy suggestion?
[23:52:37] <mrcoolbp> he coded a solution that we are testing now
[23:52:39] <xlefay> The general gist of it seems fine, I'd post it to the mailing list though, give others a chance to chip in
[23:52:55] <mrcoolbp> I will, I wanted your input first if you sign off I will do that
[23:53:27] <xlefay> MrBluze should sign off on it, he's the task-force's commander
[23:53:39] <mrcoolbp> yup, I'm talking with him
[23:53:39] <MrBluze> i just did
[23:53:41] <mrcoolbp> thanks
[23:53:42] <xlefay> I'm just a guy sitting here, doing random stuff :P
[23:53:52] <mrcoolbp> we value your feedback
[23:54:18] <xlefay> I say it looks good but I'm rather preoccupied atm so I'm not the best person to ask :p
[23:54:27] <mrcoolbp> okay, I need to get to work on the proposal to staff
[23:54:31] <xlefay> :)
[23:54:52] <mrcoolbp> MrBluze, I'll send you a draft in a few minutes
[23:54:57] <MrBluze> ok
[23:55:05] <mrcoolbp> BRB
[23:55:05] <MrBluze> then put it up for submissions
[23:55:14] <MrBluze> they can email it to style@ ...
[23:55:19] <mrcoolbp> okay
[23:55:32] <mrcoolbp> first you, then mailing list, then submission to main site
[23:55:36] <MrBluze> yes
[23:55:42] <mrcoolbp> okay BEV
[23:55:44] <mrcoolbp> BRB
[23:55:45] <mrcoolbp> lol
[23:55:53] <MrBluze> if u want, put up the already owned domains up as an example list
[23:56:08] <MrBluze> or just dont put up any
[23:56:14] * MrBluze doesnt mind
[23:56:21] <mrcoolbp> okay, send me the original long list
[23:57:46] <MrBluze> oh.. yeah ok gimme a few min
[23:57:57] <MrBluze> but it contains fuck and so on
[23:57:57] <mrcoolbp> k
[23:58:06] <mrcoolbp> whatever I edit it then
[23:59:37] <MrBluze> ok hmm