#dev | Logs for 2014-07-30

« return
[01:26:58] juggs is now known as juggs|afk
[11:06:16] juggs|afk is now known as juggs
[12:57:55] <TheMightyBuzzard> paulej72, still doing data dumper of the txns? i could use one over here to get an idea of field sizes.
[12:59:33] <paulej72> use this url to trigger a pdt dump: http://dev.soylentnews.org
[13:00:10] <TheMightyBuzzard> done
[13:01:43] -!- Tachyon has quit [Ping timeout: 244 seconds]
[13:58:23] -!- Tachyon [Tachyon!Tachyon@acznspx081.ms.mff.cuni.cz] has joined #dev
[15:27:30] -!- Bytram|away [Bytram|away!~pc@Soylent/Staff/Developer/martyb] has joined #dev
[15:27:30] -!- mode/#dev [+v Bytram|away] by SkyNet
[15:28:12] Bytram|away is now known as Bytram
[15:43:40] <TheMightyBuzzard> paulej72, you had some bit of script that split the raw paypal transaction info into a hash by using split, yes?
[15:46:28] <TheMightyBuzzard> i ask because i'm writing the logging sub to accept a hash of a raw transaction split on equal signs
[16:18:34] -!- Tachyon has quit [Ping timeout: 244 seconds]
[16:22:52] <TheMightyBuzzard> paulej72, for you (schema for paypal database logging and subs necessary to log with): http://pastebin.com and http://pastebin.com
[16:23:12] <TheMightyBuzzard> should work fine for pdt or ipn
[16:23:26] <TheMightyBuzzard> barring the inevitible typo
[16:27:28] Bytram is now known as Bytram|afk
[16:27:48] -!- Tachyon [Tachyon!~Tachyon@dgc-obm-01-40.cust.vodafone.cz] has joined #dev
[16:41:49] -!- Tachyon has quit [Ping timeout: 244 seconds]
[16:50:02] -!- Bytram|away [Bytram|away!~pc@Soylent/Staff/Developer/martyb] has joined #dev
[16:50:02] -!- mode/#dev [+v Bytram|away] by SkyNet
[16:53:11] -!- Bytram|afk has quit [Ping timeout: 244 seconds]
[16:54:21] <paulej72> TheMightyBuzzard: yes transaction is a hash split on =
[16:54:35] <TheMightyBuzzard> excellent
[16:55:00] <TheMightyBuzzard> see above for implementing logging
[16:55:39] <paulej72> TheMightyBuzzard: I want to turn this into a real sub as I will use it as well
[16:55:41] <paulej72> raw_transaction => sub { my $dumped = Dumper(%$logthis);
[16:55:42] <paulej72> $dumped =~ s/^\s+//mg; $dumped =~ s/^.VAR1 = {\n//g; $dumped =~ s/};\n//g;
[16:55:42] <paulej72> return $dumped;}
[16:55:53] <TheMightyBuzzard> go for it
[16:56:25] <TheMightyBuzzard> oh, remove the % in front of $logthis
[16:56:28] <paulej72> I am plannng on duming the raw data into the payments table as well, jsut so we see it.
[16:56:30] <TheMightyBuzzard> that's a typo
[16:56:44] <TheMightyBuzzard> paulej72, nod nod
[16:58:03] <TheMightyBuzzard> you could put it in data i guess but it's blob rather than text type
[16:58:20] <paulej72> not for long :)
[16:59:22] <paulej72> thinking about puttin it into a new text field called raw_transaction just like your new table.
[17:00:55] <TheMightyBuzzard> nod nod
[17:00:56] <paulej72> TheMightyBuzzard: i never would have expected that there was this much stuff missing from the subscription code, but this code was rally old.
[17:01:41] <TheMightyBuzzard> truth. but it's good practice for when we start accepting credit cards. should be easy with as much as we've already done.
[17:02:55] <TheMightyBuzzard> i'm dicking off for a while now. ping if you need anything.
[17:02:57] <paulej72> TheMightyBuzzard: pp will do our credit cards. We do not want to do that ourselves PCI compilance is a real bitch.
[17:03:04] <TheMightyBuzzard> oh, handy
[17:03:55] <paulej72> np about slacking off for the rest of the day. I am mostly doing real work and will not code for another 4 hours.
[17:04:05] <TheMightyBuzzard> last time i did billing work there was no such thing as PCI
[17:04:33] <TheMightyBuzzard> ya, i'll prolly kick out a few more lines after some tv and lunch.
[17:15:27] <mrcoolbp> paulej72: we're basically ready for a soft launch on the soylent store
[17:19:05] -!- Tachyon [Tachyon!Tachyon@hollhb.kolej.mff.cuni.cz] has joined #dev
[19:03:26] <Bytram|away> paulej72: looks like the main page comments are not getting updated again... main page, driverless cars says 11 comments, story page ( http://soylentnews.org ) says 18;
[19:04:16] <Bytram|away> main page "Australia Bans Reporting of Multi-Nation Corruption Case" story says 1 comment, story page ( http://soylentnews.org ) says there are 10 comments.
[19:04:25] <Bytram|away> :(
[19:10:10] <paulej72> fuck same problem as before
[19:10:22] <Bytram|away> what problem is that?
[19:15:35] <paulej72> gluster is not loading the slash file system
[19:15:48] <Bytram|away> ugh. blegh!
[19:15:58] <paulej72> NCommander: ping?
[19:16:11] <Bytram|away> looks like gluster is losing its luster. :
[19:17:48] <Bytram|away> and I'm not able to get onto icinga ( https://sentinel.soylentnews.org ) to see anything; tried twice to login; not allowed. Now I get a 403 when I try to bring up the page.
[19:19:03] <Bytram|away> okay, I gtg; Good Luck!!!!!
[19:19:40] -!- Bytram|away has quit [Quit: Leaving]
[21:15:18] <xlefay> Wow, that's odd. I get the same error. I'll have to look into it. (re: icinga)
[21:26:41] <paulej72> xlefay: i rebooted boron
[21:27:27] <xlefay> Oh.. then autossh might have been broken too, I'm a bit out of touch with how NC & I did all that. I've scribbbled it onto my notepad
[21:27:39] <xlefay> I think Icinga might just not be started
[21:28:18] <TheMightyBuzzard> i remember SOMETHING had to be manually started every time but damned if i remember what
[21:29:11] <xlefay> Yeah, I know there's something and I was going to make a startup script for it.. but.. ugh
[21:29:21] <xlefay> Too much stuff left unfixed.
[21:29:35] <xlefay> paulej72: also, I was thinking the other day... have you ever worked with containers? Perhaps even docker?
[21:29:52] <paulej72> no niethe
[21:30:01] <xlefay> I think, docker would be ideal for us. Even more so, we could even put up a few docker images of our production environment minus actual user data.
[21:30:10] <xlefay> Do you know the concept of containers?
[21:30:42] <xlefay> paulej72: take a look at: https://www.docker.com - I think you'll see what I'm aiming at
[21:31:31] <paulej72> they are lik chroot jails
[21:32:01] <xlefay> Yeah, they are. Docker's added some good stuff on top of that though. Like portfwarding built in, being able to deploy the same image on multiple places, etc..
[21:32:29] <xlefay> I can't help but think how much easier our infra would be if we were using containers like that. But putting that aside for now. Consider the development stuff.
[21:34:27] <xlefay> Instead of the VM for instance, one could simply "pull" the docker image, and has a copy (well, minus the LDAP, and gluster) of the production environment (minus actual user data, of course) - wouldn't that be much easier than a VM?
[21:39:23] <xlefay> ^ paulej72 - I think it might be worth considering for our purposes. It would keep things nice and decoupled in general & a lot easier to get up, less dependency hell, etc.. Just plug 'n (mostly) play containers.
[21:39:53] <paulej72> ok I have autossh started properly
[21:39:59] <xlefay> NC & I already discussed containers (a long time ago) - but those where just plain LXC containers for development environments.
[21:41:31] <xlefay> hmm this is odd
[21:42:34] <xlefay> paulej72: I think I know why Icinga isn't working...
[21:42:37] <xlefay> ls: cannot access /usr/lib/cgi-bin/icinga: No such file or directory
[21:42:46] <xlefay> ls /usr/share/icinga/htdocs/
[21:42:46] <xlefay> docs images
[21:42:52] <xlefay> Stuff's missing it seems
[21:44:02] <xlefay> Someone broke our icinga lol
[21:44:42] <paulej72> khyber?
[21:45:16] <xlefay> don't think so, I think someone just accidently removed stuff. Let me check though.
[21:45:35] <paulej72> xlefay: how was icigna installed
[21:45:42] <xlefay> apt
[21:45:43] <juggs> maybe gluster dropped it when it had a brain fart :)
[21:46:13] <xlefay> dpkg doesn't show it installed at all lol
[21:46:55] <xlefay> ironically, icinga files still exists.. ugh oddness
[21:47:20] <TheMightyBuzzard> sounds like it got apt-get remove'd
[21:48:28] <xlefay> I might be wrong though, it's confusing since icinga's daemon + init script still exists as well as /etc/icinga
[21:48:37] <xlefay> I need to do a better dpkg query to figure this one out
[21:48:59] <paulej72> xlefay: i can’t get to oxygen. do you have direct access to that server?
[21:49:02] <TheMightyBuzzard> remove doesn't kill configs like purge does
[21:49:37] <xlefay> paulej72: I think you can only directly access it if you've got it's IP. Try nslookup ipv4.oxygen.li694-22
[21:49:48] <xlefay> not sure if that still exists in the dns though..
[21:50:36] <xlefay> TheMightyBuzzard: ok, seems like only one package or so icinga requires was removed.
[21:52:07] <TheMightyBuzzard> has me wondering wtf happened because if it was anyone getting their malice on, they did a pretty poor and very specific job of it
[21:52:08] <paulej72> xlefay: i did some package updates, it might have been me.
[21:52:11] <xlefay> https://sentinel.soylentnews.org and it's back to life
[21:52:25] <TheMightyBuzzard> yays
[21:52:30] <xlefay> TheMightyBuzzard: it was probably just a package conflict that forced the removed all of some packages
[21:52:47] <paulej72> OK we still need to figure out oxygen
[21:52:49] <xlefay> paulej72: I just want to note, this is a fine example why containers are better ;)
[21:53:01] <paulej72> yea
[21:53:12] <xlefay> Anyway, ehm, oxygen that's a tough one
[21:53:49] <xlefay> paulej72: ssh to 2001:41d0:1:dfa9::1
[21:54:15] <paulej72> ssh: connect to host to port 22: Connection refused
[21:54:22] <xlefay> Really? Wow.
[21:54:35] <xlefay> Ehm... yeah, then I got nothing
[21:54:54] <xlefay> paulej72: icinga does report oxygen as being down for at least 4 days now
[21:55:09] <paulej72> shit
[21:55:21] <xlefay> Perhaps, the server expired and NCommander didn't renew it?
[21:55:31] <xlefay> Anyway, ehm, did you reset the autossh stuff for beryllium?
[21:56:08] <paulej72> He almost did that last month, I hope he paid it this month
[21:56:51] <paulej72> I did not go to beryllium, but this is the server outpunagios 32656 1 0 21:38 ? 00:00:00 /usr/lib/autossh/autossh -M 0 -N -T -o ControlMaster=yes -o ControlPath=/var/run/icinga/ssh.icinga@beryllium.li694-22.socket -o ServerAliveInterval 60 -o ServerAliveCountMax 3 icinga@beryllium.li694-22
[21:57:38] <xlefay> Remote command execution failed: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[21:57:48] <paulej72> NCommander: has oxygen been paid for this month?
[21:57:52] <xlefay> Ugh I hate this part where all shit goes up "CRITICAL" || "UNKNOWN"
[21:58:25] <xlefay> [21:58] xlefay@boron $ ssh beryllium
[21:58:25] <xlefay> Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[21:58:27] <xlefay> woa...
[21:58:44] * xlefay bets kerberos...
[21:59:08] <xlefay> Well this is odd, paulej72
[21:59:12] <paulej72> yes it is kerberos, because I was able to ssh from helium to bery
[21:59:25] <xlefay> I tried to ssh just now from boron, it gave me that error but... from localhost:
[21:59:26] <xlefay> ssh beryllium.li694-22
[21:59:26] <xlefay> Last login: Wed Jul 30 21:58:11 2014 from carbon.li694-22
[21:59:47] <xlefay> A bit odd
[21:59:53] <paulej72> that method bypasses kerveros
[22:00:17] <xlefay> paulej72: correct, but I'm refering to the last login time
[22:00:59] <xlefay> Unless I've been mistaken all these years, so even though my connection from boron went wrong, it still recorded it as a login?
[22:01:10] <xlefay> eh, that sentences makes no sense
[22:01:11] <xlefay> LOL
[22:01:20] <xlefay> OOH
[22:01:35] <xlefay> nvm, I get it now... boron is our ssh proxy /feels stupid now
[22:01:40] <paulej72> not it was parsable
[22:02:08] <xlefay> yeah, my brain's tired anyway, how do we fix kerberos this time
[22:02:42] <xlefay> (I'm rather getting annoyed by kerberos to be honest. It's awesome, but it's been giving us grieve at random times, which is, quite inconsiderative if you ask me.)
[22:03:28] <xlefay> TheMightyBuzzard: perhaps I could ask a favor of you, good sir?
[22:03:33] <TheMightyBuzzard> shoot
[22:03:59] <xlefay> As I just said, Kerberos has been quite inconsiderative of late. Perhaps, you could make it more, considerative?
[22:04:00] <paulej72> xlefay: most of the problems from kerberos stem form the failed upgrade of helium that NCommander has failed to fix.
[22:04:09] <xlefay> Surely, that beats doing paypal crap
[22:04:22] <TheMightyBuzzard> <---dev not sysadmin
[22:04:27] <xlefay> paulej72: ah I see. I think we all have so much shit to do that it's easy to forget stuff
[22:04:49] <xlefay> TheMightyBuzzard: I know.. but we sysops can configure it, but we can't actually code it to fail at the appropriate times...
[22:04:56] <xlefay> Thus, it goes right into your alley good sir
[22:05:15] * TheMightyBuzzard scratches his head
[22:05:21] <xlefay> (although, granted, most of us could actually code it to do just that..) I'm just messing around, happens when I get tired.
[22:05:25] <xlefay> Ignore me.
[22:05:55] <TheMightyBuzzard> i can do "worst possible" or "most inconvenient"
[22:06:36] <xlefay> anyway, paulej72 how can we correct the current mistake? Is there something that we can do to correct ourselves or do we have to wait for input from NCommander?
[22:06:40] <TheMightyBuzzard> i can fake "random" but you know nothing is ever really random.
[22:06:49] <xlefay> TheMightyBuzzard: ha. pseudo-random works for me.
[22:07:09] <xlefay> correct it ourselves* , not correct ourselves, we're fine gents so..
[22:07:19] <paulej72> xlefay: I am looking at autossh config right now to see if I can figure it out
[22:07:57] <xlefay> paulej72: our autossh configs use kerberos to ssh. So, the kerberos issue is what's stopping it from working.
[22:08:15] <TheMightyBuzzard> paulej72, what all did you reboot just now?
[22:08:22] <xlefay> e.g. not being able to ssh from boron to beryllium (unless, using ssh to proxy)
[22:08:22] <TheMightyBuzzard> or rather a few minutes ago
[22:08:38] <paulej72> yes I know, I am trying to see what the issue might be
[22:08:42] <xlefay> TheMightyBuzzard: ironically.. when I signed into boron, it's motd was complaining it requires a reboot...
[22:08:55] <TheMightyBuzzard> xlefay, i noticed that too. gave me a chuckle.
[22:09:47] <xlefay> paulej72: alright. Do you mind if I leave it in your hands? I have to do some stuff for the Swag Task Force. Feel free to message me though if you need more information but I'm pretty much useless when it comes to kerberos, it's been on my todo list for a long time.. but you know how that goes..
[22:09:49] <paulej72> I think the kernel may have been updated
[22:10:14] <paulej72> xlefay: sounds ok to me
[22:10:36] <TheMightyBuzzard> xlefay, s'actually hard to keep myself from asking to jump on the admin team too but i get enough of that in my day job.
[22:11:14] <xlefay> TheMightyBuzzard: well, I'm sure we'll be happy to have you, but, no need to spread yourself to thin (e.g. be sure to have an actual life)
[22:11:27] <xlefay> but if you ever change your mind and want to be on, just say the word
[22:11:42] <TheMightyBuzzard> ya, i take weekends off even if live catches on fire
[22:13:05] <xlefay> That's good. I gotta go do this swag task force stuff now though... has to be done
[22:13:56] * TheMightyBuzzard starts whistling Floyd's Money
[22:17:49] juggs is now known as juggs|afk
[22:19:24] juggs|afk is now known as juggs
[22:27:21] <juggs> random thought - do you have a virtual console on the troublesome box? Or would that also be knobbled by a kerberos / whatever it is issue?
[22:37:08] <xlefay> juggs: we can access every server, we're not locked out or anything. It's just, using kerberos to access beryllium doesn't work, but direct SSH'ing into it, works just fine
[22:39:14] <juggs> Ah, I see. /me stops being the idiot looking over the shoulders of people while making unhelpful comments and wanders off to get the coffees :D
[22:40:50] <xlefay> juggs: haha no worries, but while you're at it, can you bring me a cup?
[22:42:09] <juggs> sure, sure - I'm getting the round in
[22:43:41] <xlefay> haha, I'm kidding
[22:43:45] <xlefay> How are you by the way?
[22:44:34] <juggs> ok thanks. And you?
[22:45:01] <paulej72> xlefay: back when we were first having kerveros problems you found that the domains were not quit the same on helium and boron. do you remember where those setting were.
[22:45:07] <xlefay> I'm good ;)
[22:45:36] <xlefay> paulej72: domains, as in, resolv.conf domainname, or in the kerberos config files which are in /etc/krb5.../krb5...
[22:46:18] <paulej72> can’t remember, but I thought it was the krb5 stuff
[22:47:16] <xlefay> Yeah, I don't know the full path, but tab complete will get you there, it starts with /etc/krb5
[22:48:03] <xlefay> Be careful though, best to make backups before editing stuff, kerberos' is prone to the slightest configuration error as it seems
[22:48:05] <paulej72> do you recall the exact issue that you saw. I want to make sure I fix this properly
[22:49:50] <xlefay> The issue, of ssh'ing? Or, the issue we had with kerberos not updating? Or do you mean, the hostname ACLs appearing wrong on helium?
[22:50:27] <xlefay> The hostname ACLs appearing wrong on helium were in the /etc/krb5.../kpropd.acl file, but that's just how it appeared - not sure if it's actually wrong but it does appear to be missing part of our domainname.
[22:52:09] <xlefay> Is that what you meant? I'm sorry doing several things at once, so it's hard to keep track
[22:53:02] <paulej72> xlefay: yes I think that was it. the last part
[22:53:34] <xlefay> paulej72:
[22:53:35] <xlefay> root@helium:/etc/krb5kdc# cat kpropd.acl
[22:53:35] <xlefay> host/kdc-01.li694@LI694-22
[22:53:35] <xlefay> host/kdc-02.li694@LI694-22
[22:53:49] <xlefay> Notice, the 'li694@' instead of 'li694-22@'
[22:56:20] <xlefay> @ Boron, it's correct, at helium, it seems incorrect. I mentioned it to NC the other day, when we had the issue with Kerberos & kpropd, but he disappeared, so I could never confirm whether it was correct or not and since I don't have a lot of experience with kerberos, I didn't feel it right to correct it (for all I know, a slight configuration error was made and not corrected and thus the kpropd.acl was correctly formatted as per that error, but
[22:56:20] <xlefay> stuff still worked.. hence why I left it be at least, that was my plan until NC came around so I could inquire)
[22:59:37] <xlefay> paulej72: can you let me know how it goes after you've changed that ACL list?
[23:00:13] <paulej72> xlefay: Form what I am reading that file on the master is not used so it should not be an issue
[23:01:04] <xlefay> paulej72: that's also what I read; that's why I wasn't sure because the file does exists on the master, so I figured, perhaps NC was going to make a multiple-master like scenario if that's even possible with kerberos.