#dev | Logs for 2019-06-24
« return
[10:43:45] -!- NotSanguine [NotSanguine!~notsangui@xmf-69-640-580-028.nyc.res.rr.com] has joined #dev
[10:48:08] -!- NotSanguine has quit [Client Quit]
[11:00:03] -!- SoyCow4645 [SoyCow4645!~c1443663@193.68.yi.xx] has joined #dev
[11:02:11] <SoyCow4645> Hello, the site is down.
[11:03:50] <SoyCow4645> not sure who should be notified about that for help
[11:24:38] <Bytram> SoyCow4645: thanks... looking into it nowe
[11:28:53] <SoyCow4645> up now
[11:30:32] <SoyCow4645> I\ve heard about some free online monitors, not sure how "free" they really are
[11:30:46] <SoyCow4645> https://www.freshworks.com https://pingbreak.com https://www.webgazer.io
[11:30:51] -!- NotSanguine [NotSanguine!~notsangui@xmf-69-640-580-028.nyc.res.rr.com] has joined #dev
[11:30:51] <exec> ├─ 13Freshping | Free Reliable Website Monitoring software by Freshworks
[11:30:51] <exec> └─ 13Pingbreak - Free website monitoring
[11:31:01] <Bytram> I Bounced the front-ends, but am still seeing huge delay in trying to get the front page to load.... yep, timed out again.
[11:32:22] <NotSanguine> hey Bytram. I can reach the front page without issue via one ISP (Megapath) but can't reach it at all via another (Spectrum)
[11:32:39] <Bytram> oh. :P
[11:32:49] <Bytram> thanks for the info... any ideas?
[11:33:13] <Bytram> Linode dashboard shows all servers running
[11:33:35] <NotSanguine> Initially, I thought there may be an outage with an intermediate network
[11:34:03] <NotSanguine> then I got suspicious and thought that maybe Spectrum was using a transparent proxy that was having issues.
[11:34:17] <Bytram> NotSanguine: Any chance you can do a trace route over the good and bad paths?
[11:34:27] <NotSanguine> I've mostly ruled out the latter, so the former is probably true
[11:34:28] <NotSanguine> sure
[11:34:48] <NotSanguine> I've been doing that (well, mtr) for a while
[11:34:54] <NotSanguine> want me to post results here?
[11:34:57] <Bytram> awesome!
[11:34:59] <Bytram> pls do
[11:35:14] <Bytram> yep, ping to soylentnews.org dies for me.
[11:35:28] <NotSanguine> that'snot unusual
[11:35:45] <NotSanguine> traceroute or (tracert in windows) or mtr will work better
[11:36:29] <Bytram> that is *very* unusual for me... is first thing I do each time I unsuspend my lappy;
[11:36:43] <Bytram> ping soylentnews.org && ping sylnt.us
[11:37:15] <NotSanguine> My traceroute [v0.92]
[11:37:15] <NotSanguine> cartman.loveglide.com (192.168.226.57) 2019-06-24T07:36:07-0400
[11:37:15] <NotSanguine> Keys: Help Display mode Restart statistics Order of fields quit
[11:37:15] <NotSanguine> Packets Pings
[11:37:15] <NotSanguine> Host Loss% Snt Last Avg Best Wrst StDev
[11:37:15] -!- NotSanguine has quit [Excess Flood]
[11:37:46] -!- NotSanguine [NotSanguine!~notsangui@xmf-69-640-580-028.nyc.res.rr.com] has joined #dev
[11:37:54] <NotSanguine> oops
[11:38:01] <NotSanguine> got booted for "flooding"
[11:38:22] <Bytram> ouch.
[11:38:28] <NotSanguine> The "mtr" from 'cartman' is the working one
[11:38:43] <NotSanguine> and the "mtr" from venkman is the non-working one
[11:39:01] <NotSanguine> I can email it to admin@soylentnews.org or whereever you want too
[11:39:17] <SoyCow4645> Sometimes works, sometimes not... Can you check if the server is under DoS/flood?
[11:39:57] <NotSanguine> ouch
[11:40:26] <NotSanguine> the paste only put in the first of the two mtr's. Grr. Shall I email you bytram?
[11:40:46] <Bytram> pls do. martyb (at) soylentnews (dot) org
[11:41:02] <Bytram> actually, admin would be better
[11:42:22] <NotSanguine> on its way to you now. Will send to admin as well
[11:42:53] <Bytram> tx
[11:43:37] <NotSanguine> done
[11:43:55] <NotSanguine> darn it!
[11:43:56] <Bytram> looks like beryllium went off net at about 0930 UTC
[11:44:01] <Bytram> rcvd
[11:44:06] <NotSanguine> I sent the same one twice
[11:44:12] <NotSanguine> the working one no less
[11:44:16] <NotSanguine> argh
[11:44:20] <NotSanguine> please stand by martyb
[11:44:22] <Bytram> coffee++
[11:44:22] <Bender> karma - coffee: 20
[11:44:25] <Bytram> =)
[11:44:31] <Bytram> rx
[11:45:48] <Bytram> hmmmm
[11:45:49] <Bytram> http://internettrafficreport.com
[11:45:52] <exec> └─ 13Details for North America /// Internet Traffic Report
[11:46:25] <Bytram> I see several Routers with "0" zero thoughput
[11:47:48] <Bytram> ugh. And I need to be *at* work in just over an hour and haven't done *any* of my "morning stuff"
[11:48:17] <NotSanguine> It doesn't appear to be one of Soylent's servers
[11:48:33] <NotSanguine> Nor does it appear to be Linode's network
[11:48:33] <Bytram> just found this: https://www.cloudflarestatus.com
[11:48:36] <exec> └─ 13Cloudflare Status - Route Leak Impacting Cloudflare
[11:49:02] <SoyCow4645> Thanks for taking care, looks like the world routers are not really in your control to fix.
[11:49:33] <SoyCow4645> Perhaps another rogue BGP announcement or smth
[11:49:33] <NotSanguine> The timing fits, but does Linode use Cloudflare?
[11:50:07] <Bytram> no idea, but it is related to BGP if I read it correctly, so means all the other AS routers are gonna be dealing with issues, too. No?
[11:50:21] <NotSanguine> Based on the traceroutes (note that they are from the US east coast)
[11:50:33] <NotSanguine> one network is affected and another isn't
[11:51:09] <NotSanguine> Depending on how the peering and locations of the ISP interconnects are set up, that could certainly be the case
[11:51:11] <Bytram> takes a while for routes to propagate... both bad ones, and any corrections, too.
[11:51:27] <Bytram> https://linode.statuspage.io
[11:51:29] <exec> └─ 13Linode Status
[11:52:43] <NotSanguine> True enough, but unless the issue is widespread, I'm guessing something more specific than a BGP issue.
[11:53:06] <NotSanguine> Regardless, there likely isn't anything you can do about it bytram. Why don't you take care of your real life?
[11:53:23] <Bytram> sounds like a plan.
[11:53:32] <NotSanguine> Have a good day!
[11:53:48] <Bytram> On a positive note, I saw no e-mails from Linode, but it looks like we have several server reboots coming up.
[11:54:01] <Bytram> see the status page I linked above
[11:54:17] <Bytram> first one up is fluorine in about 18 hours.
[11:54:21] <SoyCow4645> http://internettrafficreport.com
[11:54:24] <exec> └─ 13Details for Europe /// Internet Traffic Report
[11:54:29] <SoyCow4645> http://internettrafficreport.com
[11:54:32] <exec> └─ 13Details for Asia /// Internet Traffic Report
[11:54:36] <SoyCow4645> not better than NA
[11:54:44] <SoyCow4645> have a good day :)
[11:54:49] <NotSanguine> Right, but that Linode maintenance hasn't begun yet,right?
[11:56:04] <Bytram> not that I can tell, right.
[11:56:36] <Bytram> thing is, they were supposed to send us emails about upcoming planned maintenance and I have seen *nothing* from them.
[11:56:44] <NotSanguine> Accorfing to the trafficreport site, alternet is having issues in the US -- which is where my traceroute dies
[11:57:42] <NotSanguine> Do you know the physical location of the SN VPS'?
[12:00:51] <Bytram> IIRC, they are in Dallas
[12:01:09] <Bytram> fwiw, was able to use a bot to submit a story... see #editorial
[12:01:38] <NotSanguine> Which would jive with alternet's problems in Colorado for connectivity from the east coast
[12:01:54] <Bytram> okay, was just able to reload: https://soylentnews.org
[12:02:04] <exec> └─ 13SoylentNews: Hall of Fame
[12:02:16] <Bytram> and just loaded main page okay, too.
[12:02:22] <NotSanguine> me too
[12:02:32] <SoyCow4645> https://livemap.pingdom.com
[12:02:48] <Bytram> is taking a long time to open a page to submit a new story, though.
[12:03:12] <Bytram> timed out
[12:03:24] <Bytram> maybe I should turn off my "evil bit" ??
[12:03:27] <Bytram> ;)
[12:03:37] <NotSanguine> Still seeing issues with traceroutes via Spectrum too
[12:03:46] <Bytram> nod nod
[12:04:00] <NotSanguine> And traceroutes via Megapath are still solid
[12:04:10] <Bytram> k
[12:05:07] <NotSanguine> so intermittent issue. Could be hardware at an intermediate network or software/routing configs at one or more. I imagine we'll be hearing more about it as time goes on, unless it's a narrow issue
[12:06:15] <Bytram> nod nod
[12:06:43] <Bytram> keep an eye on the #rss-bot channel to see if anyone announces anything and we pick it up
[12:06:50] <Bytram> really gtg, afk, biab
[12:07:59] <NotSanguine> mtr shows lots of packet loss on Cogent's network (http://cogentco.com/en/)
[12:08:12] <NotSanguine> catch you later
[12:10:16] <SoyCow4645> hmmm.... https://soylentnews.org :)
[12:14:52] <NotSanguine> SoyCow4645: according to TFA, Cogent was at issue during 512k day. Perhaps that is what's going on
[12:28:57] -!- SoyCow4645 has quit [Quit: Web client closed]
[12:39:12] -!- SoyCow4645 [SoyCow4645!~c1443663@193.68.yi.xx] has joined #dev
[12:39:20] -!- SoyCow4645 has quit [Client Quit]
[12:44:49] <Bytram> NotSanguine++ thanks so much for the help!
[12:44:49] <Bender> karma - notsanguine: 1
[12:44:51] <Bytram> gtg
[12:44:53] <Bytram> laters!
[13:01:20] <NotSanguine> Sorry I missed you bytram. and you're entirely welcome. Catch you later.
[13:01:47] -!- NotSanguine has quit [Quit: Nettalk6 - www.ntalk.de]
[14:19:37] -!- fungus has quit [Remote host closed the connection]
[14:20:28] -!- fyngyrz_ has quit [Quit: Leaving...]
[14:23:26] -!- fyngyrz [fyngyrz!~fyngyrz@Soylent/Staff/Editor/fyngyrz] has joined #dev
[14:24:37] -!- fungus [fungus!~ben@66.171.jo.vll] has joined #dev
[22:32:15] -!- NotSanguine [NotSanguine!~notsangui@xmf-69-640-580-028.nyc.res.rr.com] has joined #dev
[22:32:46] -!- NotSanguine has quit [Client Quit]