#soylent | Logs for 2024-12-19

« return
[04:21:53] -!- norayr has quit [Ping timeout: 272 seconds]
[04:21:53] -!- aqu4bot has quit [Ping timeout: 272 seconds]
[04:22:31] -!- Ingar has quit [Ping timeout: 272 seconds]
[04:22:31] -!- kolie_ has quit [Ping timeout: 272 seconds]
[04:22:31] -!- cosurgi has quit [Ping timeout: 272 seconds]
[04:23:09] -!- Xyem has quit [Ping timeout: 272 seconds]
[04:23:09] -!- progo has quit [Ping timeout: 272 seconds]
[04:23:09] -!- AlwaysNever has quit [Ping timeout: 272 seconds]
[04:23:09] -!- fffc has quit [Ping timeout: 272 seconds]
[04:23:09] -!- halibut has quit [Ping timeout: 272 seconds]
[04:23:09] -!- fab23 has quit [Ping timeout: 272 seconds]
[04:23:09] -!- [R] has quit [Ping timeout: 272 seconds]
[04:23:09] -!- SSLbits has quit [Ping timeout: 272 seconds]
[04:23:47] -!- systemd has quit [Ping timeout: 272 seconds]
[04:23:47] -!- Runaway1956 has quit [Ping timeout: 272 seconds]
[04:23:47] -!- robc207 has quit [Ping timeout: 272 seconds]
[04:23:47] -!- chromas has quit [Ping timeout: 272 seconds]
[04:23:47] -!- fliptop has quit [Ping timeout: 272 seconds]
[04:23:47] -!- lilitsyunetsi has quit [Ping timeout: 272 seconds]
[04:23:47] -!- Deucalion has quit [Ping timeout: 272 seconds]
[04:23:47] -!- jje has quit [Ping timeout: 272 seconds]
[04:25:05] -!- fab23 [fab23!~fabian@2001:8a8:izvs:s::i] has joined #soylent
[06:24:21] -!- halibut [halibut!~halibut@CanHazVHOST/halibut] has joined #soylent
[12:42:51] <fab23> site / irc is not reachable for may, backup communication on libera.chat in ##soylentnews
[12:48:15] <schestowitz[TR3]> ok
[12:48:46] <schestowitz[TR3]> at 4am today more than half of the IRC users in SoylentNews site fell offline. There are also DNS issues, but unclear what goes on. Kolie left TR IRC around the same time.
[12:49:02] <schestowitz[TR3]> can anyone explain here too?
[12:49:11] <fab23> IP reachability is strange
[12:50:04] <fab23> traceroute from multiple location in stop on the local ISP, looks like that the networks we are using are not routet any more
[12:53:31] <fab23> it looks like only IPv4 is down, but IPv6 works: https://smokeping.home4u.ch
[15:59:10] <kolie> whats up
[15:59:39] <kolie> I'm seeing ipv4 routing.
[16:02:07] <kolie> looking whats going on im off site, close by, but I can hit ipv4.
[16:04:09] -!- fliptop [fliptop!~fliptop@69.43.kn.gu] has joined #soylent
[16:04:13] -!- [R] [[R]!~rms@r15-744-213-821.abhsia.telus.net] has joined #soylent
[16:04:14] -!- Runaway1956 [Runaway1956!~OldGuy@the.abyss.stares.back] has joined #soylent
[16:04:15] -!- cosurgi [cosurgi!~cosurgi@153.19.lw.noz] has joined #soylent
[16:04:30] -!- kolie_ [kolie_!~kolie@208.91.qqu.m] has joined #soylent
[16:04:31] -!- NotSanguine [NotSanguine!~Thunderbi@dgp-169-243-882-749.biz.spectrum.com] has joined #soylent
[16:04:50] -!- fffc [fffc!~m@ujtww.bad.mn] has joined #soylent
[16:05:21] -!- Xyem [Xyem!~xyem@yu801-49.members.linode.com] has joined #soylent
[16:05:22] -!- Xyem has quit [Changing host]
[16:05:22] -!- Xyem [Xyem!~xyem@Soylent/Staff/Developer/Xyem] has joined #soylent
[16:05:22] -!- mode/#soylent [+v Xyem] by Imogen
[16:05:23] -!- robc207 [robc207!~robc207p2@62-19-913-880.static.firstlight.net] has joined #soylent
[16:05:51] -!- SSLbits [SSLbits!~SSLbits@CanHazVHOST/SSLbits] has joined #soylent
[16:05:53] -!- Deucalion [Deucalion!~Fluff@188.226.nuy.w] has joined #soylent
[16:05:53] -!- Deucalion has quit [Changing host]
[16:05:53] -!- Deucalion [Deucalion!~Fluff@Soylent/Staff/IRC/juggs] has joined #soylent
[16:05:53] -!- mode/#soylent [+v Deucalion] by Imogen
[16:06:52] <kolie> Looks look the routes weren't being accepted by upstreams. I called cogent and had them reapply their RPKi/filtering policy and its being accepted now.
[16:09:01] <fab23> yes, works again
[16:09:23] -!- cosurgi has quit [Changing host]
[16:09:23] -!- cosurgi [cosurgi!~cosurgi@Soylent/Staff/Misc/cosurgi] has joined #soylent
[16:09:45] <kolie> yea only one route was affected, it was the blanket /22 we have.
[16:10:35] <kolie> I have some more specific routes out and those were fine, .30 was covered by the /22 though and it looks like that's the one that got withdrawn about 4:00 this morning.
[16:14:16] <NotSanguine> Thanks kolie!
[16:14:32] <NotSanguine> I stepped away for a few minutes and everything is working. Hooray!
[16:14:32] -!- drussell [drussell!~drussell@2604:3d08:ghnm:vozx::zxir] has joined #soylent
[16:16:32] <drussell> Yep, IPv4 all working normally again from over here...
[16:16:33] <kolie> I put a hard coded route into the outbound advertisement and cogent has on file we are netblock owner and has removed filtering on that route.
[16:16:46] <drussell> kolie++
[16:16:46] <bender> kolie: 19
[16:16:53] <kolie> idk what caused it, but shouldn't happen again.
[16:18:07] <kolie> I don't think it was a route hijack but it's weird.
[16:19:16] -!- NotSanguine [NotSanguine!~Thunderbi@dgp-169-243-882-749.biz.spectrum.com] has parted #soylent
[16:25:02] <janrinok> still not working for me
[16:26:04] <kolie> ping soylentnews.org
[16:27:39] <fab23> janrinok: negativ DNS caching may cause such issues, try again later (~1h our such).
[16:28:01] <kolie> dns shouldn't have cached anything bad.
[16:28:30] <drussell> Yeah, the DNS responses looked correct here the whole time, just the IPv4 routing was broken
[16:29:25] <janrinok> OK - a complete reboot of router was required. How strange!
[16:29:32] <drussell> From here, I was getting as far as 64.59.169.157, which responded with a Destination net unreachable
[16:29:35] <janrinok> back on though now
[16:29:58] <kolie> yea it looks like the route for SN 192.163.30.0/24 was completely out of the tables in a good chunk of isps.
[16:30:37] -!- NotSanguine [NotSanguine!~Thunderbi@dgp-169-243-882-749.biz.spectrum.com] has joined #soylent
[16:30:46] <Fnord666> working for me now.
[16:30:51] <janrinok> kolie - just for interest the spf information is still wrong but only affects email. It is still using the Linode server addresses
[16:31:14] <kolie> I thought we added the other subnets ill verify thx for the heads up.
[16:31:56] -!- NotSanguine has quit [Client Quit]
[16:32:04] <drussell> Yes, 64.59.169.157 is the main router IP for Shaw in Kelowna, one IP hop upstream from my cable modem, which was saying NET unreachable...
[16:32:26] -!- chromas [chromas!~chromas@Soylent/Staph/Infector/chromas] has joined #soylent
[16:32:26] -!- mode/#soylent [+v chromas] by Imogen
[16:32:50] <drussell> so right there it simply didn't even know which way to route the packet, obviously missing BGP entries at all for that net.
[16:33:49] <janrinok> You are a star kolie - thanks for that!
[16:33:57] <janrinok> kolie++
[16:33:58] <bender> kolie: 20
[16:34:09] <kolie> I'm looking at the SPF record, it lists the old ips and the new.
[16:34:12] <kolie> What do we want to clean up.
[16:34:56] <chromas> Could take out the old ips if we don't have them anymore
[16:35:13] <chromas> kolie++
[16:35:13] <bender> kolie: 21
[16:35:49] <janrinok> doesn't take the spammers long to get restarted...
[16:40:35] <kolie> Anyone knows any ips being used for mail outside of the dc?
[16:41:52] <chromas> not I. You turned off the linode boxes right?
[16:44:13] <janrinok> One of the Linode IPs was registered in Mauritius
[16:45:13] <kolie> They deleted them awhile ago yea.
[17:01:47] -!- lilitsyunetsi [lilitsyunetsi!~lilitsyun@37.252.sp.xyt] has joined #soylent
[17:18:44] -!- Xyem has quit [Quit: ZNC - http://znc.in]
[17:18:44] -!- prg has quit [Quit: leaving]
[17:18:44] -!- kolie_ has quit [Quit: ZNC 1.8.2 - https://znc.in]
[17:18:44] -!- robc207 has quit [Read error: -0x7880: SSL - The peer notified us that the connection is going to be closed]
[17:18:44] -!- Runaway1956 has quit [Read error: -0x7880: SSL - The peer notified us that the connection is going to be closed]
[17:18:53] -!- Xyem [Xyem!~xyem@yu801-49.members.linode.com] has joined #soylent
[17:18:55] -!- Xyem has quit [Changing host]
[17:18:55] -!- Xyem [Xyem!~xyem@Soylent/Staff/Developer/Xyem] has joined #soylent
[17:18:55] -!- mode/#soylent [+v Xyem] by Imogen
[17:19:26] -!- robc207 [robc207!~robc207p2@62-19-913-880.static.firstlight.net] has joined #soylent
[17:19:27] -!- kolie_ [kolie_!~kolie@208.91.qqu.m] has joined #soylent
[17:19:30] -!- Runaway1956 [Runaway1956!~OldGuy@the.abyss.stares.back] has joined #soylent
[17:19:52] -!- prg [prg!~prg@ryfhuvb.de] has joined #soylent
[17:22:01] -!- ingar [ingar!~ingar@51.15.nj.ri] has joined #soylent
[17:22:42] <ingar> down deep deep deep down
[17:22:48] ingar is now known as Ingar
[17:38:16] <Ingar> was IRC down all day or was it the internets ?
[17:41:13] <chromas> it was all of sn
[17:41:48] <chromas> apparently the ISPs conspired against us and stopped routing packets, except a few stragglers
[17:57:35] <drussell> Well, IPv6 worked fine the whole time... It was just the routing for IPv4
[17:58:27] <drussell> I didn't notice anything was amiss until I tried to load the IRC logs via logs.sylnt.us as that one seems to not do IPv6...
[17:58:42] <drussell> but the logs loaded fine going to logs.soylentnews.org
[18:00:27] <drussell> Hmm, interestingly logs.sylnt.us does seem to do IPv6... 2602:fd63:12::2
[18:00:41] <drussell> Well, whatever... It's fixed now. :)
[18:01:27] <drussell> I thought maybe there was no IPv6 DNS for the sylnt.us domain or something.
[18:02:07] -!- Runaway1956 has quit [Read error: -0x7880: SSL - The peer notified us that the connection is going to be closed]
[18:02:24] -!- Runaway1956 [Runaway1956!~OldGuy@the.abyss.stares.back] has joined #soylent
[18:10:23] <kolie> Found the root cause
[18:10:34] <kolie> Failed battery backup.
[18:16:55] <Ingar> that was unexpected
[18:35:11] <janrinok> drussell, I use IPv6 all the time and have no problems in Europe. But it seems that to cross the pond a lot of traffic gets redirected in Paris and changed to IPv4. There is nothing I can do to prevent this. I presume it is to meet the requirements of many sites in the US who simply refuse to switch. I could not reach the site no matter what I tried to do.
[18:40:49] <drussell> Well, obviously it is a YMMV situation. If your provider or any of their upstream links play shenanigans or only transit IPv4, then it obviously wouldn't be working...
[18:41:53] <drussell> I can only report my personal observations and be able to say with certainty that the servers themselves were still actually up and running (just no IPv4 connectivity), as I could access them from here, but that obviously doesn't mean that everyone else worldwide would experience the same results. :)
[18:45:47] <Fnord666> So the back up is back up now?
[18:46:21] <ted-ious> I stayed connected the whole time.
[18:46:33] <kolie> yea its been up for about 2:30 hours for everyone
[18:47:09] <kolie> There was some routing infrastructure that was taken out by a BBU, and it handled some IPv4 traffic for the majority of ISPs.
[18:47:28] <kolie> The IPv6 routing wasn't disrupted.
[18:48:14] <Fnord666> systemd, systemd wherefore art thou systemd?
[18:48:47] <ted-ious> chromas hasn't added the automatic reconnection code to it yet.
[18:48:58] <drussell> Gaaa... Don't say that name three times or you'll summon a horrible daemon! ;)
[18:49:06] <ted-ious> I think he's waiting for the time to port it all to rust.
[18:51:34] <janrinok> well, looking at the first 6 spam accounts that have been created I have won in excess of 1,145,000 euros already today. I might not show up tomorrow .....
[18:53:00] <janrinok> One of them at least had the grace to suggest that I "might not remember entering the draw as it was some time back. They got that right!
[18:54:57] <janrinok> But if it turns out to be true I will do a Trump, and invite you all to a slap up feast provided by McDonalds. And all the Coke or 7up you can drink!
[19:13:23] -!- Plenumkakos [Plenumkakos!~Plenumkak@65.20.ztn.zio] has joined #soylent
[19:24:38] <janrinok> ... and now the village idiot appears. Let us see what words of wisdom he has for us....
[19:25:35] <Plenumkakos> Site goes down, and the only thing janrinok can think of is spammers.
[19:32:19] <Plenumkakos> =submit https://fortune.com
[19:37:39] -!- Plenumkakos has quit [Quit: Client closed]
[19:42:29] <janrinok> it hasn't arrived....
[20:00:23] <drussell> Yeah, plenum has the opposite of wisdom...
[20:37:51] -!- robc207 has quit [Ping timeout: 272 seconds]
[20:37:56] <chromas> Oh noes, rich people get affirmative action too?
[22:43:06] -!- Plethorum [Plethorum!~Plethorum@149.40.xw.n] has joined #soylent
[22:44:02] <Plethorum> Opacity is the new transparency, on SoylentPhlumoxed.
[22:55:16] -!- Plethorum has quit [Quit: Client closed]
[23:19:47] -!- fliptop has quit [Quit: Ex-Chat]
[23:54:10] -!- Plethorant [Plethorant!~Plethoran@200.25.xq.kkl] has joined #soylent
[23:56:05] <Plethorant> =submit https://www.bbc.com
[23:56:41] -!- Plethorant has quit [Client Quit]