#soylent | Logs for 2025-10-03
« return
[01:00:11] -!- bender has quit [Remote host closed the connection]
[01:00:15] -!- bender [bender!bot@Soylent/BotArmy] has joined #soylent
[01:00:16] -!- bender has quit [Changing host]
[01:00:16] -!- bender [bender!bot@Soylent/Bot/Bender] has joined #soylent
[01:00:51] -!- devbot has quit [Remote host closed the connection]
[01:00:52] -!- devbot [devbot!devbot@Soylent/BotArmy] has joined #soylent
[01:01:51] -!- Loggie [Loggie!Loggie@Soylent/BotArmy] has joined #soylent
[05:07:46] -!- c0lo [c0lo!~c0lo@124.190.mg.jlq] has joined #soylent
[05:07:56] <c0lo> Error 503 Backend fetch failed
[05:07:56] <c0lo> Backend fetch failed
[05:07:57] <c0lo> Guru Meditation:
[05:07:57] <c0lo> XID: 283377692
[05:07:58] <c0lo> Varnish cache server
[05:11:30] <c0lo> =sub https://www.gurufocus.com
[05:11:48] <systemd> ✓ Sub-ccess! "03Just a Moment..." (1p) -> https://soylentnews.org
[05:47:42] -!- aqu4bot [aqu4bot!~aqu4bot@pptdbxlc9.us] has joined #soylent
[05:58:01] -!- donaldo [donaldo!~donaldo@315.38.1.669.dynamic.jazztel.es] has joined #soylent
[06:32:33] <janrinok> 503 - 16 seconds
[06:56:23] <janrinok> c0lo thanks. If you have the time can you report as many 5xx or other problems that you see? It seems that kolie is throwing more resources at the problem but that isn't showing much improvement. I guess it is a software issue - but where?
[07:43:50] <Ingar> currently just timing out
[07:44:03] <Ingar> 503
[07:44:12] <Ingar> really smells liek a teimout issue with something
[07:44:21] <Ingar> *like a timeout
[07:45:11] <Ingar> with a timeout, it's less dangerous. it's a backend, fetch that failed
[07:47:32] <janrinok> We know it is a timeout, but where? Yesterday, kolie discovered that the db was running out of connections. Something is keeping the connections open far too long. However, this was resolved by increasing the size of the connection pool available to the db, which solved that problem,
[07:48:56] <janrinok> What I have noticed this morning is that the response to a request is often being built up on my screen in parts. Normally this would happen so quickly as to not be noticeable, but even some elements are being delayed during the request processing.
[07:49:11] <fab23> could be caused of some crawler requesting db intense pages too quickly
[07:50:57] <janrinok> That might give kolie an idea where to start looking. One problem is that there is no test harness for Rehash. Over the years several options in the code have been removed for very good reasons at the time. The problem is that they have never been tested as a complete entity other than letting the users get at them.
[07:52:40] <janrinok> Or, perhaps more correctly, the various elements have never been fully tested for corner and edge cases before being included in the final build. It is possible that some of these are now conflicting which each other, causing the problems to appear now - although the code changes were made a long time ago.
[07:54:15] <janrinok> fab23 - agreed. But overnight it looks like kolie has re-enabled that part of the firewall that takes care of that problem, but the 5xx are still appearing.
[07:56:03] <janrinok> The requests by AI bots was still within the capabilities of the server, but we are now blocking some of those sources anyway.
[07:57:59] <janrinok> Another problem with blocking the bots is that some of them use the same vpns that our users do. Blocking the source could well prevent our users from accessing the site. If they have paid for a vpn service they might be unimpressed to find that we are intentionally blocking it so we have to tread gently.
[07:59:35] <janrinok> Some of our users have already complained of this (although not very many of them). They think that the block is directed at them personally and not the bots which are abusing our site. We have to be very careful not to wield too big a hammer to crack a relatively small nut.
[08:00:29] <fab23> I see
[08:22:54] <Ingar> oth, if blocking the great china firewall solves 99.9% of your spam issue, what do you do
[08:23:02] <Ingar> (been there actually)
[08:26:38] -!- c0lo has quit [Ping timeout: 245 seconds]
[08:26:58] <janrinok> Unfortunately, it isn't China in our case. These are often acceptable sites which unfortunately also host some bots. And some of the bots are also of value. We want people to be able to search for SN in their favourite search engine - if we don't appear there they will never find us.
[08:29:37] <Ingar> the Hetzner I assume :-)
[08:38:28] <janrinok> Nah, it isn't a single company - you are safe
[08:43:51] <janrinok> rebooting
[08:45:51] <janrinok> I had to force a re-read of my environment after changes
[09:08:03] <Ingar> darn I wanted to post a post and it got 503'ed
[09:17:49] <janrinok> how long did it take to recover - can you estimate?
[09:25:51] -!- c0lo [c0lo!~c0lo@124.190.mg.jlq] has joined #soylent
[09:35:50] <Ingar> usually a few seconds
[09:35:59] <Ingar> take 10
[09:40:57] -!- nosler [nosler!~nosler@216.238.yx.rp] has joined #soylent
[09:44:05] -!- nosler has quit [Client Quit]
[09:53:07] <janrinok> I have experienced pages arriving in parts over 10-15 seconds. Pages are always constructed in parts but usually they arrive almost immediately and appear as a complete page. Has anyone else seen this effect?
[10:07:55] <Ingar> no just timeout, white screen, 504 after a while. reload and ok
[10:08:01] <Ingar> *503 obviously
[10:08:52] <Ingar> right now, it took almost 30sec to laod the front page
[13:21:01] <donaldo> I'm going to pass opinion beer-in-hand and sitting on the bar stool: I BLAME DOCKER
[13:37:47] <donaldo> Is per chance Docker Compose being used?
[13:40:55] <janrinok> I don't follow your logic. The problem has only surfaced recently. We have been using Docker for well over a year. But I am sure that kolie will see your comment, thank you.
[13:41:53] <janrinok> Can you point us to any documented (and proven) examples of Docker, Docker Compose or any other aspect of Docker that has shown this sort of behaviour?
[13:43:18] <donaldo> As I said I am beer in hand right now
[13:45:48] <donaldo> Docker, if implemented as a solution comprised of several containers working "together" where the "app" is the "whole", is very sensitive to "configuration drift"
[13:47:29] <janrinok> You statement is not evidence. If you are the only one to have seen it, it doesn't look very convincing. I have brought your comment to kolie's attention nevertheless.
[13:48:54] <janrinok> We know what docker is. You have specifically highlighted Docker Compose. If you are suggesting that we should go back to what we had over 18 months ago then you will need to have a better argument than that.
[13:51:00] <janrinok> Rehash (i.e. the site) operates in a single container.
[13:52:05] <donaldo> I don't think "going back" is good course for action
[13:52:35] <donaldo> I think the problem is in the Docker "orchestration"
[13:53:05] <janrinok> I can't find any other reports of this problem yet. I will keep looking.
[13:53:05] <donaldo> something is amiss there, I suspect
[13:54:32] <janrinok> OK, noted. kolie has several ways of testing this potential cause.
[13:54:35] <donaldo> Other reports? It's now a problem of Docker per-se, is a problem of Docker as-orchestrated-here
[13:54:43] <donaldo> *it's not
[14:17:31] <janrinok> Understood, thank you
[14:38:18] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[14:38:38] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@2a00:23c8:7480:zijk:qqxj:ktn:uxll:ikin] has joined #soylent
[15:07:03] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[15:26:01] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@2a00:23c8:7480:zijk:qqxj:ktn:uxll:ikin] has joined #soylent
[15:34:33] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[15:37:03] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@swuj40-76-144-95.range88-95.btcentralplus.com] has joined #soylent
[15:46:38] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[15:53:46] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@swuj40-76-144-95.range88-95.btcentralplus.com] has joined #soylent
[15:58:43] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[16:03:49] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@2a00:23c8:7480:zijk:qqxj:ktn:uxll:ikin] has joined #soylent
[17:17:03] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[17:29:42] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@swuj40-76-144-95.range88-95.btcentralplus.com] has joined #soylent
[17:30:21] -!- jje has quit [Quit: ZNC - https://znc.in]
[17:34:08] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[17:38:58] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@2a00:23c8:7480:zijk:qqxj:ktn:uxll:ikin] has joined #soylent
[17:43:43] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[17:45:08] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@2a00:23c8:7480:zijk:qqxj:ktn:uxll:ikin] has joined #soylent
[17:48:18] -!- jje [jje!jje@ddjffrbpjin.info] has joined #soylent
[17:49:33] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[17:52:02] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@swuj40-76-144-95.range88-95.btcentralplus.com] has joined #soylent
[17:56:13] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[18:06:39] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@2a00:23c8:7480:zijk:qqxj:ktn:uxll:ikin] has joined #soylent
[18:11:13] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[18:13:43] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@swuj40-76-144-95.range88-95.btcentralplus.com] has joined #soylent
[18:19:58] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[18:20:54] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@2a00:23c8:7480:zijk:qqxj:ktn:uxll:ikin] has joined #soylent
[18:54:04] -!- soylentil5 [soylentil5!~soylentil@2600:1005:b155:siqs:vxik:jpyn:yyvr:ypxk] has joined #soylent
[19:09:48] -!- soylentil5 has quit [Quit: Client closed]
[19:09:59] -!- soylentil5 [soylentil5!~soylentil@2600:1005:b155:siqs:vxik:jpyn:yyvr:ypxk] has joined #soylent
[19:12:23] -!- soylentil5 has quit [Client Quit]
[19:31:45] -!- nosler [nosler!~nosler@200.25.uk.sgi] has joined #soylent
[19:35:13] -!- nosler has quit [Client Quit]
[20:45:13] <c0lo> https://www.universetoday.com
[20:45:14] <systemd> ^ 03JWST Finds An Exoplanet Around A Pulsar Whose Atmosphere Is All Carbon
[20:47:55] <c0lo> =sub https://www.universetoday.com +https://arxiv.org/abs/2509.04558 +https://www.youtube.com/watch?v=KtzKyCTYZ8A
[20:48:01] <systemd> ✓ Sub-ccess! "03JWST Finds an Exoplanet Around a Pulsar Whose Atmosphere is All Carbon" (18p) -> https://soylentnews.org
[20:54:58] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[20:55:18] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@2a00:23c8:7480:zijk:qqxj:ktn:uxll:ikin] has joined #soylent
[20:59:33] -!- schestowitz[TR] has quit [Ping timeout: 245 seconds]
[21:02:02] -!- schestowitz[TR] [schestowitz[TR]!~schestowi@swuj40-76-144-95.range88-95.btcentralplus.com] has joined #soylent
[22:41:20] -!- Runaway [Runaway!~OldGuy@the.abyss.stares.back] has joined #soylent
[22:44:58] -!- Runaway1956 has quit [Ping timeout: 245 seconds]
[23:21:38] -!- progo has quit [Ping timeout: 245 seconds]