#soylent | Logs for 2024-03-04
« return
[03:53:42] -!- NotSanguine has quit [Quit: NotSanguine]
[04:06:27] -!- NotSanguine [NotSanguine!~Thunderbi@357-481-070-982.biz.spectrum.com] has joined #soylent
[07:37:24] -!- aristarchus [aristarchus!~aristarch@149.88.pw.voo] has joined #soylent
[07:39:05] <aristarchus> When is the community vote on the new bylaws to form the new non-profit corporation that will allow us to get rid of janrinok? Soon, eh? Or have we once again given up?
[07:45:15] -!- aristarchus has quit [Quit: Client closed]
[08:34:06] -!- Runaway1956 [Runaway1956!~OldGuy@the.abyss.stares.back] has joined #soylent
[09:34:17] <Ingar> and if the vote doesn't turn out the way he wanted, he'd call election fraud
[09:37:10] <janrinok> Hi Runaway1956, you just missed asi!
[09:37:17] <janrinok> *ari
[10:00:13] <Runaway1956> didn't miss anything then
[10:01:03] <Runaway1956> Up early today - gotta take a kid to surgery, will be gone most of the day
[10:27:58] -!- aristarchus [aristarchus!~aristarch@149.40.yr.zyk] has joined #soylent
[10:29:49] <aristarchus> Internal server error! Could it be that all the deleted comments are backing up, and causing the server error? Or is this the end? (Oh, I have no dog in the race, for a Soylentil erection. I really don't believe there will be one. But janrinok will tell us how it went.
[10:54:08] <Ingar> blame aristarchus
[10:55:51] <janrinok> best wishes for you 'kid' - I hope it is neither serious nor expensive.
[10:56:06] <aristarchus> You could do that, but it wasn't my policy.
[10:56:46] <janrinok> Site appears to be having a bit of a wobbly....
[10:57:28] <Ingar> janrinok: it worked fine this morning until ari apepared
[10:57:32] <Ingar> *appeared
[10:57:59] <janrinok> lol - I wouldn't give him the credit, he will start to believe that he is a leet hacker
[10:58:18] <janrinok> 500 Internal Server Error
[10:58:24] <Ingar> oh definitely not hacked, more like, evil aura
[10:58:56] <janrinok> I think you misspelled aroma
[10:59:17] <aristarchus> Blaming aristarchus is like being a Trump patriot.
[10:59:36] <Ingar> janrinok: I almost choked on my coffee laughing :)
[10:59:59] -!- dx3bydt3 [dx3bydt3!~|dx3bydt3@129.224.qpo.uzq] has joined #soylent
[11:00:56] <fab23> abuse of the reload button
[11:01:46] <Ingar> I bet ari tastes like artichoke, and I don't like that either
[11:04:32] -!- pTamok [pTamok!~pTamok@46.252.i.oxz] has joined #soylent
[11:04:54] <dx3bydt3> The site is giving an "Internal Server Error" I assume the admins are aware?
[11:06:02] <janrinok> yes - thanks
[11:16:13] <aristarchus> Ides of March approacheth.
[11:16:46] -!- aristarchus has quit [Quit: Client closed]
[11:17:35] <pTamok> It's worth looking at the irc logs to see if things are in hand. https://logs.sylnt.us One should also note that the admins being aware of a problem might not be useful if they do not have the permissions/access and technical ability to make the necessary changes to solve problems - especially if it involves monetary expenditure.
[11:17:36] <pTamok> I'm hoping the governance issues can be sorted out, and a sane system put in place for managing the infrastructure can be put in place.
[11:17:36] <systemd> ^ 03#Soylent | Logs
[11:31:21] <fab23> pTamok: are you also used to do ssh root@server directly?
[11:33:05] <pTamok> I'm sorry, I don't entirely understand your question.
[11:34:22] <fab23> As far as I know access to the servers at SN is complicated thourgh a jump host with user and then with user on the system and then an additional password to get to the root user
[11:38:41] <janrinok> The system of access to our servers is that we normally ssh into 1 specific server, and from there use kerberos to move around. Kerboros was broken during the 'update' and moving around is now quite difficult. We have lost many of our sys-admin staff over the last 6 months. Finding someone with the appropriate access who is also available is where we now have a problem.
[11:39:47] <fab23> :(
[11:40:13] <pTamok> Ahh, I see the confusion. I'm not an admin on soylent, so I'm not familiar with the actual technical details of the set-up.
[11:41:11] <chromas> https://www.bbc.co.uk
[11:41:12] <systemd> ^ 03Nissan accused of dumping its electric car pioneers
[11:41:32] <chromas> Cars made in the 2010s using 2G? That's like 2Gs older than what was current
[11:43:38] -!- pTamok [pTamok!~pTamok@46.252.i.oxz] has parted #soylent
[11:45:24] <chromas> That's okay; nobody else is an admin either ;)
[11:45:42] <fab23> Not sure, but I guess in Switzerland some of the mobile carrier already want 3G to be turned off, as far as I know 2G is already off on all 3 mobile carriers.
[11:46:24] <chromas> I can see turning off ancient 2G being a problem for also ancient installations, but 2G was already ancient when those cars were made
[11:46:55] <chromas> Saving a buck I guess
[11:47:05] <chromas> or whatever Brits use. Metric buxks
[11:47:09] <chromas> quid
[11:48:33] <chromas> How many quidlings to a dollarydoo?
[11:48:37] <fab23> a few years ago the telcos here where turning of classic POTS and migrated to VoIP (of course with their own ADSL/VDSL), at that point many elevator alarm buttons got converted to 2G or 3G :)
[11:49:58] <chromas> Modern installations would probably spin up an ec2 instance when the alarm button is pushed
[11:49:59] * fab23 stopped to care about currency, as CHF/EUR/USD are all the same, so its just e.g 1.- money now. :)
[11:50:06] <chromas> but would fail on a missing left_pad()
[11:50:50] * chromas deposits 35 moneys into the jar
[11:54:41] * fab23 does mention the backup irc channel at irc.libera.chat/6697 in ##soylentnews (yes with 2x #)
[11:56:48] <fab23> or for the lazy without own client: https://web.libera.chat
[11:56:49] <systemd> ^ 03Kiwi IRC
[13:16:37] -!- soylentil15 [soylentil15!~soylentil@2600:1700:bef0:knts:itwy:wwrr:ylml:lxki] has joined #soylent
[13:17:17] <drussell> I guess after wobbling along for the last few days, the site's backend finally fell over...
[13:17:50] <janrinok> it does look a bit that way
[13:21:50] -!- soylentil15 [soylentil15!~soylentil@2600:1700:bef0:knts:itwy:wwrr:ylml:lxki] has parted #soylent
[13:29:21] <janrinok> I have put out an email trying to find someone who can restart the site but I have not had any response. I cannot give an estimate on how long it will be before the site is recovered.
[13:32:03] <fab23> it is probably still early in the morning in the west :)
[13:37:45] <janrinok> Monday is never a good day - everyone has work to do to get the week started
[13:40:25] -!- drussell has quit [Ping timeout: 252 seconds]
[13:41:23] -!- drussell [drussell!~drussell@a4627691kd3g1a0z4.wk.shawcable.net] has joined #soylent
[13:55:47] <Ingar> as far as mondays go, I'm firmly in Garfield's camp
[14:00:30] <fab23> I liked the one from last Monday: https://assets.amuniversal.com
[14:21:37] -!- Ox0000 [Ox0000!~Ox0000@ysig-694-71-0-819.washdc.fios.verizon.net] has joined #soylent
[14:22:41] <Ox0000> Anything going on with the main soylentnews.org site? I keep getting 500's...
[14:23:50] <fab23> its down and staff knows
[14:24:16] <Ox0000> ack
[14:24:33] <Ox0000> That second part was why I came here :) . Thank you!
[14:27:48] <Ox0000> Out of interest, does the staff have anyone with devops experience or background?
[14:28:17] <fab23> the question should be, do they have access? :(
[14:28:39] <Ox0000> Oh, it's that bad?
[14:30:09] <Ox0000> I wasn't asking to admonish about their potential lack of such experience, just wondering about it
[14:30:13] <requerdanos> There are few administrators these days with such knowledge + access, I wouldn't say it's "that bad" but it's less than ideal of course.
[14:30:50] <Ox0000> Who is working on the development of the new site? Is that still a thing?
[14:31:06] -!- pTamok [pTamok!~pTamok@46.252.i.oxz] has joined #soylent
[14:32:22] -!- pTamok [pTamok!~pTamok@46.252.i.oxz] has parted #soylent
[15:12:42] <janrinok> We do have some access, but we have also lost a lot of staff. The site needs extensive rebuilding after the 'upgrade', which was not well documented. Nobody who is still interested in the site knows which software is on which server.
[15:13:41] <janrinok> A lot of the homegrown stuff that took care of the routine has been lost (I'm not going to explain that here - but it is not good). So tasks which should be relatively simple to do are not.
[15:15:49] <janrinok> Finally, after 8 months of meetings, many staff have become disillusioned with what was being proposed and have simply lost interest in giving their support. We have new sysadmins lined up for the new site but we are waiting for a response from the US IRS so that we can create the business.
[15:17:31] <janrinok> What few staff we now have all have real lives to live and Mondays are never a good day - they have businesses which they have to tend to as that puts food on the table for their families.
[15:18:33] <janrinok> Ox0000, ^^^
[15:19:43] -!- Freeman [Freeman!~Freeman_S@CanHazVHOST/Freeman] has joined #soylent
[15:20:16] <janrinok> hi - still no good news yet. Nobody is responding to the email that I sent out
[15:22:22] <Freeman> Hello. :-)
[15:22:29] <Freeman> I just read the logs.
[15:22:48] <Freeman> Hopefully things are resolved for the positive!
[15:23:23] <janrinok> Not sure what you are reading but the site is still down....
[15:23:42] <Freeman> Yeah, I was just checking to see about the site being down. xD
[15:24:04] <janrinok> I have sent out an email but no response from anyone who can do anything
[15:24:37] <janrinok> Even a simple reboot would be better than nothing at all
[15:26:36] <janrinok> we have stories queued for at least 36 hours so it should have content if we can just get it active again.
[15:33:01] <Freeman> Thanks for kicking the tires! Hopefully things are up and going soon.
[15:33:08] -!- Freeman has quit [Quit: Leaving]
[15:38:15] <Ox0000> janrinok thank you for the detailed explanation, much appreciated
[15:39:02] -!- Ox0000 has quit [Quit: Client closed]
[16:35:27] -!- soylentil71 [soylentil71!~soylentil@116.75.nll.tpg] has joined #soylent
[16:38:59] -!- soylentil71 has quit [Client Quit]
[17:01:41] -!- soylentil57 [soylentil57!~soylentil@gff-60z355b2.dyn.optonline.net] has joined #soylent
[17:02:47] <soylentil57> Just an FYI - in case you don't know (which you probably do) - soylentnews.org is throwing a 500 error
[17:05:11] <fab23> hm, janrinok it may be a good idea to mention the outage in the channel topic :)
[17:05:16] <drussell> Yeah, thanks. Staff knows. It's been stuck for about 6h now, we're just waiting for one of the people with access to go in and prod it.
[17:05:20] <fab23> soylentil57: the staff knows
[17:05:50] <drussell> I was just going to mention that is anyone has ops, they could temporarily update the topic.
[17:05:58] <soylentil57> OK. Good luck wrangling with it. Have a good day (or at least better than being down)
[17:06:24] <drussell> Indeed. Thanks. :)
[17:06:32] -!- soylentil57 has quit [Client Quit]
[17:14:20] <janrinok> .op
[17:14:20] -!- mode/#soylent [+o janrinok] by Imogen
[17:14:54] janrinok changed topic of #soylent to: The Site is Currently DOWN - please be patient.
[17:15:02] <janrinok> .deop
[17:15:02] -!- mode/#soylent [-o janrinok] by Imogen
[17:22:59] -!- halibut has quit [Quit: Timeout]
[17:28:20] -!- halibut [halibut!~halibut@CanHazVHOST/halibut] has joined #soylent
[17:32:31] <fab23> https://www.geekculture.com
[17:32:32] <systemd> ^ 03The Chatbot Therapist.
[17:39:35] <requerdanos> Working. nice.
[17:44:03] <drussell> It's janrinok's fault... He changed the topic here, so the site started to work again... :)
[17:45:16] <requerdanos> .op
[17:45:16] -!- mode/#soylent [+o requerdanos] by Imogen
[17:45:25] requerdanos changed topic of #soylent to: SN Main Channel | Keep discussions civil | https://soylentnews.org | Impersonating another user's nick is forbidden | Some PISG charts: https://stats.sylnt.us | This channel IS logged and publicly displayed here https://logs.sylnt.us
[17:45:30] <requerdanos> .deop
[17:45:30] -!- mode/#soylent [-o requerdanos] by Imogen
[17:46:15] <requerdanos> checking... yep, still works :)
[17:50:31] <janrinok> thanks requerdanos - I had just sat down to do that :)
[17:50:53] <requerdanos> just a little celebration of happiness.
[17:51:15] <janrinok> I am quite please too. I wonder if anyone knows what the problem actually was...?
[17:52:02] <requerdanos> I am hoping there was a restart of some sort (of the httpd, of the server, something) and not just random fluctuations in the aether.
[17:52:44] <janrinok> Maybe, but a restart doesn't necessarily correct the underlying cause.
[17:53:25] <requerdanos> wonder what the errorlog says
[17:57:16] <requerdanos> not working again. I promise to keep my hands off the topic this time.
[18:00:34] -!- dx3bydt3 has quit [Ping timeout: 252 seconds]
[18:01:03] -!- dx3bydt3 [dx3bydt3!~|dx3bydt3@129.224.qpo.uzq] has joined #soylent
[18:01:16] <janrinok> down - up - down again
[18:01:56] <drussell> Did anyone actually DO anything, or did it just happen to respond for a moment or two?
[18:02:21] <janrinok> I am not aware of anyone doing anything. There are no comms elsewhere that I have seen.
[18:03:14] <janrinok> drussell, its life Jim, but not as we know it....
[18:03:50] <drussell> Yeah, that's kinda what I figured... It likely just happened to respond for a bit, especially since it has been up and down randomly for the past several days...
[18:04:09] <drussell> janrinok, LOL :)
[18:16:02] <janrinok> OK, it seems that when the site was 'updated' they didn't include a check for the disk full when doing backups. It was MYSQL throwing a fit because it had run out of room. We had fixed this problem once!
[18:16:30] <janrinok> My thanks to audioguy for sorting it out.
[18:21:20] <drussell> That's what caused the database corruption when the site lost months of data, back even before the "upgrade"
[18:22:02] <janrinok> Yes, it was. And as a result we had fixed it. The fix didn't survive the update....
[18:31:44] <janrinok> The update saved us quite a bit of money by reducing the number of servers. The software that we had produced to manage certain things knew where the various elements were running. The restructuring changed a lot of that but the software that had be produced over the previous 9 years wasn't updated. As I understand it (and I am NOT a sys-admin) the software was swapped across to the new servers but it would no longer work. The sys-admins were trying to get
[18:31:45] <janrinok> this resolved early last year but were suddenly blocked from the system.
[18:33:09] <janrinok> We need to go back to a basic system and then start adding the enhancements which are nice but not essential. Plus it all needs documenting and, where possible, automating too.
[18:34:51] <janrinok> No documentation in the tech-wiki was updated, so we had nowhere to look to find out what the new structure was.
[19:18:13] -!- Ox0000 [Ox0000!~Ox0000@ysig-694-71-0-819.washdc.fios.verizon.net] has joined #soylent
[19:20:26] -!- Ox0000 has quit [Client Quit]
[19:40:00] <Bytram> Sorry I missed the "fun". Unfortunately, though I have the necessary privs, I've forgotten a *lot* and can't remember much at all. :(
[19:41:26] <Bytram> Whoever restarted the system, thank you! Now how can *I* do that?
[19:45:19] <Bytram> The system *could* have been restarted ~3 hours ago if I knew how.
[19:59:56] <Bytram> ----
[20:03:09] <Ingar> sudo reboot
[20:34:01] <Ingar> or if there is a management console, from there
[20:34:08] <Ingar> (the hard way)
[20:34:18] <requerdanos> control-alt-big red switch
[20:46:23] <Bytram> Ingar: Thanks so much!
[20:46:36] <Bytram> Regurgitator: LOL!
[20:47:08] <requerdanos> :)
[20:52:19] <Bytram> The BRS was an often-used option way back when. (Some would argue is has not yet made obsolete!)
[20:52:43] <Bytram> <grin>
[20:53:23] <requerdanos> the big red switch is plug and play, no drivers required
[20:57:58] <Bytram> LOL! I doubt it will be entirely obsoleted :^)
[20:59:37] <Bytram> Speaking of which, it looks I'll need to install some updates :/
[21:00:00] <Bytram> (I'll do it later)
[21:02:14] <fab23> Bytram: as I understood the disk of the database run full, so reboot wont resolve it.
[21:06:14] <Bytram> fab23: Understood. That's in THIS case. I needed the: sudo reboot in the other, more common scenarios. :)
[21:08:47] <fab23> reboot is kind of the Windows way of "fixing" problems :)
[21:10:26] <requerdanos> That's probably because in windows, there are many things that are too arcane or deeply-imbedded to fix individually, but a reboot puts them right; in the [L][inu][uni]x world there is usually a tool or two to put things right, but the reboot still works in many cases to resolve issues.
[21:11:14] <requerdanos> sometimes a reboot can be a faster, less impressive fix than studying something for an hour and fixing it on the fly, maintaining one's uptime
[21:13:13] <fab23> it really depends on the server and what other services would be impacted with a reboot
[21:16:40] <requerdanos> Yes it does. there's something to be said for good uptime.
[21:57:04] -!- drussell has quit [Ping timeout: 252 seconds]
[21:57:57] -!- drussell [drussell!~drussell@a4627691kd3g1a0z4.wk.shawcable.net] has joined #soylent
[22:49:49] <chromas> "GIMP 3.0 will be released, and everyone will be in awe. At once, all of humanity will throw off the shackles of proprietary software, and Richard Stallman's vision will finally be fulfilled. Definitely before May 9."