#soylent | Logs for 2026-08-30
« return
[02:25:11] -!- Loggie [Loggie!Loggie@Soylent/BotArmy] has joined #soylent
[02:25:17] -!- bender [bender!bot@Soylent/Bot/Bender] has joined #soylent
[02:25:32] -!- jje [jje!jje@ddjffrbpjin.info] has joined #soylent
[02:25:38] -!- Deucalion [Deucalion!~Fluff@Soylent/Staff/IRC/juggs] has joined #soylent
[02:25:39] -!- mode/#soylent [+v Deucalion] by Imogen
[02:25:42] -!- Bytram [Bytram!~Bytram@Soylent/Staff/Developer/martyb] has joined #soylent
[02:25:42] -!- mode/#soylent [+v Bytram] by Imogen
[02:25:48] -!- prg [prg!~prg@ryfhuvb.de] has joined #soylent
[02:25:48] -!- SSLbits [SSLbits!~SSLbits@CanHazVHOST/SSLbits] has joined #soylent
[02:25:49] -!- Xyem [Xyem!~xyem@Soylent/Staff/Developer/Xyem] has joined #soylent
[02:25:49] -!- mode/#soylent [+v Xyem] by Imogen
[02:26:43] -!- Fnord666 [Fnord666!~Fnord666@Soylent/Staff/Editor/Fnord666] has joined #soylent
[02:26:43] -!- mode/#soylent [+v Fnord666] by Imogen
[02:27:07] -!- janrinok [janrinok!~janrinok@Soylent/Staff/Editor/janrinok] has joined #soylent
[02:27:07] -!- mode/#soylent [+v janrinok] by Imogen
[02:27:09] -!- fab23 [fab23!~fabian@ysxbxfyv.wenks.ch] has joined #soylent
[02:28:28] -!- inz [inz!~inz@wbi.fi] has joined #soylent
[02:29:13] -!- progo [progo!~progo@eegc-73-589-96-43.nwrknj.fios.verizon.net] has joined #soylent
[02:39:49] -!- AlwaysNever [AlwaysNever!~donaldo@315.38.1.669.dynamic.jazztel.es] has joined #soylent
[02:48:39] -!- halibut [halibut!~halibut@CanHazVHOST/halibut] has joined #soylent
[03:02:53] -!- ted-ious [ted-ious!~tedious@ted.ious] has joined #soylent
[03:04:41] -!- Core6508 [Core6508!~Core6508@Soylent/Staff/Management/kolie] has joined #soylent
[03:04:41] -!- mode/#soylent [+o Core6508] by Imogen
[03:04:50] <Core6508> hellllo
[03:07:50] -!- lld [lld!~lld@llvm.link.editor] has joined #soylent
[03:38:31] -!- Core6508 has quit [Ping timeout: 265 seconds]
[07:12:54] -!- Ingar [Ingar!~ingar@2001:bc8:1640:mwzr:pxpv:wq:jjgg:moil] has joined #soylent
[07:13:14] <Ingar> In Soylent Veritas
[07:27:45] -!- halibut has quit [Quit: Timeout]
[07:34:11] -!- halibut [halibut!~halibut@CanHazVHOST/halibut] has joined #soylent
[07:38:17] <janrinok> kolie has been working hard to fix the problem. The main site seems to be ok now, but lots of the peripheral stuff is still not working, but it is less important.
[07:43:30] <Ingar> Soylent runs on blood, sweat and kolie.
[07:43:43] <janrinok> true
[07:50:54] -!- chromas [chromas!~chromas@Soylent/Staph/Infector/chromas] has joined #soylent
[07:50:54] -!- mode/#soylent [+v chromas] by Imogen
[13:23:55] <AlwaysNever> what was the cause of the latest breakage?
[13:26:36] <AlwaysNever> today I was feeling retro, and bought the Sunday's newspaper together with its supplement the "weekend magazine", the last time I did was probably 20 years ago
[13:27:12] <AlwaysNever> 3,50 Euros, by the way
[14:00:24] <janrinok> AlwaysNever, For all the dirty details, see LiberaChat #soylentnews. Basically the site had been ransomed by hackers.
[14:10:03] <AlwaysNever> I visited the "official" SN presence un X.com, but it was latest updated on 2025 - I will try to dig info in LiberaChat #soylentnews, but will I be able to see past chats?
[14:25:02] <fab23> probably not
[14:26:34] <AlwaysNever> indeed not :-(
[14:29:15] <fab23> AlwaysNever: see query
[14:31:21] <AlwaysNever> fab23: thanks a lot!
[14:32:33] <fab23> you're welcome
[14:50:07] <AlwaysNever> that was an interesting read!
[14:51:54] <AlwaysNever> so SN was on a shared, Proxmox-based, hosting; some other tenant got hacked, then the hacker escalated privileges into the host, and encrypted the whole host demanding ramson.
[14:52:42] <AlwaysNever> Linux security is beginning to look like Windows XP
[14:54:26] <janrinok> fab23, where is the 'query' that you are referring to? I also missed quite a bit of the discussion.
[14:56:09] <AlwaysNever> that the backups survived is a feat, when faced with a root-level compromise. Kudos for that resilience!
[14:57:09] <AlwaysNever> I would like a lessons learned about this, with a note about the backup estrategy (but a careful note, as that must remain "obscure" for the hackers).
[14:58:18] <AlwaysNever> specifically, I would like to know whether the surviving backup was a Proxomox-native one, or the Veeam-based one.
[15:07:32] <AlwaysNever> because we at work are evaluating Proxmox as an exit strategy out from VMware, so I am interested on what is the best backup strategy for Proxmox
[16:42:07] -!- kolie [kolie!~kolie@Soylent/Staff/Management/kolie] has joined #soylent
[16:42:07] -!- mode/#soylent [+o kolie] by Imogen
[16:42:09] <kolie> sup.
[16:42:27] <janrinok> hi kolie
[16:48:41] <AlwaysNever> hello kolie
[16:50:16] <AlwaysNever> kolie: did the hackers move laterally to other hosts in your infrastructure?
[17:04:32] <kolie> no.
[17:05:32] <kolie> what it appears like is one of four vms on that device was used to jailbreak to the host and then target all vm disks on the host, they were encrytpted so they wouldnt bppt
[17:11:22] <kolie> Janrinok - you are not able to login because the VM cluster I am using is a different vm host.
[17:16:15] <kolie> I'm restoring the staff box
[17:25:55] <kolie> I don't think the VMs were accessed but they did crypt them.
[17:50:23] <AlwaysNever> it's good to know that the attack was contained to a single host.
[17:51:20] <AlwaysNever> kolie: the backups are the Proxmox-native ones, or the Veeam-based ones?
[17:52:18] <kolie> its pbs.
[17:52:46] <kolie> we dont have veeam, we had an offsite remote host we rsync'd stuff too, but thats gone.
[19:14:11] <AlwaysNever> do you deploy PBS as a physical host? the backup storage is local, iSCSI/NAS of something else?
[19:14:28] <AlwaysNever> *or something else?
[20:00:53] <kolie> There is a seperate machine running PBS in the same datacenter.
[20:18:55] <AlwaysNever> Aren't you worried that a compromise of the Proxmox Datacenter admin credentials could jeopardize the integrity of the PBS machine, provided that the PBS machine is part of the Datacenter?
[20:21:17] <kolie> Every proxmox machine has its own credentials, and I don't suspect a compromise of any of the admin credentials.
[20:22:03] <kolie> None of the logs in the datacenter show any evidence of anything outsode of that one host
[20:23:24] <kolie> So no, I don't believe the PBS host to be compromised.
[20:23:54] <kolie> It's not a virtualized host, there is no path for vm escalation on it, and it doesn't share credentials with anything else.
[20:24:18] <kolie> The logs on it are intact and don't even show that the threat actor is awware of it or even attempted to compromise it.
[20:28:05] <AlwaysNever> very nice, that's great!
[20:29:06] <kolie> Yea I feel pretty good that they are off the network, It's a pretty good chance pve-68 is clear of all persistence and threat activity, and there is no active network connections anywhere there should be across the board.
[20:29:21] <kolie> I'm still going to wipe the host
[20:29:37] <kolie> Which I can do some time wednesday time permitting bit of a busy week.
[20:31:16] <AlwaysNever> I hope your court affair goes well that next Twesday
[20:31:42] <kolie> I'm dealing with fresh accusations as of this morning, so fun.
[20:32:43] <AlwaysNever> why attempting to build a family has to be such a risk-ladden endeavour?
[20:33:29] <kolie> Seperation started in late 2021. Still in court, nothing decided at all, so fun.