Slrpnk.net is unexpectatly back online 🥳
After a month and a half downtime all the users will have moved on to other instances. This is essentially a death sentence for the instance and its communities.
You underestimate the userbase. I made a temp account in the mean time, but we are a hyper tight knit community. We will probably lose accounts - no question - but the core userbase will return
Time to come back, it is working again. Lucky break 😅
Thanks for the speed and the work !
The local communities are such a great mix of uplifting and informative posts that I am sure to return.
I dont think it will be. I’ve had two accounts for a while to deal with unexpected issues & will happily return to my slrpnk account once it’s back up and running :)
Plus for the communities, people who were subscribed to them before will likely stay subscribed & once the instance is back up the posts will be in their feeds without an issue.
Being able to accommodate issues like this is one of the major upsides to a decentralized platform.
Alt text:
Slrpnk.net is currently offline due to an unforseen hardware failure in combination with the main system-administrators having no physical access to the server location until mid July due to work or summer-holiday related travels.
We are very sorry for this unforseen down-time, but slrpnk.net will return for sure and we already have some plans for a nice relaunch, so stay tuned!
“unforseen hardware failure”
somebody didn’t check their backups!
The backups are most likely fine 😅
yes, that’s why it’s been down for this long…
Why is this a jpg?
Just be happy that the jpg is not embedded in docx
Itself embedded in an auto extractable zip executable.
Archived once more in RAR, just for sure.
And then tar.bz2 on top so that no basic users of sidewalks can ever open it.
ikr
svg, lossless webp, or png are way better options for an image of text
JPEG XL would be the true future format, unfortunately Google killed it
Not yet they haven’t
They’re doing their best though
Because fuck blind people, that’s why
What is so special about piefed? I see a few communities moving there. The interface looks different from the original lemmy interface.
For regular users there is not a huge difference, but the web-interface is significantly faster and you can subscribe to topics that combine multiple communities. The disadvantage is less mobile app support. Right now only Interstellar supports Piefed.
The real advantage is for moderators and admins, as the Piefed developers actually listened to community feedback and implemented a lot of nice moderation features that require bots or annoying work-arounds on Lemmy.
And there are a few technical differences that make Piefed easier to administrate and troubleshoot from a sysadmin perspective.
Our sysadmin explained some technical advantages here: https://feddit.org/post/13613230/7063696
Postgres slowing factor
I’m pretty sure one of the best optimised free/libre DBMS’s is faster than Python 💀EDIT: skimmed it wrongly, see corrections below
Also no one knows how Piefed scales, since it only has like 350 MAU
Reread it. They’re saying “The slowest part is postgres, so even if python is slower than rust it won’t make a difference.”
That is, in theory. In practice, it can make a big difference. Source: Worked professionally with large services built in Python and Rust.
Yes, I also suspect that at a certain scale it would make a difference, but with a single Postgres database and no advanced clustering or so, the real-life performance metrics of a medium sized Lemmy instance strongly point to Postgres being the bottleneck and not the Python or Rust codebase of the rest of the software.
Don’t pull a feddit.de on people, alright?
What exactly happened there? It was the big thing, then I didn’t use it for a month or so and then it was gone.
The admin basically ran it as a one man show with only one other admin who had very limited privileges. He then went on a “business trip” or workaction or longterm vacation - there were different stories. Anyway, the database went belly up, the other admin couldn’t do a thing and none could contact the admin. There are some rumours that he wasn’t who he claimed he was and actually was a Chinese national who simply returned home, but who knows that. As a matter of fact none had any meaningful contact with him for months then and it appears he did not return. (But is alive)
A Austrian NGO who amongst others does host some mastodon instances,etc. took over and now feddit.org is on a very productive, professional and transparent level.
Thanks for the summary! That sounds freaky!
Well, the trade-off between trusting a huge corporation or a single dude on the internet.
XKCD #2347
Ya gotta give a line from it or something, I don’t just see numbers and know what they are.
from context, it’s probably the “single dude in nebraska holding up the entire internet” one
e: holy shit I got the state right
Thank you so much for the update - I hadn’t had access for about a week and was getting worried. Couldn’t reach them on XMPP either.
Yeah, the xmpp server is down too. That is something that bugs me quite a bit and I will probably move that one to an external small VPS to retain a more secure backup communication channel.
Assuming slrpnk stands for solar punk - that produces funny little theories in my mind about the server location.
Their servers are located in Portugal I believe and do indeed run on solar power! They gave details of their set up on their wiki page but… that was on the server that’s gone down so… have a read in July I guess?
it’s on archive.org
Server hosting This is a work in progress Slrpnk.net is hosted by F-hub.org, a volunteer driven and non-commercial effort to host federated community services in a resource efficient and ethical way. F-hub.org grew out of a community of open-source game developers (freegamedev.net) that has existed since the early 2000s. All the servers are operated and maintained (as a hobby) by one of the founding members and are currently located on the Azores in Portugal. Connectivity is provided through a dedicated high-speed fibreglass connection. Hardware The F-hub.org servers are based on second-hand consumer PC and data-centre hardware, but optimized for low energy consumption. Battery backup power is provided and data is stored with triple redundancy (off-site backups are still a work in progress). Electricity is currently provided by the utility grid (about 60% green-energy, mostly from a geothermal power-plant) but it is a work in progress to upgrade to a on-site solar PV system for near 100% renewable power. Slrpnk.net itself currently runs on a dedicated 6th gen Intel CPU server with 8 threads, 16GB RAM and SATA SSD storage in raid configuration. Image uploads are stored on a large HDD raid array. All data is snapshotted and transferred for backup to a second shared server on daily basis. External services Given how difficult it is to have outgoing email accepted by the large email providers, all emails are currently routed through an external SMTP server hosted by OVH in France (same as the domain registration and DNS routing). There are backup plans to move the servers to a co-location data-center owned by Altice, should this be required for scaling. However, monthly rent of rack-space is quite costly (starts at around 300€/month), thus this will require a substantial regular donation base to be possible.I think they’re located in the Azores, and yes, running their own servers on solar (of course) and other renewable energy. Coolest instance out there.
Wondering if a volunteer could go to the physical location necessary to restore service. If it’s in Athens Greece, I can make the trip.
Edit: The server is located in Azores, the flight would cost 415eur, I cannot afford that, maybe someone in Portugal could help?
Is it in a data centre or someone’s house? If the latter, would they let a stranger in?
Surely they would need a backup and replicate db to so in case of hardware failure they switch over.
Sounds like they could improve their setup.
Too much of a single point of failure.
Slrpnk.net admin here.
The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.
There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time…
An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.
I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I’ll know more when I have physical access again.
Is it run out of a private residence? How could it happen if it’s in a real data center…?
It is run from a private residence in the DIY punk spririt (and this also allows us to run of a local solar PV system), but more or less the same would happen if you rent rack-space in a “real” data-center. Only if you rent a managed server or VPS someone else will be responsible to fix such issue and this comes at a significantly higher cost at the scale we operate at (slrpnk is part of a bigger project that also hosts other services).
If you rent rack space, you can at least call the DC to send someone to reboot it or smt.
Sure, but they will not replace hardware and set up remote access for you, which seems like this would have been required in this case.
