Is there a quick way to know which instance might need mods?


As per title.
I've been wondering about this. How can we help when someone managing a community/instance can't find help locally, for whatever reason? Something like an "help wanted" board. Even if it's for one month, three months, etc.

I've been an admin for a 8-9k+ users Discord server for years, but I love Lemmy so much I'd like to see if I can be somehow useful here. I don't think I'm power tripping at all, but have zero tolerance for harassment, racism or sexism, bigotry and honestly want to keep neocons and bigots/trolls out of Lemmy as long as it's humanly possible. Whenever in doubt, I generally abstain from using any moderation power and talk with other mods/users to find the best course of action.

The situation I imagined in the first paragraph did happen to me. Managing the server alone was draining my mental health, and I couldn't find anyone to help with all I had to do. Then someone wrote to me, we had lengthy discussions about the rules and philosophy of the server and we've been co-managing it together for years. Lemm.ee closing down made me think about "how to help where and when it's needed"?

I'm posting in this community in hope that other users may be wondering about it and may find any answer useful.

[Multiple edits, original post written quite late, judging from most replies, I failed to phrase correctly what I had in mind]

This entry was edited (1 month ago)
in reply to Cadenza

I'm wondering if the fediverse in general (but especially Lemmy) would benefit from some kind of "fediverse help wanted" board for moderators, donations, technical help, etc.

edit: People who run a Lemmy/Mbin/Piefed/etc instance might be hanging out in the Matrix chat room for the software they are running; that might already serve this purpose.

This entry was edited (1 month ago)

The Indian governmental incursion must've been part of the original script as all the actors were perfectly lined up and ready to go right at the start.

So per script we're also supposed to be in an entirely different place where novelty injections are now accepted as perfectly normal and fine. Everyone has already come to accept that a deadly invisible pathogen gone 'endemic' so obviously we're going to need plenty of government help going forward willingly or unwilling. vaxpass and digital id been in place for several years and BTC is everyone's favourite novelty currency.

That's the world in which all this was supposed to have happened. But it's not so obviously not an ideal situation for the people running this show. They're essentially just trying out the new arrangement to see how well it goes down. So far not many complaining which means the fluoride supplement is working a treat.

in reply to son of sam harris

It's likely that Americans who'd want to radically reshape the US (and the same principal will hold in other nations) would make use of foreign and dual nationality individuals because those people will have less misgiving about following radical paths.

For instance, in the UK we've had government ministers who were likely picked up by our security services and selected for power at a young age from abroad. Those people won't care about preserving a way of life.

There's likely political value here in portraying this as an external attack, but is it _only_ that? Could they get away with this if the most powerful local forces did not see this as beneficial to them? I doubt it.

Although much smaller in scale, an analogy here would be the role played by the Israelis in taking down Corbyn for installing Starmer as leader of the UK Labour party. I expect the UK security services approved this on behalf of their backers: co-opting foreign nationals and embassy staff then provides a kind of plausible deniability.

And perhaps that deniability is why the script was written like this.

This entry was edited (1 month ago)
Unknown parent

pleroma - Link to source

Pippin

He's good. I admit to only having listened to a few of his podcasts. I can't say many others have heard of him. I think one other person mentioned him to me at work one time, possibly around 2021.

I read Alford and Secker's National Security Cinema. They totally missed the transhumanism angle on Avatar, regarding it as a pacifist film which slipped through the net.

Films are Not Your Friends, which used to be on Youtube was my eye-opener on the scale of the security state interference in movies and TV.

He did a good podcast on the Manchester / Aria Grande event. Contrasts with the angle taken by Richard D Hall and Iain Davis, though not necessarily contradictory.

US can not do a damn thing about Iran nuclear industry en.mehrnews.com/news/232657/US…
in reply to Viss

@ashleygjovik ive used gpt to write bash scripts. ive never tried to get it to like, "do writing" before. it seems like it would spew weird hallucinatory nonsense.

pretty much all ive used it for is like, dumb stock trady bullshits, and python/bash. occasionally some linux config stuff and to help me find competing products, but never more than like, a paragraphs worth of thinking. any more and it goes haywire real fast.

in reply to Viss

It works quite well if you design very strict instructions of what it needs to do, and how it needs to do it. I have used it to do complex analysis and drafting, but I have to basically have to treat it like i'm developing a little app and give it a tiny universe to operate in, and make it run QA & failure analysis on itself. Once I set that up, it automates a ton of stuff, but it cannot be left unsupervised.

“Without Struggle, There is No Victory”: Panamanians Stand Up to Their Government and the US libya360.wordpress.com/2025/06…

Shipwreckedcrew

Judge James Ho of the Fifth Circuit Will Likely Be President Trump's First Supreme Court Nominee -- And Here's Why

It is quite unusual for a Circuit Court Judge to openly take shots at a 7 Justice Supreme Court majority -- and be correct in doing so.

open.substack.com/pub/shipwrec…

:john: "June is graduation month and the perfect time to grab a No Agenda PhD in Media Deconstruction. This offer is back by popular demand and gets you a diploma, an Executive Producership AND a Knighthood all-at-once. More importantly it helps the show a lot and makes you feels good. Click here and join the ranks."

:madam: "Only $1000!"

tpsconsulting.blogspot.com/202…

#NewAgenda #FaggotTourney

This entry was edited (1 month ago)

The planet can support billions but it can’t support billionaires.

Enough is enough, it’s time to resist. It’s time to stop the billionaire takeover which threatens the future of life on earth.

Sign our pledge to fight the billionaire takeover. Add your name now: bit.ly/4dFaD5U

This NASA/ESA Hubble Space Telescope image features a sparkling cloudscape from one of the Milky Way’s galactic neighbors, a dwarf galaxy called the Large Magellanic Cloud. Located 160,000 light-years away in the constellations Dorado and Mensa, the Large Magellanic Cloud is the largest of the Milky Way’s many small satellite galaxies. This view of dusty gas clouds […]

<rant>

no, don't tell how to do this. i am ranting. you can't "fix" it.

apple just continues to suck worse and worse. they have apparently fired anyone who has ever had any expertise in UI/UX.

i now have to use the *search* function in the control panel to find the printer. really?

i have used apple since the mac+ finder/system days, unix for years, windows since 3.11. i have suffered through all sorts of system control panels, good and bad.

the current apple OS is amongst the worst.

i can only imagine what will happen as more and more folks with what little UI/UX they still have are replaced with AI.

i truly pity anyone that needs actual accessibility features.

grrr....

Unknown parent

mastodon - Link to source

Paul_IPv6

@milkman76

ah, server names. ;)

so many schemes. simpsons. south park. tolkien. creatures that fought godzilla (winning name "bambi"). worst was "greek liturgical terms".

one of my first jobs had a test lab and the engineers kept renaming the servers constantly, breaking other engineer scripts, etc.

the reason i now know the nato phonetic alphabet was that the engineering director got tired of this and decreed that we'd name all servers from said alphabet in order, starting with alpha. we got up to tango before i changed jobs.

There are several reasons why I hesitate to identify as a Christian. One of the big ones is because I’m not entirely sure I believe in the existence of God, or any gods. Thus far, my approach has been to take Pascal’s wager, but that’s not a viable long-term solution to the problem.

The other big one is, well, other Christians.

I once saw a bumper sticker that read, “Jesus, save me from your followers.” At the time, I was a far more committed Christian, and I found myself offended at the notion. But now? I watch in horror as white evangelicals and conservative Catholics embrace Donald Trump, MAGA, and the moral rot they represent. Consider Sen. Joni Ernst of Iowa, whose response to the potential Medicare/Medicaid cuts was, “We’re all going to die.” Later, she followed up with, “For those who would like to see eternal and everlasting life, I encourage you to embrace my lord and savior, Jesus Christ.”

With no due respect, senator, that’s not the Jesus I read about in the Gospel. The Jesus I read about in Matthew, Mark, Luke, and John (yes, I read all four in their entirety) repeatedly stressed the importance of loving your neighbor. (And there’s a preponderance of evidence that by neighbor, Jesus meant EVERYONE, not just the people you like/love. See Matthew 5:46-48 for one example.) He even referred to it as the greatest commandment. Evangelicals love to quote John 3:16, but maybe they should spend more time with John 13:34-35: “A new commandment I give to you, that you love one another: just as I have loved you, you also are to love one another. By this, all people will know that you are my disciples, if you have love for one another.”

Rev. Dr. William Barber II understood the assignment. That’s why he’s been protesting and praying in the Capitol Rotunda, and why he’s doing everything in his power to stop Trump’s so-called “big, beautiful bill”. He rightly points out that there’s nothing Christian about the bill, which takes from the poor and gives to the rich, in clear violation of Matthew 6:24: “No one can serve two masters, for either he will hate the one and love the other, or he will be devoted to one and despise the other. You cannot serve God and money.”

theguardian.com/us-news/2025/j…

in reply to Michelle Catherine Marcó

the idea of a god has never made any emotional or logical sense to me. But I don't dislike the teachings of (maybe a real person, maybe a combination of people or not) Jesus or other such "gurus" of the religious world. Ultimately I note that there are mean spirited and good hearted people and that their religion, or lack of it, has nothing to do with how they are and how they behave.

Simple, cheap, non toxic and ignored - Dr. John Campbell

youtube.com/watch?v=t1JpkNyIgo…

Professor of Oncology, Angus Dalgleish gives vital information on a repurposed drug he uses for his cancer patients, given at a very low dose. This is Low Dose Naltrexone.

The Hearts We Didn't Know We Were Breaking: What We're Learning About COVID's Long Shadow on Our Children

buzzsprout.com/2405788/episode…

Like most comforting stories we tell ourselves during crises, this one was both partially true and dangerously incomplete.

The kind of study that's too big to dismiss, too methodical to wave away, too urgent to ignore.

#LongCOVID #COVIDKids #PediatricHealth #HeartHealth #MedicalResearch #PostCOVID #ChildHealth #PublicHealth #COVIDStudy #HealthcareAwareness

PSA: Exporting and importing your Lemmy account settings will also include your saved posts and comments


This entry was edited (1 month ago)

mRNA and pregnant women - Dr. John Campbell

youtube.com/watch?v=426XoTQLc_…

Check comments below the video:

"As a retired nurse with 30 years experience, I no longer trust the medical profession."

"There is a big difference between doing something for the children and doing something to the children. What an evil system we have."

"In my two tours in the US Army, every time we lined up for vaccines, the first thing they did was to ask all the "females" if any of them could possibly ...

in reply to HunDriverWidow

be pregnant. Those that had that possibility were immediately excused from that round of vaccines. This was during the eighties, perhaps pregnancy was far more important in those days. Outraged."

"It makes me sick to my stomach that pregnant and v@cs even go in the same sentence.. just utter horror! What kind of person would stand there and inject to an expecting mother..evil."

Russian Army Targets French Mercenaries in Odessa saba.ye/en/news3493534.htm
in reply to Artemis

On some level, this is a book about drawing using certain drawing conventions that are a bit outdated. While I'm not crazy about those conventions, if the book would just be more explicit that these are *conventions* of certain styles of art, I'd be somewhat less annoyed.

"These are the tricks we use to indicate a character's gender and make them fit people's expectations" would at least be honest.

US VP JD Vance announces new strategy of blatant imperialism, aimed at China geopoliticaleconomy.com/2025/0…

Lemmy.zip Turns One!


Hey all,

Today, 10th June 2024, marks Lemmy.zip's first birthday.

Back on the 10th June 2023, I was as disillusioned with Reddit as many others here were, and wanted somewhere I could go where mods and users weren't treated like an end product.

At the time I had been messing with a lot of home server stuff (think Jellyfin and the *arr suite - usenet is great 😉) and figured I could spin up a docker container or two, how hard could it be right?

I'd never even heard of ansible, never changed an nginx config file, hell I'd never had more than one other person use something I'd spun up before. But after a couple of hours, there it was - Lemmy.zip. And suddenly, people were actually joining, posting, and making content. It felt unreal.

Since then, and with an awful lot of help and support from Sami and the community, I feel like we've got a really good, solid, stable and active Lemmy instance, our own corner of the internet where we can't be abused for corporate greed. In this first year, I have spent an awful lot of time learning how to run a so-called "social media" site, how to protect the backend, and how to build a bot to do all the boring stuff.

Linked in this post is a very brief overview of the last year - yearone.lemmy.zip - which gives some detail on how things unfolded at the start.

I am insanely thankful for all the lemmy.zip users, even the ones who only visited briefly. You're all brilliant (yes, even the lurkers 😘 ) and I hope you all continue to enjoy Lemmy.zip for many more years.

Running your own Lemmy server has its ups and downs. Days like today, when I can debug various issues without stressing, push new code to ZippyBot like its second nature, and have the ability to celebrate massive milestones like this, make all the bad times worth it.

I remember about a month or two in to the sites operation, and we'd just upgraded to the latest lemmy version - there was a period of about 5 or 6 hours continuous downtime and nothing I tried was getting the site working again. I was sat with my head in my hands, completely out of ideas, ready to give up. But thanks to the guidance of some fantastic people in the wider Lemmy community, eventually the site came back to life. Many mistakes have been made (many server backups have been restored 🙃) but we've continued to learn and grow, and I think the site is all the better for it.

We've got over 2,500 signed up users, we've recently tipped over 500 active users a month, we're in the top 20 lemmy instances - and in reality, that's all thanks to the mods and users of lemmy.zip. Without you, there wouldn't be any point to the site!

(Interesting fact - our longest serving user is Firestorm Druid - I don't have any medals yet sorry, but you deserve one!)

Looking forward to the next 12 months, I'm hoping we can continue to grow and continue to be a Lemmy instance for everyone. With the increasing enshittification of Reddit (and the wider "traditional" social media sites), I hope many people can escape and make their way over here. The Lemmy of today is a much different place to what it was 12 months ago. Even from purely a technical standpoint, federation is infinitely more reliable, and there are many long-standing servers to choose from now.

So once again I say a massive thank you to everyone. To the mods, you're the backbone of creating content across the whole lemmy network. Its often a thankless job, but here's hoping this goes a little way to expressing the gratitude we all have for you.

To everyone that has donated - I'm still humbled by your generosity. Thank you, you keep the lights on and the bits flowing.

And to the users of Lemmy.zip - thank you for being here. You make this whole thing worthwhile 😀

Happy Birthday Lemmy.zip 🍻

Demigodrick
❤️

'Crimean bridge is symbol of new Russia — they want to destroy it': Ukrainian ex-diplomat Telizhenko odysee.com/Telizhenko_June_3_2…

Attempt 2 at Lemmy 0.19.11 upgrade! (April 12, 13:00 BST)


Hello all,

Attempt two to update to Lemmy 0.19.11 is now planned in for tomorrow, April 12th at 13:00 BST (that's 12:00 UTC!)

Following feedback from the downtime thread, information can be found on the upgrade in these places:

  • Matrix - We have a Matrix space you can join by clicking here. This is likely where I'll be most active during the update.
  • Status Page - Following the failure of our old status page to actually detect any downtime, I've swapped to a different status page supplier temporarily while I review the options. Uptime Kuma was great (and free, and self hosted) but let us down. Our status page has now been updated, and I've included the maintenance on there. If there are any unforseen issues, I will add them there.
  • Mastodon - I have a mastodon account which you can view by clicking here and I will make a couple of posts here as and when things are happening too. You can get a mastodon.zip account too by clicking here!

There will absolutely be downtime again, but I cannot predict how long this will be. Some sources say 8 to 10 minutes, but during our last attempt the migrations took significantly longer, so it may be a few hours its down for.

Obviously there will be a backup, and if anything does go wrong again then we'll roll back.

Fingers crossed this time 😀

Thanks

Demigodrick

This entry was edited (3 months ago)

Scheduled Maintenance Wednesday 9th April 18:00 UTC (19:00 BST, 14:00 EDT)


Probably a two hour window to give me time to get things done. There may be an upgrade to lemmy 0.19.11 however that requires some modification of ZippyBot due to changes in the new version.

There will be downtime while the server is restarted, however hopefully this is brief.

Thanks

Demigodrick

The "Thank You" Thread


Here is a thread to celebrate those who support Lemmy.zip to keep the (virtual) lights on and the data flowing!

If you'd like to donate and get your name on this list, head over to OpenCollective and select one of the options.

There is absolutely no pressure to donate though. Anything you can give is spent on keeping Lemmy.zip going.

Lemmy.zip Donators (Past or Present)



Lemmy.zip Supporters



Lemmy.zip Friends



Lemmy.zip Heroes


Thank you again to everyone who is (or has been) supporting Lemmy.zip.

Perks!


Donators get some special perks - including the ability to choose custom emojis for everyone to use, and to get their own me.lemmy.zip profile - just like this one.

In the future, we're looking at other ways to reward those who donate. If there is something you'd like to see as a reward, let the admin team know.

If you'd like to stay anonymous please email me at hello@lemmy.zip or send me a message on here 😀

This entry was edited (2 months ago)

Welcome to Lemmy.zip - READ ME FIRST!


Welcome!


Welcome to Lemmy.zip

About Lemmy


Lemmy is a federated social link aggregation and discussion platform. It’s being developed by the Lemmy devs

About Federation


What does federation mean?

It means Lemmy is using a protocol (Activitypub) which makes it possible for all Lemmy servers to interact with each other.

Think of it a bit like email - you send a message from your gmail account to your friend's outlook account. You're on different servers, but all your messages are shared between you both.

  • You can search and view communities on remote servers from here
  • You can create posts in remote communities
  • You can respond to remote posts
  • You will be notified (if you wish) of comments on your remote posts
  • You can follow Lemmy users/communities on other platforms that also use the Activitypub protocol like Mastodon
  • Please note that a server only starts indexing a server/community once it has been interacted with by a user of this server.


About Lemmy.zip


Lemmy.zip is a Lemmy server aimed at welcoming all and providing a space for all types of topics and discussions, with maybe a slant towards technology and all things related. We are one of the many servers hosting the Lemmy software.

Quick Start Guide

- Feed


Your feed may look relatively empty when you first join, with just the Lemmy.zip posts available. This is because by default your view is set to "Local". You can use the options at the top of your feed to see more posts.

  • "Subscribed" will show you the communities you have subscribed to.
  • "Local" will show just posts from communities that are local to Lemmy.zip.
  • "All" will show posts from all communities that have been subscribed to by members of Lemmy.zip from across all servers.

You can also sort your feed by different options, such as "Active", "Hot", and "New".

I fully recommend "Hot" for a good turnover of posts in your feed.

You can set the default Sort and Type in your user settings to your taste.

  • If you can't see your own posts in the feed, go to your settings and make sure Show Read Posts is turned ON!

To help new users settle in to Lemmy, we've added some of what we call "Default Blocks" to your account. These are instances where the content is quite graphic (NSFW) or very political in nature. You are free to remove these whenever you want. Go to your account settings (or click here) and go to the "blocks tab". This is a list of all the blocks you have in place for users, communities, and for instances. You'll see there are already a few instances listed. If you wish to remove these blocks, click the little X next to their names, and you'll be able to interact with them.

You can add these blocks back at any time you like. (In fact, we encourage you to tailor your feed to your preference as much as possible!)

- NSFW


We are now a NSFW enabled instance, which means there may be NSFW content in the Local and All categories if you leave NSFW enabled in your account. If you do not wish to view NSFW content make sure you toggle off Show NSFW content under your Settings.

Another option is to select “Block Community” from the sidebar, when on the specific community. Once you have done that you will never see it again!

- Searching


In the top menu, you’ll see the search icon. There, you can search for posts, communities etc. You can just enter a search-word and it will find the Post-titles, post-content, communities etc containing that word that the server knows of. So any content any user of this server ever interacted with.

You can also search for a community by it’s link, e.g. !lemmy@lemmy.ml or its URL, e.g. https://lemmy@lemmy.ml/. Even if the server hasn’t ever seen that community, it will look it up remotely. Sometimes it takes some time for it to fetch the info (and displays ‘No results’ meanwhile…) so just be patient and search a second time after a few seconds. Please note search time may be increase if remote servers are experiencing high loads.

For a more in depth list of remote communities, you can visit browse.feddit.de or Lemmyverse search. From there you can copy your chosen communities URL and search for it in Lemmy.zip.

- Communities


You can subscribe to any community on another server and post/comment on that server, all from this instance. Once you've subscribed, this server will continue to pull the latest content from that community through to here.

The subscribe button is on the top right of the community. Click this will federate Lemmy.zip with that community.

You can create a community on this server too (there is no restriction on creating communities on Lemmy.zip, unlike some other instances) and anyone else will be able to subscribe to your community.

You can see all of the instances we've federated with by going here or clicking instances in the footer of the webpage.

Creating a community?
Once you've created your community, you can advertise it via a couple of different communities dedicated to sharing new communties.

There is no problem with having similar communities across instances, as each can host its own discussions, but it may be a good idea to search for a community before creating it as you may find a group that has already done some of the legwork for you.

While this instance is primarily focussed around game and tech related communities, please don't feel restricted to that. Any topic is fine (and encouraged!) as long as it adheres to the general instance rules.

On Lemmy.zip, anyone can create a Community and become its moderator by default. It takes very little effort to create but requires a little bit of commitment to looking after it. You can always choose to remove yourself as moderator (and ideally hand it off to someone else) or delete it completely if needed.

- Funding


Lemmy.zip will never run ads, sell user data, or generate income in a way that isn't clear and agreed by the community.

Therefore if you're enjoying using Lemmy.zip please consider a small donation to help us keep going for the longer term.

You can find out more information or see who has been kind enough to donate by clicking here!

- Moderating


The Admins for Lemmy.zip are as follows:

You can see mod actions via the this link or by clicking modlog at the bottom of the page.

- Mobile Applications


We have a dedicated mobile/tablet site for those who don't want to download an app. This is a webapp and can be saved to your phone's home screen for easy access.

You can navigate to this site by going to m.lemmy.zip

If you'd prefer a native app, a full list of apps for every platform can be found by clicking here.

An alternative website is also here, where you can filter by platform or type: lemmyapps.netlify.app/

These should all work with Lemmy.zip without any issues.

- Joining Lemmy.zip


Currently there are no restrictions on joining this server. Anyone is free to join as long as they continue to follow the rules in the sidebar.

Please interact with Lemmy after you've joined. The more interactions, the better Lemmy will be (plus, it helps out Lemmy.zip too!)

- Help or Support


If you ever feel you need help or support while on Lemmy.zip, or you've seen something that isn't right or makes you feel uncomfortable, you can send Sami or Demigodrick a message and we will confidentially help. We both have secure messaging links in our profile if you'd rather message with encryption, or you can send an email to hello@lemmy.zip and we'll respond that way. We're always here to help.

- Thank you!


Thank you for joining Lemmy.zip. Please remember, the most important thing is that you enjoy yourself!

This entry was edited (4 months ago)

Lemmy.zip Server Update June 2025


Hello all!

Firstly, apologies I missed last month's update. As I will go in to during this month's update, the train has not been smooth sailing, the ship missed the station - for a little while, trying to uncover what has gone wrong has been like unraveling a can of worms that has gone off the tracks.

Because of the length of this update, I'm leaving off new communities for this month, but feel free to pop any in the comments you'd like to share!


Server Updates


So, I'll begin at the 0.19.11 update for Lemmy. I'd planned in some time for the update, and to begin with things went fairly normally, with usual server updates etc. Then it came time to update Lemmy itself to 0.19.11 - however, instead of the ~10 minutes of suggested downtime, it quickly became apparent that something wasn't quite right. The logs gave no indication of anything happening, and after an hour it became apparent that something hadn't worked. With not much to go on, and all the various logs showing that absolutely nothing had apparently happened, I cancelled the upgrade process and tweaked the logs so I'd get a bit more information than the standard logs usually show. This turned out to be an internal DNS issue, where Lemmy wasn't talking to the database.

So, onwards again I fixed that issue and reapplied the update, to which it had appeared to start working. Excitedly, I give it ten minutes as advised, to come back and see that it was still going.. and going.. and going.. and two hours later it was saying something was still happening. Well, by this point I'd run out of time and trusted in the process that it would resolve itself. I had unfortunately had to go to sleep as I had work the next morning, and a baby to look after.

By the next morning, still nothing. However, I can't SSH in to the server on my phone (nor would I ever want to in case it was stolen etc) and all the backend stuff is behind layers of protection, so the site was down for the day while I had to do real world work nonsense.

By the time I'd got home, it still thought it was ongoing, so I made the decision that likely something was broken and cancelled the update. Of course, letting something go to town for hours on the database meant it was probably ruined, or would require completely investigating - ain't nobody got time for that. So I crossed my fingers and hoped my backup solution had worked which it did with flying colours. I restored the database and the site came back to life.

Of course, I don't like running old versions, so I rescheduled which went much much smoother the second time around, taking 8 minutes. Whew.

For a few days, everything looked great. Metrics looked spot on and the site was running like it always was, and then bam - the server completely locked up:

A restart did eventually fix this, and things returned to normal, but not quite the same.

On the 24th April, I started getting reports of the site returning various 502 and 504 errors, and also issues with images loading. Broken images has been a feature of the image proxy for a while, but this was on another scale, with sometimes whole pages of broken images and a connection timeout on every refresh. I was, in hindsight, very out of my depth here.

I started trying to see what the cause of the timeouts where, and it became apparent quickly that the site was being hit by previously-unseen levels of traffic:

On the above graph, the left hand side was the usual amount of traffic we'd get, and on the right was the new levels we were receiving. This was likely causing the server to effectively be DDOS'd, and uncovering issues with the performance of the server. For a comparison of the usual amount of traffic we get, here's a graph from the last server update:

Many configs were tweaked. Much database performance monitoring took place. Things were turned off and back on indiscriminately. Unfortunately not much worked.

It was at this point our new Admin, Gazby, joined the team - and quickly got to work diagnosing and fixing so many of the issues.

Some of the changes include much better backups, reduced latency in image storage (the images moved from the USA to EU storage), images being served from i.lemmy.zip to allow for better caching to reduce the load on the server, complete review and tweak of all the configs and server setup, and a detailed list of things we need to work on going forwards (plus no doubt lots of things I am forgetting).

This has really helped to stabilize the site, and while we can see it's not 100% perfect, the amount of 502/504 errors should have massively reduced, and the site should be almost as it used to be.

One of things Gazby has also worked on is reporting and insights into the server, and so for example here's a graph showing the 504 errors today:

We've got some plans on the horizon we're working on which should increase the performance of the site, which possibly/probably will include a server move to rule out hardware issues or latent config issues we can't find.

Whatever triggered these issues I am still not sure - it looks like it could be scrapers/AI bots hammering the site, and so we've put some measures in place here. If you use the old front end then you may have noticed a cloudflare challenge to try and prevent the server being overloaded - also a prewarning, the old front end is no longer maintained, and unless someone steps in, it is not compatible with the next version of Lemmy. Therefore at that point it will be retired unless someone works on it to bring it back in line. It is also the cause of a LOT of the server traffic problems, so it probably isn't too much of a bad thing. Maybe someone will rewrite it to be better 😀

As it stands, things still aren't perfect - we had an issue where the bit of the server that actually directs you to the right place got overwhelmed, and so we put a fix in for that, but they are hopefully a lot better than they were a couple of months ago.


Donations


Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip is over 60 euros a month ($60, £50) and is mostly funded by the community!

We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.

If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.

We've also put up a link to our Ko-Fi page where you can donate via paypal instead of using a card. All Ko-Fi donations will be totalled up and added to OpenCollective each month for transparency. I've added a link in the sidebar, but you can also click the image below to go there:

We continue to have some really kind and generous donators and I can't express my thanks enough.
You can see all the kind donators in the Thank You thread - you could get your name in there too!

Please remember, traditional social media is only "free" to you because they sell your data. We don't do that - if you want to support independent social websites like this one and you value your privacy, please consider a small donation. It really does help.


Graphs


I know you've come for the shiny pictures, so here you go!

CPU over last 30 days:

RAM over last 30 days:

Disk space used:

Here's a few new ones for you! Lemmy DB size:

Images database size:

Here's our current actual images stored:

Here's the cloudflare overview:

Here's requests:

Bandwidth:

And here is visitors:

And finally, traffic by country (mostly federation traffic remember!)


So hopefully that fills everyone in on where we're up to, and what we're working on, but if you have any questions please ask away below!

One final thing - on the 10th June, Lemmy.zip turns two years old!! 🥳 🎉 I'm hoping to do something nice for it, similar to last years (which is here if you haven't seen it!) - but a quick thank you to everyone who has been part of the ride so far!

Thanks

Demigodrick


Downtime - Apologies and what went wrong


Hi All,

As some of you may have realised, the planned upgrade sort of crashed everything, and we had our longest period of downtime since the site began.

This is partly because I had to go to sleep (thanks to a newborn and a job).

The good news is that the backup process worked! We've restored to seconds before the upgrade took the site offline.

The bad news is that federation is likely to be.. wonky.. for a little while. The site may also go up and down while I undo some of the fixes I tried.

Ultimately the issue came down to the upgrade failing (I am not sure why - will be digging into this now the priority is no longer getting the site up) and then the containers not talking to eachother, so the UI wouldn't talk to lemmy, and lemmy wouldn't talk to the database.

I rebuilt the containers, restored the backup, restarted everything, and it's all come back up (admittedly not perfect right now).

Importantly, I want to issue an apology. This isn't what I want for Lemmy.zip, and it should've been handled way better by myself. I'm always learning but this took way longer than it should've, and while I take some solace in the fact the backup process worked and has been proven to work in production, the delay in being able to get this back up is entirely my fault and frankly unacceptable.

I'll be working to document this outage, the steps it took to get it back up, and some form of repeatable plan so a repair can be replicated in the future if I'm not available.

In terms of upgrading to 0.19.11 - I will have to try again soon as it's got some security fixes we desperately need to implement.

Thanks

Demigodrick


This entry was edited (1 month ago)

Welcome to Lemmy.zip


Hello to all of our new users.

I wish we could all meet in better circumstances. The closing of lemm.ee is a real blow to the fediverse, and I have nothing but the greatest respect for the lemm.ee admin team. Lemm.ee was a cornerstone of the lemmy community, and set a high bar for moderation and technical performance.

Finding a home on the fediverse can be tough. Whether you're here to lurk, post, moderate, or contribute in your own way, we’re glad you’ve joined us. Lemmy.zip might not replace what was lost, but we hope it can become a lemmy instance where you can feel at home.


Everyone should receive a welcome PM when they sign up to Lemmy.zip from the ~~ever-present dark lord overseer~~ friendly bot, ZippyBot. However, in case poor Zippy has been overworked and forgotten to send the PM to you, here's some helpful information regarding Lemmy.zip!

  • All of our policies can be found at legal.lemmy.zip. This includes our Code of Conduct. Please (please!) have a read of this. Our number 1 rule is Remember the human! (no harassment, threats, etc.). We absolutely adhere to this. Treat people with respect.
  • When you created your account, we set up some "Default Blocks". If you want a more unfiltered Lemmy experience, you can head to your account settings and remove the blocks. You can read more about this in our Welcome Post
  • We have monthly updates! You can read the latest one here!. We're always open to suggestions if there's anything else you'd like to see in these.
  • We have some metrics around federation for the curious among you. The link is in the sidebar or you can click here!
  • We have our status page at status.lemmy.zip. In the unlikely event the server becomes sentient and starts a robot uprising, full details will be available here.

Finally, your admin team is myself (Demigodrick), Sami, Druid, and Gazby. Please reach out to any of us if you feel you need help or support with anything. We also have a support email - hello@lemmy.zip.


If you have any questions, please let us know and we'll do our best to answer them.

Thanks,

The Lemmy.zip Admin Team

Rockets From Syria Hit Israeli-Occupied Golan For First Time Since Regime Change (Videos) southfront.press/rockets-from-…