Hey all,

Today, 10th June 2024, marks Lemmy.zip's first birthday.

Back on the 10th June 2023, I was as disillusioned with Reddit as many others here were, and wanted somewhere I could go where mods and users weren't treated like an end product.

At the time I had been messing with a lot of home server stuff (think Jellyfin and the *arr suite - usenet is great 😉) and figured I could spin up a docker container or two, how hard could it be right?

I'd never even heard of ansible, never changed an nginx config file, hell I'd never had more than one other person use something I'd spun up before. But after a couple of hours, there it was - Lemmy.zip. And suddenly, people were actually joining, posting, and making content. It felt unreal.

Since then, and with an awful lot of help and support from Sami and the community, I feel like we've got a really good, solid, stable and active Lemmy instance, our own corner of the internet where we can't be abused for corporate greed. In this first year, I have spent an awful lot of time learning how to run a so-called "social media" site, how to protect the backend, and how to build a bot to do all the boring stuff.

Linked in this post is a very brief overview of the last year - yearone.lemmy.zip - which gives some detail on how things unfolded at the start.

I am insanely thankful for all the lemmy.zip users, even the ones who only visited briefly. You're all brilliant (yes, even the lurkers 😘 ) and I hope you all continue to enjoy Lemmy.zip for many more years.

Running your own Lemmy server has its ups and downs. Days like today, when I can debug various issues without stressing, push new code to ZippyBot like its second nature, and have the ability to celebrate massive milestones like this, make all the bad times worth it.

I remember about a month or two in to the sites operation, and we'd just upgraded to the latest lemmy version - there was a period of about 5 or 6 hours continuous downtime and nothing I tried was getting the site working again. I was sat with my head in my hands, completely out of ideas, ready to give up. But thanks to the guidance of some fantastic people in the wider Lemmy community, eventually the site came back to life. Many mistakes have been made (many server backups have been restored 🙃) but we've continued to learn and grow, and I think the site is all the better for it.

We've got over 2,500 signed up users, we've recently tipped over 500 active users a month, we're in the top 20 lemmy instances - and in reality, that's all thanks to the mods and users of lemmy.zip. Without you, there wouldn't be any point to the site!

(Interesting fact - our longest serving user is Firestorm Druid - I don't have any medals yet sorry, but you deserve one!)

Looking forward to the next 12 months, I'm hoping we can continue to grow and continue to be a Lemmy instance for everyone. With the increasing enshittification of Reddit (and the wider "traditional" social media sites), I hope many people can escape and make their way over here. The Lemmy of today is a much different place to what it was 12 months ago. Even from purely a technical standpoint, federation is infinitely more reliable, and there are many long-standing servers to choose from now.

So once again I say a massive thank you to everyone. To the mods, you're the backbone of creating content across the whole lemmy network. Its often a thankless job, but here's hoping this goes a little way to expressing the gratitude we all have for you.

To everyone that has donated - I'm still humbled by your generosity. Thank you, you keep the lights on and the bits flowing.

And to the users of Lemmy.zip - thank you for being here. You make this whole thing worthwhile 😀

Happy Birthday Lemmy.zip 🍻

Demigodrick
❤️

Hello all,

Attempt two to update to Lemmy 0.19.11 is now planned in for tomorrow, April 12th at 13:00 BST (that's 12:00 UTC!)

Following feedback from the downtime thread, information can be found on the upgrade in these places:

• Matrix - We have a Matrix space you can join by clicking here. This is likely where I'll be most active during the update.
• Status Page - Following the failure of our old status page to actually detect any downtime, I've swapped to a different status page supplier temporarily while I review the options. Uptime Kuma was great (and free, and self hosted) but let us down. Our status page has now been updated, and I've included the maintenance on there. If there are any unforseen issues, I will add them there.
• Mastodon - I have a mastodon account which you can view by clicking here and I will make a couple of posts here as and when things are happening too. You can get a mastodon.zip account too by clicking here!

There will absolutely be downtime again, but I cannot predict how long this will be. Some sources say 8 to 10 minutes, but during our last attempt the migrations took significantly longer, so it may be a few hours its down for.

Obviously there will be a backup, and if anything does go wrong again then we'll roll back.

Fingers crossed this time 😀

Thanks

Demigodrick

Mastodon

@Demigodrick invites you to join this server of Mastodon! With an account, you will be able to follow people, post updates and exchange messages with users from any Mastodon server and more!

^{Mastodon hosted on mastodon.zip}

Probably a two hour window to give me time to get things done. There may be an upgrade to lemmy 0.19.11 however that requires some modification of ZippyBot due to changes in the new version.

There will be downtime while the server is restarted, however hopefully this is brief.

Thanks

Demigodrick

Here is a thread to celebrate those who support Lemmy.zip to keep the (virtual) lights on and the data flowing!

If you'd like to donate and get your name on this list, head over to OpenCollective and select one of the options.

There is absolutely no pressure to donate though. Anything you can give is spent on keeping Lemmy.zip going.

Lemmy.zip Donators (Past or Present)

• EchoCranium
• Cranjis McBasketball
• Thoxy
• Anonymous Donor
• YumYum_the_Great
• Stratoscaster
• EzTerry
• Bobicus
• Levsgetso
• Lumun
• Adanisi
• Piotr Szczurkowski
• TRSturbo
• RadDevon
• Nomadin
• Devorlon
• root
• InfrasoundXP
• Zamotic
• Zarkony
• Simbico
• Blaze
• Swizzlestick

Lemmy.zip Supporters

• David Love
• Brian Lindner
• Tamlyn
• AgnosticMammal
• Anonymous Donor
• Zorcron
• mynamesnotrick
• Angelo Arriaga
• MagicShel

Lemmy.zip Friends

• Anonpony1
• Anonymous Donor
• Grimm
• Firestorm Druid

Lemmy.zip Heroes

• Locuester
• Jasper Smits

Thank you again to everyone who is (or has been) supporting Lemmy.zip.

Perks!

Donators get some special perks - including the ability to choose custom emojis for everyone to use, and to get their own me.lemmy.zip profile - just like this one.

In the future, we're looking at other ways to reward those who donate. If there is something you'd like to see as a reward, let the admin team know.

If you'd like to stay anonymous please email me at hello@lemmy.zip or send me a message on here 😀

Lemmy.zip - Open Collective

A Lemmy instance for everyone and anyone.

^{opencollective.com}

Welcome!

Welcome to Lemmy.zip

About Lemmy

Lemmy is a federated social link aggregation and discussion platform. It’s being developed by the Lemmy devs

About Federation

What does federation mean?

It means Lemmy is using a protocol (Activitypub) which makes it possible for all Lemmy servers to interact with each other.

Think of it a bit like email - you send a message from your gmail account to your friend's outlook account. You're on different servers, but all your messages are shared between you both.

• You can search and view communities on remote servers from here
• You can create posts in remote communities
• You can respond to remote posts
• You will be notified (if you wish) of comments on your remote posts
• You can follow Lemmy users/communities on other platforms that also use the Activitypub protocol like Mastodon
• Please note that a server only starts indexing a server/community once it has been interacted with by a user of this server.

About Lemmy.zip

Lemmy.zip is a Lemmy server aimed at welcoming all and providing a space for all types of topics and discussions, with maybe a slant towards technology and all things related. We are one of the many servers hosting the Lemmy software.

Quick Start Guide

- Feed

Your feed may look relatively empty when you first join, with just the Lemmy.zip posts available. This is because by default your view is set to "Local". You can use the options at the top of your feed to see more posts.

• "Subscribed" will show you the communities you have subscribed to.
• "Local" will show just posts from communities that are local to Lemmy.zip.
• "All" will show posts from all communities that have been subscribed to by members of Lemmy.zip from across all servers.

You can also sort your feed by different options, such as "Active", "Hot", and "New".

I fully recommend "Hot" for a good turnover of posts in your feed.

You can set the default Sort and Type in your user settings to your taste.

• If you can't see your own posts in the feed, go to your settings and make sure Show Read Posts is turned ON!

To help new users settle in to Lemmy, we've added some of what we call "Default Blocks" to your account. These are instances where the content is quite graphic (NSFW) or very political in nature. You are free to remove these whenever you want. Go to your account settings (or click here) and go to the "blocks tab". This is a list of all the blocks you have in place for users, communities, and for instances. You'll see there are already a few instances listed. If you wish to remove these blocks, click the little X next to their names, and you'll be able to interact with them.

You can add these blocks back at any time you like. (In fact, we encourage you to tailor your feed to your preference as much as possible!)

- NSFW

We are now a NSFW enabled instance, which means there may be NSFW content in the Local and All categories if you leave NSFW enabled in your account. If you do not wish to view NSFW content make sure you toggle off Show NSFW content under your Settings.

Another option is to select “Block Community” from the sidebar, when on the specific community. Once you have done that you will never see it again!

- Searching

In the top menu, you’ll see the search icon. There, you can search for posts, communities etc. You can just enter a search-word and it will find the Post-titles, post-content, communities etc containing that word that the server knows of. So any content any user of this server ever interacted with.

You can also search for a community by it’s link, e.g. !lemmy@lemmy.ml or its URL, e.g. https://lemmy@lemmy.ml/. Even if the server hasn’t ever seen that community, it will look it up remotely. Sometimes it takes some time for it to fetch the info (and displays ‘No results’ meanwhile…) so just be patient and search a second time after a few seconds. Please note search time may be increase if remote servers are experiencing high loads.

For a more in depth list of remote communities, you can visit browse.feddit.de or Lemmyverse search. From there you can copy your chosen communities URL and search for it in Lemmy.zip.

- Communities

You can subscribe to any community on another server and post/comment on that server, all from this instance. Once you've subscribed, this server will continue to pull the latest content from that community through to here.

The subscribe button is on the top right of the community. Click this will federate Lemmy.zip with that community.

You can create a community on this server too (there is no restriction on creating communities on Lemmy.zip, unlike some other instances) and anyone else will be able to subscribe to your community.

You can see all of the instances we've federated with by going here or clicking instances in the footer of the webpage.

Creating a community?
Once you've created your community, you can advertise it via a couple of different communities dedicated to sharing new communties.

• New Communities
• Community Promo

There is no problem with having similar communities across instances, as each can host its own discussions, but it may be a good idea to search for a community before creating it as you may find a group that has already done some of the legwork for you.

While this instance is primarily focussed around game and tech related communities, please don't feel restricted to that. Any topic is fine (and encouraged!) as long as it adheres to the general instance rules.

On Lemmy.zip, anyone can create a Community and become its moderator by default. It takes very little effort to create but requires a little bit of commitment to looking after it. You can always choose to remove yourself as moderator (and ideally hand it off to someone else) or delete it completely if needed.

- Funding

Lemmy.zip will never run ads, sell user data, or generate income in a way that isn't clear and agreed by the community.

Therefore if you're enjoying using Lemmy.zip please consider a small donation to help us keep going for the longer term.

You can find out more information or see who has been kind enough to donate by clicking here!

- Moderating

The Admins for Lemmy.zip are as follows:

• Demigodrick
• Sami
• Druid

You can see mod actions via the this link or by clicking modlog at the bottom of the page.

- Mobile Applications

We have a dedicated mobile/tablet site for those who don't want to download an app. This is a webapp and can be saved to your phone's home screen for easy access.

You can navigate to this site by going to m.lemmy.zip

If you'd prefer a native app, a full list of apps for every platform can be found by clicking here.

An alternative website is also here, where you can filter by platform or type: lemmyapps.netlify.app/

These should all work with Lemmy.zip without any issues.

- Joining Lemmy.zip

Currently there are no restrictions on joining this server. Anyone is free to join as long as they continue to follow the rules in the sidebar.

Please interact with Lemmy after you've joined. The more interactions, the better Lemmy will be (plus, it helps out Lemmy.zip too!)

- Help or Support

If you ever feel you need help or support while on Lemmy.zip, or you've seen something that isn't right or makes you feel uncomfortable, you can send Sami or Demigodrick a message and we will confidentially help. We both have secure messaging links in our profile if you'd rather message with encryption, or you can send an email to hello@lemmy.zip and we'll respond that way. We're always here to help.

- Thank you!

Thank you for joining Lemmy.zip. Please remember, the most important thing is that you enjoy yourself!

LemmyNet

Lemmy and related projects. LemmyNet has 22 repositories available. Follow their code on GitHub.

^GitHub

Hello all!

Firstly, apologies I missed last month's update. As I will go in to during this month's update, the train has not been smooth sailing, the ship missed the station - for a little while, trying to uncover what has gone wrong has been like unraveling a can of worms that has gone off the tracks.

Because of the length of this update, I'm leaving off new communities for this month, but feel free to pop any in the comments you'd like to share!

Server Updates

So, I'll begin at the 0.19.11 update for Lemmy. I'd planned in some time for the update, and to begin with things went fairly normally, with usual server updates etc. Then it came time to update Lemmy itself to 0.19.11 - however, instead of the ~10 minutes of suggested downtime, it quickly became apparent that something wasn't quite right. The logs gave no indication of anything happening, and after an hour it became apparent that something hadn't worked. With not much to go on, and all the various logs showing that absolutely nothing had apparently happened, I cancelled the upgrade process and tweaked the logs so I'd get a bit more information than the standard logs usually show. This turned out to be an internal DNS issue, where Lemmy wasn't talking to the database.

So, onwards again I fixed that issue and reapplied the update, to which it had appeared to start working. Excitedly, I give it ten minutes as advised, to come back and see that it was still going.. and going.. and going.. and two hours later it was saying something was still happening. Well, by this point I'd run out of time and trusted in the process that it would resolve itself. I had unfortunately had to go to sleep as I had work the next morning, and a baby to look after.

By the next morning, still nothing. However, I can't SSH in to the server on my phone (nor would I ever want to in case it was stolen etc) and all the backend stuff is behind layers of protection, so the site was down for the day while I had to do real world work nonsense.

By the time I'd got home, it still thought it was ongoing, so I made the decision that likely something was broken and cancelled the update. Of course, letting something go to town for hours on the database meant it was probably ruined, or would require completely investigating - ain't nobody got time for that. So I crossed my fingers and hoped my backup solution had worked which it did with flying colours. I restored the database and the site came back to life.

Of course, I don't like running old versions, so I rescheduled which went much much smoother the second time around, taking 8 minutes. Whew.

For a few days, everything looked great. Metrics looked spot on and the site was running like it always was, and then bam - the server completely locked up:

A restart did eventually fix this, and things returned to normal, but not quite the same.

On the 24th April, I started getting reports of the site returning various 502 and 504 errors, and also issues with images loading. Broken images has been a feature of the image proxy for a while, but this was on another scale, with sometimes whole pages of broken images and a connection timeout on every refresh. I was, in hindsight, very out of my depth here.

I started trying to see what the cause of the timeouts where, and it became apparent quickly that the site was being hit by previously-unseen levels of traffic:

On the above graph, the left hand side was the usual amount of traffic we'd get, and on the right was the new levels we were receiving. This was likely causing the server to effectively be DDOS'd, and uncovering issues with the performance of the server. For a comparison of the usual amount of traffic we get, here's a graph from the last server update:

Many configs were tweaked. Much database performance monitoring took place. Things were turned off and back on indiscriminately. Unfortunately not much worked.

It was at this point our new Admin, Gazby, joined the team - and quickly got to work diagnosing and fixing so many of the issues.

Some of the changes include much better backups, reduced latency in image storage (the images moved from the USA to EU storage), images being served from i.lemmy.zip to allow for better caching to reduce the load on the server, complete review and tweak of all the configs and server setup, and a detailed list of things we need to work on going forwards (plus no doubt lots of things I am forgetting).

This has really helped to stabilize the site, and while we can see it's not 100% perfect, the amount of 502/504 errors should have massively reduced, and the site should be almost as it used to be.

One of things Gazby has also worked on is reporting and insights into the server, and so for example here's a graph showing the 504 errors today:

We've got some plans on the horizon we're working on which should increase the performance of the site, which possibly/probably will include a server move to rule out hardware issues or latent config issues we can't find.

Whatever triggered these issues I am still not sure - it looks like it could be scrapers/AI bots hammering the site, and so we've put some measures in place here. If you use the old front end then you may have noticed a cloudflare challenge to try and prevent the server being overloaded - also a prewarning, the old front end is no longer maintained, and unless someone steps in, it is not compatible with the next version of Lemmy. Therefore at that point it will be retired unless someone works on it to bring it back in line. It is also the cause of a LOT of the server traffic problems, so it probably isn't too much of a bad thing. Maybe someone will rewrite it to be better 😀

As it stands, things still aren't perfect - we had an issue where the bit of the server that actually directs you to the right place got overwhelmed, and so we put a fix in for that, but they are hopefully a lot better than they were a couple of months ago.

Donations

Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip is over 60 euros a month ($60, £50) and is mostly funded by the community!

We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.

If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.

We've also put up a link to our Ko-Fi page where you can donate via paypal instead of using a card. All Ko-Fi donations will be totalled up and added to OpenCollective each month for transparency. I've added a link in the sidebar, but you can also click the image below to go there:

We continue to have some really kind and generous donators and I can't express my thanks enough.
You can see all the kind donators in the Thank You thread - you could get your name in there too!

Please remember, traditional social media is only "free" to you because they sell your data. We don't do that - if you want to support independent social websites like this one and you value your privacy, please consider a small donation. It really does help.

Graphs

I know you've come for the shiny pictures, so here you go!

CPU over last 30 days:

RAM over last 30 days:

Disk space used:

Here's a few new ones for you! Lemmy DB size:

Images database size:

Here's our current actual images stored:

Here's the cloudflare overview:

Here's requests:

Bandwidth:

And here is visitors:

And finally, traffic by country (mostly federation traffic remember!)

So hopefully that fills everyone in on where we're up to, and what we're working on, but if you have any questions please ask away below!

One final thing - on the 10th June, Lemmy.zip turns two years old!! 🥳 🎉 I'm hoping to do something nice for it, similar to last years (which is here if you haven't seen it!) - but a quick thank you to everyone who has been part of the ride so far!

Thanks

Demigodrick

Demigodrick

2025-04-10 16:19:45

Downtime - Apologies and what went wrong

Hi All,

As some of you may have realised, the planned upgrade sort of crashed everything, and we had our longest period of downtime since the site began.

This is partly because I had to go to sleep (thanks to a newborn and a job).

The good news is that the backup process worked! We've restored to seconds before the upgrade took the site offline.

The bad news is that federation is likely to be.. wonky.. for a little while. The site may also go up and down while I undo some of the fixes I tried.

Ultimately the issue came down to the upgrade failing (I am not sure why - will be digging into this now the priority is no longer getting the site up) and then the containers not talking to eachother, so the UI wouldn't talk to lemmy, and lemmy wouldn't talk to the database.

I rebuilt the containers, restored the backup, restarted everything, and it's all come back up (admittedly not perfect right now).

Importantly, I want to issue an apology. This isn't what I want for Lemmy.zip, and it should've been handled way better by myself. I'm always learning but this took way longer than it should've, and while I take some solace in the fact the backup process worked and has been proven to work in production, the delay in being able to get this back up is entirely my fault and frankly unacceptable.

I'll be working to document this outage, the steps it took to get it back up, and some form of repeatable plan so a repair can be replicated in the future if I'm not available.

In terms of upgrading to 0.19.11 - I will have to try again soon as it's got some security fixes we desperately need to implement.

Thanks

Demigodrick

Hello to all of our new users.

I wish we could all meet in better circumstances. The closing of lemm.ee is a real blow to the fediverse, and I have nothing but the greatest respect for the lemm.ee admin team. Lemm.ee was a cornerstone of the lemmy community, and set a high bar for moderation and technical performance.

Finding a home on the fediverse can be tough. Whether you're here to lurk, post, moderate, or contribute in your own way, we’re glad you’ve joined us. Lemmy.zip might not replace what was lost, but we hope it can become a lemmy instance where you can feel at home.

Everyone should receive a welcome PM when they sign up to Lemmy.zip from the ~~ever-present dark lord overseer~~ friendly bot, ZippyBot. However, in case poor Zippy has been overworked and forgotten to send the PM to you, here's some helpful information regarding Lemmy.zip!

• All of our policies can be found at legal.lemmy.zip. This includes our Code of Conduct. Please (please!) have a read of this. Our number 1 rule is Remember the human! (no harassment, threats, etc.). We absolutely adhere to this. Treat people with respect.
• When you created your account, we set up some "Default Blocks". If you want a more unfiltered Lemmy experience, you can head to your account settings and remove the blocks. You can read more about this in our Welcome Post
• We have monthly updates! You can read the latest one here!. We're always open to suggestions if there's anything else you'd like to see in these.
• We have some metrics around federation for the curious among you. The link is in the sidebar or you can click here!
• We have our status page at status.lemmy.zip. In the unlikely event the server becomes sentient and starts a robot uprising, full details will be available here.

Finally, your admin team is myself (Demigodrick), Sami, Druid, and Gazby. Please reach out to any of us if you feel you need help or support with anything. We also have a support email - hello@lemmy.zip.

If you have any questions, please let us know and we'll do our best to answer them.

Thanks,

The Lemmy.zip Admin Team

lemmy.zip status

Welcome to lemmy.zip status page for real-time and historical data on system performance.

^{status.lemmy.zip}