In the last weeks Lemmy has seen a lot of growth, with thousands of new users. To welcome them we are holding this AMA to answer questions from the community. You can ask about the beginnings of Lemmy, how we see the future of Lemmy, our long-term goals, what makes Lemmy different from Reddit, about internet and social media in general, as well as personal questions.
We’d also like to hear your overall feedback on Lemmy: What are its greatest strengths and weaknesses? How would you improve it? What’s something you wish it had? What can our community do to ensure that we keep pulling users away from US tech companies, and into the fediverse?
Lemmy and Reddit may look similar at first glance, but there is a major difference. While Reddit is a corporation with thousands of employees and billionaire investors, Lemmy is nothing but an open source project run by volunteers. It was started in 2019 by @dessalines and @nutomic, turning into a fulltime job since 2020. For our income we are dependent on your donations, so please contribute if you can. We’d like to be able to add more full-time contributors to our co-op.
We will start answering questions from tomorrow (Wednesday). Besides @dessalines and @nutomic, other Lemmy contributors may also chime in to answer questions:
Here are our previous AMAs for those interested.
No questions right now. Just wanted to say thank you for your hard work.
I know y’all catch a lot of shit and get hammered with requests/demands, so I wanted to let you know that your work is greatly appreciated.
Thanks for dedicating your time and energy to making a non-corporate, federated social environment possible.
Being on Lemmy has been a breath of fresh air.
Thanks a lot for the work you do! How do you get by with such a limited amount of funds? How sustainable is your financial situation if donations don’t pick up considerably?
Just wanted to say I LOVE lemmy! It’s a really positive community, the atmosphere is great and I like how it’s unique but also familiar. I really appreciate your work on it. I know this is AMA… what’s your favourite animal?
Do you plan on moving away from GitHub to something else like Forgejo?
+1 on registration experience being the #1 issue.
Would also be cool if we could stop 404/500ing deleted posts and instead display some indication it has been deleted. See https://en.m.wikipedia.org/wiki/Principle_of_least_astonishment.
Thanks for Lemmy! 💙
When will users be able to frictionlessly migrate between instances without losing their posts, their comments, their history, their relationships, their reputation etc? (Without requiring the consent of the exiting instance owner, or that this server still even exists, as they sometimes don’t)
How are you?
A bit tired because my whole family is half sick. Luckily the kids are still okay to go to school.
Otherwise Im excited for this AMA, because I rarely have such direct conversations with users about Lemmy. The discussions on Github are usually quite technical.
Not bad, the swiss chard and spinach I planted recently are sprouting, so that’s got me excited.
Spinach is a finicky bastard in my experience, take good care of it
Chilling in the morning before I start my day job.
What have been the biggest challenges with the project over the years, both in terms of technical and non technical aspects. I’d be interesting to hear a bit of retrospective on how has the stack’s been working out, and what surprises you might’ve run into in terms of scaling and federation. What recommendations you’d make based on that and what you would’ve done differently knowing what you know now.
2nding @nutomic, that I’m really happy with the stack.
The one that seems really magical to me, is diesel. With it we get a compile-time-checked database, that’s tightly integrated to the rust objects / code.
Every single join, select, insert, etc is checked before lemmy is even run, and it eliminates a whole category of errors resulting from mismaps.
Its made adding columns, and changing our data structures so much less error-prone than when I lived in the java-world.
Whenever we find that we’d want to do things differently, we usually do a refactor ASAP so as not to keep rolling spaghetti code. We’ve had to do this many times for the federation and DB code, and even have 2 major refactors that also add features ongoing. But luckily we’ve been able to stay in the rust eco-system for that.
As for UI, leptos didn’t exist when I built lemmy-ui, so I went with a fast react-like alternative, inferno. Its showing its age now, so @sleepless1917 is working on lemmy-ui-leptos, which hopefully will supercede lemmy-ui.
I briefly worked with Hasura which does this sort of magic to produce GraphQL API on top of Postgres. Incidentally, also written in Rust. I do like Leptos approach, it sounds similar to HTMX approach where you just treat the DOM as a dumb terminal.
The stack is great, I wouldnt want to change anything. Postgres is very mature and performant, with a high focus on correctness. It can sometimes be difficult to optimize queries, but there are wizards like @dullbananas@lemmy.ca who know how to do that. Anyway there is no better alternative that I know of. Rust is also great, just like Postgres it is very performant and has a focus on correctness. Unlike most programming languages it is almost impossible to get any runtime crashes, which is very valuable for a webservice.
The high performance means that less hardware is required to host a given number of users, compared to something like NodeJS or PHP. For example when kbin.social was popular, I remember it had to run on multiple beefy servers. Meanwhile lemmy.ml is still running on a single dedicated server, with much more active users. Or Mastodon having to handle incoming federation activities in background tasks which makes the code more complicated, while Lemmy can process them directly in the HTTP handler.
Nevertheless, scaling for more users always has its surprises. I remember very early in development, Lemmy wasnt able to handle more than a dozen requests per second. Turns out we only used a single database connection instead of a connection pool, so each db query was running after that last one was finished, which of course is very slow. It seems obvious in retrospect, but you never notice this problem until there are a dozen or so users active at the same time.
With the Reddit migration two years ago a lot of performance problems came up, as active users on Lemmy suddenly grew around 70 times. You can see some of that in the 0.18.x release announcements. One part of the solution was to add missing database indexes. Another was to remove websocket support, which was keeping a connection open for each user. That works fine with 100 users, but completely breaks down with 1000 or more.
After all there is nothing I would do different really. It would have been good to know about these scaling problems earlier, but thats impossible. In fact for my project Ibis (federated wiki) Im using the exact same architecture as Lemmy.
It’s great to hear things mostly worked out. Stuff like scaling bottlenecks is definitely tricky to catch until you have serious loads on the system, but sounds like the fixes very mostly trivial validating overall design. It also looks like you managed to get a way with a fairly simple stack by leveraging Postgres and Rust. I’ve had really good experience with using pg myself, and really don’t see a point in using anything else now. You can use it both as a relation db and a document store, so it’s extremely flexible on top of being highly performant. Keeping the stack simple tends to be underappreciated, and projects often just keep adding moving pieces which end up adding a lot of overhead and complexity in the end.
The stack is overall amazing, but not perfect. Waiting for the Rust code to compile is sometimes very annoying, but I wouldn’t want to use a different language. And we had to implement somewhat complicated things that existing Rust libraries did not do. For example, I made the “i_love_jesus” library so Lemmy could have cursor pagination that uses indexes well and allows bringing back the “back” button, we have a few custom QueryFragment impls because of diesel’s limitations, and we have a custom migration runner to do fancy stuff (see crates/db_schema/src/schema_setup.rs).
I made the “i_love_jesus” library so Lemmy
Could I ask if there’s any meaning behind that name?
Nothing related to the library. I love Jesus. I’m Catholic and a little silly. Also my GitHub profile picture is Jesus.
I don’t know what I expected lol. Cheers.
Edit: I don’t mean this disrespectfully, it was just a very direct and obvious answer.
Being a Lisper, the idea of waiting for the compiler is very jarring. :)
Oh, to be able to develop Lemmy with something like SLIME or Geiser, now that would be a dream. Too bad the CL’s library ecosystem is so much worse than Rust’s.
Clojure has a lot better story in that regard living on the JVM, but the overhead of using the JVM is a downside of its own. It’s a good platform, but definitely not what you’d call lightweight.
Yeah, if I was building something production ready in Lisp, Clojure would be my choice even though I prefer CL. Ecosystem is ultimately king.
For sure, having mature and battle tested libraries is really hard to beat.
What Is the airspeed velocity of an unladen swallow?
How would you improve it?
a way to filter out posts that have no engagement or comments from others would be helpful since the larger instances flood my feed w hundreds/thousands of news links that flood out the discourse on lemmy.
- From a code architecture perspective, how close is Lemmy/ActivityPub to reaching its maximum capacity for posts/comments per second? Are there any ways to 10x the load ActivityPub can handle?
- With Nicole in everyone’s DMs, what does the future of spam filtering look like on Lemmy?
- There is no specific maximum capacity, in theory it can scale indefinitely with horizontal scaling. Also see my reply here regarding scaling.
- 0.19.10 already includes a fix to remove private messages when a user gets banned which should help a lot. There is an issue about disabling private messages by default, but Im not sure if that will be necessary. Also 1.0 will include a plugin system, so other devs and instance admins can write their own checks. That way spam waves can be fought in a more flexible way, without having to get a change merged into Lemmy and then waiting for a new release.
What are your thoughts on blocking AI scraper access? Any attempts to improve that on the side of Lemmy? Basic things like allowing to customize the robots.txt easily would already help.
I also recently tried this new AI block tool called Anubis with Lemmy, but for some reason it fails with Lemmy-ui. Might be interesting to investigate further.
You can load a different robots.txt in your nginx config, something like this:
location /robotx.txt { index /path/to/my/robots.txt; }Additionally 1.0 will change the “private instance” to work with federation enabled (see https://github.com/LemmyNet/lemmy/pull/5530). Then only logged-in users will see content, while AI scrapers wont see anything except the login page.
Anyone that wants to scrape Lemmy would have an easier time setting up their own server, federating with everyone, and reading straight from their DB. No web scraping required. Though, web scraping defenses would be useful against general web scrapers/crawlers.
That would require the authors of these AI scrapers to actually give a f*ck. The problem is that they don’t, and just scrape what ever they can find repeatatly almost like a ddos attack on the open web.
Yup, same as they could clone git repos in one shot, but they instead crawl every single page.
I just set up Anubis today. Specifically I’m only testing it for Lemmy-ui, and it seems to work fine.
It looks like the distributed waves that keep bringing the service down hit exclusively our lemmy-ui subdomain, so maybe non-SSR photon is also a good defense, heh.
Hmm, that is odd. I guess I need to double check my Nginx config for lemmy-ui then. You have your setup documented somewhere?
Edit: ah, you run Photon as the main UI and lemmy-ui somewhere else? I think specifically the split between frontend and backend on the root domain somehow makes Anubis fail to set the correct cookie.
I don’t think it should be a problem, but I’m not that sure either. Lemmy.fedi.zutto.fi also runs it and that’s just a normal lemmy-ui installation. I think Zutto simply forwarded all traffic to Anubis and then fixed federation. There was some discussion and config shared in sopuli’s finnish matrix room.
I’ve previously worked in anti-scraping. There is a negative 0% chance the Lemmy devs have the resources to effectively do this without tanking the server for everyone else.
Do you plan to introduce some kind of post tags into Lemmy, preferably something that will behave like Hashtags on Mastodon and other activitypub platforms? I know that Lemmy has been embedding community name as a hashtag for a while now, though having tags that can be populated by users would help discovery greatly.
Are there any plans to deal with the most common annoyances regarding Lemmy? In my opinion these are all based on federation:
- Some completely automated way for users to join Lemmy. Yeah, it’s not hard to select a server and it’s a “good thing to do”, but it’s still better to give people the option to go for convenience instead of the “proper” path. Maybe some kind of system where instances sign up for this general, convenience way of signing up, and the registered users just get automatically distributed evenly across those instances.
- Duplicate post aggregation. The nature of federation will always make it make sense to have duplicate communities, but this will also make posts with the same links, same images, same videos, etc show up in people’s “all” feeds multiple times. It is technically possible to algorithmically detect these duplicates and offer users a UI option (not actual backend merge) to merge them all visually into one post.
- A way to backup your whole user data and completely restore it on any instance you want. If an instance goes under, it should be possible to keep all subscriptions, all your posts, all your comments, and migrate them to a new instance.
Yeah, it’s not hard to select a server and it’s a “good thing to do”, but it’s still better to give people the option to go for convenience instead of the “proper” path.
We could add a “fast join” button to the signup dialog on join-lemmy.org , where it takes you to a random instance’s signup page.
Overall though, we should ignore the “advice” from reddit that tells us that people are too stupid now to sign up for anything now. People did this for every forum and every other site all until ~2005 when US tech gobbled up most services, and ppl continue to show us that yes, they do know how to type in a username, password, and email to sign up for something.
Duplicate post aggregation. The nature of federation will always make it make sense to have duplicate communities, but this will also make posts with the same links, same images, same videos, etc show up in people’s “all” feeds multiple times.
In lemmy-ui we have a post-deduplicator for feeds, but unfortunately not a lot of other apps (including jerboa, that’s my bad) have added something similar.
A way to backup your whole user data and completely restore it on any instance you want. If an instance goes under, it should be possible to keep all subscriptions, all your posts, all your comments, and migrate them to a new instance.
Settings export already exists. Copying historical content and rewriting history isn’t possible in a federated system, but we do have an open issue for data export.
we should ignore the “advice” from reddit that tells us that people are too stupid to sign up for anything
Definitely agree. The problem is just when someone in the past said “you should join <forum x>!”, you were always able to just immediately go to forum x’s signup page and sign up. But if someone hears of Lemmy, and goes to join-lemmy.org, there is no way to go to a signup page directly. They have to first learn about the multiple servers, and choose one. I think a “fast join” button like you say should be fine, and immediately next to it something to catch all the advanced actually curious users with something like a “advanced sign-up”
In lemmy-ui we have a post-deduplicator for feeds
That’s weird, because that’s exactly from where I’m coming from, I’m always using the lemm.ee website directly on all my devices, and I constantly see duplicate posts.
Copying historical content and rewriting history isn’t possible in a federated system
I have less knowledge of this topic so I’ll defer to you, but I have the feeling this may not be true. You might of course not be able to ensure consistency between all instances, ensure that it’s been changed everywhere, but I really can’t see why this is any different than “editing” a comment’s content or a post title, which is already possible. Why wouldn’t it be possible to “edit” the comment/post author in exactly the same way?
Thanks for your response and all you’re doing!
That’s weird, because that’s exactly from where I’m coming from, I’m always using the lemm.ee website directly on all my devices, and I constantly see duplicate posts.
I was thinking about what you were saying yesterday and I had another look.
- on the subscribed feed, using “New Comments”, crossposts seem to indeed aggregate
- on the All feed, using “Top of the Day”, crossposts would indeed appear multiple times, depending on their respective scores
I like to use “Top of X” a lot 🤔
Then that’s where the issue comes from. I guess you might open an issue on the Lemmy GitHub so that crossposts are grouped in the Top views too rather than separated by their own upvote scores
The problem is just when someone in the past said “you should join <forum x>!”, you were always able to just immediately go to forum x’s signup page and sign up. But if someone hears of Lemmy, and goes to join-lemmy.org, there is no way to go to a signup page directly.
People should probably stop saying “join Lemmy”, and instead link directly to the site/instance they suggest
it’s still better to give people the option to go for convenience instead of the “proper” path.
https://phtn.app/signup gives a prepopulated list
show up in people’s “all” feeds multiple times.
Which interface do you use? Crossposts only show up once on the default UI
A way to backup your whole user data and completely restore it on any instance you want. If an instance goes under, it should be possible to keep all subscriptions, all your posts, all your comments, and migrate them to a new instance.
You can already export and import your subscriptions between instances (account settings - import/export)
Posts and comments can’t be migrated, but Mastodon doesn’t allow it either.
Mastodon currently does not support importing posts or media due to technical limitations, but your archive can be viewed by any software that understands how to parse Activity Streams 2.0 documents.
gives a prepopulated list
The official one also does that. I’m talking about choosing a username, password, and email maybe, and then clicking register, and being done. No thinking involved.
Crossposts only show up once on the default UI
False, you get links to the other posts, of which you posted a screenshot, but each post is handled as being completely separate. If you are in the subscribed, local or all feeds, you would see all of these posts separately. Have you really never noticed scrolling by “the same” post multiple times? You have to go to each post manually to get all the comments to the “same” thing.
but Mastodon doesn’t allow it either […] due to technical limitations
Yes, I know that. But I’m also a programmer and I know that “technical limitations” is mostly a term for “that’s how we started it and it would be too costly to solve now, so we’ll just dismiss it” and not for actual limitations (i.e. not technically possible). It’d maybe require breaking changes of some kind or some kind of annoying backwards compatibility workaround, but that is why I’m asking. I’m not completely familiar with activity pub, but there’s likely some key used to verify posts/messages are made by a certain user, and there’s currently no way to transfer or change that key to a new account. But it seems very technically possible to me, and also possible without massive security issues. So that was my question, is there any plans to do this or no?
The official one also does that. I’m talking about choosing a username, password, and email maybe, and then clicking register, and being done. No thinking involved.
This should probably be handled more by people when they talk about Lemmy. Instead of explaining what Federation is, just point people to https://vger.app/settings/install so that they can install an app.
Voyager by default suggests Lemm.ee as the instance to register, so no thinking indeed: https://vger.app/profile
That’s the recommended approach nowadays on !fedibridge@lemmy.dbzer0.com
False, you get links to the other posts, of which you posted a screenshot, but each post is handled as being completely separate. If you are in the subscribed, local or all feeds, you would see all of these posts separately.
I only see them once in my subscribed feed. You may indeed see them multiple times if you are not subscribed to all the communities.
You have to go to each post manually to get all the comments to the “same” thing.
This one I agree, and it’s why I generally suggest to consolidate similar communities to solve that issue. !fedigrow@lemm.ee has quite a few examples of successful consolidations.
I’m not completely familiar with activity pub, but there’s likely some key used to verify posts/messages are made by a certain user, and there’s currently no way to transfer or change that key to a new account.
You’re looking for https://activitypods.org/ . I haven’t been following their recent progress, not sure how mature their solution is, and how many Fediverse platforms support them.
False, you get links to the other posts, of which you posted a screenshot, but each post is handled as being completely separate. If you are in the subscribed, local or all feeds, you would see all of these posts separately.
I understand your frustration, however these can be multiple posts but to different communities with varying focuses and moderation styles.
Simply consolidating all the comments in one introduces its own problems.
That’s why no one suggested “simply consolidating”. I didn’t suggest any solution at all. I’m just posing a question of if this actually pretty big problem is attempted to be handled.
I guess it’s just been mentioned too much in the past that it still comes to mind when I hear this. Sorry.
Multispam is one of the things that is genuinely a threat to Lemmy’s usability. If you follow certain topics, you start seeing 2-5 copies of every post. It’s a genuine spam problem and “just block” or “just scroll” is as much of a non solution as it is with other spam.
Some users use it as an ideological war on instances they don’t like, which is extremely funny.
Personally when I want to share something to multiple communities, I deliberately space out the posts in time. It has the slight downside of potentially showing up for a person repeatedly for days, but I think this is outweighed by the upside of them not seeing the same post multiple times within a few minutes, which is really annoying. And it has the added advantage of being more likely to be seen by people who weren’t online at the right time to see it the first time.
Sometimes. But other times the instance/moderation vibes of each post will be different. Sometimes I enjoy seeing how different groups respond.
- A way to backup your whole user data and completely restore it on any instance you want. If an instance goes under, it should be possible to keep all subscriptions, all your posts, all your comments, and migrate them to a new instance.
This would be great… also even if the “restore” part were not possible (yet?) I feel offering a way to extract your data might even be a requirement for a server to be fully GDPR compliant (though I could be wrong on that, IANAL), reddit does allow you to download your data after all.
Native push notifications would be awesome for Lemmy! I’m keeping my fingers crossed.
Unified Push support would be great.
https://github.com/LemmyNet/lemmy/issues/2631
It’s coming in the next release
Thanks for the info! It looks like we’re almost there - 81% completion means the final release of 1.0 update is just around the corner!
Let’s go! No more email notifications
There is an issue open requesting this… I been following it for a while.
