Well damn, I had no idea this was going on. I might have to jump on a bus and go check it out.

Thanks for the information.

4:00 I get the problem about huge hyperscalar datacenters being a menace to their neighbors, and I really think that a lot of LLM usage would be much more acceptable to the general public if users were running local models instead. Of course then developers would have a set of dual RTX 4090 GPU's heating up the office instead, but for some reason nobody seems to care if you are playing Cyberpunk 2077 for four hours every night on your PS5 (well, except for your wife, perhaps...), even if that definitely takes a similar toll on your electricity bill.

And no - not every development task can be done reliably on a local setup, I know that - but a huge amount of the chatbot usage that people do could easily be run locally on a modest setup, dispersing the energy usage and heat disspation to a point where it does not become a localized problem any longer.

There is still a major issue with training on illegally obtained data, but twisting LLM answers to fit a particular political world view would at least be much more difficult. You could either use an abliterated version of the model (although that comes with its own can of worms, security wise) or simply use another model - there is a huge amount of very capable models for just about every possible scenario, and with the latest Gemma and Qwen models, most people will be able to find something that suits their need and hardware.

I run models with everything from 2B to 31B active parameters on a humble Radeon780M iGPU on my Ryzen7-based PC (granted, with 64 GB of system RAM), and many will generate answers faster than I can read them. What we do need, though, is a way to make the setup procedure a lot easier and with much better help for new users. We already have local systems like AnythingLLM and LocalAI that takes care of the infrastructure for you, but what model to download and use for what scenario is as much of a learning curve as the initial setup.

Browsing models on Hugging Face is intimidating at first, and model descriptions are way, way, WAY more technical than most users have the knowledge to appreciate, and if AI in general are to be much more distributed and local, that needs to change.

About Linux kernel, not all of them share the opinion that LLMs are really helpful in every way. There is a talk about how they affect the kernel work on a daily basis, the first 35 minutes is the talk, rest questions and answers: inv.nadeko.net/watch?v=NnV_cWe… -> I don't want to know how much energy is only wasted for all these thousands of bug reports that are duplicated, already fixed, stolen from researchers that were using LLM chats and so on. How can Linus still make such a strong claims when it is clearly not that helpful? When patches even include human fails from 20 years ago, because the data is trained by the whole history of human mistakes, too?

Sad to hear about Bitwarden, but never used it. I always had the feeling this will happen. Even more sad about KeepassXC, because I used it and was happy, but never realized the vibecoding policy and I don't trust those who are not only using them on such important security applications, but especially that early. Was looking deeper into it and it seems there is an anti-AI fork called KeepassChi that forked the version before XC was introducing the new policy. I will look a bit deeper next days, but currently I am 80% sure to switch over to it. The browser integration for KeepassXC is not allowing LLMs and support Chi, so no issues at this front. Chi is hosted on Codeberg for those also want to give a look.

About Linux kernel, not all of them share the opinion that LLMs are really helpful in every way. There is a talk about how they affect the kernel work on a daily basis, the first 35 minutes is the talk, rest questions and answers: inv.nadeko.net/watch?v=NnV_cWe… -> I don't want to know how much energy is only wasted for all these thousands of bug reports that are duplicated, already fixed, stolen from researchers that were using LLM chats and so on. How can Linus still make such a strong claims when it is clearly not that helpful? When patches even include human fails from 20 years ago, because the data is trained by the whole history of human mistakes, too?

Sad to hear about Bitwarden, but never used it. I always had the feeling this will happen. Even more sad about KeepassXC, because I used it and was happy, but never realized the vibecoding policy and I don't trust those who are not only using them on such important security applications, but especially that early. Was looking deeper into it and it seems there is an anti-AI fork called Chipass that forked the version before KeepassXC was introducing the new policy. I will look a bit deeper next days, but currently I am 80% sure to switch over to it. The browser integration for KeepassXC is not allowing LLMs and support Chi, so no issues at this front. Chi is hosted on Codeberg for those also want to give a look.