friendica.eskimo.com

Friendica Bottleneck

I previously had Friendica on a server with an i7-6850 CPU 6 core 12 thread at 4.2Ghz, 128GB of RAM. This was clearly insufficient for the traffic we have.

So I upgraded to an i9-10980xe overclocked at 4.9Ghz, 18 cores 36 and 256GB of RAM, database is on two WD black nvme SSD's raid 1, most of the rest of the file system is on 14TB drives.

Load was heavy even though CPU was 95% idle, and disk I/O was nowhere near saturation. I finally found the issue was net.core.somaxconn which Ubuntu defaults to 4096, on this machine this was not enough, bumbed this up to 65536, socket connections immediately jumped to 13,000+, load dropped way down, response way up. About 1000 of those connections were to the database.

Because this item did not show up in any Friendica tuning docs, but did seem highly relevant here I thought I'd share.

Shoreline, WA, USA
Just a follow up, after this was fixed and the machine caught up with the backlog, it is now running with a load of about five, and this is with a friendica, hubzilla, mastodon, yacy, and roundcube web mail. So on average now 13 cores are completely idle. There are other things I will be adding to this but it just goes to show how important identifying and tuning bottlenecks is. I thought perhaps it might be a good idea to illicit how I found this. I noticed with a ps aux that many processes were sitting in state Sl which means it's a multi-threaded process in interruptible sleep. I wanted to find out WHAT it was waiting on, so I did strace -p pid to trace one of the sleeping processes, I found it sleeping on trying to open a socket that belonged to mariadb. I knew from previous testing that mariadb was capable of around 13,000 transactions per second and it was only doing slightly over 1000, so this lead me to this kernel variable being the issue.
Shoreline, WA, USA