Posted on June 4, 2022

I am planning on doing kernel upgrades tonight. If they go as smooth as last time they will complete by 11:30 but may take another half hour to check NFS/NIS bindings.

This will affect all Eskimo North services including our Fediverse websites:

friendica.eskimo.com, hubzilla.eskimo.com/, nextcloud.eskimo.com/, and our main site eskimo.com/

The downtime for any given service should not exceed 10-15 minutes.

Posted on June 3, 2022

About a month ago I posted about upgrade issues with Nextcloud. I did not realize until a few days ago that it broke NIS logins because I had created my account on nextcloud before hooking it into the systems NIS authentication system. A ticket I received a few days ago alerted me of this fact.

In order to fix Nextcloud properly I am essentially going to re-install it except the database and files will remain in place. Then I will need to re-install all the apps including the one that provides authentication to the majority of Eskimo’s users via NIS.

Because the install instructions tell me to stop our web server during the duration but this does not make a lot of sense and I don’t wish a multi-hour and potentially multi-day interruption, I am going to instead disable the NextCloud config in the web server. This will cause any calls to Nextcloud to go back to our home page, then when the new version is installed, undo the web configuration disable, at which point it will be operational but without the applications and you still will not be able to login unless you, like me, created your login before I had NIS wired in.

Because the NIS connection actually involves an application, this will not work until I get the apps re-installed.

As a consequence tonight at some point if you go to nextcloud you will just get our home page. When nextcloud initially comes back you won’t be able to login right away, it may take a day or two before this capability returns because there are a lot of applications and some require some configuration before logins will be operational again.

I got something super awesome working on my computer tonight. I regularly run Ubuntu Linux, also have arch installed, slowly working on building that environment, and that's where the majority of my work gets done.

But I also like to play a 3D MMORPG called flyff (Flying for Fun), so named because you can fly around the game on various vehicles.

Anyway, prior to tonight to actually play I had to dual boot into windows, which meant I couldn't play between work items or whatever because I would have to boot the computer back and forth.

I had a win10 virtual machine but the virtio drivers only managed about 5fps and a lot of lag between action and display getting updated, not adequate for game play, ok to sell off items in a store or something but not for actual battles and navigation was VERY painful in that mode.

Well, my machine is an i7-9700k based systems, and in addition to the usual vt-x and vt-d virtualization technologies they also have vt-g. What is that? It is a virtualized GPU, actually it allows you to create as many virtual GPUs as you like within the contraints of memory.

I had to build a kernel to support this because the one Ubuntu supplies does not, but that's okay I have a zillion other reasons I build my own kernels anyway.

So once setup, and that was some work, initially I got a black screen, then fixed that but couldn't see boot messages or my cursor and keyboard was super lagged.

I discovered that they had wrapped libvirtd in apparmor and the apparmor profile did not include many of things qemu needed to access to implement virtual gpu's in guests hosts. So since I'm the only one using this machine, I did not feel a need to protect the host from the guest in this manner and simply disabled the profile and now everything works.

It's not QUITE bare metal but it's close enough for decent game play AND qemu and the virtual GPUs work together to share the physical GPU with the host and guest just as the normal emulated display would.

Another reason to love Intel graphics! With any other vendor I would need TWO gpus, one for the host and one for the guest in order to go pass through to the guest. But this let's me share ONE gpu in a way that is very seemless and transparent between the host and guests. Great solution, would be nice if Ubuntu would implement it natively into their network manager, I had to hand code the xml file for the virtual machine because virt-manager didn't know how to do it.

I did not get the kernel upgrade done last night, between the fact that I broke a tooth and so had to go to bed early so I can get to the dentist today and the fact that 5.18.0 had some compiler errors, I’ve got patches from the developers that fix them but haven’t had time to make and install the new kernels yet, the kernel upgrade didn’t happen. So far I haven’t had anyone express connection difficulties, but if you experience them in the meantime, mosh is a possible work-around. Else I’ll at least get the guest machines done tonight or tomorrow. Host servers (physical machines) will have to wait until I know I’ve got a working car.

With respect to the new router, I have it here, still in the process of figuring out how to configure it. The interface is quite different than the old machine owing to vastly expanded capabilities. This is really a full fledged multi-media box not just a router although in our case it will be used just as a router. It is also a video recorded, network controller, etc.

This affects all eskimo.com services:

I am planning an emergency kernel upgrade of at least the most used shell servers, the mail server, and the web server tonight. May not be able to get to everything as I can not stay up late as I broke a tooth last night and need to get up to see a dentist tomorrow.

This upgrade mostly involves the fallback of MCTCP to TCP. MCTCP is a multi-path TCP protocol that allows a connection to be maintained even when the end point IP addresses change. These are mostly used with mobile phones but any device that does not talk this protocol needs to fall back to TCP, these would mostly be older devices without wireless capability.

The existing kernel has a bug in this fall-back code. Because some of my customers have antique computers that may be affected, I want to try to get the publicly facing computers upgraded if possible tonight but I can’t stay up late to do it. I’m going to focus on the physical servers in case they fail to reboot. I will not have a car available after tonight until sometime next week as my wife’s car needs to go into the shop and she will need mine to get to work.

I also have the new router but have not had time to install the drive and get it configured. This should happen in the next week or so.

Posted on May 22, 2022

Now that duckduckgo.com and swisscows.com/ch are censoring, as bing.com and google.com have always done, and since what else is out there, save for yandex.com which censors but on different content, there is nothing else out there that does not censor except for yacy.com. Searx DOES only proxy requests from the others above, it provides privacy but does not provide uncensored results.

So to that end, I’ve installed a federated Yacy server and made it publicly available.

You can reach it at yacy.eskimo.com/ or from eskimo.com/ (web apps) menu.

It’s just started indexing so the local database isn’t very complete yet but it will return results from ALL of the federated Yacy peers so you will see good results even now. They will only get better with time.

However, use with caution, remember this is NOT a censored search engine so no doubt you will find things in the Not Safe For Work and NOT Family Friendly category as well as ALL political views, even those that offend you.

Posted on May 21, 2022

This pertains to friendica.eskimo.com, hubzilla.eskimo.com, nextcloud.eskimo.com, and www.eskimo.com.

I finally received the info I needed from Ubiquiti and we have a new router ordered. It should be here in 7-10 days and will provide 6.8x the CPU power of the existing router.

Our existing router is sometimes saturating between 6pm-9pm Pacific Daylight Time, even though the bandwidth is not anywhere near maxed out. This causes latency and dropped packets.

The existing router handles high throughput if TCP traffic if packets are large, but it does not handle high UDP traffic or small TCP packets. The new router should improve this substantially.

I also bought an optional 1TB hard drive for the router (it has 128GB of flash internally) which will allow me to install a nice desktop and more security software.

For anyone looking for a router, I really love Ubiquiti products because of their excellent firewall capabilities and interface and the fact that they are Debian based so you can basically install anything from Debian as long as it is present in the MIPS ports, and if it isn’t you can self-compile as it also includes the normal Debian port of GCC and libs.

Posted on May 20, 2022 Pacific Daylight Time (GMT-0700)

Kernel upgrades and reboots are completed.

All services are active. All NFS mount points working. All NIS binding successful.

Only server where there was a problem was one customers private virtual server and that because he had configured SSL but the SSL certificates were not present.

Not really sure what changed, but all the machines booted significantly faster this time around. Not sure if this kernel boots better or if some changes Ubuntu made improved efficiency.

This impacts all eskimo.com services including shared web hosting, virtual private servers, e-mail, shell access, friendica.eskimo.com/, hubzilla.eskimo.com/, and nextcloud.eskimo.com/ federated social media services.

Posted on May 19, 2022

We will be doing kernel updates on all systems Friday May 20th starting at 11pm Pacific Daylight Time (-0700 GMT). I expect interruptions of individual services not to be more than about ten minutes and it should be completed by 12PM (midnight).

This will impact all of Eskimo North’s services including:

eskimo.com/

nextcloud.eskimo.com/

friendica.eskimo.com/

hubzilla.eskimo.com/

This kernel upgrade is a point release that fixes some bugs. It does not change capabilities or interfaces so should have no impact except to prevent crashes and processes hanging. We haven’t experienced either of these with the current kernel but others have.

I use tickless kernels built from upstream sources on kernel.org in my own organization. Tickless kernels are especially valuable for laptops where wasted CPU cycles equal wasted battery life and on servers hosting virtual machines where all the time spent wasted on clock interrupts where no work is scheduled adds up to a significant portion of CPU cycles on the host machine. The downside to tickless kernels is there might be slightly more latency but not enough to affect audio or video on workstations as near as I can tell.

These kernels are available at eskimo.com/kernel/linux-(versi…

These Linux kernels are provided with no warantee. Use them only if you agree to accept all risk for any damage to your machine or it's contents.

The are several differences between these kernels and those provided by Ubuntu, Debian, Mint, Redhat, etc.

First, these are all upstream kernels, they have no vendor patches.

Second, they are tickless kernels. CONFIG_NO_HZ is set, CONFIG_NO_HZ_FULL is not set. On the client kernel, CONFIG_HZ_1000=y, on the server kernel, CONFIG_HZ_100=y.

Thirdly, these kernels all provide support for both UDP and TCP protocol NFS versions, 2, 3, 4, 4.1, and 4.2. Ubuntu has chosen to eliminate support for NFS UDP and version 2, and effectively version 3 since there is no provision in the original version 3 specification for TCP.

The server kernel is non-premptive, the client kernel is fully preemptive. This provides maximum processing ability on the server, minimal latency on client machines.

Unlike in the past where I compiled only '.deb' packages for debian based systems, I am also now providing '.rpm' packages for Redhat based systems.

There are also RPM only packages in a subdirectory called centos6, this kernel is compiled specifically for centos6 which does not work on the default Ubuntu configuration from which my configuration is derived.

There is a set of .deb files in "debian", these are the same as the server kernels except that zstd compression is NOT used in the packages because of debians version of dpkg being unable to support it and they are signed so that debian package loader doesn't squak.

In the case of Debian, download all three '.deb' files for the kernel of your choice and install with: dpkg -i *.deb

In the case of Redhat based systems, everything from Centos6 to CentosStream, Fedora all modern flavors, and Scientific Linux, install by downloading both .rpm packages and install with "rpm -i *.rpm"

Note, in the case of RPM some systems will complain that it conflicts with existing header packages installed, in which case remove the existing package first with:

rpm --nodeps -e kernel-header