Access Denied
You don't have permission to access "http://support.microsoft.com/ems/auth/landing" on this server.

Reference #18.d5d9c217.1728052365.2a2135c9

errors.edgesuite.net/18.d5d9c2…

This outage, I learned three resources were particularly important to have available on more than one machine:

DNS – Without this mail will returned no such address.
SSL Certificates – Without this no encrypted services, mail, web, databases, can be started.
Mail spool – Without no mail services.
Home directories – Without no mail folders other than INBOX, no customer websites, and no shell services.

All of these are single points of failure that bring important services down entirely. Here are some general priorities I try to achieve.

No loss of customer data, e-mail or home directory contents. I have been very successful at this objective.
No loss of incoming mail. Not 100% on this mostly because of network outages making DNS unavailable.
The uninterrupted ability to send and receive mail, more severe issues in this area.
Uninterrupted availability of your websites.
The availability of some shell servers. I do not strive to have all of them up all of the time because there is enough overlap that services available on any one server are also available on others. Because of the direct access to the OS on these machines, security is a more difficult challenge than on other services so I do require more time to address issues on these.
Ancillary services such as Nextcloud, Friendica, Hubzilla, Mastodon, and Yacy.

Here are plans I have to address these issues at this point. They are not fully developed which is part of the reason I am sharing this now, to get your input during the process.

My workstation at home is an 8-core i7-9700k with 128GB of RAM and about 20TB of disk. It is on 24×7 but occasionally I boot windows to play games, so not available 100% of the time but obviously I am not going to be playing games during an outage. Nothing I do really requires 128GB on this machine, I just happen to have an opportunity to acquire 128GB of RAM for about $36 so did. But now since I’m also running Linux on it, and I have five static IPs with Comcast, my idea to address not ever disappearing from the net or loosing incoming mail is to setup a virtual machine on my box here and on it install bind, postfix, and set it up as a name server that is outside of the co-lo facility so if our network connection goes down we will still have a working name server and then postfix will be setup as a store and forward server, that is it will be a lower priority MX server that if the first two are unreachable mail will come to it, store, and then when the primary servers come back online it will forward to them. This would address the second issue, not ever losing incoming mail.

The third issue is more difficult to address because the mail spool is a single point of failure. I could use rsync to maintain a near time duplicate, but the issue is if we switch to that during an outage of the primary server and then rsync then stop the primary incomings and let mail go to the store and forward server while we rsync any changed mail spools back to the original spool directory, any mail that came in between the last rsync from the spool to the secondary spool would be lost. I have to do some experimentation to determine how often rsync can reasonably be run and how minimal that time span can be made.

I can do something similar with home directories, this is less problematic than mail spool because the mail spool contains all INBOX mails for a given user in one file, but most home directory files are not subject to as rapid change and only those people who use procmail to sort into folders will risk any loss in this case, and we can rsync any files with a more current update when primary storage goes back online. If we can duplicate home directories then duplicating the web server is pretty trivial, in fact when we get the big machine stable we will have two web servers operational under normal circumstances.

So while not totally thought out I’m letting you know how I plan to address these issues but open to input. Particularly if there is some risk of losing mail between the time of last rsync and the primary system going down, is that risk worth having the ability to have access to mail during an outage of the primary server?

Now in the more immediate future, the motherboard arrived for the ice, I don’t know for 100% sure if it is the motherboard or power supply, I replaced the supply with one I had on hand but still had the same problem but I’m not 100% sure that supply isn’t also dodgy as it is from the same vendor and I do not remember it’s history. At any rate, I’m going to try to replace the motherboard tonight and if the machine works, I will return it to the co-lo facility Friday evening and take down Inuvik which has friendica, hubzilla, mastodon, yacy, and roundcube on it and take it back home to replace the power supply, and probably return it Saturday depending upon time frame. Power on that machine is kind of a nightmare but I should be able to replace it in one night.

Lastly, I am preparing kernel 6.11.1 for installation, 6.11 fixes a couple of issues. 6.10.x had an issue with some of our CPUs when it came to changing clock speeds in response to loads. It detects an error when writing the MSR register, this is a register in the CPU that controls, among other things, the clock multiplier. It actually succeeds and so it does change clock speeds appropriately but it doesn’t know it succeeds and so generates kernel splats. This is fixed in 6.11.x. I will apply this when I am at the co-lo so there will be a brief (around 2-3 minute) interruption in every service.

To the best of my knowledge, everything is back online now but not all of the hardware, thus some things will be not as fast as usual.

We have four physical hosts providing the services with numerous virtual machines on these hosts. Two hosts are i7-6700k 4 core/8 thread systems with 64GB RAM, an i7-6850k system 6 core/12 thread system with 128GB of RAM, and an i9-10980xe 18 core/36 thread system with 256GB of memory. Of these four machines the last is really the big work horse as it has more CPU and as much memory as all of the other machines combined. It also has dual nvme RAIDED and 16TB hard drives RAIDED. I had intended this really to be the model for our next generation servers. Even though this CPU design is five years old, Intel has not produced any non-Xeon systems capable of addressing this much memory since and memory is our biggest constraint.

The newest system has become unstable and it is unstable in a bad way, instead of merely rebooting and coming back up, it is hard hanging. I have had this very issue with this server before and the last time around it ended up being a bad power supply. But in the meantime the Asus motherboard also developed an issue where it would not see one of the memory channels. This is typical of a bent pin on the CPU socket except I had not had the CPU out of the machine.

So I bought a replacement Asus Motherboard, and it had exactly the same issue. Asus support told me the memory I was using was not compatible (even though it had previously worked), and so at this point I decided to try another company, and went with an Asrock Motherboard. That motherboard ran four hours and then died with a puff of smoke. Upon examination this motherboard melted the soldered connection between the power and CPU socket. The i9-10980xe CPU is an extremely hungry chip and can draw as much as 540 watts with all cores fully busy at 4.8Ghz. Even though the Asrock motherboard was designed for i9-xx9xx series of CPUs, it was designed for earlier models that had fewer cores and addressed less memory and so were not as power hungry.

So I then bought a Gigabyte board, went this route because, like Asus, they are designed for overclocking and thus have much more robust power and ground traces to handle the power requirements of monster CPUs. And initially all was well, it ran stable at 4.8Ghz all cores loaded with no issues.

However, after a bit of operation it started locking up. And when I checked the temps they were high even though I had previously tested under full load and the CPU never got hotter than 62C. What had happened is that I did not use enough thermal paste and it had developed an air gap between the CPU heat spreader and the cooler right in the middle of the heat spread so cores near that area were overheating. I fixed that but it still wasn’t entire stable.

Initially the power supply I used that subsequently died, was a Thermaltake. When it failed, I replaced it with a Gigabyte PSU, my thinking being that since they make components designed overclocking, this PSU should be, like the motherboard, more robust. Apparently my thinking is wrong. Net wisdom seems to suggest the best units are now made by Seasonic, I actually ordered through Phanteks but the supply is a rebranded Seasonic. This time around I went with a slightly higher power rated supply so it will be less taxed, the prior two supplies were 1000-watt units, which with a CPU maxing out at 540 watts, a very minimal graphics card, maybe 50 watts, perhaps 100 watts worth of drives and another 100 worth of fan, should have been enough but at full load it is running at the upper end of it’s capability. So this time around I bought a 1200 watt unit so it has a bit more overhead.

This power supply will arrive Monday. Then at 4:00AM Sunday morning we disappeared from the Internet. The machine I was using as a router which had also been rock solid, died. So I moved the network to another machine with dual NIC’s but one of those NIC’s was a Real-Tek and the Linux Real-Tek network drivers do not work well and can not operate at 1Gb/s, so had to run at 100mb/s but that proved totally inadequate, lot of packet loss and very bad performance.

I went back to the Co-Lo and took the network card out of the failed machine (Ice) and put it into Iglulik, when I powered Iglulik back up it would not boot. I took the card out, it still would not boot, so I put it into a third machine and now that machine would not boot. So now I’m in a situation where I have three dead machines, one that periodically locks up, and has an interface that only works at 100mb/s so I moved the net back to that machine and proceeded to try to diagnose the others. The easiest machine to get back online was igloo, I could get a grub prompt but not boot fully into Linux, but the fact that I could get to grub prompt suggested hardware was ok and just the boot configuration had gotten mangled, so I repaired the initramfs system and re-installed grub and it came up and ran. This at least allowed us to have DNS and to have a working incoming mail server although it could not accept SSL encrypted because the encryption certificates were not available.

I brought Iglulik back, and this machine is particularly important because it has the /home directories and the SSL certificates. I could not even get a grub prompt, but what is more I could only see six of the seven drives present on the machine. Everything on this machine is RAIDED except for the root partition because at the time I did this build, I did not know of any way to boot off software MDADM raid. I have since figured that out and so Inuvik is 100% raided except the EFI partition and even that is replicated, just manually rather than by software RAID. So of seven drives, is one of the RAIDED drives going to fail? No the drive with the root partition failed.

So I replaced the drive and then tried to restore from backups. Problem is when I mounted the partition labeled backup, there was nothing. At this point I began to wonder if I had been hit by some malicious virus but at any rate, I had backups on my home workstation as well as a guard against a Ransomware attack. I tried to restore from that but it was corrupt. Now I am faced with having to rebuild from scratch and that could potentially take weeks. But then I mounted all the partitions and found that the one labled libvirt, supposed to have images for virtual machine actually contained the backups and I was able to restore.

While it was restoring, at this point I had been up for nearly 48 hours straight, I am 65 and don’t handle this well anymore, and so I slept for four hours while it was restoring. When I got up the restoration was finished but it still would not boot. There was something wrong with the initramfs file system but I could not determine exactly what. Eventually I noticed it was trying to mount the wrong /root partition UUID, because when I restored the system the file systems had to be re-created and had a new UUID as a result. So I fixed the fstab and it still would not work, and I was up all night again last night chasing this down. I finally discovered that /etc/initramfs-tools.conf.d/resume had the wrong UUID in it and fixed that. Now that machine was bootable and ran. Because I knew it was going to have to do more than it was originally intended to do until I get the remaining machines repaired, I attempted to remove some unnecessary bloatware, for example CUPS, not needed since not doing any remote printing from this machine AND there is a security issue that makes it wise not to have on servers anyway. Also bluetooth software, the machine has no bluetooth hardware so that didn’t make a lot of sense. So removed that. Then found wpa-supplicant, didn’t know what it was for so looked it up and it said it was for managing wireless connections, well no wireless hardware either so removed it, then the machine got very very sick. What the online material didn’t tell me is that it’s also a back-end to NetworkManager and tied into the dbus daemon and when you remove it it breaks both things, and with both things broken the machine was so insane that I had a very hard time getting it re-installed, and then when re-installed it still did not work. I finally determined it was necessary to re-install NetworkManager to fix and got it working. Took it back to the co-lo facility around 7am and installed it.

This was enough to mostly get us back up into service except I had to get backups from the other machines to get their functionality up on this hardware. So I got stuff started restoring and went to sleep, got up after about four hours and started up those services which had recovered, mostly the virtual private servers, went back to bed, slept another six hours, got up and restored the remaining things to service.

Now during this time, particularly the second day into it, Tuesday, I got some calls while I am frantically trying to figure out what was wrong with Iglulik even after I had replaced the drive and restored from backups and I was somewhat rude to a feel people. I apologize for this but like I said earlier, at 65 I do not have the endurance I had at 25.

And I’ve got a bunch of hate calls and hate mail about our reliability, but here is my delimma, I have not raised prices in 30 years, but if I even whisper a hint at doing that people threaten to bolt, and by the same token the only way to provide more reliability is more redundancy. For example, having enough disk to maintain near-time copies of home directory and mail directories, it would have been possible to maintain at least minimal services. I ask people to refer more people to use because that is another way to increase income and provide some of these things but only happens minimally. And this is a rather niche operation so I do understand that.

And on the opposite side of the hate mail and calls, I also got calls from people who appreciated my efforts, and I want you to know I appreciate your patience.

If you are thinking of upgrading to IOS 18, you may want to hold off as there is a bug in the mail application. You may be able to work around this by adding just a ‘/’ for the mail path and you mail not.

If you have already upgraded to IOS 18, you can use webmail on our website (under web apps) to access e-mail IF the ‘/’ fix does not work for you. Here are the contents of a ticket sans user info:

The mail app on my iPhone and iPad stopped working a few days ago.

My subscription expires on 10/5, so I need to know if this is a fixable problem.

I reloaded the app several times with all the correct info (below). It returns the following error messages:

Updated Just Now
Account Error: Eskimo.

Cannot Get Mail The mail server "mail.eskimo.com" is not responding. Verify that you have
entered the correct account info in Mail settings.

Server message "Mailbox doesn't exist: mboxes/buy/ebay/guns/s&wjframegrips"'

--------------------------------

I checked and the referenced file does exist. I also reloaded the app several times with all the correct info:

Incoming mail server
Host name: mail.eskimo.com

Outgoing mail server
SMTP - mail.eskimo.com
Primary server - mail.eskimo.com
Host name - mail.eskimo.com
Use SSL - On
Authentication - Password
Server port - 465

Robert Dinse posted 9/22/24 17:16

It turns out this is a bug introduced into IOS and IpadOS 18 mail app. One user has found that by adding a / to the path (initially a null path) it made it work for them.

The entire thread is here: discussions.apple.com/thread/2…

Please keep me updated if status changes, and/or if this work around works for you.

I've been using Linux as my primary workstation OS since the late 90's but until now I've never really found a music player I liked. I have been using WinAmp under Wine just because it provided the functionality I wanted though a bit clunky, like it took eight minutes to load my collection. Recently developers have broken Wine to the point where the audio just wasn't good, lot of buffer underruns, skipping, but only wine.

So I went looking for a Linux player that was workable and I found Audacious. I've seen it in the past but not in classical GTK form where it is structured exactly the same as WinAmp. I didn't care for the skin, but guess what? Audacious can read and use WinAmp skins, so I just installed my old WinAmp skin and I'm mostly good!

There are a few annoying buglets. It chokes on some mp4 files even though it uses ffmpeg as an input plugin and ffmpeg groks the same file fine.

It also insists on placing the file extension of .ogg and .mp3 between title and artists but does not do this for other formats. Odd.

The sort by title doesn't work right either and that's a bit annoying but I can always find what I want by using J and searching. It also has some neat graphical extensions, one is attached.

The machine I was working on the other night, all up and running now! And by running, I do mean RUNning, 18 cores at 4.8Ghz, most things on SSD, everything on RAID, even the root partition.

The following sites are now up and running, please check them out if you haven't recently:

friendica.eskimo.com/

I started this site after I was kicked off of Farcebook in 2020 for voicing my opposition to the RNA based Covid Clot Shot.

Originally, I started on an i7-6850 machine which proved over time to be
not up to the task. Hence the new machine.

Also:

hubzilla.eskimo.com/

And Most recently:

mastodon.eskimo.com/

And lastly a Federated Search Engine:

yacy.eskimo.com/

All Free.

Last night I spent several hours fine tuning our newest machine, Inuvik, I was not able to get any faster CPU speeds, in fact I am running at 4.8Ghz now because I did find with some tests, notably long fft tests, it did show some instabilities even though in a weeks operation these have not manifested.

But I was able to significantly improve memory performance from 2133Mhz to 3400 Mhz and on an 18 core CPU every memory cycle you can get is precious.

I was also able increase the Mesh frequency slightly from 2.6Ghz to 2.8Ghz. The mesh is similar to rings in lower core count CPUs, it is used to provide communications between the cores. I don’t know how much traffic this actually is in a Linux environment so I don’t know to what degree this helps but every cycle one can get anywhere.

Ok, after that I turned to kernel upgrades, and the last machine to upgrade was the one used for our router. It had a drive that had previously shown some SMART errors but after running a diagnostic they went away and it behaved until last night. Last night it absolutely would not boot. Strangest damned thing, could read the drive, write the drive, but could not boot off of it. I’ve never seen a drive failure cause this so I assumed a software issue and re-installed grub, re-installed kernels, re-built the initramfs system, and these are pretty much all the software components you should need to be able to boot but no go. It would find and load grub but grub couldn’t find the kernel, very very odd. I was so convinced this had to be software issue that I had to re-install 25 times to convince myself otherwise. I finally stole one of the drives out of the RAID array and turned it into a new system disk, that worked. But it does not have everything it needs and it’s not a healthy young pup itself.

So for now I’ve moved the routing and all the virtual machines off this box. At present the two services are down are the NIS master which means you can’t change your password or login shell at the moment, and a DNS server, but we have six so that isn’t going to seriously impair things.

I have a new drive which I had purchased when the first drive started puking out SMART errors, and it also is a 7200 RPM drive with 4x the cache the old drive had. At present I’m copying all the data off the failed drive to the drive I stole out of the RAID array to bring the machine back up, and then I’m go replace that old drive that has failed with the new one, recover any data I need for NIS and for the name server off that drive, then reformat and return it to the RAID array.

I am working on a new video conferencing feature to add to our site shortly. I tried to get another suite working but it depended upon a message protocol that we are not able to get to work. This one uses infrastructure I am more familiar with so my chances are somewhat better.

I am also looking at RustDesk as a possible replacement for Guacamole because the developers have really turned Guacamole into an unmaintainable disaster. It really is oriented towards LDAP auth and our network isn’t, so that doesn’t work so well for us.

I previously had Friendica on a server with an i7-6850 CPU 6 core 12 thread at 4.2Ghz, 128GB of RAM. This was clearly insufficient for the traffic we have.

So I upgraded to an i9-10980xe overclocked at 4.9Ghz, 18 cores 36 and 256GB of RAM, database is on two WD black nvme SSD's raid 1, most of the rest of the file system is on 14TB drives.

Load was heavy even though CPU was 95% idle, and disk I/O was nowhere near saturation. I finally found the issue was net.core.somaxconn which Ubuntu defaults to 4096, on this machine this was not enough, bumbed this up to 65536, socket connections immediately jumped to 13,000+, load dropped way down, response way up. About 1000 of those connections were to the database.

Because this item did not show up in any Friendica tuning docs, but did seem highly relevant here I thought I'd share.

The following sites share a common server which had failed:

Social Media:

friendica.eskimo.com/
hubzilla.eskimo.com/
mastodon.eskimo.com/

A Federated Search Engine:

yacy.eskimo.com/

A webmail client:

roundcube.eskimo.com/
roundcube.eskimo.com/

And a Linux shell server:

ssh://manjaro.eskimo.com/

After four motherboard replacements, this machine is again operational. It
is an 18-core hyperthreaded (36 thread) machine with 256GB of RAM, the CPU is overclocked at 4.9Ghz. In this configuration it can consume up to 540 watts if running full-tilt and finding a motherboard that could provide the required power was difficult. The Asrock actually melted the solder connections at the CPU socket.

All of these services except the shell server are free, the shell server runs $8/month for a standard account of $84/year.

See eskimo.com/

Since elm successfully compiled on Rocky8, I decided to try metamail. It successfully compiled as well and is now installed and linked into elm.

Also, this is a reminder that friendica.eskimo.com, hubzilla.eskimo.com, mastodon.eskimo.com, yacy.eskimo.com, and manjaro.eskimo.com will all be going down for probably around six hours tonight in order to replace the motherboard.

The motherboard that is presently installed has two dead memory slots so a quarter of our system memory is not recognizable on that machine.

I'm trying to build a lemmy node from scratch. Unfortunately, I am not knowledgeble in Rust, so I dunno how to fix an error I am getting when attempting to compile pict-rs, the embedded picture store.

Compiling pict-rs v0.4.0
error[E0310]: the parameter type `F` may not live long enough
--> /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pict-rs-0.4.0/src/lib.rs:1219:5
|
1219 | HttpServer::new(move || {
| ^^^^^^^^^^^^^^^
| |
| the parameter type `F` must be valid for the static lifetime...
| ...so that the type `F` will meet its required lifetime bounds
|
help: consider adding an explicit lifetime bound

I've got like ten of these and it results in the compile not completing, and
since I know zero rust, I don't know how to fix. Any advice?

In the long term we have to get away from fossil fuels, not because there isn't plenty of carbon to burn and not because even doubling the CO2 in the atmosphere would not represent the existential crises the money changes would like you to believe it is, but because the nature of all extractive technologies is that they become increasingly expensive because first you go for the most concentrated and easy to get at ores and as you exhaust those you go to increasingly diluted and increasingly difficult to get at ores, and because our future prosperity depends upon being able to continue to increase our use of energy.

Wind and solar are not viable replacements because they are dilute and intermittent. They require way too much land as well as too many exotic and difficult to recycle materials and at the same time they do not provide the reliable energy we need. They can to some degree be a part of the mix but they can not replace fossil fuels.

Two renewable sources of energy which can to a degree are hydroelectric and geo-thermal, but hydro-electric is already largely tapped out and geo-thermal is expensive except in a handful of locations where heat is very near the surface. But new drilling techniques may change this and even now there are locations where heat is very near the surface, such as Yellowstone, that we could but are not tapping.

The most viable short-term energy source is nuclear fission, and in the longer term nuclear fusion but the technology for nuclear fusion is not yet viable and the way we are currently doing nuclear fission would be a disaster if we scaled it as is, however there is a solution to all of nuclear problems in form of molten salt breeder reactors. Molten salt reactors, in general, are inherently safe and do not pose any of the dangers of boiling or pressurized water reactors. The fast spectrum breeder versions of these reactors further can burn much more abundant fuels such as U-238 or Thorium-232 which in breeder can be bread into Pl-239 or U-233 which is fissionable. A fast spectrum breeder reactor can also burn up ALL of the actinide (heavier than Uranium) waste from existing reactors, and this is the stuff that is a long term hazard, the fission products represent about a 300 year problem and if you burn up the actinides this is all you have left.

Further because the fuel is liquid in these reactors and fission products are extracted in real-time, many medically and industrially useful short-lived radio-isotopes can be extracted. And because the fission products are continuously removed, they do not poison the reaction so the reactor can be continuously fueled, ever requiring downtime for re-fueling, and because they are removed, when the reaction is shut down, there is not ongoing heat production from their decay, unlike water based reactors.

Because these reactors do not use water as a coolant, they do not need to be located near large bodies of water. Because they aren't pressurized, they do not need an expensive containment vessel, and because the physics make them self-regulating there is no danger of a run-away reaction or of anything disasterous happening if either cooling is lost or control is lost. If we are going to scale nuclear energy, and I don't believe we really have a choice, then fuel use efficiency, not generating million year waste we can't contain, and inherent safety is something we NEED to have in our reactor fleet.

I am moving eskimo.com domain to a new registry. I have some issues doing this in that my name servers are in the same domain presently. This is problematic as it creates a circular resolution problem where to resolve a name server address here, say ns1.eskimo.com, you first need to access the name server to get the IP address of ns1.eskimo.com but since you don't yet have the IP address you can't get it.

This is presently resolved by the use of glue records. These are records maintained in the upstream name servers that resolve our name servers to physical IP addresses to get around this circular resolution problem. And in the beginning there was only one registry, Network Solutions, so this was okay but today there are multiple and Network Solutions will eliminate the glue records the instant they release the domain but the new registry might take several days to get it in their database during which time we have no glue records.

This is no good, so what I have done is placed a DNS server in another domain which I own, yellow-snow.net, and it will be one of eskimo.com's name servers, however web.com's website for changing the DNS records is not working so I have to wait for their team to change it for me. So we are holding off moving the domain, this may happen this weekend or next depending upon their timing. Anyway wanted to make you aware of that potential interruption.

Another potential interruption is that I am going to be replacing the CPU on this server, currently an I9-10900x which is 10 cores, to a I9-10980xe which is 18 cores, upgrading the CPU cooler fan from one that maxes at 1500 RPM to one that maxes at 3000 RPM to help dissipate the huge amount of heat these CPUs generate, and I'm replacing the RAM which is currently 8x32GB sticks with 4x64GB sticks, and the reason for this is that this CPU has difficulty driving two sticks per channel, just not enough drive current requiring that I slow memory down, but to feed 18 cores I want this memory I/O to be as fast as possible.

So probably next weekend there will be a period where this site is down for that hardware upgrade, but the end result should be faster performance.

With the world-wide support for Marxism, all I can think about it is that people don't know history.

I grew up in Seattle just a couple of miles south of this location. This crap did not happen 45 years ago. And now the infection is spreading North to Shit, er Shoreline where I presently live. They are turning up light rail and that's going to bring all the hooligans to our neighborhood.

I grew up just about a mile East of what is now Hookersville Seattle, and it's really sad to me to see what has become of my home town. When I grew up, hookers advertised in some specialized Seattle newspapers, they didn't walk the streets, and the streets also weren't filled with garbage, homeless camps, and human feces.

So how the fuck did we get to where we are today? Isn't the US law banning Tic-Tok exactly this?