And Then There Was One

This post is uncharacteristically long and heavy. It took three days to write. Buckle up.

I can’t say I wasn’t allowed to have a pet as a child. I had several species of freshwater fish. Nothing exotic or over-the-top, just your typical goldfish, neon tetras, and assorted gouramis over the span of many years. There were small aquatic frogs in there as well. At one point I had a half-dozen or so hermit crabs named Bob, mainly because I couldn’t tell the difference between ’em, and because they’d occasionally swap shells overnight so it was never really possible to tell which was which. However, my sister and I were never allowed a dog, point-blank. There was no discussion, no argument, just a solid wall of no.

So, of course, as soon as we each moved out and got married, we got our own dogs. My wife had several growing up, so she knew her way around caring for a puppy. We were still dating when her family got the most recent one, and I was able to see a lot of the early stages of the puppy’s upbringing, so I kind of knew what to expect. That puppy was Lilo, a red female Shiba Inu.

At that point in late 2006, I didn’t know what a Shiba Inu was. Cryptocurrency didn’t exist, and neither did the “doge” face meme that appeared alongside Dogecoin. Lilo was my first exposure to the breed, and while she had a love-hate relationship with me during the first few of years, we ended up figuring each other out, and I grew to love the Shiba Inu personality.

Julie and I got married in 2007 and, in early 2009, we made the decision to get a dog of our own. Our search ended in Gurnee, Illinois with a tiny, 6-week-old, black Shiba Inu named Saki; on June 8, 2009, we brought her home, decided “Saki” just wouldn’t do — us giving a Shiba Inu a Japanese-sounding name just seemed played-out — and renamed her Daisy.

Daisy became the focus of our lives. We found ways to be with Daisy, or to have Daisy with us, even if we couldn’t physically be in the same space: setting up a laptop on a tray table with Skype on a webcam, projecting a continuous webcam view of Daisy’s enclosure on a wall at the office for all to see (the PuppyCam), two-way conversations via whatever random software I could find that day … whatever worked. We were able to teach her that the way to the bathroom was through the sliding door downstairs, and she’d wait patiently for us to grab her leash and take her out. Daisy learned very quickly who brought her food, and she never — and I mean that in the strictest sense of the word, absolutely never — bit any of our hands when we were near her while she ate. When Julie or I put our hands into her food bowl while she was eating, she’d simply stop and wait for us to take ’em out before resuming. We played with her paws, and she had no problem with us trimming her claws, though she preferred Dremels to clippers. She would sit in my lap, give me a paw, and just stare off at her surroundings while I worked.

She didn’t care for baths, though. She’d tolerate them, sure, because she knew we wouldn’t hurt her, but they were not her favorite. Daisy preferred to lie down somewhere and lick herself clean like a cat.

We took her to some group training classes being held in the large back room of a local PetSmart so she could be exposed to other dogs and learn a couple things, and while she was about as stubborn as Wikipedia would describe a Shiba Inu to be, she did pick up some good habits and earned the privilege of wearing the silly graduation cap.

Julie and I decided to start a family in 2010. Daisy was aware something was changing due to the increased activity around the house and such, and she did not leave Julie’s side throughout her pregnancy. She was protective of whatever it was that was in Julie’s belly, but not to the point of violence. When our son arrived, Daisy was skeptical at first: one of my favorite pictures of that time was when Julie was sitting on our sofa holding our screaming infant son, and Daisy was sitting on the sofa opposite her, leaning back in shock at the noisy object. Captures the moment perfectly.

And so, that is how Daisy was introduced to our son at the age of 15 months. While she didn’t quite act like a sister, being a dog and all, she took a liking to the tiny human and watched over him as he navigated the world. Being a Shiba Inu she kept her distance and made it look like she was detached from the situation entirely, but anyone with eyes could tell that she was being protective. As our son learned to walk, she was right there, following close behind and actually propping him up with her snout when he started to falter. I wish I were making this up as it sounds like something straight out of a cartoon; however, that’s exactly how it happened. She was a surrogate broom/mop as well, happily collecting the delicious morsels of whatever our son decided to throw from his high chair. She was quite selective, though, as she left some behind if the flavor wasn’t to her liking.

The “aloof” personality of the Shiba Inu is something that I really love about the breed. Most of the time, Daisy simply could not be bothered. During the day, she’d lie on the back of the sofa and look out the window absorbing the sun. As a puppy and a young adult, fireworks and thunderstorms really didn’t faze her. If anything she was looking changed, though, she’d make all manner of weird noises, making her a fantastic watchdog. Squirrel? Bark. Mail? Bark and run to the door. Car driving by? Stand up and bark. Someone familiar shows up? Definitely don’t bark — start howling and prancing around instead. And make no mistake, I’m using the word “bark” somewhat loosely here, as Shiba Inus … vocalize. They yodel, howl, squeak, scream … I think one of the more frequent sounds out of Daisy’s mouth was “rrr-ow” — a very nicely rolled “r”, at that — often repeated so it was more like “rrr-ow rrr-ow rrr-ow.” Not as a howl, either, more like she was trying to make it into a sentence, actually, with each occurrence descending in pitch until she was finished. It’s not the easiest to describe in text.

Our second child arrived shortly before Daisy reached the age of 5, and none of her behaviors toward babies changed. Daisy was right there behind the new baby girl the same way she was behind our son, exactly as caring as before. This little dog knew her way around human babies, and our human babies saw this little dog as the integral part of the family she was.

Almost two years after our second child arrived, Lilo passed away and Daisy remained as the only Shiba Inu in our lives. Daisy noticed, and we noticed that she noticed. She slowed down a bit, and was clearly missing Lilo, but after a bit she got used to being extra-spoiled. We started taking road trips all over the place, and Daisy had her own seat of honor in the minivan. It wasn’t a fixed seat — whatever seat she sat in was the seat of honor. She had earned it.

The year 2020 rolled around with all of the hot garbage it brought to everyone. Daisy was 11 years old, her tan fur fading, white hairs showing up among the black. She couldn’t run as fast as she used to anymore, and she’d started hiding under the bed or in the bathroom for storms, but other than that she was still the same Daisy, still as gentle and silly as usual. She’d been alone too long, though, so we decided to bring home a friend: on July 3 of 2020, Daisy met Luna, a Labrador mix we adopted via Chicagoland Lab Rescue, and was instantly reinvigorated. Luna, despite being just 9 weeks old, was almost as big as Daisy but was exceedingly clumsy, as you’d expect of a puppy. She was also overly playful, so Daisy had to keep up, and she adapted masterfully. Daisy took on the role of Luna’s surrogate mother despite having no experience being a mother herself, and she handled it like a natural. They did everything together: went on walks, played in the backyard, visited the vet, had spa days, slept in our bed.

Julie and I bought the camper in 2021, and we did not leave Luna or Daisy out. They came camping with us every time. Our camper had a loft for the kids, which Luna and Daisy loved to visit whenever they could, though Daisy did so with increasing difficulty. She wasn’t getting any younger after all, and Shibas tend to have hip and knee issues as they age.

Around mid-2023, Daisy stopped being able to climb up the stairs consistently. There were times she simply stopped at the bottom and waited for us to come get her; other times she felt like she could do it, but found out halfway up that she couldn’t and fell down. Those times were the worst; she wouldn’t scream, either. In fact, the way she reacted, she seemed almost ashamed about it. There was nothing to be ashamed of at all, though: she was 14 years old. In early June Daisy started stumbling all over the place, tripping over her own paws, tilting her head to the side — we didn’t know what to make of it and thought she’d had a stroke. We were terrified, and thought the worst; I’m going to have to take credit for floating the “we may need to make a really difficult decision soon” idea first. Julie took her to the vet, where the diagnosed it as vestibular disease, basically dog vertigo, and prescribed some meds. We were instructed not to allow her to go up and down the stairs by herself at all anymore, so Julie and I carried her up and down the stairs, and lifted her on and off the bed. We gave her the medications that the vet prescribed, and for the first time in her life she disagreed with the method I used to do so, attempting to bite my hand on a number of occasions. However, after a couple weeks of this, Daisy returned to her normal self.

Last Saturday, October 21, we headed out to the camper to clean it up and get it ready for winterization. It was just going to be an overnight trip, as we only needed to get toiletries, cleaning supplies, nonperishables, etc. out of there, as we didn’t want to risk anything freezing over the winter. As usual we took Daisy and Luna with us, though they rode in separate cars: Luna rode with me, and Daisy rode with Julie. At some point during the drive out to the campsite Daisy started screaming, which lasted for the rest of the ride out there — for a 90-minute drive, that’s torture. Julie and our daughter were quite upset by this, so I grabbed Daisy out of the minivan and walked around the campsite with her to see how she was. She seemed to be alright, waddling around, peeing on stuff, giving Luna the side-eye as usual. Her head tilt had returned, though. In the late afternoon I sat out on the deck alongside the camper and trimmed up her claws with the Dremel while she looked around at the trees and sniffed at the autumn wind.

The trip was short, but we were concerned, so Julie took Daisy to the emergency vet as soon as we got home on Sunday. They recognized it as vestibular disease again and prescribed fresh meds, but also drew some blood for labs. Monday morning I gave Daisy these meds in the usual form — stuffing them into a treat — and she ate ’em right up. When Julie came home, though, she found Daisy lying on the floor in a puddle of liquid excrement. Distressed, she called me and asked if I was on my way home. I was, and was happy to take over; she’d already cleaned up the puddle, but Daisy was still dirty, so I gave her a nice warm bath.

Not 30 minutes after her bath, I heard Daisy scream and looked over to see that situation play out again: Daisy, in a puddle of her own excrement. I quickly picked her up, crated Luna, rushed Daisy back to the bathtub, cleaned up the floor, and bathed Daisy again. I told Julie what happened and we thought it was due to the new medications being stronger than the old, since after this second occurrence, Daisy returned to her bowls to eat and drink.

Tuesday morning, the 24th, started out the same as the 23rd. We got up, I took the dogs out — Luna ran down the stairs and I carried Daisy. The kids refilled the food and water bowls as usual. I stuffed Daisy’s meds into a treat as usual, but she refused to take them. I didn’t blame her after what happened the night before, so I put the treat into a small Ziploc and put it on the kitchen counter behind a bunch of stuff so Luna wouldn’t get at it. Shortly before I left to take the kids to school Daisy threw up on the rug. I again attributed it to the strength of the medication and cleaned it up, but when Julie returned home, she saw a scene similar to Monday’s: Daisy in a puddle of urine, lying on her side. That’s too much to let slide, so Julie called me to ask if I was on my way. Again, I was, so as soon as I got home and parked in the garage, we all got into the minivan and we went to the vet. All four of us were more worried than ever about Daisy at this point, and the “difficult decision” talk came up multiple times during the relatively short ride. Upon arrival they immediately took Daisy into the back and started checking her out.

When the vet told us that Daisy’s temperature was quite high, even for a Shiba Inu, that her breathing wasn’t normal, and that her heartbeat was irregular and a murmur had developed, we knew Daisy was telling us it was time. That difficult decision we’d put out of our mind from 2009 until June of 2023 finally had to be made. We had originally wanted to have someone come to our house to perform the procedure in an environment where Daisy would be comfortable, but given the past few days, taking her home and waiting another 12-24 hours would have been nothing but torture. My first dog was in pain, was dying, and Julie and I made the decision to put an end to her suffering right then and there.

The vet inserted a catheter into Daisy’s front leg, wrapped her in a blanket, and brought her to us to say our goodbyes. Given how recent this was it’s a bit difficult to write about, so I won’t get into much detail, but after approximately half an hour we called the vet into the room. She came in with the drugs necessary to end Daisy’s life, explained the procedure, and began. The four of us put our hands on Daisy while Julie held her in her lap. The vet put Daisy to sleep with the first drug, cleaned the tubes with saline, and then administered the second drug to stop her heart. It was all over in less than a minute; she looked like she was asleep, finally at peace, not in pain. The vet left the room to give us another few minutes to say goodbye before the vet tech returned to take our Daisy away forever.

The lab results from Sunday’s emergency vet visit finally came back Wednesday morning: Daisy was experiencing heart and kidney failure at that time already. We found a minor amount of comfort in the decision we’d made the night before, but … Daisy’s still gone.

Last night, the 27th of October, I returned to the emergency vet’s office to pick up Daisy’s ashes. We’ve currently placed them in the china cabinet, until we find a better place. Luna’s been sniffing around for her, and rubbing her head where Daisy used to hang out, but she knows Daisy isn’t coming back; she hasn’t been barking as much as usual, and has shown a bit less activity. The kids are wrecked. Julie and I miss her as well. The house seems empty despite Luna being an 80-pound beast. I don’t feel like doing anything for Halloween. But … life’s continuing.

Daisy gave us everything. Sometimes I wonder if I gave her enough. She was the best first dog anyone could ever ask for … she was the best dog anyone could ask for full-stop. She cannot be replaced. The family will be getting another Shiba Inu, that is 100% certain — I’m ready to get in the car right now. However, we will miss Daisy ’til the end of our days, and as unfair as it may sound, every dog we have from here on out will have to live up to the standard she set.

Disconnect

By now you know that I enjoy tech. One with a knack for self-promotion may say I’m “passionate” about it, but I think it’s more in-character for me to say I like it a lot. Technology is a thread that is woven through every aspect of my day whether my eyes are open or shut. I’ve been online practically non-stop since 1998. My kids have heard this phrase numerous times: I am the internet.

There is no way any of what I just said is healthy over the long term. By “long term” I’m talking years, not just the singular year. That’s where the title of this post comes in: disconnect. I can’t be the internet. I can’t constantly be online, and I need to shut my computer off and take my hands off the keyboard every so often.

Somehow, my wife and I stumbled upon our preferred escape route: glamping. Apparently the phrase isn’t as well-known as I once thought it was, so for those who are not familiar, it’s short for “glamorous camping” — camping with modern amenities. We don’t go all the way to the top of the scale by getting internet, streaming services, or over-the-air TV; a solid roof/floor/walls, a proper mattress to sleep on, a standard bathroom (toilet+basin+shower with heated water), air conditioning, but in the middle of nowhere. That’s how we prefer to disconnect. Sure, we’ll take some movies with us in the form of DVDs or files on a laptop we plug into a TV, but if we don’t bring it along, forget about it.

Can we get online? Sure, of course we can. We always have our phones, and there’s “LTE” coverage (3G speed), but while we can tolerate the slower speeds having grown up on 14.4k, we choose not to. It’s an easy choice to make, especially after using 10Gbps internet. Takes a bit for the kids to get used to it, but they come around after a few hours. The dogs? They couldn’t care less, they’re with us and they get to see more squirrels.

And, after a week of campfires (actual fires, outdoors, with wood … not “glampfires”), grilling, getting eaten alive by mosquitoes, getting crisped by the sun, fishing, walking around with the dogs, swimming in whatever (safe) bodies of water we encounter, we get back behind the wheel and exchange touching grass for #!/bin/bash yet again. Sure, the commands come slowly at first, but the thoughts come more clearly … at least, once the >1500 emails are cleared out of the ol’ inbox.

So, with that, it’s time to disconnect for a bit. But not before a nice round of Satisfactory. Gotta get that Thermal Propulsion Rocket factory up and running, after all.

No.

To answer the question posed by my last post: no. A server cannot be my main workstation. Or, at least, that server can’t be my main workstation. The one I’m still using — 3900X with 64GB of DDR4-3200 — is running perfectly, but most importantly, it’s running fast. From a first-hand usage perspective, the Ryzen wrecks the Xeon up and down the stack … but given that the Xeon was launched almost 10 years ago, and the Ryzen has been around for less than 5, I suppose that’s to be expected. 

With that said, my homelab server’s CPU is identical to the one in my workstation, so … maybe I’m running a server as my primary workstation. Only difference is that the workstation has 64GB of non-ECC RAM. 

The E5-2690v2 is now running TrueNAS Scale and sits quietly in a corner, absorbing data backups from various other machines around the house. It’s chock-full of Noctua fans that run completely silently in spite of the SuperMicro mainboard’s efforts to ramp them up. Seriously, if I leave the default fan settings in the BMC, the fans all cycle between minimum and maximum speed every few seconds, which is obnoxious. Only happens with Noctua fans, likely because they run at reasonably low RPMs. Overall I’m pretty happy with TrueNAS Scale on that box, but I’m pretty sure I don’t use even 25% of its functionality.

I think I can switch my parents’ file server to TrueNAS Scale, on that note. It’s a server in name only, really: it’s a mini-ITX Sandy Bridge machine with 16GB of DDR3, a 250GB SSD boot volume, and a pair of mirrored 1TB HDDs. The hardware runs Proxmox VE which, in turn, runs Pi-Hole and a Windows 10 VM that hosts a singular file share. Why bother with Windows 10, though, when TrueNAS Scale is purpose-built for storing files? And, with the k3s functionality built into Scale, I can chuck Pi-Hole on there with minimal resource impact. 

Can a server be my main machine?

We usually think of servers as powerful, expensive machines sitting in data centers, and that’s usually accurate enough. Small to medium businesses have tower servers available as an option. Back in the day Intel used to make kits including chassis, mainboards, CPUs, memory, and cooling for DIY servers; SuperMicro has something similar now.

I’ve been running a homelab for a while. I started with an old gaming rig, switched to a proper server, moved to a couple of those SuperMicro kit components, and landed on another DIY option with an ASRock “server” mainboard in it. Now, I have those SuperMicro kit components left over.

The SuperMicro components are not considered that powerful in 2023. The CPU is a Xeon E5-2690 v2, making it a 10-core, 20-thread Ivy Bridge-EP part. Generationally this is on par with my first “old gaming rig” server, an i7-4930k, but with far more grunt. It’s running 128GB of ECC DDR3 RAM. The mainboard has a decent amount of SATA ports, though most are SATA2. The PCI Express slots are also jerks — they’re all sized either 16x or 8x, but electrically there is only one 16x slot and a number of the 8x slots are PCIe 2.0.

With all of this in mind, what do I actually need to do my work? I’m running a Ryzen 9 3900X for that purpose, which means the Xeon’s core count is pretty close. I’m running 64GB of DDR4-3200, mainly because I run multiple virtual machines; the 128GB of ECC DDR3 would allow me to allocate more memory to those VMs and possibly run more of them in general. I like booting from NVMe drives because of the performance, obviously, which I can’t do using the SuperMicro board. However, I don’t have an issue booting off a SATA drive and using NVMe on PCIe 1x add-in cards as necessary; that’s what I did when this machine ran my homelab and performance was fine. The GPU I use for work was never considered high-end: it’s a FirePro W4100. PCI Express 3.0 is fine for that, but the SuperMicro board gives us a built-in display output, though it just has a single HD-15 port so its usefulness is somewhat limited.

One of the VMs I’m running is Windows 11, which requires a recent CPU and a TPM. I’ve virtualized plenty of machines using the E5-2690 v2, and I’ve gotten Windows 11 virtualization working on an Alder Lake-based KVM machine before, but I had to set the CPU model to Skylake. Virtualizing the TPM is trivial, but I’m not sure if I’d be able to do set Skylake as the CPU model with an Ivy Bridge-EP CPU.

So … I haven’t answered the question posed in the title of this post, but I think it’s worth trying. It’s an interesting thought experiment, to be sure, and I’m not expecting it to outperform the 3900X. However, as I mentioned in a previous post I’m pretty keen on keeping hardware running as long as possible, and not using a 10c/20t CPU seems like a tremendous waste. Let’s see what happens.

A Second Life

Nothing beats the feeling of opening a box of shiny new tech. Well, peeling the plastic off that piece of shiny new tech beats the pants off opening the box, but given that you have to open the box and witness the peelable plastic first, I’ll defer to the box opening as the best feeling.

Like most things, though, the tech we use has a limited lifespan. Stuff fails. Older hardware seems to grow slower and slower by the day. If we take the proper precautions, though, we can pass our older hardware on to the next user. My kids use my older mainboards, CPUs, and GPUs in their computers, for example, but only after I secure erase the SSD that goes with them.

We should not shy away from being on the receiving end of that, though. My previous laptop was a Lenovo ThinkPad T460 and, given that the previous statement is in the past tense, you can tell it didn’t end well. I didn’t buy that T460 new, and I didn’t buy its replacement — a ThinkPad X395 — new either. I love the feeling of getting a shiny new piece of tech as much as the next computer nerd, but does it have to be new new? I got pretty lucky with this X395 too, as it looked brand-new when I took it out of the box. Keyboard feels like it was never used, monitor has no smudges, and there was not a speck of dust in the cooling fan. Didn’t stop me from putting a larger NVMe drive in and re-pasting the CPU, of course.

The only problem I had was that it came with Windows, but as you are acutely aware, I have a tried and true solution to that problem. The ThinkPad lineup, especially the older models, run Linux very well and are occasionally certified to run some distributions by their maintainers. Canonical, for example, certified the ThinkPad X395 on Ubuntu 18.04. Now, I’m not going to reach for a 5-year-old OS when I get my hands on a new-to-me laptop, but it’s good to know that someone put effort into making sure the hardware works properly with my operating system (not necessarily distro) of choice.

As a side note, the ease with which I can run Linux on an all-AMD machine is unparalleled. My main machine, a Ryzen 9 5900X on an X570 mainboard with a Radeon RX 6750 XT GPU, runs perfectly with minimal intervention. This laptop has a R5-3500U, which has built-in Radeon graphics, and it’s running great. The only thing that needed extra attention was the fingerprint scanner, but even that was a single extra package I had to install.

So, the long and the short of it is, don’t discount something just because it’s been used before. There are tons of good machines out there waiting for a new lease on life, and with the way Linux has grown over the past couple years, it’s never been easier to slap an OS on there and get down to business.

IoT is a Gateway Drug

I was fiddling around in my homelab this weekend and finally decided to write this down as the thoughts have been kicking around for a while.

Years ago, when the term “Internet of Things” came to be, I thought it was pretty stupid … and let’s face it, in hindsight, it is a pretty stupid term. The internet is the internet, and while the objects connecting to it are “things” in the strictest sense of the word, it’s computers all the way down. Raspberry Pi? Adafruit Feather? ESP8266? All computers. Everything. It doesn’t have to have a Xeon or a Ryzen to be a computer, its RAM can measure in the megabytes. Let’s just lay that groundwork right now so we can go on to the next paragraph.

“Physical computing” is a bit silly as well, but to an extent it makes a little more sense. We’re grabbing data from the physical world by taking measurements of it. Temperature, humidity, proximity, brightness, presence or absence of a magnetic field, voltage, current, vibration, orientation … we’re taking all of that, putting the numbers somewhere, applying some operations to them, and putting them on a screen so we can make sense of it all. We’re gathering the data and visualizing it in a way that makes more sense to us than numbers on a spreadsheet. Maybe I’m misunderstanding, but that’s how I’ve interpreted this whole “internet of things” … thing. I’ve built some systems to do stuff with it, and based on what I’ve been reading about recently it’s become apparent that the entire economy that’s built up around it is a gateway to what folks typically call “enterprise technology.”

Now, I work with some enterprise technologies, but this isn’t a post about work. That said, there’s a pretty extensive overlap between the monitoring and reporting we use for those technologies, and what I was describing above. The stuff at the “bottom” of the ladder — lightweight stuff that’s designed to run on single-board computers, for example — implements concepts that translate really, really well to the high end. At some point the only difference becomes syntax, which may take a little practice, but nowhere near as much effort as learning something from scratch. It was the same back in college when we used different programming language for every other course. The only one that really threw me off was LISP.

Before containerization we’d spin up VMs or, before that, entire servers to handle some of these things. Right now, a tiny VM running a lightweight Linux distro, Docker, and a smattering of containers is enough for a homelab or a small to medium business environment. Given a docker-compose file, you can even class it up with a lightweight Kubernetes deployment using something like MicroK8s, K3s, or RKE2. Let’s Encrypt makes it almost trivially easy to get your hands on a certificate to secure everything properly. A really basic deployment doesn’t require too much in the way of configuration changes, but if your goal is to learn this stuff, editing some text files is going to come with the territory.

So … long story short: learning IoT, regardless of my opinions on the term, is a good way to get your hands dirty with some pretty powerful technologies. Good stuff if you’re looking for an in at some tech shops.

Is it nostalgia after only six years?

I’ve played video games for a long time. Not that I’m setting any records or anything, but the first games I ever played were on Atari 2600 and Intellivision. The Atari was destroyed in late 1986 for certain aquatic reasons, and the Intellivision was my cousin’s while we lived in their basement for a while. After that came the NES, and that’s all I had until I was able to get a job and save up enough for a PlayStation 2.

Despite the console beginnings, though, my true gaming love was the PC. Believe it or not, that started with a TRS-80 Model 4, and Horse Race. Sure, the 486 with Wolfenstein 3D, Doom, and Microsoft Flight Simulator 4 blew that thing out of the water, but that TRS-80 was the beginning. At this point I don’t even know how many PC games I’ve played, and I’ve probably forgotten the titles of more games than I remember. And, while Wolfenstein 3D and Doom still hold a special place in my memory, and the Half-Life games took my enjoyment of PC gaming to the next level, it wasn’t until Mass Effect that I realized how much I loved gaming.

Mass Effect appeared when I was in grad school. It just so happened that I was focusing on 3D graphics and dataset visualization when Mass Effect was ported to PC and released in mid-2008. The graphics were somewhat janky and the controls were a bit of a mess, but it was the story that drew me in. Story and character development were front and center, and I thought Mass Effect delivered that in spades. To date I have spent hundreds of hours playing each game in the Mass Effect trilogy and, after attending the midnight launches of Mass Effect 2 and 3, I can confidently count 2 as my favorite, especially with the improvements introduced in the Legendary Edition. However, the fact remains that Mass Effect 3 is an 11-year-old game, which doesn’t line up with the title of this post.

The year 2017 promised a fourth major entry in the Mass Effect franchise: Andromeda. Reviewers were pretty down on it at the beginning and, like a silly fan, I did the midnight launch on EA Origin for this one as well. The experience was different this time, though. There were no familiar characters, different combat mechanics, and a story that didn’t quite feel whole. But, despite this, I pushed through it and finished the game. Somehow, I didn’t find any of the characters all that memorable. Who can forget the first time they met Liara T’Soni in that cave? Battling Sovereign’s forces while climbing up the side of the tower in the Citadel? Getting killed ten minutes into Mass Effect 2, only to be resurrected by the organization you were fighting back in the first game that happened to be run by Martin Sheen? Encountering the Collectors and raiding their base near the galactic core, only to discover and fight the human Reaper? Seeing the Citadel fill with refugees while the situation at Huerta Memorial Hospital continuously deteriorated? Punching out Khalisah Bint Sinan al-Jilani because you were sick and tired of her snide insinuations/disingenuous assertions/tabloid journalism? Stabbing Kai Leng in the gut with your omni-blade for Thane? What is there to remember from Mass Effect: Andromeda? Quite literally forgettable.

The year 2017 also brought a new intellectual property from Guerilla Games called Horizon Zero Dawn. A former coworker and I were debating which one to get and, because I game on PC, I didn’t have much choice in the matter: Horizon Zero Dawn was a PlayStation 4 exclusive and I didn’t have one of those. We decided that he’d get Horizon Zero Dawn and I’d get Mass Effect: Andromeda. He seemed to enjoy Horizon thoroughly, whereas I was left with a robust “meh” feeling. Along came 2020 and Horizon Zero Dawn was released for PC, so I immediately jumped on it, fired it up, and was absolutely blown away. That feeling I’d gotten with the first Mass Effect had returned, to an extent, but with spectacular graphics and a decent keyboard & mouse control scheme.

Unlike Mass Effect: Andromeda, I’ve replayed Horizon Zero Dawn 4 times as of Tuesday of this week. The story and characters are engaging. The New Game+ mode makes you feel utterly invincible. Yes, some of the face modeling behaves kind of janky with people seeming to move their mouths quite a bit more than necessary for certain words, but that’s a small price to pay for the story, cinematics, and gameplay.

Just for kicks, though, I launched Mass Effect: Andromeda last night and my jaw nearly dropped to the floor. It was as if I’d gone from a triple-A game (which, to be fair, HZD actually is) to something I’d developed during my graduate courses back in 2007. ME:A felt lifeless, like going through the motions with muscle memory.

So … where does the nostalgia come in? Maybe it’s the feeling of anticipation before launching ME:A for the first time … or maybe it’s more like regret. Regret that I didn’t snag a PlayStation 4 and a copy of Horizon Zero Dawn instead of loading up Origin. If I could go back and do it again, or if I could give advice to my past self, I’d know exactly what not to choose this time around.

Hunt is over for now

Server’s been up and running happily since the last post. So, it was the enormous VM data volume that was causing the problem.

A different hunt has ended as well (for now, of course), completely unrelated to Proxmox: the hunt for a decent camera. Like most people these days I’ve been recording videos on my phone, which is generally enough. Then, my kids started doing sports and I started sharing the videos with family, which made me remember some feedback I received at a much younger age: my videos were not stable enough. I switched to my previous phone, a OnePlus 7T, which vastly improved video stability compared to my current iPhone 12. The videos produced by the OnePlus had better color saturation as well, but it turned out the microphone was flaky, as the left channel muted and un-muted every few seconds. As a temporary solution I recorded with both phones and combined the OnePlus’s video with the iPhone’s audio in kdenlive, but that was a garbage solution.

My next upgrade was a gimbal and a small cage for the iPhone. An add-on lens and external mic improved the video and audio quality to the point that I don’t use the OnePlus anymore, but I wasn’t able to use the entire feature set of the gimbal. There are several buttons and a joystick on the grip, and while I could use the joystick to adjust the pan and tilt, that left button and accessory panels unused. There’s a list of DSLRs and mirrorless cameras on the manufacturer’s site, so I figured I’d slap my Nikon D3100 on it to see how it works. Spoiler alert: it didn’t. At least, not very well. My main goal here is to shoot video, and the D3100 has three things working against it. Maximum video length is 10 minutes, but the things I record can last over an hour; the built-in mic is abysmal and there’s no provision for an external mic; and I can’t use my favorite lens because it’s enormous and heavy, not to mention it has manual zoom. Twisting the lens to zoom while it’s on a gimbal is super awkward, and the only other lens I have for the D3100 is a 35mm prime, which I wouldn’t use for video from stands at my kids’ sporting events. Therefore, it was time to upgrade.

I started out patiently, perusing eBay and placing bids on Nikon mirrorless bodies, as I’d be able to use my existing lenses along with lenses specifically for the mirrorless body. My patience ran out when I was outbid on 8 or 9 different auctions, so I switched brands. I’d heard good things about Sony mirrorless bodies on a Linux podcast that has a professional photographer as one of its hosts, so I started looking for Sony kits that were on the gimbal’s compatibility list. As I’d run out of patience I looked for kits with “buy it now” enabled with a maximum price around what I was bidding. Within a couple hours I found an A6400 with the 16-50mm kit lens and two batteries for about $150 less than my maximum bid, so I snapped it up. I slapped it on the gimbal, attached the control cable, and went to town. Picture quality puts both phones to shame, and the audio quality is … exactly the same because I used the same mic. First test was during the family Christmas gathering, and the videos were met with positive feedback.

The 16-50mm lens left me wanting a little more, though, so after a mishap with a used 18-200mm lens (focus didn’t work at all), I settled on an 18-100mm with electronic zoom. Finally sat that on the gimbal and balanced it, and it’s perfect. Zoom works fine using the gimbal’s control knob too, but it’s not going to set any speed records. Not going for full-on NBA “I need to zoom on this guy shooting a 3-pointer yesterday” shots here, of course. First game is tomorrow, and I can’t wait to see how it performs.

Almost had it

Turns out it wasn’t actually some weird LXC-related permissions thing, which is a bit of a bummer, but it was most definitely file system-related. My Plex server’s data volume was a single monolithic 12TB volume, and when I did anything that caused significant I/O, that resulted in high load for some reason. I split my collection apart into multiple smaller volumes, with the largest being 5TB, used rsync to transfer everything over, and deleted the 12TB volume. No issues since the rsync finished. I’m still going to keep an eye on it for a few days, though.

Now, on to the next project: improving the quality of my home videos. I started with a OnePlus 7T, which does a decent job in its own right. Its image stabilization is surprisingly good, but my specific OnePlus 7T’s microphone was not behaving properly, forcing me to combine left and right channels to prevent audio drop-outs after every recording. My iPhone 12’s audio recording is great, but the color saturation is awful compared to the OnePlus, and there is no image stabilization, so I snagged a gimbal and some accessories for the iPhone. It’s been alright so far, but without HDR, the color saturation is still a problem. My existing Nikon D3100 is far too heavy to sit on the gimbal, so I’ve upgraded to a Sony A6400. The video quality is above and beyond the iPhone’s, and with the Rode VideoMic GO II attached, the audio quality is right up there with the video. At this time the only drawback is that the kit lens is 16-50mm and I have no zoom controls from the gimbal; an 18-200mm f3.5-6.3 lens is on its way, as well as the gimbal’s zoom control module, and that’ll complete the kit. All this to shoot better video of my kids’ sporting events.

On the software side I do my editing in DaVinci Resolve or kdenlive running on EndeavourOS. This is the only distro I was able to get the proprietary OpenCL stack working with alongside the open source video drivers, as I am running an AMD GPU; without proprietary OpenCL, Resolve won’t run properly. I render them to disk with x265 encoding to maximize quality and minimize disk usage and dump them onto the Plex server, which has a Quadro P2200 passed through to it for transcoding purposes.

This is also where the VHS tapes I capture go, though I used h264 for the first 8 tapes; those files aren’t as big because they’re quite low-resolution at, only because the source medium is at quite low resolution. Unfortunately the BlackMagic Intensity Pro 4k can’t capture at that resolution, so I have to use 720×576 when ingesting the footage and scale it down in post.

Minor Stability

My home server has been running steadily for the past 10 days without any issues … is what I’d say if replacing the RAM with ECC modules fixed 100% of the problems I was encountering. The majority of the problems were indeed fixed, but I encountered some new ones now that I didn’t have to worry about services randomly crashing and preventing access to the web UI.

I started noticing startlingly high load averages on the hardware. I’m used to seeing load averages well under the number of logical cores in the system, but was constantly seeing values over 34 for 1, 5, and 15-minute averages, which is not something I’d want to see for a machine with 12 physical/24 logical cores. It didn’t take long after the system booted for these values to show up either. Proxmox’s host-level Summary page showed a pretty steady CPU utilization of 15% with a constant 6-7% IO delay, which correlates with the OS’s iowait metric. This was all a bit unusual.

First port of call was good ol’ htop, which revealed that udisks2 was eating up a lot of CPU. I didn’t install this explicitly and hadn’t seen it on other Proxmox machines before, and after a little digging it turned out that this was normally installed alongside cockpit. This is a package I tend to install on most machines because it looks pretty, but I hadn’t installed it on this host so I had to dig a bit further.

While researching iowait issues I stumbled upon a forum post that said that high iowaits can be a result of issues with the underlying hardware. I hadn’t seen any hardware-related alerts, and the SMART values looked healthy for the SSDs and spinning rust, but the output of zpool status -v revealed that the ZFS pool did indeed have one single, solitary error, and was already performing a scrub operation to clear it. The file containing the error was for an experimental VM that I didn’t need anymore, so I just deleted the VM and restarted the scrub. That cleared the error but didn’t have any effect on the high system load.

It was at that point that I connected a couple dots. I was running several LXC containers for things like Pi-Hole, Squid, Homebridge, Gitea, Ansible, and a Minecraft server, and all of these kept their files on the host’s file system. That is, unlike VMs, LXCs don’t have their own dedicated virtual hard disks carved out as one large VMDK (or QCOW2 or VHDX) file, with all of the smaller files within. So, if an unprivileged container was having trouble accessing files due to a permissions issue between the container and the host, that could also show up as high iowaits. To test this theory I disabled autostart on all containers and rebooted the host. The VMs came up and were happily running, with host system load averages peaking at 4. The Pi-Hole container was the first one I started back up, and it had no observable impact on the load; however, the squid and gitea containers made the hardware extremely unhappy, driving load averages slightly higher than they were before, to over 36.

I quickly spun up a Debian VM for squid, copied the config from the container, and … no observable system impact. Built an Ubuntu VM for gitea (one-line installer via snap) and, again, no observable system impact. At that point I figured “well, if the VMs run with this little impact on overall system performance, I’ll just move all of my LXC roles into separate VMs,” and that’s exactly what I did.

That brings us to a new count of 11 VMs and 0 LXC containers:

  1. Pi-Hole
  2. Squid
  3. Plex
  4. VM for Docker
  5. Tailscale endpoint
  6. Desktop Environment Playground VM
  7. Certbot
  8. Homebridge
  9. Gitea
  10. Ansible
  11. Minecraft

…and, with these VMs running for the past 6 hours or so, my load averages are currently sitting at 0.36, 0.27, and 0.35, with iowait values of less than 0.1%.

So, the past week was a bit of a mixed bag when it came to the overall stability of the system, which means it’s time to observe for a bit longer. This also begs the question: if I’m only running VMs on this box, why am I running Proxmox? Could I get away with running Rocky Linux 9 on the bare metal, with the VMs running in a standard KVM install? Or would I get fancy with something like TrueNAS Scale, effectively rendering my small Synology NAS redundant? For now I’ll stick with what I know, but there could be some interesting plans for the future.