My fun project of the last 2 weeks: Adventures in Linux land and debugging Win11

[viper37]

So.  It all began last year, actually.

My network was becoming a tad slower, and I was on Windows Server 2019 Essentials, an edition made for small businesses, with a license that was sold for 600$ back in the day, for the 2016 edition, later an upgrade to 2019 for I don't know much.  At that point, my intent was to sell my shares in the company, all the infrastructures, move on, be at peace.  

Didn't work out, so I kept everything. I put my own stuff on this over time, but some was scattered elsewhere too.  While I had the business, I had 7 desktop computers total, including the server + 1 shared laptop between dad's girlfriend and me. Everytime I tried to centralize things, something got in the way, but I was mostly there by the end.  Now, I only have 4 working computers, including the server.  Another one is to be sold soon and I have spare parts to assemble a DDR3 machine that I might reuse for distro-hopping (switching Linux distros) later.

It's a server OS, but it's still Windows.  And I'm not an IT guy, just an amateur.  So it constantly asks to be rebooted whenever there's an update.  Annoying.  And over time, it slows down.  And the rest of my network too.

Like I said, the idea began forming for a while ago, going back to Linux.  I had used Linux Mint for a while (also shortly Ubuntu and OpenSuse, which was too complicated for me back then).  Might go back there?  A former friend, qualified for Red Hat Linux Enterprise was telling me about OpenSuse, his favorite OS for home.  But I still had nightmare about the last time I tried to use that thing...

And I kept pushing this because I wanted to have my friend "near me", a phone call/text away in case I needed help.  Either a RHEL derivative or OpenSuse.

Sadly, he was no longer an option as of this winter (see the Backroom) as his health declined rapidly.

And this late spring the coup de grâce: I lost my network while working.  No more access to my files.  It eventually came back, but for a day and half, for urgent work, I had to rely on Gemini instead of my trusted Excel spreadsheet to do all my calculations for a new project I'm starting and house renovations.

Got tired.  Really, really, really tired.

Decided to merge everything into one computer/server, too some more info from the 'net, look at what I needed to build a Jellyfin Server on top of my business needs and I also ordered a wifi7 router with 10gbps ports, a switch, some cables and rang another accointance to remake the cables in the house.

I thought I could reuse some of the old hardware I had, but in the end, I needed a new AM4 motherboard, a new CPU, a cheap Intel GPU and... am I set?  Can I begin working now?  Ah, my PSU decided it was a good time to cross the Styx.  Manage to find one for 80$.  Grrr.

Am I done now?  Not yet.

I want to sell the old machine.   My server has 16gb ram, the old machine as a Ryzen 1700X and no ram.  The IT guy says he can find someone for the machine if I put ram in it and a newer cpu that is sure to support Win11 "without hacks".

FUCK IT.

So I find 32gb DDR4.  THAT THING IS \@¢*** EXPENSIVE NOW.  240$ for an unknown brand, apparently good, but they make garbage ssd.  Thank 'od, I had ssd and hard drives from before these price hikes.
So now, I have the machine rebuilt, I need to clean it and install Win11 on this to offset my server costs.



Back to the server.
AM4, DDR4, Intel video card supported by the latest Linux kernels, I'm all set.

Where do I go from there?

Hmm.  I'm alone.  Let's go back to Linux Mint Cinammon, always feels good to slip back in the old shoes.

So I get there.  And partition/format my hdds in BTRFS.  And try to configure the ext4 SSD for snapshots.  Some hickups.  Try to find a good gui tool.  Won't work on older kernels (6.x) like Mint is using (Linux is on 7.x now).  Grr.  Ok, I can manage.  Need another piece of software.  Not working on Mint/Debian, too old.

Fuck that shit.

Wipe everything out, start back with Fedora.

And meanwhile what happens?
MICROSOFT!!!

They push a performance patch on Windows.  My main computer, a 7800X3D becomes near-unusable.  I can barely move the mouse in Windows.  It's awefully slow.

But I got work to do, I need to rebuild the server and access my files.
So I keep working on Fedora KDE.  And later at night, I try fixing the goddam Windows computer that suffers and acute crisis of vibe coding patch.

So, I start chatting with Gemini.  Useless half of the time, it tells me to deactivate features in my bios.
Go back to working on Fedora.  Some more problems. Some packages aren't working there.  They use flatpacks now and lots of the software from their appstore won't work at all.  Lots of cool software on the 'net won't work with Fedora, even if you compile them manually without significant rework.
But the OS in itself is fast, I'll give it that. There OS file system is taking snapshots automatically, in case your bork your system, in theory you can go back in time and restart from there.

In theory.

In practice?  You need to be able to boot into Fedora KDE Plasma to access your snapshot.  Or somehow manage to access the command line from somewhere with a rescue disk, which is not really easy.

And what did happen?  While I was setting my permissions on the drives for the security system (SELinux) and Samba (the service that let Linux and Windows "talk" to one another to exchange files), I made a small syntax mistake.  It should not have damaged anything because I was not on the system drive.  But I passed the command on the root of my data drive instead of the folder where my data was.  And it borked my system.  Totally. Unbootable, unaccessible from anywhere.  Sometimes, I could go beyond the bios, most of the time not.

And so I let it rest for the night...

And the morning after, I began a new chat with Google Gemini to evaluate which distros would be good for me based on my experience and my level of profienciency with Linux (intermediate, at this point).

We passed and eliminated multiple distros, mainly a bunch of Arch-based distros, too advanced for me, as they could often break something.  We eliminated anything Debian based as too old, that could cause problem like Mint had, except for a rolling release.  I settled on a rolling release.  Investigated from there.

And I settled on OpenSuse Tumbleweed.  

It's a rolling release, meaning there's no 10,11,12, it's always OpenSuse Tumbleweed and the packages, apps, kernels are constantly updated after a short period of test.  Linux Mint may delay an update by 2 years for the sake of stability, and it's great for some people, not so good for me.  Like, for example, they don't offer much support for the BTRFS file system because it's still considered too experimental.  It's been invended in 2007 and in use in Linux since 2013...

I can still run the updates whenever I want, so it's not much of a bother.  Right now, I run them daily, there are still some issues (back in a moment).

So, OpenSuse...

I still had that Windows patch issue...  and meanwhile, my work computer downloaded and installed that patch too... became unusable.

Constantly tried to switching the power plan there from "balanced" to "High performance", it seemed to make things a little better, but it always defaulted back to "Balance" and some other performance problems appeared.
Fuck that.  I have a server OS to learn.

OpenSuse... It had a been a while.  There's a nice GUI for the installation now.  You chose everything you need/want from the get go.

So Friday afternoon I begun...  9h30-10h00 am until 3:00/3:30am every day for 6 days straight. I washed and shaved on the 4th    I emptied my stock or energy/snack bars from Amazon.  

By default, it's not optimized for modern, personal computers, unlike, say CachyOS.  And if you use the Yast partitionner, you will bork your fstab, wich gives you an unbootable system... Ah, but then here's the magic!  Remember those snapshots from Fedora? OpenSuse (German efficiency! yeah!  )as this totally integrated in the os. When you boot, it shows the last dozen or so snapshots on your system (you can create some manually).  So I break that file fstab by using the old GUI tool instead of relying on sudo nano fstab... A mistake I did not repeat twice.   

I already knew I needed to add noatime at the end of my mountpoints for the hard drives, and thanks to Gemini, I found a couple of other commands that reduced the need for the OS to constantly use the swap drive.  By default, OpenSuse does not fill your ram.  It does not recognize you have 32gb ram, it always act like you have 8gb ram, so it constantly swaps to the ssd/hard drive, which slows down things.  And IPV6 is enabled, which, in Linux somehow creates network lag (but not in Win11), go figure.

So, I fixed the performance issues, I could work.  Installed a few things.  Firefox is installed by default.  Heard about Zen.  Not sure I like it, but it's there.  Codecs, etc.  Everything I may need.
There's a GUI tool to create Samba shares and users.  But it's not really working.  So I did not it by hand.   sudo smbpasswd -a  is still the way, along with editing smb.conf to create all the network shares.  No GUI tool seems to be working there, despite Gemini reassurances before I begun.     I needed Gemini's help to configure and isolate my copier/scanner from the rest of the system as it is still using SMB1 and it's a weak point in the security of my system.  


Network is working, drives are auto mounted, the shares are there and the slooooowwww Win11 computers are seeing it.  Not bad.  Next step.

There's a new invention in the Linux world (well, Windows too  ): containers.  Docker is the most widely used, but Red Hat (IBM) came out with their own version, Podman.

It's hard to describe, but it's like a mini-virtual machine for a software.  Your software run in its own ecosystem, separate from the rest of your system, it does not install anything on your system, everything is in, well, a container!  It can't write outside the container and it can't see anything you don't specifically give it access to.




So, on Ubuntu, they say "run Jellyfin as if you were on Windows".  And for the rest of Linux they say "the preferred method of installation is through Docker (or Podman).
So at first, I installed software packages for Podman I found the app store (Discover).
But oh! What is it there?  There's this nice little shiny thing called "Cockpit" in OpenSuse.  A totally modern interface to manage your system, locally or from a distance: localhost:9090.  


This is an old view.  I actually can manage everything.  I've got system logs, user accounts, updates (not yet recommended though), hard drives, monitor temps, and more importantly: containers.

So I set to create my container: Jellyfin.

Wonderful!

Except it's not working.

Not yet at least.  That part will need refinement.  There were some access problems with the file permissions on the BTRFS hard drive pool.

I had to use the command line tools... Never done this in my life.  And it's really, really, really hard to fin distro appropriate commands to start this and set the right permissions on a web search now... Back to Gemini...

At this point Gemini thinks it's my girlfriend.  It's telling me I'm a bit stubborn, it's late and I need to rest.

A couple of hours later, when the sun rises, I end up listening to that stupid, nagging machine.

[viper37]

PART II.


So, I created my containers.  Once created they appeared in my Cockpit interface.  There I could deleted them, recreate them in the command line, stop them, restart them at will.

Next were the permissions.  A trick I used before with restorecon for Samba. But this time, had to do it from inside a container... Again with Gemini.  <sigh>.


Now, things are working well.  Jellyfin sees my movies, tv shows, music, everything.

Make it nicer.  Try a custom background.  Not working.  Try a banner image.  Not working.

2-2.5 hrs of frustration with the chatbot and different scripts, nothing would work from inside a container, and it's really difficult to insert a plugin from within a container, as I have discovered.
Forget that.  I'll live with the default look.

On the client side, Kodi is reading the media files.  This is the important part.


All of this is working, so what's left to do?  Ah, yes.  2 out 3 remaining computers are crawling on the floor, thanks to Windows.

But first, there seems to be a driver issue on the Linux PC...

Hmm.  Here I thought all drivers were in the kernel and I didn't need to touch anything?

Ok, more chat.  Gemini convinces me to use the i915 Intel driver, saying this will solve my problem.   There were weird errors in my logs.  Indeed the errors are gone, success!
....
....
....

Wait... why is my CPU climbing to 65C? 80C? 85C? 86C?  Everything is throttling!  Waaaaiiiiiiittttttttt!  Fuck.  I had to shut down the Jellyfin containers from Cockpit.

What is happening the user asked his newfound artificial friend?
"Your OS is not using hardware decoding to a mistmatch driver.  You should use the most recent xe driver provided in the kernel".


FUCK YOU STUPID MACHINE!!!!


now, revert what you made me do tonight.

2hrs later, I'm going nowhere.
I keep telling Gemini to go back to our conversation, it tried to get me to modify Grub while we're using Systemd.  I keep pushing for the updated doc, it refuse to acknowledge it, always coming back to GRUB edits.

I decide to try the Pro-Mode and move the option to "complex reasoning for math and science problems".

I re-submit the documentation and tell it to look at our conversation, the problem is there.

FINALLY!

The chatboot found the solution on how to reverse the damage.

Back on xe.  Jellyfin is working, the cpu isn't overheating too much.



But now, Firefox is crashing...

Ok, we need to force the flatpack to use hardware acceleration and take the xe driver.  Done.


No way I'm remembering all of that next week.



So, OpenSuse is working.


Or so I thought.


But that was on the 4th day, around 21:00.  I decided I needed some real food.

Grabbed an old t-shirt, punched the McD app on my phone and here I go to the car-service to grab some food.

Mind you, I had yet to wash and shave myself...

But I got to eat something else than eggs, yogurt and snack bars...

Then I went to the shower because I couldn't smell myself anymore 

[viper37]

Part III.

On 5th morning, I realized I had yet to shave when I woke up.   That's what it does when you go to bed at 3:00/3:30 every night.  So I did, right after I showered.  I looked like a modern human again.  



Back to OpenSuse.

I noticed there were crashes.  And Kodi wasn't working right on the HTPC trying to read the Jellyfin files.


Back to the server.

FFmpeg is crashing...  Hard.


Is it Gemini's fault? 

Let's ask another professional for help. And why not 2?

So I quizzed Lumo AI and Duck AI (Claude) who assured me my problem was really a kernel problem with Tumbleweed and not anything "I" had done.

Ok. 

Back to have a chat with Gemini, with all my problem logs, and my command outputs.  Pro mode, Math&complex problem solver.

-"Is it or is not something we did?"

"I have a techical confession to make..."

Aaaaaahhhhhh!!!  I fucking knew it!


But it turns out it was just about the Firefox & Zen fix it had me apply.  It fix the problem for the apps, not systemwide.

I found documentation dated from late May later on indicating someone was working on this and submitted it to Gemini who assured me there's nothing to do but wait for a kernel patch.

Got to use software decoding.

So, on the 5th day, I tried to fix my Microsoft problem...

Not much success, but I discovered why my 4th machine wasn't behaving abnormally: it had not been updated sine 2024! 
Fixed that, it installed 25H2 service pack and all the latest updates.
but this one has no gpu and it's working fine...



The other two have AMD gpus.... And telemetry disabled.
All have Ryzen cpus.


Try to fix a lot of things on my own, nothing was working. Took a break, went back to it.  In the end it was 3:45 when I went to bend...


This morning I had a flash.

But my network was no longer working.

<sigh>

Ok, so, a visit to Windows credential managers, erased old Windows Server credentials, deleted the network connections, made sure the server was always on...
Ok, got my network back.

Now, on to the fix.


Began with the gaming computer.

I had found half the solution: switching the power plan to high performance alleviates the problem.  but Windows does not keep it.  There's another round of patch from MS.  Install this.  Took an hour.  Went to the other computer to start the updates there also.

Back.  Link state power management: off.  This puts the GPU to off when unused. Things are slightly better.

Now, cpu min/max usage.  Ahh
Here it is.  What if I increase that min value?

Things get better...  Not there yet.

I look online again, find that the patch enables indexation everywhere of everything. Disable this!  Oh yeah!  Now, I can move the mouse!  Almost there!


Had to quizz Gemini again.  He said I was "obstinate" to fix MS problems.  You fucking bet!
But here we go.  Qbitorrent is another culprit.  I switched to libtorrent2 version.  This is apparently creating conflicts with the new patch.  Uninstall, reinstall the base version.

almost there...

Tweak the power plan again...
Min CPU: 80%
Max CPU: 100%


Voilà!

No more suttering.  My computer is working fine.

Well, almost.  As I'm using it (6th day now) I realize there's some hiccups with the mouse... hmm.  "USB selective suspend".  Off.

Now it's done.

One computer fixed.

But I have essentially disable everything power-management related.  That PC is running nearly full power all the time.



What the new Windows patch is doing to "speed up our computer" is what MacOS and Linux have been doing for years.  When your computer is on idle, your cpu and your gpu are not working.  They don't draw power, they are essentially shutting themselves down.

If you play a game, the game is supposed to ask for more power from the CPU and GPU and you game feels more responsive.  And the process is the same when you click on the Start Menu and wan to open your browser.  Linux and MacOS sends micro-burst to your CPU, like "Ok, wake-up buddy, I need you!" and then "It's ok, go back to sleep now".  It's all happening for a fraction of a second, so it does not show on your power bill.

But Windows was not always doing that.  In fact, on the desktop, it was always keeping your cpu half asleep.  So it felt sluggish.  And in some games, especially if they were badly optimized, the game engine wasn't always talking to the CPU&GPU, and both were kind partially shutting themselves down.

That's why Linux was often giving more performance through the Proton layer, because it accessed the resources it needed, when it needed them.


The patch is supposed to change all that in Windows.

But in my case, and in some other people's case apparently, it's totally broken.  And instead of waking my cpu and gpu, it's keeping my cpu asleep.  And my mouse.

And by constantly re-indexing files on the hard drive and ssd, it was taking all of the CPU's power to manage the hard drive...  So I had no "juice" left to do basic command.


But really now, my computer has no power saving feature. At all.

I'm doing my part to save Hydro-Quebec this summer I guess?

Thank you dearly Micro$oft.  

[viper37]

Part IV.

Office computer.

Last one to fix.


It's still on its knews/crawling on the ground.

I had done everything but "USB selective suspend setting".

And voilà.  Problem fixed.

No more stuttering.

But I need to put this one to sleep after I finished working, otherwise, it's also running at full speed all day long.



Now I've got all my running, my home server is running perfectly and I can move all my files centrally with enough storage to last me a couple of years, hopefully. 


The Linux part was really fun.

The Windows part, much less fun.