Mudkip Mud Sport

Mudkip's Mud Sport Journal

KDE Plasma 6

I have been using EndeavourOS on my Dragon Canyon NUC for months. It is my chosen Linux distribution because it offers an easy-to-install Arch Linux environment, and I am familiar with Arch Linux. I need to stay on the cutting edge for desktop environments, display protocols, and graphics drivers due to issues with NVIDIA and the Linux desktop (especially Wayland). Many developments are happening in real-time to improve them. With Arch Linux, I can easily install an AUR package from a merge request, as well as switch to beta drivers when they become available.

Wayland is important to me because I’m using a 27" 4K display. For the best visual size, it should be set to 150% fractional scaling. KDE Plasma supports fractional scaling under Wayland and has an option to “let Xwayland applications scale themselves” to prevent them from being blurry. That’s the reason I chose KDE as my desktop environment. Moreover, KDE Plasma 6 offers a nice experience both functionally and aesthetically.

Here is a note in Logseq that I use to track how I install and configure EndeavourOS. I’m interested in a declarative way to build operating systems and am slowly learning NixOS. However, in the meantime, I am very satisfied with my current EndeavourOS setup.


  • Configure the Windows Registry to use UTC hardware time:
    1
    reg add "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /d 1 /t REG_DWORD /f
  • Disable Secure Boot and Fast Boot in the BIOS. If Fast Boot is enabled, accessing the BIOS by pressing F2 is difficult. Instead, I have to use Settings > System > Recovery > Advanced startup in Windows, and after rebooting, choose Troubleshoot > Advanced options > UEFI Firmware Settings to access the BIOS.
  • Flash the EndeavourOS ISO to a USB disk using Rufus, contrary to the documentation on the EndeavourOS website. I had to choose the GPT partition and ISO mode to make my PC recognize the boot drive.
  • Change network settings in KDE settings: manually set the IP address and DNS and gateway to 192.168.1.2.
  • Install EndeavourOS using the online installer, KDE desktop environment, erasing the Samsung 970 EVO disk with btrfs and without swap.
  • Eject the USB disk and reboot into EndeavourOS.
  • Install nvidia-inst using yay and run nvidia-inst to install the GPU driver, then reboot again.
  • Disable the Dummy HDMI Plug in Display Settings and set the scale to 150%; turn off Adaptive Sync.
  • Fix the suspend issue by preserving video memory as described here:
    • sudo nano /etc/modprobe.d/nvidia-power-management.conf
      • options nvidia NVreg_PreserveVideoMemoryAllocations=1 NVreg_TemporaryFilePath=/var/tmp
    • sudo systemctl enable nvidia-suspend.service
    • sudo systemctl enable nvidia-hibernate.service
    • sudo reinstall-kernels
  • Install the xorg-xwayland-explicit-sync-git and egl-wayland-git AUR packages using yay. This will make Xwayland issues with NVIDIA more manageable.
  • Install fcitx5-im and fcitx5-chinese-addons using yay, and select Fcitx 5 in System Settings > Keyboard > Virtual Keyboard; add Pinyin as an Input Method.
  • Copy .ssh/id_rsa, .ssh/config, and .ssh/id_rsa.pub from Windows on the same device.
  • Enable mdns and kdeconnect in the Firewall configuration.
  • Configure Firefox:
    • Extensions:
      • Mainichi
      • uBlock Origin
      • Bitwarden
      • floccus
    • Always show the bookmark bar.
    • Disable Recommend extensions and features as you browse.
    • Enable Proxy DNS when using SOCKS v5.
    • Disable DNS over HTTPS.
    • Disable Ask to save passwords.
    • Enable “Tell websites not to sell or share my data” and “Send websites a Do Not Track (DNT) request”.
    • Uncheck all checkboxes in Firefox Data Collection and Use.
    • Uncheck “Block dangerous and deceptive content”.
    • Toggle identity.fxaccounts.enabled to false in about:config.
    • Toggle extensions.pocket.enabled to false in about:config.
    • Set the homepage to “Blank Page”.
  • Disable “Show small window previews when hovering over tasks” in task manager options.
  • Arch packages installed with pacman:
    • flatpak
    • thunderbird
    • obsidian
      • Create .config/obsidian/user-flags.conf with --ozone-platform-hint=wayland --enable-wayland-ime.
    • bitwarden
      • Create .config/electron28-flags.conf with the following content:
        1
        2
        --ozone-platform-hint=wayland
        --enable-wayland-ime
    • fastfetch
    • docker
      • Add the current user to the docker group: sudo usermod -aG docker $USER.
    • cuda
    • libreoffice-fresh
      • Create .config/environment.d/env-vars.conf with SAL_USE_VCLPLUGIN=gtk3. This can fix text and icon sizes.
    • btop
    • htop
    • bottom
    • nvtop
    • gst-plugins-qsv
    • gst-plugins-bad
    • libva-utils
    • intel-media-driver
    • intel-media-sdk
    • intel-gpu-tools
    • handbrake
    • gimp
    • krita
    • thunderbird
    • kdenlive
    • remmina
    • freerdp
    • discover
    • kid3
    • merkuro
    • zanshin
    • tokodon
    • kasts
    • telegram-desktop
    • ttf-jetbrains-mono
    • zeal
    • kubectl
    • kubectx
    • zsh
    • rustup
    • nextcloud-client
    • neovim
    • tmux
    • inkscape
    • pastel
    • yt-dlp
    • k3b
    • bat
    • dust
    • git-delta
    • eza
    • eva
    • fd
    • gitui
    • glow
    • gron
    • xh
    • pipe-rename
    • procs
    • yazi
    • ttf-hack-nerd
    • rclone
    • ripgrep
    • silicon
    • skim
    • tokei
    • viu
    • zellij
    • timeshift
      • Setup daily btrfs snapshots.
      • systemctl enable cronie.
    • nvidia-container-toolkit
    • speech-dispatcher
    • cmatrix
  • AUR packages installed with yay:
    • logseq-desktop-bin
      • Note: Wayland has not been enabled for Logseq due to a bug causing the window to shrink with each start and occasional crashes on close.
    • visual-studio-code-bin
      • Create .config/code-flags.conf with the following content:
        1
        2
        --ozone-platform-hint=wayland
        --enable-wayland-ime
    • vlc-git
    • xone-dkms
    • spotify
      • Create .config/spotify-flags.conf with the following content:
        1
        2
        3
        --enable-features=UseOzonePlatform 
        --ozone-platform=wayland
        --enable-wayland-ime
    • drawio-desktop
    • kopia-ui-bin
      • Set up daily backups of the home directory.
    • plex-mpv-shim
      • Open TCP 3000 and UDP 32410-32414 ports in firewall settings.
      • Set up auto-start in ~/.config/systemd/user/plex-mpv-shim.service:
        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        [Unit]
        Description=Plex MPV Shim Service
        After=network.target

        [Service]
        Type=simple
        ExecStart=/usr/bin/plex-mpv-shim
        Restart=always
        RestartSec=3
        Environment=GDK_BACKEND=x11

        [Install]
        WantedBy=default.target
        1
        2
        3
        systemctl --user daemon-reload
        systemctl --user enable plex-mpv-shim.service
        systemctl --user start plex-mpv-shim.service
  • Flatpak apps:
    • Ungoogled Chromium
      • Create ~/.var/app/com.github.Eloston.UngoogledChromium/config/chromium-flags.conf with --ozone-platform-hint=wayland --enable-wayland-ime.
    • Android Studio
    • OpenLens
    • DBeaver Community
    • Zotero
    • Discord
    • Plexamp
    • Flatseal
    • Gear Lever
  • AppImage apps:
    • Insomnia
  • Configure mpv for hardware decoding: create ~/.config/mpv.conf and ~/.config/plex-mpv-shim/mpv.conf with hwdec=auto-copy. Additionally, add the command set hwdec auto-copy in Haruna Media Player.
  • Install prezto:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    zsh

    git clone --recursive https://github.com/sorin-ionescu/prezto.git "${ZDOTDIR:-$HOME}/.zprezto"

    setopt EXTENDED_GLOB
    for rcfile in "${ZDOTDIR:-$HOME}"/.zprezto/runcoms/^README.md(.N); do
    ln -s "$rcfile" "${ZDOTDIR:-$HOME}/.${rcfile:t}"
    done

    chsh -s /bin/zsh
    • Configure Konsole command to /bin/zsh
    • Add pmodules in .zpreztorc: git, syntax-highlighting, ssh, tmux, fasd before prompt
  • Install goenv:
    1
    2
    3
    4
    5
    git clone https://github.com/go-nv/goenv.git ~/.goenv
    echo 'export GOENV_ROOT="$HOME/.goenv"' >> ~/.zshenv
    echo 'export PATH="$GOENV_ROOT/bin:$PATH"' >> ~/.zshenv
    echo 'eval "$(goenv init -)"' >> ~/.zshrc
    goenv install 1.22
  • Install fnm:
    1
    2
    curl -fsSL https://fnm.vercel.app/install | bash
    fnm install 20
  • Set up the rust toolchain and install some cargo applications:
    1
    2
    3
    4
    5
    rustup install stable
    cargo install cargo-update
    cargo install dtool
    cargo install --locked csview
    cargo install fclones
    • Append export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH" to .zshrc.
  • Append aliases to .zshrc:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    alias comet-core="ssh comet-core.lan -t 'tmux new-session -A -s main'"
    alias porygon2="ssh porygon2.lan -t 'tmux new-session -A -s main'"

    alias vim="nvim"
    alias vi="nvim"
    alias vimdiff="nvim -d"

    export EDITOR="nvim"
    export GIT_EDITOR="nvim"
  • Check that the Intel iGPU works for video encoding: LIBVA_DRIVER_NAME=iHD vainfo --display drm --device /dev/dri/renderD129.

Lenovo Legion Y700

The Google Nexus 7, released in 2012, is the first “small tablet computer” I can recall. It featured a pleasing 800p display, a powerful processor for its time, and shipped with a “stock Android” system. I used a Nexus 7 for a few years alongside an iPad Mini, which was subsequently replaced by a larger iPad Air 2. Although the device has not been with me for many years, the memories of reading manga on a perfectly sized device and the hackable Android experience, including running a bash shell and emulators, always lead me to seek the spiritual successor to the Google Nexus 7.

For the past two years, I have been using the Lenovo Legion Y700 (2022) as my primary device for content consumption. It comes equipped with a Snapdragon 870 processor, 256GB of storage, 12GB of RAM, and an 8.8-inch 120Hz 2K display. Similar to the Google Nexus 7, there is an improved “2023” version featuring a Snapdragon 8 Gen 1 processor. However, the older 2022 model suffices for my needs, and I prefer its advanced 3.5mm audio port.

While the hardware performance is excellent and the freedom to install apps from various sources—such as Google Play, F-Droid, GitHub, or directly from the developer’s website—feels right, I have never felt good about its ZUI operating system or any Android OEM system. I appreciate that Lenovo doesn’t include too much bloatware, doesn’t require me to log in to its account, and natively supports Google Play Services, but there are always moments when I encounter buggy or peculiar issues with some apps, or something just doesn’t seem right with the OS.

On the other side, for the past two years, a Google Pixel 6a installed with GrapheneOS has been my secondary phone. While the hardware is decent, it kind of underperforms when compared to an iPhone or a Snapdragon 8-equipped Android device. However, the open-source operating system is more than just great. It offers many secure features, and even Google Play Services can be run in a sandbox to protect privacy and battery life. I can disable network access for Google Camera, Photos, and Gboard while still keeping their on-device ML features. Additionally, I can place some apps that I don’t trust as much in a Work Profile with Shelter, making other apps and data invisible to these apps.

In the end, I learned that the bootloader of the Lenovo tablet can be unlocked, and I discovered Project Treble and the Generic System Image (GSI), which enable the availability of custom Android distributions on most devices without excessive difficulties. Weeks ago, I followed this video and installed crDroid on my Lenovo Legion Y700.

crDroid, a fork of LineageOS, offers a pure and customizable Android experience. After experimenting with it for a few hours, I deeply regret not replacing the OS on this tablet sooner. Using an open-source stock-like Android system is incredibly refreshing. The UI is smooth, and I can truly appreciate the 120fps screen while navigating. It brings back memories from more than a decade ago with the Google Nexus 7, making me realize that I’m holding what could be considered the spiritual successor to the Google Nexus 7.

System Experience

As is common with most custom images lacking official support, there are a few caveats when using a GSI ROM on this tablet, but I can say that I have fewer issues with crDroid than with the stock ZUI.

The settings and issues worth noting are:

  • 120fps can be enabled under Settings > Phh Treble Settings > Misc features > Force FPS, by setting it to [email protected].
  • Lock screen rotation can be activated by going to Settings, searching for “rotation,” and selecting “Rotate lock screen.” I also enabled all rotation modes here.
  • The 3.5mm audio port does not work initially, but it can be fixed under Settings > Phh Treble Settings > Qualcomm features > “Use alternate audio policy.”
  • The double-tap to wake feature is not operational yet, but, surprisingly, the Lenovo screen cover lock/unlock feature does work.
  • Adaptive brightness does not work.
  • Changing Wallpaper crashes initially, which is fixed by installing Fossify Gallery.

The crDroid launcher is a lightweight fork of the Google Pixel launcher. Initially, I preferred it as it is similar to the launcher used in GrapheneOS, so I was already accustomed to it. However, there is an issue where the background of the app drawer remains dark even when using the light theme, which may be due to some compatibility issues with the tablet size or the enabling of the work profile. As a result, I switched to Niagara Launcher, and it looks perfect.

I also learned a valuable lesson: Unlike Pixel phones, Lenovo does not allow the re-locking of the bootloader once a non-stock OS is installed. The first action I took after installing crDroid_gsi was attempting to re-lock the bootloader. Subsequently, the tablet became bricked, displaying a message that the system was corrupted and could not boot. Thankfully, the tablet could be unbricked by following this guide.

Privacy and Security

In terms of security, I’m content with the basic Android security model, but it’s important to note that the tablet is less secure than a Pixel equipped with GrapheneOS.

Keeping the bootloader unlocked is clearly a security risk, as theoretically, a thief with hardware access could install a hacked ROM to decrypt data on the device. Additionally, the face recognition feature on this device is less secure than Apple’s Face ID or the face unlock feature on the Pixel 8, meaning a thief could theoretically mimic my face to unlock this device.

I do not consider a tech-savvy thief attempting to access my data to be within my threat model. Nonetheless, I choose not to store sensitive data on this tablet, and I also disabled biometric unlock in Bitwarden after setting up most of my apps.

While crDroid does not come with sandboxed Google Play Services like GrapheneOS does, disabling the advertising ID, personalization, autofill, and location settings is sufficient for my privacy needs regarding Google.

One theoretical privacy threat when using mobile devices is the possibility of manufacturers or popular apps harvesting the list of installed apps on the device and potentially sharing this information with authorities, which could result in phone calls from “anti-fraud offices” warning about the risks of certain apps. While this is all theoretical and without neither any evidence nor my own experience, using an open-source system can definitely mitigate such threats. Similar to my setup on my Pixel phone, I use Shelter to isolate certain apps within a Work Profile.

Apps

Despite it being over a decade since the launch of Nexus 7, there are still not many Android apps optimized for tablet size. However, since Android does not restrict non-optimized apps to run in a centered window like the iPad, most Android apps function adequately on this smaller tablet. Additionally, the operating system supports basic multitasking features, such as split-screen.

I prefer to install apps from Neo Store when possible, which is a frontend for F-Droid and also supports IzzyOnDroid by default. For apps that are only available through GitHub releases, I can update them automatically using Obtainium.

Here is a list of apps worth mentioning on this tablet:

  • Cromite: The default browser, featuring built-in ad blocking and privacy enhancements.
  • DAVx5 and Etar: Used for syncing calendars with Nextcloud.
  • FocusReader: An RSS client connecting to Miniflux.
  • GitNex: A client for Forgejo.
  • Material Files: A file manager supporting SMB.
  • Mihon: A manga reader and Komga client.
  • Moe Memos: A note-taking app developed by me.
  • MX Player Pro: Used for watching local videos not in my Plex library.
  • NewPipe: Considered the best YouTube client.
  • Play Books: Used for reading ePub books, chosen for its page-turn animation.
  • Plex and Plexamp: My primary media players.
  • Squawker: Used to subscribe to voice actors from the former birdsite.
  • Stealth: A Reddit client.
  • Tasks.org: For to-do lists syncing with Nextcloud.
  • Termux: A powerful terminal emulator.
  • Transdroid: Used to manage my qBittorrent server.
  • Tusky: A Mastodon client.
  • Zoo for Zotero: For reading from my PDF library.

Gaming

The Lenovo Legion Y700 is marketed as a “gaming tablet.” Although I am not a mobile gamer—since most modern mobile games adopt a freemium, casino-like business model that I feel detracts from the concept of video games as works of art—Android remains an excellent platform for emulation and game streaming. This tablet has the capability to emulate PS2 and Wii games at full speed with upscaled textures, as well as stream PC games via Moonlight, or PS5 games using PSPlay or Chiaki, all with negligible latency.

Gaming on Android devices represents a larger story for me, but it’s not particularly tied to this tablet or the Pixel phone. I’m keeping it a secret for now and may discuss it in more detail in the future.

Conclusion

The journey to find the spiritual successor to the Google Nexus 7 led me to the Lenovo Legion Y700, a device that, while initially not perfect due to its ZUI operating system, transformed into a nearly ideal tablet experience after installing crDroid. This shift back to a stock-like Android experience reminiscent of the Nexus 7 days has been both nostalgic and enlightening, proving that the essence of a great tablet is not just in its hardware but in the freedom and flexibility of its software. The Lenovo Legion Y700, with its powerful specs and the newfound joy of a pure Android experience, serves as a testament to the enduring appeal of customizable, user-centric technology. It reminds us that sometimes, the true successor to a beloved piece of technology is not about finding an identical replacement, but about recapturing the spirit of innovation and openness that made the original so special.

Homelab, Illustrated By DALL·E

Homelab is a place where you can store all your family’s data, self-host applications and services, locally stream media, and experiment with various technologies.

A Homelab can start with low-power devices such as a Raspberry Pi, or with hacking your router to install open-source firmware and run custom applications. It can also involve easy-to-use consumer NAS solutions like Synology. Alternatively, you might find yourself overwhelmed by used data center hardware.

Throughout 2023, I have been tinkering with my home network and computing setup, which has been both fun and rewarding.

Hardware

Various devices operate continuously at my home. Some were shelved last year, but there are a few newcomers worth mentioning.

The router “Rotom,” which I use for my home’s internet, is an iKoolCore R1. This compact device is powered by a Celeron N5105 processor and is equipped with 8GB of RAM, a 128GB SSD, and four 2.5GbE Ethernet ports. It runs Proxmox VE, hosting an OpenWRT VM as my primary firewall, and a Debian VM for several networking applications.

This summer, I noticed that the SSD became read-only on several occasions due to overheating. Fortunately, the manufacturer offers an R1 Pro upgrade kit. This kit replaces the case with a heatsink that directly contacts the NVMe SSD, effectively mitigating the overheating issue. Additionally, it includes a PWM fan, which keeps this compact computer silent in my closet.

My Network Cabinet in Augest 2023

I have a 15U network cabinet situated in the corner of my living room. At the bottom lies a Santak TG-BOX 850 UPS, safeguarding the devices within the cabinet. Positioned in the middle is my NAS, acquired in 2020 – a Synology DS1621+ equipped with six 8TB HGST drives and 12GB of ECC RAM. It’s set up in a RAID-6 configuration, providing a total storage capacity of 32TB.

At the top of the cabinet is the Homelab server “Porygon,” which I assembled in 2023. Inspired by Tao of Mac, it’s built around an ASRock DeskMeet B660 box. The configuration includes an Intel Core i7 12700 CPU, 128GB of DDR4 RAM, two NVMe SSDs (a WD SN770 2TB and an older Samsung 970 Evo Plus 1TB), and a Geforce RTX 3060 GPU.

The DeskMeet B660 also runs Proxmox VE. The integrated Iris Xe GPU is dedicated to a Fedora Workstation VM for transcoding videos and offering a remote desktop. The RTX 3060 switches between a Windows VM, which I use for game streaming, and a Debian VM for doing AI experiments.

However, I’ve encountered heat issues with the “Porygon” server inside the network cabinet. The CPU, cooled by a Noctua NH-L9i-17xx, can quickly reach 100°C during moderately intensive tasks. The SSD, even with a heatsink, often heats up to 70°C. To address this, I implemented two solutions. Firstly, I replaced the PSU fan with a Noctua NF-A12, which reduced the CPU temperature by 15°C, both at idle and under load, at the cost of injuring both my hands. Secondly, I installed a dedicated cabinet fan controlled by a Xiaomi Smart Plug. When the SSD temperature hits 70°C, Home Assistant activates the fan, which turns off once the temperature drops below 65°C. Ultimately, I’m quite content with the new Homelab server. It’s usually quiet and energy-efficient, yet it can be exceptionally powerful when needed.

Applications and Services

In my Homelab, I run a wide array of applications, catering to various needs. This includes network applications that ensure connectivity for my devices at home and allow me to access my home network remotely. I also use note-taking apps, read-later services, home automation systems, media management services, file synchronization programs, RSS readers, as well as a git server complemented by CI/CD pipelines. Moreover, I have dashboards in place to navigate and monitor all these applications efficiently. It’s fair to say that a significant portion of my digital life depends on these machines, which are entirely under my control at home.

Homepage Dashboard and my Applications

On my router “Rotom”

On the compact yet capable machine, I have set up two virtual machines and one LXC container to manage various services and applications.

One VM operates as my primary router OS, running OpenWRT. I’ve configured it with passthrough for three NIC ports. The WAN port connects to the modem provided by my ISP, while the LAN ports, in conjunction with the VM net port, establish a bridged LAN network. The majority of OpenWRT’s configuration revolves around DHCP settings and firewall rules.

In addition to the routing functions, “Rotom” hosts several other programs:

  • Jellyfin: This serves as my secondary media center. I’m contemplating a switch from Plex to Jellyfin since it is fully open-source and doesn’t depend on a remote server.
  • frpc: This is for exposing certain services, like an SSH service that enables me to connect back to my home network. It’s useful because I have a CG-NAT IPv4 network. If I encounter a lack of IPv6 support in an external Wi-Fi network, I rely on frp to access my home network.
  • cloudflared: This acts as an alternative method for exposing services via Cloudflare Tunnels.
  • cloudflare-ddns: A dynamic DNS service that maintains an AAAA record, facilitating direct IPv6 connections back to my home.
  • traefik: This reverse proxy manages services not running on “Porygon.” I’ll detail the services running on the Homelab PC separately.

Jellyfin operates within an LXC container, with the Intel iGPU passed through for video transcoding. The remaining services are housed in Docker containers running on a Debian VM. The Debian VM also functions as a side router for my Homelab PC and devices such as the Apple TV and Sonos One.

On my Synology NAS “Uxie”

As my storage center, I need my NAS to be as robust as possible, so I haven’t installed too many applications on it.

In addition to standard NAS services like SMB, NFS, and File Station, the applications I use include:

  • Synology Photos: Backs up the photos of my family. I’m considering switching to an open-source solution like Immich, but the last time I tried, Immich didn’t integrate well with the iCloud Photo Library.
  • qBittorrent and VueTorrent: Mostly used to download Linux ISOs.
  • aria2: Another tool for downloading Linux ISOs.
  • Forgejo: A self-hosted Git server; it’s lightweight and a community fork of Gitea.
  • Sonatype Nexus Repository: My private package manager for Docker and npm, and also a cache proxy for docker.io and ghcr.io to ensure all my applications can start without an internet connection.
  • MinIO: An S3-like object storage service for my other applications.
  • Navidrome: A lightweight music manager compatible with the Subsonic API.
  • Sonarr: An automatic media manager for anime and TV shows.
  • Radarr: An automatic media manager for movies.
  • Bazarr: A subtitle downloader for content in Sonarr and Radarr.

Apart from first-party Synology apps, all third-party programs are running in Docker containers.

A Home Kubernetes Cluster in “Porygon”

I’ve migrated the k3s control plane and worker VMs from Frost Canyon NUC “Comet” to my new Homelab PC. Most of my applications are running in this k3s cluster because it’s easy to maintain, upgrade, and recover. Building this has also been a great way to learn.

Here’s a list of applications running in this cluster:

  • Memos: My primary note-taking app for capturing my thoughts and ideas. I also developed the iOS and Android client titled “Moe Memos.”
  • Miniflux: My RSS reader. It supports the Fever and Google Reader API and can be connected to various RSS clients.
  • Vaultwarden: An unofficial server written in Rust for the open-source password manager Bitwarden. It’s more lightweight than the official server.
  • Wallabag: My link manager and “read later” app.
  • MicroBin: A text and file-sharing application for when I’m transferring things without Universal Clipboard.
  • NocoDB: An AirTable-like smart spreadsheet. I use it for personal project management and to track games and movies.
  • Komga: My manga library. It supports OPDS clients like Panels and Mihon.
  • Paperless-ngx: A document management system for organizing scanned documents.
  • Stirling PDF: An all-in-one PDF toolkit.
  • Audiobookshelf: I use it as an automatic podcast downloader.
  • Photoview: I use it to manage collected pictures not in my photo library, such as CD and Blu-ray booklets.
  • Prowlarr backed by FlareSolverr: An indexer manager for Sonarr/Radarr.
  • YoutubeDL-Material: Downloads videos from YouTube via yt-dlp.
  • draw.io: The most powerful diagramming tool.
  • Kimai: A self-hosted time tracker.
  • Firefly III: A self-hosted finance manager. I haven’t used it much yet.
  • Node-RED and n8n: Powerful automation tools, but I haven’t used them much yet.
  • Woodpecker CI: A Kubernetes native continuous integration solution.

The manifest of my Kubernetes cluster is managed in a Git repository and is automatically deployed via a GitOps tool named Flux CD. When I push changes to the repository, such as adding a new application or upgrading Docker images, the deployment occurs within a few minutes.

Recently, I discovered a powerful tool named Renovate Bot. It scans my Kubernetes manifest repository every few hours to identify which Helm releases or container image tags require upgrades and then creates pull requests in Forgejo, similar to how Dependabot operates on GitHub.

In addition to the applications I’m running, my setup includes the Ingress NGINX Controller, MetalLB, and cert-manager to facilitate service access and manage certificates. The data for these applications is stored on Longhorn volumes, which are replicated across all three worker VMs and two SSDs. I’ve also integrated Kube Prometheus Stack and Grafana Loki to monitor the cluster and review pod logs.

Another tool I use in parallel with kubectl for managing resources in the cluster is Portainer. I’ve also configured Portainer environments to manage Docker Compose stacks on my NAS, router, and several cloud servers. For convenient navigation through all the apps in my Homelab, I utilize a highly customizable dashboard named Homepage. It seamlessly integrates with Kubernetes and is capable of discovering services based on Ingress annotations.

All k3s nodes operate under Ubuntu 22.04, and I’ve configured the system-upgrade-controller to automatically update k3s and apt.

Remote Development and Desktop

Like my Kubernetes cluster, The Fedora Workstation VM “comet-core” is also migrated from the Frost Canyon NUC. The Intel Iris Xe iGPU is passed through it. I use it mostly as a remote development server, and when I need GUI applications, I can access the GNOME desktop with RDP via xrdp and xorgxrdp.

Notable applications running in this VM contains:

  • Plex: My primary media library. I have lifetime Plex Pass and I still prefer Plex for a variety of unique features. But I’m kinda worried about Plex’s future and its troublesome handling of privacy. Maybe I’ll switch to Jellyfin some day.
  • code-server: A web development environment. However, I often develop on this VM via the Remote Development feature in Visual Studio Code.
  • calibre and calibre-web: My book library. I also use calibre to DeDRM and convert digital books I bought.
  • Handbrake: I use it to transcode videos from my Blu-ray collection to H.265 with Intel Quick Sync.

Gaming and AI Experiments

Having a dedicated GPU has significantly justified upgrading my Homelab PC. I’ve set up a Windows VM “Porygon-Z” and a Debian VM “Porygon2,” both configured to utilize the RTX 3060 GPU. However, only one VM can be powered on at a time since GPU passthrough is restricted to a single VM.

In the Debian VM, I’ve been experimenting with a few smaller LLM apps such as Ollama and ChatGLM-6B, as well as creating art with the Stable Diffusion web UI. I’ve also explored vector databases such as Chroma.

On the Windows VM, I’ve installed Sunshine, primarily to stream games from PC Game Pass, and titles that are either unsupported or don’t perform well on the Steam Deck, to my handheld console with Moonlight. The ability to play games at 1080p 60fps with the highest settings on a handheld device was a dream I never thought would be possible until now.

Other VMs and LXC Containers

Last but not least, I’d like to highlight more applications.

Nextcloud serves as my file synchronization server. It used to perform poorly, particularly when syncing a large number of small files. However, with recent upgrades to both Nextcloud versions and my hardware, its performance has significantly improved, offering super-fast file transfers. I’ve set up Nextcloud with Nextcloud AIO in an LXC container on “Porygon.” Beyond file syncing, Nextcloud also manages my calendar and tasks through CalDAV and syncs browser bookmarks using Floccus.

Home Assistant OS orchestrates all my IoT devices and bridges them to HomeKit. For devices that don’t natively support HomeKit, I can access them through hass-xiaomi-miot. Home Assistant also automates various tasks, such as controlling the network cabinet fan I mentioned earlier and activating the Aqara M1S night light when someone enters the living room at night.

Backups

I loosely adhere to the 3-2-1 backup strategy for safeguarding my data. My NAS serves as the primary backup target for various devices, including my family’s iMac via Time Machine and two Proxmox VE devices using a Proxmox Backup Server VM. Additionally, my older Synology DS916+ NAS, which has been operational for 8 years, secures the data from my main NAS through rsync and Hyper Backup.

For offsite backup, I utilize Backblaze B2 and OneDrive. Backblaze B2 houses backups from Hyper Backup, restic, Longhorn, and Arq, while selected folders from my NAS are one-way synced to OneDrive using Cloud Sync. I’ve opted not to have a remote backup for my media library due to the storage costs, considering that most of the content can be reacquired by re-ripping physical media or re-downloading.

Homelab in 2024

That’s all for my 2023 Homelab setup. I’m very satisfied with my current setup, and there are a few areas I’d like to explore.

Currently, I only have Gigabit Ethernet, even though I have a few 2.5GbE devices. Perhaps I should add a few 2.5GbE switches to access my data faster.

I’d like to make the most of my devices. I’m considering reconfiguring my Frost Canyon NUC as a backup device in case one or two pieces of hardware in my Homelab fail. It happened before with the Noctua CPU fan, and I lost access to “Porygon” for a few days. I also have a spare Raspberry Pi 4 and a Zero 2 W, but I haven’t figured out how to utilize them effectively.

I’m also interested in open-source NAS solutions such as mergerfs, SnapRAID, and openmediavault, but I’m not yet motivated to build one. I’m looking into VLAN setup as well, although I don’t think I really need it yet.

Overall, I have enjoyed my Homelab journey and am looking forward to more building and learning. I hope you enjoyed reading this post, and if you have any suggestions or ideas, please reach out to me on Mastodon.

Background

Hundreds of thousands of image files are stored on 52Poké Wiki. Previously, we used AWS S3 to store all these images, AWS Lambda for image processing (generating thumbnails and converting to WebP format), and two layers of cache (a container on Linode and Cloudflare CDN) for image distribution.

Our monthly AWS bill increased gradually due to the growing number of images and traffic each month. Even with two layers of cache, more than 280GB of traffic defaulted to AWS S3, incurring a $20 expense every month.

Fortunately, there are many S3-compatible services available at much more competitive prices. We considered switching to Cloudflare R2, Backblaze B2, or Linode Object Storage.

Choice

Currently, we have 150GB of files. It costs $2.25 every month to store on Cloudflare R2, $0.9 for Backblaze B2, and $5 for Linode Object Storage. Egress traffic charges are zero for all services: Backblaze B2 doesn’t charge for 3x the monthly data stored, Linode uses a traffic pool that sufficiently covers our usage, and Cloudflare never charges for egress bandwidth. If the data storage increases to 250GB, it will cost $3.75 on Cloudflare, $1.5 on Backblaze, and remain $5 on Linode.

However, since the main services of 52Poké are located in Linode’s Tokyo datacenter, there is significant latency when proxying from the other side of the Pacific Ocean. Linode recently announced Object Storage availability in their Osaka datacenter, and with a latency of only 8ms between Tokyo and Osaka, this is an optimal choice performance-wise. We can also eliminate one cache layer (20GB) backed by block storage, saving $2 monthly. Ultimately, by migrating from AWS S3 to Linode Object Storage, we can save $17 every month.

Steps

Creating a Bucket on Linode and Setting Up the Bucket Policy

We can create an Object Storage bucket in Akamai Cloud Manager with the bucket name media.52poke.com.

Previously, we configured a bucket policy on S3 to allow access solely from the IP addresses of 52Poké’s cloud servers. While Akamai Cloud Manager doesn’t support direct bucket policy configurations, they can be set up using s3cmd as per Linode’s documentation.

The bucket policy looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::media.52poke.com/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"..."
]
}
}
}
]
}

Full Migration

To minimize disruption to 52Poké Wiki during this migration, we’ll implement a two-step mechanism. First, we’ll migrate all files from the AWS S3 bucket and allow users to upload or update files. Later, we’ll conduct an incremental migration, which should take considerably less time.

A Kubernetes job will be used to run rclone sync to sync all files to the new Linode Object Storage bucket. For the access key ID and secret access key of Linode Object Storage, we’ll use SOPS to encrypt and store them in the Git repository. These can then only be decrypted with the private key in the 52Poké Kubernetes cluster and deployed automatically.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: batch/v1
kind: Job
metadata:
name: migrate-media
namespace: default
spec:
ttlSecondsAfterFinished: 100
template:
spec:
containers:
- command:
- /bin/sh
- -c
- rclone sync awss3:media.52poke.com los:media.52poke.com
env:
- name: RCLONE_CONFIG_AWSS3_TYPE
value: s3
- name: RCLONE_CONFIG_AWSS3_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: accessKeyID
name: aws-s3
- name: RCLONE_CONFIG_AWSS3_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: secretAccessKey
name: aws-s3
- name: RCLONE_CONFIG_LOS_TYPE
value: s3
- name: RCLONE_CONFIG_LOS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: accessKeyID
name: linode-object-storage
- name: RCLONE_CONFIG_LOS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: secretAccessKey
name: linode-object-storage
- name: RCLONE_CONFIG_LOS_ENDPOINT
value: jp-osa-1.linodeobjects.com
image: rclone/rclone:latest
imagePullPolicy: Always
name: migrate-media
restartPolicy: Never

Disabling Uploads

In the second step, we must disable uploads on MediaWiki to prevent data loss. This can be configured via LocalSettings.php.

1
$wgEnableUploads = false;

Incremental Migration

We’ll now run the Kubernetes job again to sync updates from the old AWS S3 bucket to the new one. Additionally, we’ll manually check that recently uploaded files from 52Poké Wiki have been synced to the new bucket.

Updating Malasada

Malasada is an AWS Lambda Serverless function designed to generate image thumbnails and convert images to the WebP format to conserve bandwidth.

Previously, we used an IAM policy to grant the Serverless function access to the S3 bucket. Now, with the shift from AWS S3, we’ll need to manually provide the access key ID, secret access key, and the endpoint of Linode Object Storage to the Serverless function using environment variables.

1
2
3
4
5
6
7
8
const s3 = new S3({
credentials: {
accessKeyId: process.env.S3_ACCESS_KEY_ID,
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
},
endpoint: process.env.S3_ENDPOINT,
region: process.env.S3_REGION,
});

Configuring Nginx

The Nginx configuration for the domains media.52poke.com, s0.52poke.wiki, and s1.52poke.wiki must be updated to replace the AWS S3 domain with that of Linode Object Storage. We also removed the persistent volume claim backed by block storage from the nginx deployment.

1
proxy_pass http://media.52poke.com.jp-osa-1.linodeobjects.com$request_uri;

After updating the nginx configuration, we noticed all requests were failing due to permission issues. However, there was no issue when requesting the upstream Object Storage URL with curl in the nginx container.

Upon further investigation, we determined that Linode Object Storage was blocking requests containing “X-Forwarded-For” and “X-Real-IP” headers because these headers didn’t align with the bucket policy. Removing these headers from the request resolved the issue.

1
2
proxy_set_header X-Real-IP '';
proxy_set_header X-Forwarded-For '';

Configuring MediaWiki

We now need to adjust the AWS S3 MediaWiki extension to use Linode Object Storage instead of AWS S3. This adjustment can be quickly made via the endpoint parameter. Afterward, we can re-enable file uploading on MediaWiki.

1
2
$wgFileBackends['s3']['endpoint'] = 'http://jp-osa-1.linodeobjects.com';
$wgEnableUploads = true;

Configuring WordPress

The 52Poké homepage uses the WP Offload Media Lite plugin to upload files to S3. While the plugin’s admin page doesn’t offer functionality to change the object storage endpoint, our investigation of the source code revealed hooks that allow modifying the S3 connection configuration. This can be adjusted in wp-config.php.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
add_filter( 'as3cf_aws_s3_client_args', function ( $args ) {
$args['endpoint'] = 'https://jp-osa-1.linodeobjects.com';
$args['region'] = 'jp-osa-1';
$args['use_path_style_endpoint'] = true;
return $args;
} );
add_filter( 'as3cf_aws_get_regions', function ( $regions ) {
$regions = array(
'jp-osa-1' => 'Osaka',
);
return $regions;
} );
add_filter( 'as3cf_aws_s3_bucket_in_path', '__return_true' );
add_filter( 'as3cf_aws_s3_domain', function ( $domain ) {
return 'linodeobjects.com';
} );

Backup

As the migration nears completion, it’s important to remember that we have a cron job set up to back up files from AWS S3 to Backblaze B2. We’ll need to update the backup source to Linode Object Storage, which can be achieved through the environment variables of rclone.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
env:
- name: RCLONE_CONFIG_MEDIA_TYPE
value: s3
- name: RCLONE_CONFIG_MEDIA_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: accessKeyID
name: linode-object-storage
- name: RCLONE_CONFIG_MEDIA_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: secretAccessKey
name: linode-object-storage
- name: RCLONE_CONFIG_MEDIA_ENDPOINT
value: jp-osa-1.linodeobjects.com

Conclusion

52Poké Wiki successfully transitioned from AWS S3 to Linode Object Storage for image storage. Despite the availability of lower-cost options like Cloudflare R2 and Backblaze B2, Linode Object Storage was the optimal choice given its low latency benefits and the site’s primary services being located in Linode’s Tokyo datacenter. With this change, we anticipates a monthly savings of $17 and enhanced performance. Through a careful migration process, including addressing challenges with permissions and configurations, the transition was executed smoothly. This move underscores the importance of regularly evaluating and optimizing infrastructure choices to ensure both cost-effectiveness and performance for online communities.

Home Projects

As my family moved to a new place, I have been rearranging our network and computer equipment.

The first new hardware to arrive at my home in 2023 is an iKOOLCORE R1. It is a tiny x86 computer with four 2.5GbE NICs pre-installed with Proxmox VE and OpenWrt. I named this device “rotom” and use it as a router. I also set up a Debian VM called “rotom-core” as a “side gateway.” This addition has vastly improved my home network experience, in terms of both responsiveness and functionality.

I have tinkered with a few configurations of the open-source router system. One benefit of OpenWrt is its ability to configure IPv6 firewalls. I wanted to open a specific port on rotom-core to allow me to connect back to my home network while blocking all other incoming connections. However, most home networks have dynamic IPv6 address prefixes, and only the suffix is static. Fortunately, this can be easily set up in OpenWrt:

1
2
3
4
5
6
7
8
9
config rule
option target 'ACCEPT'
option src 'wan'
option dest 'lan'
option name '<rule name>'
option family 'ipv6'
option proto 'tcp'
option dest_ip '::<IPv6 address suffix>/::ffff:ffff:ffff:ffff'
option dest_port '<Port I want to open>'

I wanted certain devices on my home network to use rotom-core as the gateway; however, some devices (like Sonos) do not allow for configuring static IP and gateway. To address this, I added an entry in the DHCP settings to pin the DHCP response to these devices.

1
2
3
4
5
6
7
8
9
config host
option name 'SonosOne'
option dns '1'
option mac '<mac address>'
option ip '192.168.1.31'
option tag 'tag1'

config tag 'tag1'
option dhcp_option '3,192.168.1.2 6,192.168.1.2'

One issue I faced previously was the inability to change or disable the IPv6 DNS server on Android, rendering the DNS server settings useless in a dual-stack network environment. OpenWrt allows me to disable the announcement of the IPv6 DNS server address.

1
2
3
config dhcp 'lan'
...
option dns_service '0'

In rotom-core, besides the critical services I prefer not to mention, there are also containers running frpc and cloudflared to enable connection to my home network from IPv4-only networks. Additionally, there is cloudflare-ddns to update the AAAA record in DNS.

After receiving this new tiny PC, I benchmarked it using p7zip and compared it to my Frost Canyon NUC (“comet”, named after Comet Lake). Surprisingly, it outperformed the i7-10710U by 200% in both single-core and multi-core tests, which indicated an issue with the NUC.

Two possibilities came to my mind. I thought it could be related to how VMware ESXi handles turbo boost, or the cooling system in the NUC had melted down. After turning off the NUC and waiting for half an hour, the CPU speed greatly improved, but it soon hit rock bottom. Thus, it must be the cooling.

There are various videos suggesting that the thermal silicone between the CPU and cooler needs to be replaced. I purchased a small Honeywell phase change pad and prepared to fix the NUC. However, I made a mistake at the first step; I broke a screw on the motherboard, and it was impossible to pull out.

I eventually gave up and only cleaned the air outlet of the cooler. Surprisingly, it worked. Now, the NUC is much faster and makes significantly less noise than before. I regret not noticing this problem earlier.

A k3s worker node VM in comet has been replaced by a Fedora Workstation VM named “comet-core”, since the services in the home Kubernetes cluster don’t require too much memory. I want to experiment with different Linux distributions, and the VM allows me to work on tasks that the Raspberry Pi can’t handle.

I also migrated the hypervisor of comet from VMware ESXi to Proxmox VE. Initially, I wanted to try GVT-g Split Passthrough to enable both the Windows VM and the Fedora VM to access the iGPU. However, the split performance was not sufficient to provide a smooth RDP experience. Nevertheless, I appreciate Proxmox VE’s flexibility and LXC features.

52Poké

52Poké Wiki has been upgraded to MediaWiki 1.39, featuring numerous improvements under the hood. I am a fan of the new Vector-2022 skin[1] (with the max-width option disabled). It is clear, vibrant, responsive, and fully-fledged.

The Kubernetes cluster of 52Poké is now synchronized with this git repository via GitOps, using FluxCD. Additionally, the database server has been moved into the cluster, thanks to NVMe Block Storage. All nodes have been upgraded to 4 CPU core 8GB memory instances, and the number of nodes has been reduced (currently only 2), helping to decrease the total cost.

One issue that occasionally stresses the 52Poké Wiki is when editing a template used by hundreds of pages. MediaWiki attempts to purge the cache of every page using this template. When hundreds of pages load without cache, the MediaWiki instance may freeze. I am developing a new web server to address this issue. It will implement a persistent cache layer, inspired by the concept of Cloudflare Cache Reserve. This approach allows the cache purging process to work in an asynchronous queue, updating the persistent cache instead of deleting it. Ideally, this will enable the backend MediaWiki instance to handle much fewer requests while ensuring that page edits are updated for all visitors as quickly as possible.

Since April 2023, Linode (now Akamai Connected Cloud) has increased its computing prices by 20%. This has prompted me to consider how to maintain the long-term sustainability of 52Poké Wiki. Up until now, I have been covering all the monthly bills from Linode, but there is always a risk of facing financial difficulties at some point in the coming years. I hope that 52Poké Wiki can eventually be partly supported by its community and visitors; however, I have not yet found a viable solution to achieve this.

Moe Memos

Initially, Moe Memos was a niche open-source project developed to meet my personal needs and maintain my hands-on experience with app development. However, it has since become a much larger success. As of now, over 3,500 users have installed the app from App Store and Google Play (excluding GitHub and F-Droid downloads). It has evolved into a feature-rich app for Memos while maintaining its minimalistic design.

I still have many ideas for this charming project, ranging from implementing major features such as local storage (hoping for a revamp of Core Data this June) and multi-server/user support, to refactoring the project structure and adding a tipping feature to motivate me to continue working on it and explore more ideas for the self-hosted community.

AI Experiments

The past few months have undoubtedly been an AI Summer, with news related to AI projects such as ChatGPT, New Bing, Office 365 Copilots, Stable Diffusion, and Midjourney flooding my mind every few hours.

I initially tried to resist the hype, but I eventually succumbed after the release of the GPT-3.5-Turbo API and GPT-4 model. I experimented with various things involving LLMs and AI art generators. I read a letter that Yui from Sword Art Online sent to Asuna and Kirito and asked her what everyday life is like as an AI in the VRMMO world. I interviewed a Pokémon Trainer in Hoenn about her epic journey. I asked ChatGPT to write the first commit of the web server I mentioned earlier. I even allowed it to be a Pikachu with no knowledge of human language, experiencing the atmosphere of Snowpoint City and battling a Glalie. In addition, I have written stories such as “Takami Chika Meets Her Real World Counterpart Inami Anju” and “The Birth of Amane Suzuha in the Steins;Gate Timeline”. I also used Midjourney to design alternative icons for Moe Memos.

Another entertaining project stemmed from an idea inspired by Persona 5. I asked ChatGPT, “You are the main character of Persona 5. You found a palace belonging to Donald Trump. What does Donald Trump’s palace look like, and how will your team obtain the treasure within it?” The response was both hilarious and reasonable. I tried a few other names, ranging from real-world villains to absurdities like “MacBook Pro with Butterfly Keyboard.” Eventually, I created a Persona 5 Palace Generator in about an hour with ChatGPT’s help. I have more intriguing ideas stemming from this, but I prefer to keep them secret for now.

I am still uncertain about how LLMs, generative AI, and future AGI will change our world, or whether the change will be overwhelmingly positive. I remain concerned that widespread poverty may result from these advances. I still prefer to enjoy technologies I can reason with, predict, and reproduce variants of. I am annoyed by the racial/regional discrimination that makes it difficult to use and pay for OpenAI and other services. I hope humanity can find and imagine a path to a better future.


  1. I understand that there are many controversies surrounding the Vector-2022 skin in the Wikipedia community, and perhaps within 52Poké/Encyclopaediae Pokémonis as well. ↩︎

0%