I’m having trouble staying on top of updates for my self hosted applications and infrastructure. Not everything has auto updates baked in and some things you may not want to auto update. How do y’all handle this? How do you keep track of vulnerabilities? Are there e.g. feeds for specific applications I can subscribe to via RSS or email?
You must log in or register to comment.
I don’t.
Yeah, hot take, but basically there’s no point to me having to keep track of all that stuff and excessively worry about the dangers of modernity and sacrifice the spare time I have on watching update counter go brrrr of all things, when there’s entire peoples and agencies in charge of it.
I just run
unattended-upgrades(on Debian), pin container image tags to only the major version number where available, run rebuild of containers twice a week, and go enjoy the data and media I built the containers and installed for software for.I think the problem is that a lot of people are just running flatpaks, dockers, and third party repos which might not be getting timely updates.
I try to stick to debian packages for everything as much as possible for this reason.
Regarding things like dockers and flatpaks, I mostly “solve” it by only running official images, or at least images from the same dev as the program, where possible.
But also IMO there’s little to no reason to fear when using things like flatpaks. Most exploits one hears of nowadays are of the kind “your attacker needs to get a shell into your machine in the first place” or in some cases evn “your attacker needs to connect to an instance of a specific program you are running, with a specific config”, so if you apply any decent opsec that’s already a v high barrier of entry.
And speaking of Debian, that does bring to mind the one beef I have with their packaging system: that when installing a package it starts the related services by default, without even giving you time to configure them.
GitOps + Renovate.
Tools that allow you to work GitOps (everything is defined in text files in Git) are:
- Kubernetes
- NixOS
- to a lesser degree, Ansible
Here’s a nice starter template for running your own Kubernetes cluster via GitOps with Renovate pre-configured: https://github.com/onedr0p/cluster-template
95% of things I just don’t expose to the net; so I don’t worry about them.
Most of what I do expose doesn’t really have access to any sensitive info; at most an attacker could delete some replaceable media. Big whoop.
The only thing I expose that has the potential for massive damage is OpenVPN, and there’s enough of a community and money invested in that protocol/project that I trust issues will be found and fixed promptly.
Overall I have very little available to attack, and a pretty low public presence. I don’t really host any services for public use, so there’s very little reason to even find my domain/ip, let alone attack it.
You should try wireguard if you haven’t before, like a breath of fresh air
Does badly count as a way?
I kinda keep an eye on that https://selfh.st/ post that does a weekly roundup of stuff to know when I need to do patching.
No doubt there is a container I could run that would do it for me. I just can’t remember the name of it.
There are some tools to help, but things are sort of specific to particular aspects. Lynis for general systems, ntopng for networks, and such.
For 90% of stuff, though, you can just stick to stable repos and upgrade on a schedule and you’ll be alright.
- VPN only, nothing exposed
- Host runs openSUSE MicroOS which updates itself daily
- Watchtower updates the containers daily and if something blows up so be it, except for Nextcloud as everyone says it’s brittle as hell.
i subscribe to the release page of the repo in my rss reader. simple and effective.
That is a fantastic idea. Wtf how is this not commonplace? Or am I just way behind 😅
I put all my data on a NAS with non-writable snapshots so hackers cannot destroy my data.
What happens if they compromise your device in secret and use it as part of a botnet? Lots of state backed attacks rely on traffic relays provided by compromised devices.
My NAS is behind a firewall and doesn’t normally run the types of things you would compromise. (no web browser). They need to break many things at the same time to compromise it. I’m not saying it would be impossible to compromise my NAS, but is is very unlikely just because of how difficult it is. If I’m target of a state level attack I’m sunk anyway.
though offline backups are always a good idea. However they by definition need several days to restore (if they take less than that they are too easy for an attacker to target)
Unless you have actual tooling (i.e. RedHat erratas + some service on top of that), just don’t even try.
Stop downloading random shit from dockerhub and github. Pick a distro that has whatever you need packaged, install from the repositories and turn on automatic updates. If you need stuff outside of repos, use first party packages and turn on auto updates. If there aren’t any decent packages, just don’t do it. There is a reason people pay RedHat a shitton of money, and that’s because they deal with much of this bullshit for you.
At home, I simply won’t install anything unless I can enable automatic updates. Nixos solves much of it. Two times a year I need to bump the distro version, bump the nextcloud release, and deal with depreciations, and that’s it.
I also highly recommend turning on automatic periodic reboots, so you actually get new kernels running…
Thank you everyone for the helpful replies!
upgrades:
- distribution packages: unattended-upgrades
- third party software: subscribe to the releases RSS feed (in tt-rss or rss2email), read release notes, bump version number in my ansible playbook, run playbook, done.
vulnerabilities:
That’s the neat part. I don’t!
I have automatic updates on everything, but if I actually spent time managing updates and vulnerabilities I’d have no time to do anything else in my life.
For my docker containers I use what’s up docker which not only alerts me when there is an update but also give a link to the changes, so I can have a look what’s happening !
For my system itself… Just doing
sudo pacman -Syu. Though that’s not great, cause some updates can potentially break my EndeavourOS system… I keep sometimes an eye on the forum when I see some critical changes like the kernel itself or nvidia updates though.I just update every month or two, or whenever I remember. I use Docker/podman, and I set the version to whatever minor release I’m using, and manually bump after checking the release notes to look for manual upgrade steps.
It usually takes 5 min and that’s with doing one at a time.
upgrade all things by default
This is also a great way to just break everything you’ve set up.
that’s a lot of FUD, topgrade just upgrades using all package managers you have, it doesn’t do the upgrades itself bypassing the manager that installed it, or package authors.
The issue is more that trying to upgrade everything at the same time is a recipe for disaster and a troubleshooting nightmare. Once you have a few interdependent services/VMs/containers/environments/hosts running, what you want to do is upgrade them separately, one at a time, then restart that service and anything that connects to it and make sure everything still works, then move on to updating the next thing.
If you do this shotgun approach for the sake of expediency, what happens is something halfway through the stack of upgrades breaks connectivity with something else, and then you have to go digging through the logs trying to figure out which piece needs a rollback.
Even more fun if two things in the same environment have conflicting dependencies, and one of them upgrades and installs its new dependency version and breaks whatever manual fix you did to get them to play nice together before, and good luck remembering what you did to fix it in that one environment six months ago.
It’s not FUD, it’s experience.
I’ve been doing that for years. Rollbacks are very rare, to the point that it doesn’t make much of a difference whether I do them all at once or not, other than spending more time to do it.
If I wasn’t using containers for everything, sure. Otherwise it’s a bit of an excessive concern.
