New Issue: I’ve been on 0.18 since the day of release without any major issues… After a reboot of my VPS after seeing some very high CPU utilization and errors about postgress queries panic in the lemmy service. I rebooted, my server, I made a few posts after during that time and they were not appearing in any communities I posted to. However, on my profile I could see them, and in the communities I could see them if I sorted by “top, hour, day, etc…”. I could tell some others were seeing them as I got upvotes and a comment or two. About 8 hours later, the issue persists for those posts. However, I made a new post now and it is viewable with normal sorts.

So, best guess, my lemmy service was FUBAR for awhile after the reboot, I had a persisting high CPU utilization that I can only describe as “odd” that I need to dig more into. I need to grep some lemmy logs to see what happened. I also changed my root password to be safe. Be vigilant y’all.

Just a report.

Logs that were similar:

lemmy_1     | thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'thread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'thread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rsthread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'thread 'actix-rt|system:0|arbiter:0' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
lemmy_1     | thread 'actix-rt|system:0|arbiter:1' panicked at 'read local site data: QueryBuilderError(Timeout(Wait))', crates/apub/src/lib.rs:43:8
  • RoundSparrow@lemmy.mlEnglish
    2·
    1 year ago

    lemmy_server kicks off a bunch of vote counting and such on startup, then they run as scheduled repeat jobs as it keeps running. It can really put a heavy load on the database all at once.

    • Rick@thesimplecorner.orgOPEnglish
      2·
      1 year ago

      No doubt, I was watching the logs as I restarted the containers last night. What’s interesting is this all started because my instance just stopped responding after being up for (days or weeks). I’m not sure what to think of it to be honest. As of now everything is working fine, but those posts that I made during that period are gone from the feed if not sorted by Top. I can also see them on my profile. Weird stuff.

  • Rick@thesimplecorner.orgOPEnglish
    1·
    1 year ago

    Just to confirm, I can see this post in normal sorts, so the issue seems resolves but… for about 15-30 mins posts were there but not viewable by normal sorts (new, hot, active, etc.)