Giac El Vecio shared a comment of this post/conversation

Giac El Vecio

Sat, 20 Jul 2024 06:44:28 -0300 de Diversi Spiritus

Update test

Haakon Meland Eriksen (Els Mussols)

Sat, 20 Jul 2024 07:13:17 -0300 de Streams at els Mussols

Is the test updated?

Giac El Vecio

Sat, 20 Jul 2024 08:13:41 -0300 de Diversi Spiritus

@Haakon Meland Eriksen (Els Mussols) until two weeks ago everything was going smoothly...
Then a series of major updates, like 4 or 5, and every time I make a post involving activitypub here I have 10 minutes of panic on my site.
But yesterday I did experiments: I made a post for a list of 50 or 60 people, but without activitypub, just zot nomad, and it all worked perfectly. Then a post test for a small list of 18 activitypub people, then a very small list that even the stupidest server would manage without difficulty, and again 10 minutes of panic on my site.
Now, honestly, I'm tired, obviously something is wrong, at the same time I'm also tired of listening to bullshit about the ability of my site administrator friend, because he is a person with a lot of experience, who has managed various sites including friendica red hubzilla... .

Giac El Vecio

Sat, 20 Jul 2024 07:56:03 -0300 de Diversi Spiritus

A new one now. Updated again

Giac El Vecio

Sat, 20 Jul 2024 08:38:37 -0300 de Diversi Spiritus

More than 10 minutes of shit timeline black-out....

Haakon Meland Eriksen (Els Mussols)

Sat, 20 Jul 2024 17:24:23 -0300 de Streams at els Mussols

Sorry I cannot help.

What do the logs say?

Is there any help watching journalctl live with

journalctl -f

Bill Statler

Sat, 20 Jul 2024 17:34:26 -0300 última edição: Sat, 20 Jul 2024 17:47:09 -0300 de Bunny of Doom

This is really frustrating -- I mean, more frustrating for you than for me, but I want to make a suggestion and I can hardly think of anything.

Mike made bunch of changes in the last ~15 hours, so an update to 24.07.20.1 might help. Maybe. Oops, I see from your posting time that you probably already have those updates.
Other than that, my only clue is that your instance has this bug, and no other instance has noticed it. Why? What makes your instance different from the others?

Mike Macgirvin 🖥️

Sat, 20 Jul 2024 18:55:39 -0300 de Y

@Bill Statler echoes my sentiment. Over the years I've gotten a lot better at finding issues buried somewhere in megabytes of code based on some simple description and no access to logs or database on the affected system; but I'm still not clairvoyant. I have no idea what's happening here and I cannot duplicate it locally, and I don't know of any changes in the last 5-6 months which would affect this. And I've changed a whole lot of code in the last 5-6 months. The nomadic identity changes were massive. So were conversation containers before that.

So here's what you do. Turn on full logging at the maximum log level if you haven't already. Create an access list with 5 ActivityPub folks in it. Two might work, but 5 would be better. Then send them a message. Just before sending the message, delete your current logfile. Send the message. After a minute or two, copy the logfile somewhere and email it to me. Then turn your logging level back down to something reasonable again so it doesn't fill your disk - unless you have the logrot addon installed.

Describing what's happening is sometimes helpful, but the reason we have logs is because they actually tell you what (nearly everything) the software is doing behind the scenes and what data they are working with currently. This is the only way possible to solve this kind of problem. I actually don't want to read your logs. There is nothing more mundane than reading megabytes of somebody else's logfiles looking for a needle in a haystack. But it beats playing these guessing games.

Giac El Vecio

Sun, 21 Jul 2024 04:16:01 -0300 de Diversi Spiritus

I can certainly do this, but it may be useful to know that, after I send, the first 40 seconds or 70, everything seems normal, the sent post appears in my timeline, the delivery report is normal as well, but immediately after, and for about 7 or 10 minutes, that the site gets "stuck in a loop".

Giac El Vecio

Sun, 21 Jul 2024 05:50:38 -0300 de Diversi Spiritus

"After a minute or two, copy the logfile somewhere and email it to me. Then turn your logging level back down.."

This part is perhaps difficult or impossible if I use that ActivityPub list of 18 people, because exactly after a minute and a few seconds I lose control of the site and can't do anything for x minutes.
If I just manage for 5, but with 5 people the site doesn't seem to have any problems, and therefore I don't think it's useful, then someone might say "but if with 5 it's fine but with 18 it's not, then it's a problem of your server", at this point I can reply "then explain to me why if I send for 100 zot people the site behaves perfectly?"

Haakon Meland Eriksen (Els Mussols)

Sun, 21 Jul 2024 11:51:55 -0300 de Streams at els Mussols

You should be able to monitor and log what is happening when you do send and receive to your AP connections. Do that.

How about these random action items:

ping the site of your AP your AP contact one by one. Is it reachable? Is it responding very slowly?
is the local disk full, i.e. 80%+, it happens often that temporary files in the background actually fill and swamp the disk, so you have no input/output to speak of, and no evidence without active monitoring of temp files.

Giac El Vecio

Sun, 21 Jul 2024 15:03:28 -0300 de Diversi Spiritus

I'll see what I can do, I'm on holiday in Italy, and the other person with access to the server is in Brazil, there are 5 hours difference, and we need to agree and find an appropriate time.

elvecio

Mon, 22 Jul 2024 04:35:24 -0300 de mitra

@giacelvecio @haakon my site is now completely broken, I can't post, I can't see the connections, the lists/circles only open blank pages.

Mike Macgirvin 🖥️

Sun, 21 Jul 2024 17:11:53 -0300 de Y

Let's start over. I don't have a clue what your server is doing or why. And I'm really tired of playing guessing games.

If you look at admin/logs, there's a setting for the log level. Set it to Normal. Send a post to your ActivityPub friends.

When the server goes wonky, note the time.

Find that time in the log. What was your server doing? Tell me. If you can't figure it out, send me a few hundred or few thousand lines starting at that time.

This might be enough to solve the problem. If it isn't, turn the log level to 'Debug' and repeat. This will provide some details you didn't see on the 'Normal' setting. Again, this will probably solve the problem, without burying either of us in data structures.

But if you still can't figure out what it's doing, turn it all the way up to 'All' and do it again. This will also provide all the data structures that the software was processing at this time. It's a huge amount of data. I doubt that you'll be able to process this or even guess what was going on. Send it to me. But I don't need or want it all. Just send a minute or two starting when you noticed it going wonky.

elvecio

Tue, 23 Jul 2024 08:17:34 -0300 de mitra

@giacelvecio It's the database that explodes the CPU after we post for x number of activitypub people.
What can cause this? Apparently the solicitation is much greater when we send something to activitypub people. How can we record and adjust the server so as not to drive the database and consequently the server crazy? Any useful tricks? Any special numbers in the configurations to self-regulate better?

Waitman Gobble

Tue, 23 Jul 2024 15:43:14 -0300 de Get Ready To Rumbly

I made a script once that would only run the cron task if it wasn't already running.

You can also fiddle with the cron task frequency.

However the problem could be traffic from mastodon incoming, as i recall it can be quite lot of requests.

Bill Statler

Tue, 23 Jul 2024 16:07:56 -0300 de Bunny of Doom

Traffic from Mastodon, interesting possibility!

Or, what if it is a single misconfigured ActivityPub instance hammering your site?

Giac El Vecio

Tue, 23 Jul 2024 16:41:09 -0300 de Diversi Spiritus

in my case, however, it manages incoming traffic well, the problem is that sending for 18 people of activitypub blocks the site for x minutes

Bill Statler

Tue, 23 Jul 2024 16:52:54 -0300 de Bunny of Doom

I'm wondering if Mastodon (or some other AP platform) does something when it receives a post/comment from your site. Like, does it fetch your profile, or some other data from your site? Does it tell all its Mastodon friends to fetch this data at the same time?

I remember a few months ago, owners of small websites were complaining that if somebody shared a link to their site on Mastodon, they immediately got thousands of requests for preview images and text, which acted like a DDoS attack and locked up their webserver. So I'm wondering if something similar could be happening to you? Maybe it was a Mastodon update that caused this, and not a streams update.

elvecio

Tue, 23 Jul 2024 16:52:07 -0300 de mitra

@hello to be more clear:
a single comment that arrives on my post is then sent from my site to all recipients (so more than 18 activitypub people).

Haakon Meland Eriksen (Els Mussols)

Tue, 23 Jul 2024 11:25:37 -0300 de Streams at els Mussols

This is a long read, but maybe you can transfer some of the analysis? https://www.sqlservercentral.com/articles/a-spectacular-case-of-100-cpu-caused-by-online-reindexing

elvecio

Tue, 23 Jul 2024 13:50:11 -0300 de mitra

@giacelvecio Well, I discovered a new problem: if a person from the mastoverse comments on something on my site, what does my site do? What we expect, collects the comment and himself distributes it to the recipients. By doing this my site goes haywire, and not through my fault, but because an external person commented on a post originating on our site. It is a very unsustainable situation.