Ideally we’d use a single common HTTP request error format handling
for _all_ HTTP requests (including non-ActivityPub requests, e.g. NodeInfo).
But for the purpose of this commit this would create too much noise
and it is significant effort to go through all error pattern matches etc
too ensure it is still all correct or update as needed.
The Signature module now handles interaction with the HTTPSignature library
and the plug everything related to HTTP itself. It now also no longer needs to be public.
To achieve this signatures are now generated by a custom
Tesla Middleware placed after the FollowRedirects Middleware.
Any requests which should be signed needs
to pass the signing key via opts.
This also unifies the associated header logic between fetching and
publishing, notably resolving a divergence wrt the "host" header.
Relevant spec demands the host header shall include a port
identification if not using the protocols standard port.
Fixes: https://akkoma.dev/AkkomaGang/akkoma/issues/731
The remote part is included in federated emoji names by e.g.
Iceshrimp.NET ever since remote emoji support was added in
4d21aa1670
and as of writing it still continues to do so.
It adds no value for us though; we add the remote part automatically
based on the URL and it makes it more difficult to correctly coalesce
the original reaction (from a user for whom the moji was local)
and the subsequent reactions with the identical emoji from users of
other instances. Additionally the remote part can cause issues when
later used with our REST API.
For non-reactions this is unproblematic and thus
there’s no need to change anything there.
Use a migration to fix up existing activities.
This will cause some (further) desync from the inlined reactions
array, but will be fixable with the resync mix task and avoids
issues when running the resync without first fixing existing activities.
This was already removed from mix.exs in
ea5a2a9f21
but as it turns out it was also re-set
during runtime.
Since we never set it outside of CI in
the first place there’s no need to
force-disable it here.
ActivityPub spec demands each actor has at least an inbox and outbox.
Furthermore, the current representation wouldn’t even be accepted by
ourselves, since our processing requires objects to be flagged with a
sensible type else we don't know what to do with it.
Including the nickname is just a peemptive measure.
There were no reports of this causing problems in real-world deployments
and at least for federation with other Akkoma instances we should have
never run into this, since we _always_ expose the full representation of
the instance actor and atm also always use the latter for fetching
remote content (which prevents us from fetching followers-only content).
Nonetheless, serving something which violates spec and we wouldn’t even
accept ourselves seems obviously bad, so fix it and add tests to prevent
this from reoccuring.
Fixes bug introduced in 8f322456a0
Most HTTPS requests actually fall into the single-digit millisecond
range or below on average. Even the more costly endpoints almost always
average around the lower third of the millisecond magnitude.
Only endpoints doing synchronous remote HTTP fetches (e.g. for signing
keys) occasionally spike into the order of seconds.
As is, the bucket resolution is completely unfit to reason about
anything and even just averages are better indications.
Most database queries take less than a millisecond and even in total
almost all take less than 50ms for me. Decode time is but a tiny
fraction of that and queue time usually only takes a small part of total
time too (but may spike on high load).
Shift the buckets down to be able to
give insight into all relevant cases.
In particular this allows to determine whether high averages
are the result of generally high processing times or just a few
outliers lifting the whole average up (e.g. slow network fetches).
Exact numbers are biased towards my setup for lack of other comparison
data, but at least the order of magnitude should be ok everywhere.
The old code was unnecessarily complicated, full of unused and/or
duplicated functions making it hard to understand what will actually
happen and for whom at runtime.
Since we only support a single HTTP backend this can be greatly simplified.
Now everything gets default options from a single place and only
functions to modify parts actually difffering across calls are exposed.
No HTTP3/QUIC support yet.
Note, allowing both here means we don't actually profit from HTTP2 multiplexing
due to Finch(? or maybe a dependency of Finch?) limitations. But it means we can
now interact with HTTP2-only instances (if such exist) and still may get minor
gains from header compression etc
Adventurous admins can change the config to allow only HTTP2,
thus profiting from multiplexing (but breaking federation with
HTTP1-only instances which are in fact observed to exist).
Resolves interop issue with a (reverted but possibly returning) bridgy change
as was reported in the comments of
https://akkoma.dev/AkkomaGang/akkoma/issues/831.
This won't change anything for the problem originally reported there.
Notably we now always fetch the full collection (up to the configured
item count limit) instead of only using the first page if its link was
inlined.
- This adds extra tests to be sure that scrubbing still happens.
- When doing this I notices that the htmlMfm key wasn't stored in the database when comming through the federator. This has been now been fixed too.
- We also test that values true, false or no attribute all work for incomming messages.
Previously all such requests led to '401 Unauthorized'
whih might have triggered retries.
Now, to not leak any MRF info, we just indicate an
accept for POST requests without actually processing the object
and indiscriminately return "not found" for GET requests.
Notably this change also now causes all signed fetch requests from
blocked domains to be rejected even if authorized_fetch isn’t enabled.
Fixes: https://akkoma.dev/AkkomaGang/akkoma/issues/929
To make it usable in scenarios without a draft.
The next commit adds a user for the new function.
This does technically change behaviour a bit, since
"private" relies to "direct" messages no longer implicitly
address the parent post’s actor, but this seems like a contrived
scenario and was likely never intended to actually occur anyway
as cocorroborated by the absence of tests for it.
A pool timeout shorter than the receive timeout
makes race conditions leading to active connections
being killed more likely and laso just doesn’t make
much sense in general.
See: https://github.com/sneako/finch/pull/292
It may still crash due to a race condition between checking for file
existence and opening/streaming, but File.stream! has no safe version
we can use to avoid this completely.
Just not deleting such files during a reload is easy enough.
Until now only a limited number of self-replies were inlined as an
anonymous, unordered ActivityPub collection. Notably the advertised
replies might be private posts.
However, providing all (non-private) replies allows for better thread
consistency across instances if the remote server cooperates.
The collection existing as a stndalone object has two advantages
for this. For one, if it was still anonymous, _all_ replies would need
to be inlined, which might be too bloated in pathological cases.
Secondly, it allows remote servers to update the thread by traversing
the reply collection independent of the original post. (If the remote
part knows about chronological ordering, it can in theory
even efficiently resume from where it previously stopped)
An OrderedCollection uses orderedItems instead of the items key.
So far inlined orderedCollections thus failed to be processed.
Ordered replies collections are used e.g. by IceShrimp.NET and Sharkey,
while Mastodon uses a partially inlined plain Collection.
Not all endpoints use OpenAPISpex’ string-to-atom mapping
and they’ll end up with path params being promoted to
query params in pagination next/prev links.
Fix this by never including path params in the first place
Ecto.cast is will convert valid string keys to atoms, but can
only deal with inputs which use either string keys everywhere
or atom keys everywhere.
Since :id_type is used before the case it must be an atom,
thus it was impossible to use it with string paramteres before.
Up until now queries were always forced into descending ID order
(reverse chronological order with our ID schemes).
Now it’s possible to request the reverse by passing `oder_asc: true`.
The initial info message listing all found packs ought to be sufficient
and with many packs installed thiscan create multiple pages of log
messages on each emoji reload or server start.
Any errors or non-indexed packs are still logged to higher levels.
This requirement was originally added together with splicing the
inbox owner into the non b* addressing fields to make bcc transports
work in https://git.pleroma.social/pleroma/pleroma/-/merge_requests/390.
Later on this was relaxed to always allow deliveries devoid of any
addressing at all in f6cb963df2
and always allow deliveries from actors the owner is following in
750b369d04 to fix interop issues with
Mastodon and Honk respectively.
The justification for both the filtering and splicing comes from
one sentence in AP spec’s inbox section:
> In general, the owner of an inbox is likely
> to be able to access all of their inbox contents.
While this may provide plausible justification for splicing the owner
into cc, it is less clear how this requires or justifies the set of
filtering rules employed here.
Surveying a few other implementations no similar
filtering or splicing appears to be employed.
Furthermore, spec-compliant servers will strip bto/bcc _before_
delivery to remote servers, meaning any compliant bcc transport
out there will NOT contain any explicit addressing of the inbox owner.
Thus the addressing requirement directly opposes
the goal of the original patch.
Currently the requirement for the owner to be addressed once again
is causing interop issues. It turns out to be the root cause of
a long-standing (2+ years) bug preventing meaningful federation.
Bridgy sends e.g. Follow activities and Accepts for Follows directly
to the affected user’s personal inbox while solely addressing
the public scope in the to field. Notably follow relations never
getting established prevented the "accept if followed" allow rule
to ever come into effect.
To make matters worse non-addressed messages simply lead to a
vague "internal server error" response being sent back
which likely slowed down locating the issue.
Furthermore additional issues wrt to signatures cropped up after
the 500-response issues wa first reported, but they seem to have
already been fixed in the meantime, possibly with the signature
handling overhaul in Akkoma.
Given it repeatedly caused issues, does not appear to align with common
practice in the wider fedi ecosystem and apparently contradicts its
original intention, simply remove the requirement.
This is confirmed to fix bridgy interop.
The addressing splicing actually should also add the inbox owner to bto
or bcc instead of cc, but for now this is not changed and in practice
bto/bcc delivery appears to be basically unused anyway.
Most headers are automatically checked by the library after this
upgrade. But since digest is only required for requests with a body
and body processing is handled outside the lib atm, we need to
explicity pass the presence or absence along or not get feedback
about creating broken signatures.
This makes bugs in our signatures more apparent
allowing faster discovery and fixing
This property was introduced as a way to gauge whether and
how much enabling authfetch might break passive federation in
https://akkoma.dev/AkkomaGang/akkoma/pulls/312.
However, with the db field defaulting to false, there’s no distinction
between instances without valid signatures and those which just never
attempted to fetch anything from the local instance.
Furthermore, this was never exposed anywhere and required manually
checking the database or cachex state via a remote shell.
Given the above it appears this doesn't actually
provide anything useful, thus drop it.
The most common permanent receiver error arises for likes/boosts
when we don’t yet know the rlevant object and can't fetch it
due to the remote being overwhelmed or otherwise down.
Before this changes all retries were rather rapid
thus not giving the remote enough time to recover
and usually all failing. Now the remote has about 20
minutes to recover before we give up.
Transient errors from race conditions and (presumably)
weird database-cache interactions also occur regularly.
However, they resolve within the first one or two retries
and those intial retries still happen relatively quickly.
