A member count of zero was the only guard on the local and network warms,
and a warm that finds no eligible posts leaves the count at zero, so a new
instance, or one whose network feed has aged out, ran the full candidate
query on every single timeline request. Both warms now record an empty
outcome and both read paths consult that record as well as the count, so
the query runs once per hour instead of once per request. Any write to
either feed puts the warm straight back in play.
The lock closures return a value as well, so a request that lost the lock
is distinguishable from one that completed a warm; both used to come back
as null.
The feed is ranked by created_at, so the lowest id in a window is not the
member at the bottom of it: a remote post gets its id when we first store
it, not when it was published, so it can be the oldest id and the newest
post at once. The empty-page branch picked that numeric minimum, which
handed the client a cursor past everything between the remote post and
the real bottom of the window.
The cursor is now the display-order oldest member the request fetched. A
min_id client gets a prev cursor and no next, because it asked for newer
content and an older cursor points it at posts it already has. An empty
fetched window emits no Link at all, which is the end-of-feed signal;
previously the whole branch was unreachable because an empty window
returned early with no headers.
With boosts switched off, which is the default, a window of 51 members
that happens to be all boosts produced an empty page and stopped. One
followed account that mostly boosts is enough to do that indefinitely.
The same shape happens behind a muted account's run of posts or a block
of members that were just pruned.
The read now takes up to three further windows while the page is empty
and at least one member was dropped as unrenderable, as a boost, or by a
viewer filter. A page filtered out by the viewer's own rules does not
continue: they asked for that. Direction follows the request: a min_id
read deepens the same bounded window, because Redis enforces the floor
inside the query and walking older would hand a polling client posts it
already has, while any other read walks older from the last member it saw.
The Link cursor is derived from every member the request fetched rather
than the last pass alone, so it cannot point back into a block the client
has already walked.
A member whose status has been deleted, or which cannot be resolved for
any other reason, was dropped from the page and left in the sorted set.
The feed then went on reporting hundreds of members while serving an
empty page, and nothing in the system ever removed them. The hydration
pass now removes such a member as it finds it and logs once per request
when it pruned anything.
Removing the last member makes Redis drop the set, so the feed becomes
genuinely absent and is rebuilt in the same request. That cannot loop:
the key is gone, so the branch is unreachable a second time.
Drops are also counted by reason now, because the reason decides what the
response is allowed to do. A page the viewer explicitly filtered out is
not a defect to recover from, so it still prunes nothing and runs no
query.
The read path had three ways of answering a request whose window came
back empty, and all three could leave a user staring at a blank timeline.
It returned early whenever the set had any members, so a feed full of
unresolvable ids never got looked at. It only attempted a rebuild for a
request that sent no cursor, so a client holding a cursor got nothing on
every retry. And it marked the rebuild as done before knowing whether it
had worked, in a key nothing ever evicted, so a warm that threw
suppressed the next five minutes of attempts, and the throw itself
escaped the synchronous dispatch and turned the response into a 500.
A missing cached set is now rebuilt in-request behind a non-blocking
30-second lock: a reader that loses the lock serves nothing rather than
running a second full warm, one that wins releases it in a finally, and a
warm that throws is logged and leaves the feed eligible for the next
request. The page is then re-read through the same window helper, so the
cursor the client sent survives the rebuild. The rebuild goes 100 members
deep and queues the deepening job only when it filled that depth.
Three existing cases asserted the old bookkeeping rather than the
behaviour it was meant to protect. They keep their 200s and their empty
bodies and now assert the single-warm guarantee directly.
A feed with nothing in it and a feed that was never built look the same
from Redis: neither has a key. That is why the read path could not tell
"nothing to show" from "needs building", and why the only bookkeeping it
had was a 300-second marker written before the outcome was known and
evicted by nothing.
The trait now carries an empty-warm guard keyed on the resolved feed key,
holding the timestamp at which it lapses with a matching cache TTL. It is
written only by a warm that published nothing and resolved every
candidate it found: a candidate that would not resolve means the warm did
not finish, so the feed stays eligible and the failure is logged. A
forced warm never consults it.
Eviction is automatic for the instance-wide feeds through setAdd, and
explicit for home feeds, which have no key for a write to reach: the
author's own post, a followed local or remote account posting, a new
follow, and a followed hashtag getting a post all clear it. None of them
creates the feed key, so a follower who has never loaded a timeline still
does not get one.
The warm scoped candidates to statuses newer than a six-month-old
snowflake and excluded boosts outright. Those are reasonable defaults
for the common case, but they were also the definition of "this feed is
empty": a viewer whose follows are dormant, or who follows only accounts
that boost, got no candidates and therefore no feed key, and because
every insert path is gated on that key existing, nothing could ever
repair it.
The candidate selection is now a parameterised pass. The first pass is
unchanged, including the per-account fairness that stops a prolific
account crowding out a capped feed. Only when it publishes nothing does
a second pass run, dropping the id floor, admitting shares, and
re-reading a following list that resolved to the viewer alone.
Every cached timeline is trimmed, so a client that persisted a cursor
from before the last trim sends an id whose score sits under every
remaining member. The strict getter correctly finds nothing older, the
endpoint returned an empty page, and the client kept sending the same
dead cursor on every retry. A separate getRankedMaxIdOrRecent getter
leaves getRankedMaxId's strict contract intact: when the window is empty,
the feed still has members and the cursor resolves below the oldest of
them, the newest window is served instead. A cursor at or above the
oldest member is genuine end-of-feed and still answers with an empty page.
Re-derived on staging on top of the non-positive-cursor helpers
(normaliseHomeCursor/fetchHomeWindow), which it extends.
Verified standalone off staging:
HomeCursorRecoveryTest 6 passed; tests/Feature/Timeline + HomeFeed 77 passed.
A forced warm that came up with no publishable candidates deleted the
live feed key. For a per-user feed that is destructive in exactly the
wrong direction: the warm can come up empty because the viewer's cached
following list is stale or because the statuses behind the candidates
could not be resolved, and the user who was reading their timeline a
moment ago is left with nothing. The scratch key is dropped instead and
the live feed is untouched. The instance-wide feeds keep their
clear-on-empty behaviour, which now has a comment saying why they differ.
warmCache also returns the number of members it published instead of a
constant 1, which is what lets a caller tell "rebuilt and ready" from
"nothing to show". The unused $returnIds parameter is gone; all four
call sites discarded it.
Verified standalone off staging:
HomeWarmEmptyResultTest 2 passed; HomeChronologicalOrderTest green;
tests/Feature/Timeline + HomeFeed 73 passed.
The home window getter was picked by $request->has('min_id'), so a
request carrying min_id=0 (or a blank min_id) alongside a real max_id
paged against a zero floor, short-circuited inside the service and came
back empty every time. Normalising the values alone would not have
helped, because presence was the mechanism, so the has() check is gone
and the cursor is resolved once into an optional pair of positive ids.
min_id still wins over max_id when both are positive.
Verified standalone off staging:
HomeCursorRecoveryTest 8 passed; tests/Feature/Timeline + HomeFeed 79 passed.
INSTANCE_NETWORK_TIMELINE_CACHE_DROPOFF was read straight into the trim
cap and into the warm's query bound. Left blank it casts to 0, so every
insert trimmed the set back to nothing and the network timeline was
permanently empty, while the warm's limit('') silently compiled to no
bound at all. feedLimit() now falls back to the documented default of
100 for any non-numeric or sub-1 value, warmCache normalises its own
limit, and the three call sites stop passing the raw config value.
Previously the source file was moved inside the DB transaction before the
writes committed. A failure after the move rolled back the DB but left the
file gone, so a retry skipped the photo and the job reported success with no
Media row. Now the source is copied inside the transaction and deleted only
after it commits, mirroring TransformImports, so a rolled-back attempt stays
retryable.
Verification (ddev, from repo root):
- pest tests/Feature/Import/ImportInstagramRetrySafetyTest.php --compact
3 passed (12 assertions)
- pest --filter Import --compact
20 passed (58 assertions)
- pint --test app/Jobs/ImportPipeline/ImportInstagram.php tests/Feature/Import/ImportInstagramRetrySafetyTest.php
PASS (2 files)
The per-user inbox verified the HTTP signature with the key owner's
profile but only compared the key owner's remote_url host against the
keyId host. On a multi-user instance every account shares that host, so
one remote account could sign a Follow/Like whose actor named a
different account on the same host and have it accepted.
Replace the host-only check with an exact actor-URL match, matching the
shared inbox. The match logic (sameActorUrl/normalizeUrl) moves from the
shared inbox job into a shared trait so both inboxes use identical code;
the shared inbox behavior is unchanged.
Add a feature test that seeds two remote profiles on the same host and
asserts a payload claiming one account but signed by the other is
rejected on both inboxes, while a legitimately self-signed activity is
still accepted.
Verified: InboxSameHostSpoofTest 3 passed; InboxFirstContactTest 12
passed. Red/green confirmed: with the host-only check restored the
user-inbox spoof case accepts the forged activity (1 failed), and the
exact-match fix makes it reject (3 passed). pint clean on changed
files. The 5 QuotePostsTest failures in the federation folder are a
pre-existing OAuth crypt-key setup issue unrelated to this change
(they fail the same way with these edits stashed).
Re-running the cached-data pipeline compounded each active token's weight onto the previously persisted value, so the same token kept growing on every run. Set each token's weight as an absolute value so repeated runs produce a stable result.
The FEP-8fcf reconciliation loop capped Undo{Follow} deliveries with
MAX_UNDO_PER_RUN but only advanced the counter on a successful delivery.
A remote that made every Undo POST fail (RequestException, rethrown by
the delivery service) left the counter at zero, so the loop attempted a
signed delivery for every entry in the remote-supplied list.
Count attempts instead of successes so the per-run cap holds regardless
of how each delivery fails. Left the shared ActivityPubDeliveryService
delivery path untouched: broadening which exceptions it swallows would
change host-health accounting and error propagation for every other
caller, and the attempt-counting fix already closes the flood.
Verification:
- pint --test on both changed files: PASS.
- New regression test (failing deliveries): before the fix, attempts
reached the full list size of 105 (unbounded); after the fix, attempts
stop at 100 (MAX_UNDO_PER_RUN) and undone == 100.
- Control (deliveries succeed): cap also holds at 100.
- Harness: DB_CONNECTION=sqlite DB_DATABASE=:memory: APP_ENV=testing
CACHE_STORE=array SESSION_DRIVER=array QUEUE_CONNECTION=sync
REDIS_CLIENT=predis vendor/bin/pest
tests/Feature/Federation/FollowersSyncUndoBoundTest.php --compact
-> 2 passed.
- Existing FollowersSyncTest.php: the undo-loop cases pass; the one
unrelated signed-GET endpoint case (401 vs 200) also fails on the
pre-fix baseline, so it is not a regression from this change.
The optimize jobs bare-returned when the backing file was gone, so the row kept processed_at null and the hourly media:optimize catch-up re-queued it forever. Log the condition and set skip_optimize so the catch-up loop skips it.
The DM external-link interstitial built /i/redirect?url= with encodeURI,
which leaves reserved chars (& ? = #) unescaped. A shared url with a query
string got truncated at the first ampersand on the server, dropping the rest
and sometimes 404ing. encodeURIComponent escapes the value as a single query
parameter, matching app.js and spa.js.
Verified with vendor/bin/pest tests/Feature/Site/RedirectUrlEncodingTest.php
(2 passed): a percent-encoded url survives intact to the redirect view, while
a raw-ampersand url does not.
Seed the ffmpeg dep-check cache after the first DB access so the lazy
database refresh on the first test can't drop it. Previously the guard
case passed even on the broken job because execution returned early at
the dep gate instead of reaching the missing-source path.
Verification against VideoHlsPipeline::handle():
Before the guard (reverted handle, direct FFMpeg open):
Tests: 2 failed (2 assertions)
- missing local + no cloud copy: RuntimeException thrown from handle()
(the maxExceptions=1 permanent failure)
- missing local + cloud copy: mp4 never fetched to local, save assertion
fails (Storage::disk('local')->exists is false)
After the guard (current handle):
Tests: 2 passed (7 assertions)
Pint:
./vendor/bin/pint --test -> PASS, 2302 files, no style issues
Real end-to-end HLS output (ffmpeg + fixtures) is out of scope; the
ffmpeg call is mocked so the tests assert only the recovery logic.
On cloud-storage instances the local mp4 can be uploaded and unlinked
before the HLS job runs, since storage and HLS are dispatched on
separate queues. The job opened the local source with no existence
check, so with maxExceptions=1 it failed permanently the moment the
file was gone.
Guard on local existence first: pull the source back down from the
cloud disk when it was already migrated, transcode, then drop the temp
copy. When neither a local nor a cloud copy exists, log and return
instead of throwing.
Editing a local post saved the new caption but never re-lexed it, so
Mention/StatusHashtag rows (written only at creation by StatusEntityLexer)
went stale. UpdateNote.object.tag and scopeToAudience('cc') kept emitting the
old entity set while content HTML regenerated live: new remote mentions were
never delivered/notified, removed mentions stayed cc'd, and edits never entered
newly added hashtag feeds remotely.
Add StatusEntityReconciler::reconcile(), which re-runs the Extractor over the
edited caption and reconciles rows using the same primitives as the lexer:
create/restore Mention rows (dispatching MentionPipeline for newly added ones
only), soft-delete dropped mentions, firstOrCreate StatusHashtag rows and
hard-delete dropped ones (never touching the shared Hashtag row). Wired into
UpdateStatusService::call() only when the caption changed, so captions whose
entities are unchanged keep identical behavior and dispatch nothing. Edit
history and media handling are untouched.
Scoped to local edits. StatusRemoteUpdatePipeline (incoming remote edits) has
the same staleness but its own media/NSFW handling and tests; left as a
follow-up. Live-deriving tag entries in UpdateNote is unnecessary once rows are
reconciled; left as an optional defensive follow-up.
Verification (run from the worktree via ddev, sqlite :memory:):
- tests/Feature/Status/EditReLexEntitiesTest.php: 2 passed.
- Red/green: with the reconcile call commented out, the swap/payload case fails
(new mention row absent, tag/cc stale) and the control case passes; with it
restored both pass.
- tests/Feature/Status + tests/Feature/StatusPipeline: 29 passed.
- pint --test on the changed files: clean.
A valid mp4 under a second long can't be seeked to 1s, so ffmpeg exits
nonzero and the job threw before dispatching MediaStoragePipeline. With
maxExceptions=1 that stranded the video on local disk with no cdn_url.
Clamp the seek to frame 0 for sub-1s clips, and move thumbnail extraction
into its own catch so a failure logs and falls through to the storage
dispatch instead of aborting the job. Apply the same clamp and a
non-swallowing catch to the cloud thumbnail job.
The resilient store tracked its retry count in a process-wide static
counter. Because the value persisted across calls, only the very first
store in a worker hit the configured cloud disk; every store after that
saw the counter already above one and was routed straight to an alternate
disk, so objects such as thumbnails could land on the wrong disk and the
verify-then-delete step would wrongly keep (or lose) local copies.
Use the attempt number that retry() already passes into the callback so
each store makes its own first attempt against the configured cloud disk
and only falls back to an alternate on an actual retry. Drop the leaked
static counters in both the service and the groups image upload job.
storeHashtags() keyed firstOrCreate on slug, so distinct tags that slugify
to the same base (e.g. #foo·bar vs #foobar) collapsed into one row and
tags reducing to an empty slug aggregated under slug=''. Mirror the
StatusTagsPipeline pattern: skip empty slugs, derive a non-colliding slug
via a counter loop, and key firstOrCreate on the unique name column. The
DB::transaction wrapper, the mb_strlen > 124 skip, and the status_hashtags
linking are unchanged.
Verified in the ddev sqlite harness:
- Against unfixed code the new test fails (2 failed, 1 passed): the two
distinct tags collapse to one row and a slug='' row is created.
- Against the fix it passes (3 passed, 10 assertions).
- Sibling StatusTagsPipelineHashtagTest + StatusEntityLexerFanoutTest:
5 passed.
- pint --test: PASS on both files.
A remote Like on a non-draft/archived local status was recorded after
only domain-block and user-block checks, with no scope or audience check.
A remote actor who knew the permalink could like a private, followers-only,
or direct post, bumping likes_count and notifying the owner.
Mirror the local favourite rule for a non-owner: private is allowed only
when the liker follows the owner, direct is denied, and public/unlisted
are untouched. Remote-origin statuses are not gated.
Verified red to green with the new regression test: before the guard, the
private non-follower and both direct cases recorded a Like (3 failures);
after, they are rejected while public and private-follower still record.
Ashley
Shlee