Re-apply the story cloud-migration work on a clean staging base (it was
reverted earlier to land and verify the emoji work first).
- StoryExpire: archive expiring story media on the same explicit disk the
media lives on (S3 move = server-side copy+delete), with error handling
- admin:StoryMoveStorageLocalToCloud: migrate local story media (active +
story_archives) to cloud — copy, verify by size, delete local; --orphans
relocates untracked story_archives files
- Schedule it hourly alongside the media migration when cloud storage is on
Back out all emoji cloud-storage work from staging so it can be reworked and
re-landed separately (the URL resolution flips to cloud on a global config
flag, which created a broken-URL window, and the migration approach needs
revisiting).
Reverts to pre-emoji state:
- CustomEmoji model URL/storage helpers (urlForPath, storageTarget, storeMedia,
storeMediaFromFile, deleteMedia, url) and callers in ImportEmojis,
CustomEmojiService, AdminController
- admin custom-emoji blade views back to local /storage URLs
- Remove admin:EmojiMoveStorageLocalToCloud command
- Remove the deploy migration and its scheduler entry
Media (and the already-reverted story) scheduler entries are untouched.
Process-level workers plateaued at ~7.5 uploads/sec against Fastly Object
Storage because each PUT is high-latency and only a handful ran concurrently.
Add --concurrency=N which uses the AWS SDK CommandPool to keep N PutObject
requests in flight from a single process. A successful PutObject response is
the confirmation (no separate HEAD verify), and the local file is deleted on
success. Commands are yielded lazily so memory stays flat over large runs.
--no-acl escape hatch for S3-compatible stores that reject the ACL header.
- Live 'up/s' rate shown on the progress bar during single-worker runs
- Final summary reports elapsed time and uploads/sec
- Parallel runs tally moved across workers and report aggregate uploads/sec
Makes it easy to compare --workers counts and decide whether async S3
(option 2) is worth pursuing.
The migration was ~1-2s/file due to sequential S3 round-trips (HEAD + PUT +
verify HEAD). Speed it up:
- --workers=N spawns N child processes, each handling a strided slice of the
files (index % N == shard) for real concurrency on the I/O-bound uploads
- --skip-cloud-check skips the upfront HEAD (always upload, idempotent)
- --skip-verify skips the post-upload size re-check
- --offset for manual chunking
Storage/Flysystem has no batch or async upload API, so process-level
concurrency is the pragmatic lever here.
The frontend renders a hardcoded /storage/emoji/missing.png local onerror
fallback for emoji, so that placeholder must stay on local disk. Skip it in
the migration so it is never moved to cloud or deleted locally.
The migration was DB-driven (whereNull('uri')), which excluded federated
emoji whose media is stored locally but have a uri set -> the disk was never
scanned, resulting in moved=0.
- Drive the migration by enumerating local files under public/emoji/ instead
of a DB query; the local file is the source of truth for what needs moving
- Add --debug to print config, custom_emoji table breakdown, local emoji dir
contents, and per-file decisions
Back out the story local->cloud migration so we can land and verify the
emoji cloud work first, one change at a time.
Reverts:
- 9f110bb74 feat: migrate local story media to cloud storage
- 842681b99 fix: schedule StoryMoveStorageLocalToCloud command
Removes StoryMoveStorageLocalToCloud command, its scheduler entry, and the
StoryExpire explicit-disk changes. Remcache and emoji work are untouched.
Will revisit story once emoji is confirmed.
Two issues prevented emoji from serving/migrating correctly on cloud:
- Admin custom-emoji views hardcoded url('storage/'.media_path), so they
always showed local URLs and bypassed cloud resolution. Use $emoji->url().
- The migration/command guard relied solely on config_cache('pixelfed.cloud_storage'),
which is DB/12h-cached and can read stale-false right after cloud is
enabled, causing the migration to silently no-op. Treat cloud as enabled
when either live config() or config_cache() is true.
Change --limit default to 0 (no limit) so the emoji migration processes
every local emoji in a single run instead of capping at 1000, and drop the
limit from the scheduled invocation. Avoids a multi-run window where
not-yet-migrated emoji resolve to missing cloud URLs.
Custom emoji were always written locally and served via hardcoded /storage
URLs, so they never used S3 even on cloud instances.
- CustomEmoji: centralize URL + storage on the active disk (cloud when
pixelfed.cloud_storage is enabled, else local public/ disk) via
urlForPath/url/storageTarget/storeMedia/storeMediaFromFile/deleteMedia
- Route emoji writes/deletes and URL generation (scan, CustomEmojiService::all)
through those helpers in ImportEmojis, CustomEmojiService::import and
AdminController
- Add admin:EmojiMoveStorageLocalToCloud to migrate existing local emoji to
cloud: copy, verify by size, delete local, bust caches
- Schedule it daily when cloud storage is enabled
Ensure story media lands on and stays on cloud storage for S3 instances.
- StoryExpire: archive expiring story media on the same explicit disk the
media lives on (S3 move is a server-side copy+delete), with error handling
- Add admin:StoryMoveStorageLocalToCloud to migrate local story media
(active + story_archives) to cloud: copy, verify by size, then delete local
- --orphans option relocates untracked story_archives/ files to cloud using
the same copy/verify/delete flow (media is moved, never discarded)
- Schedule it hourly alongside the media migration when cloud storage is on
The remote avatar/media fetchers wrote temp files to storage/app/remcache/
and only unlinked them on the happy path. Any exception between the write
and the unlink (e.g. a cloud upload failure) leaked the file, and nothing
swept the directory.
- Wrap post-write logic in fetchAvatar() and remoteToCloud() in try/finally
so the temp file is always removed, even on failure
- Add gc:remcache command to delete stale remcache files (default >24h old,
preserves .gitignore, supports --hours and --dry-run)
- Schedule gc:remcache daily to clean up any stragglers
StoryFetch already handled cleanup via try/catch and was left unchanged.
- Add return type hints (void) and final class markers
- Guard null returns from newestBackup() and putFileAs() in BackupToCloud
- Type ask() default values as strings
- Fix uses_left fallback condition for null/zero max_uses
- Annotate AdminInvite::whereInviteCode and cast Str::uuid() to string
- Ignore local redis-data and mysql-9-data dev directories
The resynced summary printed all three counts unconditionally, which
made an untouched metric (e.g. an already-correct comments count) look
like it had been resynced. Drive the summary from the drifted set and
show before->after values, so it matches the drift detection exactly.
reply_count is a nullable column, so NULL rendered as an empty string in
the resync summary. Cast the summary output to int so a null/absent
comment count prints as 0. No behavior change to the reconcile logic.
FixProfileCounts and FixPostCounts use the admin: signature prefix and
are operator-run maintenance tools, so move them from FixBugs/ to Admin/
(namespace updated) and refresh the README tables to match.
Add a FixPostCounts command mirroring admin:fixProfileCounts (single-id,
--all --scope, --active, --type, --dry-run, --force). It reconciles the
statuses likes_count, reblogs_count, and reply_count columns against
source-of-truth tables.
Add canonical recompute helpers and reconcileStatusCounts() to
StatusService (mirroring AccountStatService), busting the status cache
only when a column actually drifted.
status:dedup and fix:avatars address historical data states that can no
longer occur (unique statuses.uri index since 2019; SVG identicon avatars
no longer generated). Move both to a Deprecated/ folder and update the
README audit accordingly. media:fix stays in FixBugs/ since image filters
are still an active feature.
Group console commands into Admin, Dev, FixBugs, Install, Internal, and
User subfolders (matching the earlier reorganization), and add a new
Status subfolder for the status:user, status:profile, and status:post
debug commands. Namespaces updated to match; command signatures and the
total command count are unchanged.
Rename user:status, profile:status, and post:status console commands
to status:user, status:profile, and status:post. Rename the command
files and classes to match (StatusUser, StatusProfile, StatusPost) and
update the cross-reference tip in StatusProfile.
Cold-migrate existing media from an old S3 bucket to the current cloud
bucket, one media row at a time (like MigrateLocalS3MediaURL):
- Source = --sourceDisk (default s3-old, reads AWS_OLD_*); destination = the
current cloud disk (config filesystems.cloud). No .env editing: operators
point AWS_* at the new bucket first (restarting workers as usual) so new
uploads/downloads land on the new bucket, then run this to backfill old data.
- Copies media (+thumbnail) source->destination, verifies by size and by
sha256 of the freshly-written destination object (against original_sha256),
rewrites cdn_url/optimized_url/thumbnail_url to the destination host, and
GCs the source objects (unless --keep-source). Busts caches.
- Only touches rows whose cdn_url still points at the source host; idempotent.
- --sourceDisk / --limit / --dry-run / --force.
- Adds the s3-old disk (AWS_OLD_*) to config/filesystems.php and feature tests.
Add admin:MediaMoveStorageLocalToCloud and admin:MediaMoveStorageCloudToLocal:
- Copy media (+thumbnail) between local and cloud disks, verify by size (and
sha256 against original_sha256 when present) before deleting the source.
- Integrated GC: delete the verified source copy (local on upload, cloud on
download), set version=4 / reset to 3, and bust MediaService/StatusService
caches. --keep-local / --keep-cloud opt out.
- Manage PF_ENABLE_CLOUD in .env AND the live runtime + config cache so new
uploads route to the correct backend mid-migration on a hot server. Uses the
installer's atomic .env writer (shared ManagesMediaStorageEnv trait).
- --limit / --dry-run / --force.
Replaces media:migrate2cloud (CloudMediaMigrate) and media:s3gc
(MediaS3GarbageCollector); scheduler now runs MediaMoveStorageLocalToCloud
hourly for straggler upload + GC. Keeps media:fix-nonlocal-driver.
Adds feature tests (download+GC, --keep-cloud, dry-run, env-flag flip both
directions, unknown-disk guard).
Rename the command (and test) to admin:MigrateLocalS3MediaURL to reflect its
scope: rewriting stale S3/cloud media URLs only. Remove avatar handling and
the --avatars option; the command now focuses solely on status media
(cdn_url, thumbnail_url, optimized_url).
Rebuilds stale local media URLs (cdn_url, thumbnail_url, optimized_url) and
avatar cdn_urls from their storage paths via the configured cloud disk.
- Default target host comes from the configured cloud disk (AWS_URL);
requires confirmation (or --force) and can be overridden with --newDomain.
- Optional --oldDomain filters to a single old backend host; by default all
stale hosts are rewritten.
- Refuses to run when PF_ENABLE_CLOUD is false (local storage) and, when
auto-detecting, refuses a target equal to the app domain — so local-storage
instances are never rewritten.
- Single status id / post URL, --all, --avatars; --dry-run; busts
MediaService/StatusService caches for affected statuses.
- Removes the superseded media:cloud-url-rewrite command.
- Adds feature tests covering rewrite/skip/dry-run/oldDomain/newDomain/
remote-skip/local-storage-refusal.
Dumps a Status and its media for debugging. Accepts a post id or URL
(/p/username/ID). Shows status columns, author, every media row's storage
fields (media_path, thumbnail_path, cdn_url, thumbnail_url, optimized_url,
remote_url, etc.), computed url()/thumbnailUrl()/expected-from-path, a URL
health check comparing stored URL hosts against the configured cloud disk
host (flags stale hosts), and the cached MediaService media_attachments
actually served to clients.
Bulk --all reconciliation previously scanned both local and remote profiles
implicitly. Now --all requires an explicit --scope of local, remote, or
both. --active stays local-only and rejects a non-local --scope. Adds the
BelongsTo return type to Profile::user() so the whereHas('user') scope
filter passes Larastan, and adds tests for scope requirement/validation and
local/remote filtering.
- Rename command signature fix:profilecounts -> admin:fixProfileCounts.
- --active is now its own bulk mode (recently-active local accounts),
mutually exclusive with --all and a single id.
- Add --type=followers|following|statuses to restrict reconciliation to a
single metric (validated).
- Update/extend tests for the new name, --type restriction and invalid-type
rejection.
- Add --force flag to fix:profilecounts for unattended runs and schedule
'fix:profilecounts --all --force' weekly (Sun 03:37) as a safety-net
reconcile. Kept as a low-frequency full scan rather than a new event-driven
dirty-set; it only writes profiles that actually drifted.
- Add Feature tests for AccountStatService recompute helpers and
reconcileProfileCounts (media-type status_count semantics, follower/
following counts, drift/no-drift/no-write, metric restriction, missing
profile) plus fix:profilecounts command behavior (silent-when-synced,
dry-run makes no changes).
Extract canonical source-of-truth count logic into AccountStatService:
recalculateStatusCount/FollowerCount/FollowingCount and a
reconcileProfileCounts() that fixes only drifted columns and busts caches.
Both the scheduled app:account-post-count-stat-update (status-only, its
correct scope) and fix:profilecounts now use these instead of duplicating
the SQL. Also corrects the status_count definition to match the actual
increment logic in StatusEntityLexer/StatusDelete (media post types only:
photo/video albums), rather than the previous inconsistent all-statuses /
scoped counts that could themselves cause drift.
The scheduled updater keeps its incremental, dirty-set design and remains
status-only; follower/following stay owned by FollowServiceWarmCache.
Consolidate cached-count reconciliation into a single fix:profilecounts
command that resyncs followers_count, following_count and status_count
from the source-of-truth tables for one profile or --all. Only reports
profiles with actual drift (silent when in sync); supports --dry-run and
--dispatch (queues FollowServiceWarmCache and rebuilds Redis sets).
Removes the superseded manual commands fix:followercount, fix:statuscount
and fix:rpc. Keeps app:account-post-count-stat-update, which is scheduled
(runs every 6 hours) and queue-driven.
profiles.followers_count/following_count are cached columns reconciled
lazily by FollowServiceWarmCache (throttled up to 7 days), so they can
drift from the followers table. This command recomputes them from the
source-of-truth table for a single profile or --all drifted local
profiles, with --dry-run to report and --dispatch to queue the warm-cache
job (which also rebuilds the Redis sets). Mirrors the existing
fix:statuscount convention.
Unlike user:status (local users only, keyed on the users table),
profile:status keys on the profiles table so it works for remote/
federated actors too. Resolves id, username, user@domain, @user@domain,
webfinger, or remote_url. Shows full column dump, derived/federation
metadata, linked local user (local) or Instance row (remote), and
health checks for orphans, missing keys, and count desync.
Dump every profiles column dynamically (keys redacted, long text trimmed),
add derived metadata (local/remote type, urls, live vs cached follower/
following/status counts, avatar, federation fields), and profile health
checks (soft-delete, id mismatches, missing crypto keys, count desync).
Your Name
Shlee