original_sha256 is the pre-optimization upload hash and can never match the
optimized local file, so comparing against it is not a valid integrity
check for migration. Drop the --verify-sha256 option and the checksum step
entirely; verify now relies on cloud-object existence and size parity.
The cloud migration failure (stale original_sha256 verify) is resolved, so
the temporary default-on debug is no longer needed. Restore --debug to a
bare, off-by-default flag and remove the debugEnabled() string-parsing
helper.
original_sha256 is the hash of the file as originally uploaded, but the
async optimize pipeline (ImageResize/ImageUpdate) rewrites the local file
in place and never updates that column. Verifying the current local bytes
against it made every optimized image fail with sha256_mismatch, so the
migration reported moved=0 and exited 1.
Add a --verify-sha256 flag (off by default) that gates the checksum step.
By default verify relies on cloud-object existence and size parity, which
is the only signal that actually describes the uploaded copy. Add tests
covering both the default (migrates) and opt-in (fails) paths.
The scheduled command only surfaced 'exit code 1' with no cause. Add
structured Log::error entries at every failure path (cloud disk
unresolvable/unconfigured, cloud storage disabled, invalid --before-id,
candidate fetch failure, per-media failure with full context, and a
run-level summary when failures occur).
Also make --debug default to true (now a valued option) so production
runs emit verbose routing detail while this is investigated; pass
--debug=false to silence.
Aligns with the app's dominant convention (171 static Str:: calls vs
24 Str::of() chains). Uses Str::afterLast() for the repeated
"segment after last slash" pattern, Str::matchAll() where a
Collection return is needed, and native explode()/substr() where a
plain array/string suffices.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
users.storage_used only ever grew: uploads incremented it but no deletion
path decremented it, so users hit the account size limit even when their
real media usage was well below it.
- Decrement storage_used in MediaDeletePipeline when media is removed
- Add UserStorageService::increaseStorageUsed / decrementStorageUsed as the
fast, symmetric hot-path counter updates (floor-based, clamped at zero)
- Refactor the 6 upload call sites to use increaseStorageUsed instead of
duplicated inline writes (also fixes ceil/floor drift vs the reconciler)
- Add (user_id, size) covering index so per-user SUM(size) is not a full
table scan (INPLACE/LOCK=NONE, skipped on sqlite)
- Add user:storage:recalculate command to repair affected accounts, with a
daily --stale=168 scheduled reconciler to correct any drift
- Add regression tests for the pipeline and UserStorageService
- Relocate translation files from resources/lang to lang/ via git mv
- Update PHP references to use the lang_path() helper
- Update crowdin.yml source/translation paths
- Update phpstan.neon translationDirectories
The media storage migration commands read/parsed the .env file directly to
check and flip PF_ENABLE_CLOUD. In containerized deploys there is no .env on
disk (config is injected via env vars), so updateEnvFile() threw
'file_get_contents(.env): Failed to open stream' and the scheduled command
exited 1.
- Check the live setting via config_cache('pixelfed.cloud_storage') like the
rest of the app, instead of parsing .env.
- Make the .env write best-effort in ManagesMediaStorageEnv: skip gracefully
when the file is missing or read-only, and still apply the runtime + DB
config-cache updates (the load-bearing changes on a hot server).
- Apply the same fix to the sibling unstable:MediaMoveStorageCloudToLocal.
- Add a regression test covering the no-.env container scenario.
Diagnostic commands mirroring status:statuses/status:media:
- status:instance {id|domain|url|@user@domain}: instance row, moderation
state (banned/unlisted/auto_cw), sync timestamps, local profile count.
- status:avatar {avatar_id|profile_id} [--check]: avatar row, storage state
(local/cloud existence), owning profile, optional live HEAD on remote_url.
- status:emoji {id|:shortcode:|filename} [--check]: emoji row, origin
(local/remote), local file existence, optional live HEAD on image_remote_url.
Also includes the status:post -> status:statuses rename.
Rename the command signature (media:maintenance -> media:filtercleanup), class
(MediaMaintenance -> MediaFilterCleanup), and file to match. Behavior
unchanged.
Orphaned media never references a live status, so --status only accepts soft
and hard. --profile still accepts live/soft/hard. Removes the now-redundant
live short-circuit and makes valid values per-option.
--status live|soft|hard and --profile live|soft|hard narrow orphaned media by
the lifecycle state of the referenced status/profile row. Filters are applied
at the SQL level (whereExists/whereNotExists on deleted_at) so they compose
correctly with --limit. --status=live short-circuits since orphaned media never
has a live status. Options are validated up front.
- TODO.md: capture follow-ups (centralized status media teardown, DM leak fix,
no-DB-cascade rationale, remote-edit orphaning, media:gc re-check).
- media:maintenance: add --server remote|local|both (default both) to filter
orphaned media by origin.
- Annotate each row with status state (live/soft-deleted/hard-deleted) next to
status_id and profile state (live/soft-deleted/hard-deleted) next to
profile_id, in both dry-run table and verbose run output. States are resolved
in batched, trashed-aware queries.
With -v, print per-row detail (media_id, original status_id, remote_media,
profile_id, mime, size, path) as each orphaned row is processed instead of the
progress bar, and expand the dry-run table with extra columns. Uses Laravel's
built-in verbosity flag.
media:maintenance --scope orphanedMedia cleans up media whose status_id
references a status that no longer exists (hard-deleted) or is soft-deleted.
These dangling references predate the delete-path fix and MediaDeletePipeline's
attached guard would otherwise refuse to delete them, leaking files.
Detaches (status_id = null) before dispatching deletion via MediaStorageService,
so the guard sees a genuinely orphaned row. Supports --limit (batched),
--dry-run, and --force. The --scope map is extensible for future routines.
MediaDeletePipeline skips deletion when media->status_id is set. status_id has
no FK/cascade, so deleting a status never clears it, and the delete jobs
dispatched by the status-delete paths were always skipped, leaking media files.
Detach media (status_id = null) before dispatching the delete in StatusDelete,
RemoteStatusDelete and DeleteRemoteStatusPipeline, so the row is genuinely
orphaned by the time the guard checks it and the deletion proceeds.
Also adds a status:media diagnostic command that dumps all metadata for a
media id (DB columns, computed URLs, attachment state, parent status including
the dangling status_id case, owner, metadata, and an optional live URL check).
Adds CustomEmojiService::resync() which re-fetches a remote custom emoji's
media from its origin (image_remote_url) via the SSRF-hardened
SecureMediaFetchService and stores it locally under public/{media_path},
reusing the existing headCheck validation and cache busting.
The admin:resyncemoji command takes a comma-separated list of emoji
filenames, looks each up by media_path, and resyncs remote ones. Supports
--missingonly, --dry-run and --force.
- Drop --except (--only already covers task selection)
- Quiet by default; per-root/per-item and summary lines now require -v/--verbose
- Errors are always shown regardless of verbosity
- Document the command in the console README
Replace the remcache GC (GarbageCollectorRemcache / gc:remcache) with a
broader storage:maintenance command that sweeps stale remcache temp files and
recursively removes the random empty directories accumulated under the media,
story, avatar and import trees (--hours/--only/--except/--dry-run), scheduled
daily.
Fix the root causes so flows clean up after themselves rather than relying on
the sweep:
- MediaDeletePipeline removes its own emptied m/_v2 leaf dir
- AvatarOptimize logs the previously-swallowed exception, still cleans up the
old avatar on failure, and removes the old file's now-empty splayed dir
- AvatarController::deleteAvatar removes the emptied splayed dir
- StoryExpire/StoryDelete remove the story's own emptied leaf dir
- TransformImports removes imports/{userId} once its files are moved out
- StoryFetch cleans up its remcache temp file in a finally block
MediaMoveStorageLocalToCloud is stable, so move it back out of the Unstable
namespace: restore App\Console\Commands\Admin\MediaMoveStorageLocalToCloud and
its admin:MediaMoveStorageLocalToCloud signature, and update the scheduler,
README, and feature test. CloudToLocal and CloudToCloud remain under unstable:.
- Rename App\Console\Commands\Admin\MigrateLocalS3MediaURL to MediaUpdateS3CDNUrl
(class + filename only; the admin:MigrateLocalS3MediaURL signature is unchanged)
- Move the three MediaMoveStorage{LocalToCloud,CloudToLocal,CloudToCloud} commands
into App\Console\Commands\Admin\Unstable and change their signatures from
admin: to unstable:
- Update the scheduler in bootstrap/app.php, the command README, the
config/filesystems.php reference comment, and the affected feature tests
Back out all emoji cloud-storage work from staging so it can be reworked and
re-landed separately (the URL resolution flips to cloud on a global config
flag, which created a broken-URL window, and the migration approach needs
revisiting).
Reverts to pre-emoji state:
- CustomEmoji model URL/storage helpers (urlForPath, storageTarget, storeMedia,
storeMediaFromFile, deleteMedia, url) and callers in ImportEmojis,
CustomEmojiService, AdminController
- admin custom-emoji blade views back to local /storage URLs
- Remove admin:EmojiMoveStorageLocalToCloud command
- Remove the deploy migration and its scheduler entry
Media (and the already-reverted story) scheduler entries are untouched.
Process-level workers plateaued at ~7.5 uploads/sec against Fastly Object
Storage because each PUT is high-latency and only a handful ran concurrently.
Add --concurrency=N which uses the AWS SDK CommandPool to keep N PutObject
requests in flight from a single process. A successful PutObject response is
the confirmation (no separate HEAD verify), and the local file is deleted on
success. Commands are yielded lazily so memory stays flat over large runs.
--no-acl escape hatch for S3-compatible stores that reject the ACL header.
- Live 'up/s' rate shown on the progress bar during single-worker runs
- Final summary reports elapsed time and uploads/sec
- Parallel runs tally moved across workers and report aggregate uploads/sec
Makes it easy to compare --workers counts and decide whether async S3
(option 2) is worth pursuing.
The migration was ~1-2s/file due to sequential S3 round-trips (HEAD + PUT +
verify HEAD). Speed it up:
- --workers=N spawns N child processes, each handling a strided slice of the
files (index % N == shard) for real concurrency on the I/O-bound uploads
- --skip-cloud-check skips the upfront HEAD (always upload, idempotent)
- --skip-verify skips the post-upload size re-check
- --offset for manual chunking
Storage/Flysystem has no batch or async upload API, so process-level
concurrency is the pragmatic lever here.
The frontend renders a hardcoded /storage/emoji/missing.png local onerror
fallback for emoji, so that placeholder must stay on local disk. Skip it in
the migration so it is never moved to cloud or deleted locally.
The migration was DB-driven (whereNull('uri')), which excluded federated
emoji whose media is stored locally but have a uri set -> the disk was never
scanned, resulting in moved=0.
- Drive the migration by enumerating local files under public/emoji/ instead
of a DB query; the local file is the source of truth for what needs moving
- Add --debug to print config, custom_emoji table breakdown, local emoji dir
contents, and per-file decisions
Back out the story local->cloud migration so we can land and verify the
emoji cloud work first, one change at a time.
Reverts:
- 9f110bb74 feat: migrate local story media to cloud storage
- 842681b99 fix: schedule StoryMoveStorageLocalToCloud command
Removes StoryMoveStorageLocalToCloud command, its scheduler entry, and the
StoryExpire explicit-disk changes. Remcache and emoji work are untouched.
Will revisit story once emoji is confirmed.
Daniel Supernault
Your Name
Shlee