Commit Graph

6 Commits (67ae799cb45faaaf374aabb6cb66d5eff9ddba52)

Author SHA1 Message Date
Your Name 00564ac221 perf: parallelise emoji cloud migration with worker processes
The migration was ~1-2s/file due to sequential S3 round-trips (HEAD + PUT +
verify HEAD). Speed it up:

- --workers=N spawns N child processes, each handling a strided slice of the
  files (index % N == shard) for real concurrency on the I/O-bound uploads
- --skip-cloud-check skips the upfront HEAD (always upload, idempotent)
- --skip-verify skips the post-upload size re-check
- --offset for manual chunking

Storage/Flysystem has no batch or async upload API, so process-level
concurrency is the pragmatic lever here.
4 weeks ago
Your Name 9e5fcb9a90 fix: skip missing.png in emoji cloud migration
The frontend renders a hardcoded /storage/emoji/missing.png local onerror
fallback for emoji, so that placeholder must stay on local disk. Skip it in
the migration so it is never moved to cloud or deleted locally.
4 weeks ago
Your Name 79eef56be0 fix: make emoji cloud migration disk-driven + add --debug
The migration was DB-driven (whereNull('uri')), which excluded federated
emoji whose media is stored locally but have a uri set -> the disk was never
scanned, resulting in moved=0.

- Drive the migration by enumerating local files under public/emoji/ instead
  of a DB query; the local file is the source of truth for what needs moving
- Add --debug to print config, custom_emoji table breakdown, local emoji dir
  contents, and per-file decisions
4 weeks ago
Your Name aa9bb868dd fix: emoji admin URLs and cloud-migration guard
Two issues prevented emoji from serving/migrating correctly on cloud:

- Admin custom-emoji views hardcoded url('storage/'.media_path), so they
  always showed local URLs and bypassed cloud resolution. Use $emoji->url().
- The migration/command guard relied solely on config_cache('pixelfed.cloud_storage'),
  which is DB/12h-cached and can read stale-false right after cloud is
  enabled, causing the migration to silently no-op. Treat cloud as enabled
  when either live config() or config_cache() is true.
4 weeks ago
Your Name 979df6e39e feat: migrate all local emoji to cloud in one pass by default
Change --limit default to 0 (no limit) so the emoji migration processes
every local emoji in a single run instead of capping at 1000, and drop the
limit from the scheduled invocation. Avoids a multi-run window where
not-yet-migrated emoji resolve to missing cloud URLs.
4 weeks ago
Your Name fa76e1014a feat: store custom emoji on cloud storage when enabled
Custom emoji were always written locally and served via hardcoded /storage
URLs, so they never used S3 even on cloud instances.

- CustomEmoji: centralize URL + storage on the active disk (cloud when
  pixelfed.cloud_storage is enabled, else local public/ disk) via
  urlForPath/url/storageTarget/storeMedia/storeMediaFromFile/deleteMedia
- Route emoji writes/deletes and URL generation (scan, CustomEmojiService::all)
  through those helpers in ImportEmojis, CustomEmojiService::import and
  AdminController
- Add admin:EmojiMoveStorageLocalToCloud to migrate existing local emoji to
  cloud: copy, verify by size, delete local, bust caches
- Schedule it daily when cloud storage is enabled
4 weeks ago