53 Commits (45202f79ef45f0d141b1e2911b4fa34950777cad)

Author	SHA1	Message	Date
ThibG	b364175e1b	Fix link crawler not specifying accepted content-type (#12646) ... The link crawler expects HTML documents, so set the `Accept` header accordingly. Fixes #12618	5 years ago
ThibG	c656cc2191	Fix FetchLinkCardServices crashing on a tags without a target (#12159) ... * Add test for links without targets * Fix FetchLinkCardServices crashing on a tags without a target	5 years ago
Eugen Rochko	d14e74eff5	Add cache for OEmbed endpoints to avoid extra HTTP requests (#12403) ... * add youtube oembed endpoint * add check for oembed endpoint * change unless for a more readable if * clear blank lines * endpoint via https * Fix string literal in condition * use cache for endpoints * use cache for endpoints * clean up and adding check * clean up and remove redundant return * add html check * add false to return * use double quotes * use double quotes * Clean up	5 years ago
nightpool	9762fe382c	microformat mentions can have an implicit property (#12189) ... See the first example here: http://microformats.org/wiki/microformats2#hyperlinked_person	5 years ago
Eugen Rochko	9ba40a6bfd	Remove HEAD request from fetching link previews (#12028) ... It is not really necessary and we need to reduce requests	5 years ago
Eugen Rochko	3ec80c7aec	Fix preview card image not being re-fetched even if link is re-posted (#11981) ... Fix #11956	5 years ago
Eugen Rochko	5d3feed191	Refactor fetching of remote resources (#11251)	6 years ago
Eugen Rochko	b851456139	Remove Atom feeds and old URLs in the form of `GET /:username/updates/:id` (#11247)	6 years ago
Daniel Aleksandersen	10bdd912d6	Treat meta[property] as a space-separated list (#10604) ... The @property attribute in HTML is a space-separated list of values. This change normalizes whitespace and finds the desired value in the list instead of requiring an exact single-value match. More details: https://www.ctrl.blog/entry/rdfa-socialmedia-metadata.html	6 years ago
ThibG	e709b8da0d	Ignore low-confidence CharlockHolmes guesses when parsing link cards (#9510) ... * Add failing test for windows-1251 link cards * Ignore low-confidence CharlockHolmes guesses Fixes #9466 * Fix no method error when charlock holmes cannot detect charset	6 years ago
ThibG	9b89c62d43	Check that twitter:player is valid before using it (#9254) ... Fixes #9251	6 years ago
ThibG	a03d506626	Fix Pleroma mentions being fetched as preview cards (#9158)	6 years ago
Eugen Rochko	795f0107d2	Include preview cards in status entity in REST API (#9120) ... * Include preview cards in status entity in REST API * Display preview card in-stream * Improve in-stream display of preview cards	6 years ago
ThibG	7fee968e9f	Do not fetch preview card for mentioned users (#6934)	6 years ago
Eugen Rochko	f4d549d300	Redesign forms, verify link ownership with rel="me" (#8703) ... * Verify link ownership with rel="me" * Add explanation about verification to UI * Perform link verifications * Add click-to-copy widget for verification HTML * Redesign edit profile page * Redesign forms * Improve responsive design of settings pages * Restore landing page sign-up form * Fix typo * Support <link> tags, add spec * Fix links not being verified on first discovery and passive updates	6 years ago
ThibG	bd9e47e9be	Handle relative URLs when fetching OEmbed/OpenGraph cards (#8669)	6 years ago
Yamagishi Kazutoshi	e3fb528d12	Skip processing when HEAD method returns 501 (#7730)	7 years ago
Akihiko Odaki	55fd55714a	Raise Mastodon::RaceConditionError if Redis lock failed (#7511) ... An explicit error allows user agents to know the error and Sidekiq to retry.	7 years ago
Yamagishi Kazutoshi	d2ee48977c	Rescue Mastodon::LengthValidationError in FetchLinkCardService (#7424)	7 years ago
Eugen Rochko	cb5b5cb5f7	Slightly reduce RAM usage (#7301) ... * No need to re-require sidekiq plugins, they are required via Gemfile * Add derailed_benchmarks tool, no need to require TTY gems in Gemfile * Replace ruby-oembed with FetchOEmbedService Reduce startup by 45382 allocated objects * Remove preloaded JSON-LD in favour of caching HTTP responses Reduce boot RAM by about 6 MiB * Fix tests * Fix test suite by stubbing out JSON-LD contexts	7 years ago
Akihiko Odaki	40e5d2303b	Validate HTTP response length while receiving (#6891) ... to_s method of HTTP::Response keeps blocking while it receives the whole content, no matter how it is big. This means it may waste time to receive unacceptably large files. It may also consume memory and disk in the process. This solves the inefficency by checking response length while receiving.	7 years ago
Akihiko Odaki	54b273bf99	Close http connection in perform method of Request class (#6889) ... HTTP connections must be explicitly closed in many cases, and letting perform method close connections makes its callers less redundant and prevent them from forgetting to close connections.	7 years ago
Eugen Rochko	f7765acf9d	Fix #5173: Click card to embed external content (#6471)	7 years ago
abcang	7e5c433dfc	Fix saving of oEmbed image (#6409)	7 years ago
Eugen Rochko	a8deb6648b	Fix redundant HTTP request in FetchLinkCardService (#6002)	7 years ago
Akihiko Odaki	f9f6918148	Store preview image for embedded photo in preview cards (#5986) ... The preview image would be useful to embed in timeline.	7 years ago
Renato "Lond" Cerqueira	19257d91bf	Return false if object does not respond to url (#5988) ... Avoid error when the service returns a mostly valid oembed, but has no url in it, causing a MethodError: undefined method `url' for #<OEmbed::Response::Photo:0x000056505def9620>	7 years ago
Yamagishi Kazutoshi	e7d55df38d	Ignore HEAD method if does not support (#5949)	7 years ago
Akihiko Odaki	c083816c24	Add embed_url to preview cards (#5775)	7 years ago
abcang	3dce6cbbd7	Fixed duplicating URL of photo type of oEmbed (#5763)	7 years ago
unarist	3bc8924940	Don't capture scheme-less URLs in the status (#5435) ... Specifically, this fixes status length calculation to be same as JS side. BTW, since this pattern used in not only preview card fetching, we should extract it (with twitter-regex?) and write tests I think.	7 years ago
unarist	35ec1c91e3	Fix some failure cases on FetchLinkCardService (#5347) ... * If OEmbed response doesn't have a required property `type`, ignore it. e.g. `NoMethodError: undefined method 'type' for ...` * If we failed to detect encoding, fallback to default behavior of Nokogiri. e.g. `KeyError: key not found: :encoding`	7 years ago
unarist	5c8ca024ef	Improve error handling on LinkCrawlWorker (#5250) ... * Improve error handling on LinkCrawlWorker * Ignore TimeoutError and InvalidURIError too * Record errors to debug log * Enable dead job queue on LinkCrawlWorker Since most of acceptable errors were already ignored, only our side issue should go to dead job queue. * Ignore all http gem errors	7 years ago
ふぁぼ原	3816943e6b	Enable to recognize most kinds of characters as URL paths (#4941)	7 years ago
Eugen Rochko	596dab06e9	Support OpenGraph video embeds (#4897) ... * Support OpenGraph video embeds It's not really OpenGraph, it's twitter:player property, but it's not OEmbed so that fits. For example, this allows Twitch clips to be displayed as embeds. Also, fixes glitch-soc/mastodon#135 * Fix invalid OpenGraph cards being saved through attaching and revisit URLs after 14 days	7 years ago
Eugen Rochko	7dc5035031	Make PreviewCard records reuseable between statuses (#4642) ... * Make PreviewCard records reuseable between statuses **Warning!** Migration truncates preview_cards tablec * Allow a wider thumbnail for link preview, display it in horizontal layout (#4648) * Delete preview cards files before truncating * Rename old table instead of truncating it * Add mastodon:maintenance:remove_deprecated_preview_cards * Ignore deprecated_preview_cards in schema definition * Fix null behaviour	7 years ago
Eugen Rochko	1618b68bfa	HTTP signatures (#4146) ... * Add Request class with HTTP signature generator Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06 * Add HTTP signature verification concern * Add test for SignatureVerification concern * Add basic test for Request class * Make PuSH subscribe/unsubscribe requests use new Request class Accidentally fix lease_seconds not being set and sent properly, and change the new minimum subscription duration to 1 day * Make all PuSH workers use new Request class * Make Salmon sender use new Request class * Make FetchLinkService use new Request class * Make FetchAtomService use the new Request class * Make Remotable use the new Request class * Make ResolveRemoteAccountService use the new Request class * Add more tests * Allow +-30 seconds window for signed request to remain valid * Disable time window validation for signed requests, restore 7 days as PuSH subscription duration (which was previous default due to a bug)	8 years ago
nullkal	007ab330e6	Use charlock_holmes instead of nkf at FetchLinkCardService (#4080) ... * Specs for language detection * Use CharlockHolmes instead of NKF * Correct mistakes * Correct style * Set hint_enc instead of falling back and strip_tags * Improve specs * Add dependencies	8 years ago
abcang	5e6acf9601	Fix Nokogiri::HTML at FetchLinkCardService (#4072)	8 years ago
abcang	a6d02cff36	Rescue exceptions caused by FetchLinkCardService (#4045)	8 years ago
ThibG	f8fe394e7a	Fix an error when TagManager.local_url? is called with a bad URI (#3701) ... TagManager.local_url? was sometimes called with an URI with a nil host, leading to a crash in TagManager.local_url?. This fixes moves the already-existing uri.host.blank? check in front to avoid this case.	8 years ago
Yamagishi Kazutoshi	79ef8b3653	Fetch remote image using http.rb (#3114)	8 years ago
Eugen Rochko	1efda1c453	Fix #2572 - Resolve preview cards for remote statuses as well as local ones (#3088)	8 years ago
Eugen Rochko	a231f915a0	Fix #2955 - Send HEAD request ahead of GET when fetching URL previews (#2972)	8 years ago
Yamagishi Kazutoshi	e95983f5df	Decode IDNA in PreviewCard (#2781)	8 years ago
Eugen Rochko	88725d6ce8	OEmbed support for PreviewCard (#2337) ... * OEmbed support for PreviewCard * Improve ProviderDiscovery code failure treatment * Do not crawl links if there is a content warning, since those don't display a link card anyway * Reset db schema * Fresh migrate * Fix rubocop style issues Fix #1681 - return existing access token when applicable instead of creating new * Fix test * Extract http client to helper * Improve oembed controller	8 years ago
Eugen	17c591ffba	Punycode URI normalization (#2370) ... * Fix #2119 - Whenever about to send a HTTP request, normalize the URI * Add test for IDN request in FetchLinkCardService * Perform IDN normalization on domains before they are stored in the DB	8 years ago
Yamagishi Kazutoshi	d670f72830	Show multibyte URI preview card (#2363)	8 years ago
ik-fib	47a3036ea6	Add Mastodon version to user agent (#2253)	8 years ago
ik-fib	23ecc681c4	Add "Mastodon" to user agent (#2073) ... Remove non-const version string Freeze option is unnecessary	8 years ago