Commit Graph

200 Commits (938cd2875b14db3655a6c9f82f672f4baf7720a3)

Author SHA1 Message Date
Akihiko Odaki (@fn_aki@pawoo.net) c7af8cbc90 Remove some arguments of Formatter.instance.format and spec (#3541)
* Remove some arguments of Formatter.instance.format

* Improve spec for Formatter
8 years ago
Akihiko Odaki (@fn_aki@pawoo.net) ce7c0def88 Remove redundant code in AtomSerializer and spec (#3528)
* Remove redundant status.reply? in AtomSerializer#object

* Cover AtomSerializer more
8 years ago
Clworld dab8fc4584 Execute PushUpdateWorker only for accounts who uses StreamingAPI just now. (#3278)
* Add redis key "subscribed:timeline:#{account.id}" to indicate active streaming API listeners exists.

* Add endpoint for notification only stream.

* Run PushUpdateWorker only for users uses Streaming API now.

* Move close hander streamTo(Http/Ws) -> stream(Http/Ws)End (Deal with #3370)

* Add stream type for stream start log message.
8 years ago
Matt Jankowski d010e270e6 Remove usernames and hashtags from language detection (#3503)
* Add failing specs for hashtag and username extraction in language detector

* Remove usernames and hashtags from text before language detection

* Handle multiple instances of special case, and reduce whitespace
8 years ago
Matt Jankowski b25e42a77f Misc tidying and clean ups (#3445)
* Remove trailing whitespace in i18n mailers

* Use query methods instead of #present? on AR attributes

* Delegate Status#account_domain method

* Delegate Mention #account_username and #account_acct methods
8 years ago
Matt Jankowski 3576fa0d59 Improve api oembed controller (#3450)
* Add StreamEntryFinder class to parse URLs

* Use StreamEntryFinder and clean up api/oembed controller
8 years ago
Atsushi Yamamoto 402c19a924 Add preference setting for delete toot modal (#3368)
* Set delete_modal preference to true by default
* Does not show confirmation modal if delete_modal is false
* Add ja translation for preference setting page
8 years ago
alpaca-tc 08dce5e607 Refactor Setting (#3302)
* @object is not needed

* Remove unneeded dependencies

* Do not call private method

* Prefer #respond_to_missing? over #respond_to?

`#respond_to?` doesn't support `User.settings.method(:method_name)`

* Use find_or_initialize_by instead of
8 years ago
masarakki 3b59f9c6c2 fix-cache-returns-nil (#3213) 8 years ago
Yamagishi Kazutoshi f937cad68f Skip formatting for cashtag in status text (#3275)
Resolve #3270
8 years ago
abcang d22cec81fb Unify the method of extracting tags (#3138) 8 years ago
Matt Jankowski 6e4c7d6211 Conditional validations no longer accept strings for if/unless (#3124) 8 years ago
Eugen Rochko 620d0d8029 Account domain blocks (#2381)
* Add <ostatus:conversation /> tag to Atom input/output

Only uses ref attribute (not href) because href would be
the alternate link that's always included also.

Creates new conversation for every non-reply status. Carries
over conversation for every reply. Keeps remote URIs verbatim,
generates local URIs on the fly like the rest of them.

* Conversation muting - prevents notifications that reference a conversation
(including replies, favourites, reblogs) from being created. API endpoints
/api/v1/statuses/:id/mute and /api/v1/statuses/:id/unmute

Currently no way to tell when a status/conversation is muted, so the web UI
only has a "disable notifications" button, doesn't work as a toggle

* Display "Dismiss notifications" on all statuses in notifications column, not just own

* Add "muted" as a boolean attribute on statuses JSON

For now always false on contained reblogs, since it's only relevant for
statuses returned from the notifications endpoint, which are not nested

Remove "Disable notifications" from detailed status view, since it's
only relevant in the notifications column

* Up max class length

* Remove pending test for conversation mute

* Add tests, clean up

* Rename to "mute conversation" and "unmute conversation"

* Raise validation error when trying to mute/unmute status without conversation

* Adding account domain blocks that filter notifications and public timelines

* Add tests for domain blocks in notifications, public timelines
Filter reblogs of blocked domains from home

* Add API for listing and creating account domain blocks

* API for creating/deleting domain blocks, tests for Status#ancestors
and Status#descendants, filter domain blocks from them

* Filter domains in streaming API

* Update account_domain_block_spec.rb
8 years ago
beatrix 3dcb5fa28f Revert HTML CW changes (#3020)
* selectively Revert "Fix regressions from #2683 (#2970)"

This reverts commit 72698bc3b4.

* Revert "Handle hashtags in spoiler_texts (partial fix for #699) (#2683)"

This reverts commit e2491680e6.
8 years ago
Eugen Rochko 5abdc77c80 Add conversation model, <ostatus:conversation /> (#3016)
* Add <ostatus:conversation /> tag to Atom input/output

Only uses ref attribute (not href) because href would be
the alternate link that's always included also.

Creates new conversation for every non-reply status. Carries
over conversation for every reply. Keeps remote URIs verbatim,
generates local URIs on the fly like the rest of them.

* Fix conversation migration

* More spec coverage for status before_create

* Prevent n+1 query when generating Atom with the new conversations

* Improve code style

* Remove redundant local variable
8 years ago
Eugen Rochko b5a9c6b3d2 Prepend reblogs' wrapper content with "RT @original_author", (#3013)
so that when a reblog parse fails on another instance, it doesn't
look like a misattributed/stolen text
8 years ago
Eugen Rochko 43d754eb42 Fix not rejecting remote URIs when parsing out local IDs (#3012) 8 years ago
Eugen Rochko 58bcd50f7f Fix #1426 - Trim long usernames in public follower/following lists (#2993)
Fix #2221 - Catch OpenSSL exceptions when loading remote avatars/headers/attachments
Don't strip "rel" attribute from <a> tags when sanitizing (microformats)
8 years ago
Eugen Rochko a2c8da0185 When avatar/header are missing, do not include the missing file into Atom (#2988)
Receiving instances will then use their own missing image

Also, add <content /> to deleted statuses, since there was a reported
problem with the deletes and GNU social
8 years ago
Eugen Rochko 72698bc3b4 Fix regressions from #2683 (#2970)
* Fix regressions from #2683

Properly format spoiler text HTML, while keeping old logic for blankness intact
Process hashtags and mentions in spoiler text
Format spoiler text for Atom
Change "show more" toggle into a button instead of anchor
Fix style regression on dropdowns for detailed statuses

* Fix lint issue

* Convert spoiler text to plaintext in desktop notifications
8 years ago
beatrix 57a3d71c90 fix a regression introduced by e2491680e6 (#2966)
that caused user pages to break when showing toots with CWs
8 years ago
R Tucker e2491680e6 Handle hashtags in spoiler_texts (partial fix for #699) (#2683)
* services: scan spoiler_text for hashtags (#699)

* views: link hashtags from spoiler_texts

This covers linking hashtags from within the spoiler
text on the server-generated pages.

* services: fix string concat going into hashtag RE

Cleaner Ruby syntax, may handle immutable strings better
8 years ago
Akihiko Odaki d5cabfe5c6 Use CLD3 (#2949)
Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.

Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.

CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.

On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.

CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.

The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.
8 years ago
abcang 08e94d1b19 Fixed bug that timeline can not be displayed by InvalidURIError (#2947) 8 years ago
Matt Jankowski 7a7d12d27f Delegate Account#user_locale method and allow nil (#2927) 8 years ago
masarakki 267ed3d74b raise-when-oembed-not-found (#2882) 8 years ago
alpaca-tc 59804abc3d Optimize MuteService and AfterBlockService (#2836) 8 years ago
masarakki d08f1112d5 use Twitter::Extractor for creating links (#2502) 8 years ago
Eugen Rochko 81584779cb More robust PuSH subscription refreshes (#2799)
* Fix #2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron

Fix an issue where / in domain would raise exception in TagManager#normalize_domain

PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through

Fix #2702 - Correct user agent header on outgoing http requests

* Add test for SubscribeService

* Extract #expiring_accounts into method

* Make mastodon:push:refresh no-op

* Queues are now defined in sidekiq.yml

* Queues are now in sidekiq.yml
8 years ago
Matt Jankowski 8c5ad23b24 Language improvements, replace whatlanguage with CLD (#2753)
* add failing en specs

* add cld2 gem

* Replace WhatLanguage with CLD
8 years ago
Yamagishi Kazutoshi 4a5f73c8ae Add target=_blank to user note (#2622)
* Add target=_blank to user note

Open new window when click link from user profile in remote instance.

* fix rubocop
8 years ago
Eugen Rochko 7406404fa3 Hotfix remote status formatting (#2543) 8 years ago
Yamagishi Kazutoshi 831ff60698 Fix broken oEmbed provider (#2537) 8 years ago
Eugen Rochko 88725d6ce8 OEmbed support for PreviewCard (#2337)
* OEmbed support for PreviewCard

* Improve ProviderDiscovery code failure treatment

* Do not crawl links if there is a content warning, since those
don't display a link card anyway

* Reset db schema

* Fresh migrate

* Fix rubocop style issues
Fix #1681 - return existing access token when applicable instead of creating new

* Fix test

* Extract http client to helper

* Improve oembed controller
8 years ago
Yamagishi Kazutoshi 3ea5b948a4 Decodes URL containing IDN (#2436) 8 years ago
Eugen 17c591ffba Punycode URI normalization (#2370)
* Fix #2119 - Whenever about to send a HTTP request, normalize the URI

* Add test for IDN request in FetchLinkCardService

* Perform IDN normalization on domains before they are stored in the DB
8 years ago
178inaba 1244630ab4 Add simple_format to simplified_format (#2198) 8 years ago
ThibG 89dc29affb Allow webfinger controller to reply to user@WEB_DOMAIN in addition to user@LOCAL_DOMAIN (#2351)
This provides a hotfix for outbound salmon requests to other Mastodon instances
as they currently will try to resovle user@WEB_DOMAIN instead of user@LOCAL_DOMAIN
(see #2012 and #20312).

Furthermore, this should ease transition from users switching from
LOCAL_DOMAIN = WEB_DOMAIN to another LOCAL_DOMAIN when WEB_DOMAIN does not change.
8 years ago
Matt Jankowski 629d35e6f5 [WIP] Html lang on statuses (#2297)
* Add html lang attributes around statuses

* Remove urls from language detection
8 years ago
Eugen 9d3be5579a Improve bio compatibility (#2278)
* Fix #1057 (close #1819) - Move HTML-formatted bio from <poco:note /> to <summary type="html" />

* Ensure <poco:note /> is plaintext for remote accounts, also, by stripping out HTML
8 years ago
Mingye Wang 5c9aa2b732 Fix mangling of ##tag matches (#2194) (#2247)
This commit fixes hashtag_html so it correctly handles matches with multiple hash-signs.

Bug located by @over9001, initial fix suggested by @nightpool.
8 years ago
Matt Jankowski 2dda356e3f Clean up settings/preferences controller (#2237)
* Add missing fields group on preferences page

* Clean up settings/preferences controller

* Extract a UserSettingsDecorator
8 years ago
abcang 70891a99a9 Fix html escape characters in the URL (#2138)
* fix character escaping in URL

* add tests

* put a comma after the last item

* add HTML escape test
8 years ago
Matt Jankowski 297c11dba2 Language detection refactor (#2099)
* Extract detect_language to separate class

* Use default locale, not just en

* Add spec to confirm that whatlanguage cant identify empty string

* Allow account locale to override default in language detector

* PostStatusService supplies an account to detect language
8 years ago
Eugen 15ec4ae07b Fix #1972, fix #1870 - Fix special characters in XML, add tests (#1988)
Also improve efficiency of the mastodon:maintenance:add_static_avatars task
8 years ago
Eugen f902a335f9 Fix #1870 - Strip control characters out of strings in AtomSerializer (#1876)
* Fix #1870 - Strip control characters out of strings in AtomSerializer

* Adjust according to comment by @alpaca-tc
8 years ago
Eugen e4af4898de Add language detection (#1772)
* Add language detection via WhatLanguage and (de)serialization of it through Atom

* Fix default language in ProcessFeedService

* Re-add newline before 'react-rails' Gem to fix groupings

Fixes Code Climate issue
8 years ago
Eugen 95bcbaa434 Fix #1852 - Ensure feeds have valid <title> tags (#1875) 8 years ago
ThibG a9529d3b4b Allow running mastodon on a different domain as the one used for identifying users (#1267)
* Allow running mastodon on a different domain as the one used for identifying users

* Alter documentation of WEB_DOMAIN to make clear it shouldn't be used unless the admin knows what they are doing

* Compare to web_domain instead of local_domain when dealing with feeds/API

* Correctly identify mentions to local accounts

Mentions URLs point to the person's web profile, i.e., the user page served on WEB_DOMAIN.
8 years ago
Effy Elden 8321884eef Change usage of gsub to delete, as per Code Climate/Rubocop recommendation (#1753) 8 years ago
Eugen ac54da9394 Fix #1220, fix #1671 - Hook up comment box to the Redux comment value (#1699)
Fix username styling regression introduced in #1063
Fix report screen background regression introduced in #1415
8 years ago
Eugen 043862f411 Fix #1609, fix #1628 - Revert #1397 (#1700)
When transmitting data in a HTML-encoded element like <content type="html" />,
relying on newlines being preserved is not wise, since HTML by itself
does not care for newlines - it cares for <p> and <br>

Additional fix: reset NSFW toggle after sending toot
8 years ago
Matt Jankowski b330d1f000 Organize coverage dirs (#1695)
* Add `Presenters` group to SimpleCov configuration

* Move validators to app/validators, add to simplecov config
8 years ago
Ben Roberts 0254ee9795 significant improvement in microformats markup (#1063)
* significant improvement in microformats markup

This is a huge improvement and I believe will close #965.

Had these microformats reviewed by others in the community to help
ensure they are at least correct, if not complete.

I did not want to change the structure of the page, and so there it does
not fully mark up the entire ancestry chain, or reply chain, only the
direct decendants and direct ancestors are correctly associated, but
this is likely fine as the most important bit is to have access to the
urls for those toots which are now correctly fetchable.

* improve code climate

* trying to pass code climate tests

* code climate

* fix p-summary for content warning posts

* fix error introduced when merging via github
8 years ago
Matt Jankowski c44a700252 Quick best practice cleanup of views/helpers (#1546)
* Remove trailing whitespace

* Use query methods instead of explicit .blank? checks
8 years ago
Matt Jankowski aa90798386 Webfinger resource to extract username from resource string (#1607)
* Add WebfingerResource class to extract usernames

* Use WebfingerResource in xrd#webfinger
8 years ago
Matt Jankowski b57eed4584 Remove order prior to .find_in_batches (#1470)
The `Status` class has a default order on it, so when this query gets built and
gets all the way to `find_in_batches` there is an order already there.

When `find_in_batches` is run it discards any existing order on the query, and
emits a warning to the logs if there is one there.

This change removes the order prior calling `find_in_batches`, which will stop
the logged warning from occurring as well.
8 years ago
Matt Jankowski 0687ab8ae3 Clean up generation of account webfinger string (#1477)
* Consolidate webfinger string creation under Account#to_webfinger_s

* Introduce Account#local_username_and_domain for consolidation
8 years ago
Rachel H f690320fb9 Keep newlines in xml (#1397) 8 years ago
Eugen c172919745 Fix #1339 - better Atom titles (#1343) 8 years ago
Joël Quenneville d4c94fa004 DRY up reblog vs original status check
Checking reblog vs original status was happening in multiple places
across the app. For views, this logic was encapsulated in a helper
method named `proper_status` but in the other layers of the app, the
logic was duplicated.

Because the logic is used at all layers of the app, we extracted it into
a `Status#proper` method on the model and changed all uses of the logic
to use this method. There is now a single source of truth for this
condition.

We added test coverage to untested methods that got refactored.
8 years ago
Eugen Rochko 8a6d8de60a Fix nil#object_type error 8 years ago
Eugen Rochko 624a9a7136 Re-add forgotten <author> element on standalone <entry> 8 years ago
Eugen 1c351709bc Force UTF8 encoding on generated XML (#1140) 8 years ago
Eugen 6d6a429af8 Rewrite Atom generation from stream entries to use Ox instead of Nokogiri (#1124)
* Rewrite Atom generation from stream entries to use Ox instead of Nokogiri::Builder

StreamEntry is now limited to only statuses, which allows some optimization. Removed
extra queries on AccountsController#show. AtomSerializer instead of AtomBuilderHelper
used in AccountsController#show, StreamEntriesController#show, StreamEntryRenderer
and PubSubHubbub::DistributionWorker

PubSubHubbub::DistributionWorker moves n+1 DomainBlock query to PubSubHubbub::DeliveryWorker
instead.

All Salmon slaps that aren't based on StreamEntry still use AtomBuilderHelper and Nokogiri

* All Salmon slaps now use Ox instead of Nokogiri. No touch from status on account
8 years ago
Eugen Rochko 5b95be1c42 Replace calls to FeedManager#inline_render and #broadcast 8 years ago
Kurtis Rainbolt-Greene 7bed4e51db Moved to the worker 8 years ago
Kurtis Rainbolt-Greene dc5704b0b0 This method isn't used anymore 8 years ago
Kurtis Rainbolt-Greene 1e96ce378e By pushing this into a worker we can reduce the amount of time the feed manager using workers eat up a connection 8 years ago
Eugen 117b22e905 Merge pull request #852 from peterkeen/email-whitelist-817
[#817] Add email whitelist
8 years ago
Eugen 2edeb3fe1c Merge pull request #858 from krainboltgreene/patch-6
Use active record shorthand
8 years ago
Eugen Rochko 6fd865c000 Spawn FeedInsertWorker to deliver status into personal feed 8 years ago
Kurtis Rainbolt-Greene 731e650681 Use active record shorthand 8 years ago
Pete Keen e9a6da6bc7 [#817] Add email whitelist
This adds the ability to filter user signup with a whitelist
instead of or in addition to a blacklist.

Fixes #817
8 years ago
Eugen Rochko 82aaedec46 Reduce number of items in feeds, optimize regeneration worker slightly,
make regeneration worker unique, (only schedule/execute once at a time)
8 years ago
Eugen Rochko b1f3499c38 Optimize FeedManager#unmerge, and slightly optimize FeedManager#merge 8 years ago
Eugen Rochko ce9df2fa82 Optimize filter methods in FeedManager a bit, use redis pipelining on merge/unmerge feed methods,
do not re-create a dynamic class on each feed push call, make sure redis-rb uses hiredis
8 years ago
Eugen Rochko 8232f76c48 Add check for visibility.nil? even though it can't ever be, to check for race conditions 8 years ago
Eugen Rochko d6b965cf08 Fix issue with feed merge-in code as well 8 years ago
Eugen Rochko 139fc994e2 Fix #408 - link @ names in bios 8 years ago
Eugen Rochko d6ed2eb512 Prettier account and stream entry URLs 8 years ago
Eugen Rochko 74ae158c2f Add "direct" visibility level in the backend. Web UI is not yet
adjusted to allow choosing it, yet
8 years ago
Eugen Rochko 6b81d10030 Add digest e-mails 8 years ago
Eugen Rochko c64a1c25c4 Fix #231 - Muting 8 years ago
Kit Redgrave 442fdbfc53 Mute button progress so far. WIP, doesn't entirely work correctly. 8 years ago
Eugen Rochko 7f0d1b8cc0 If a status is within 40 statuses from the top of a home feed, do not
reinsert it when someone boosts it
8 years ago
Eugen Rochko 5f511324b6 Add validation of media attachments, clean up mastodon-own exception classes 8 years ago
Eugen Rochko 9d4cad6307 Improve glow 8 years ago
Eugen Rochko 3202bdd744 Fix #555 - Use a better URL parser 8 years ago
Eugen Rochko dc851c922e Mentions in private statuses allow mentioned people to see them 8 years ago
Eugen Rochko 149887a0ff Make follow requests federate 8 years ago
Eugen Rochko 6331ed16e5 Fix #614 - extra reply-boolean on statuses to account for cases when replied-to
status is not in the system at time of distribution; fix #607 - reset privacy
settings to defaults when cancelling replies
8 years ago
Eugen Rochko 6fa4e01139 Fix showing ellipsis even when link hasn't been cut off 8 years ago
Eugen Rochko fb6aa7ad5c Add tracking of delay to streaming API 8 years ago
Eugen Rochko d9ca46b464 Cleaning up format of broadcast real-time messages, removing
redis-backed "mentions" timeline as redundant (given notifications)
8 years ago
Eugen Rochko 450ad43180 Do not run FetchLinkCardService on local URLs, increase file size limit to 8MB,
fix ProcessFeedService pushing status into distribution if called a second time
while the first is still running (i.e. when a PuSH comes after a Salmon slap),
fix not running escape on spoiler text before emojify
8 years ago
Eugen Rochko 999cde94a6 Instead of using spoiler boolean and spoiler_text, simply check for non-blank spoiler_text
Federate spoiler_text using warning attribute on <content /> instead of a <category term="spoiler" />
Clean up schema file from accidental development migrations
8 years ago
Eugen f8da0dd490 Merge branch 'master' into master 8 years ago
Eugen Rochko 80cefd5b3c Fix #204, fix #515 - URL truncating is now a style so copypasting is not
affected, replaced onClick handler with onMouseUp/Down to detect text
selection not trigger onClick handler then
8 years ago
blackle bf0f6eb62d Implement a click-to-view spoiler system 8 years ago