Object Storage
Serving user-uploaded files in Mastodon using external object storage
User-uploaded files can be stored on the main server’s file system, or using an external object storage server.
By default, Mastodon will store user uploaded and federated media files on the server’s file system, under public/system
in its installation directory and the files are served at https://example.com/system
.
Configuration Options
Backend Variables
These variables specify how Mastodon connects to your backend S3 storage provider. While AWS is mentioned as the default, Mastodon can work with various providers like AWS S3, DigitalOcean Spaces, Cloudflare R2, Wasabi, MinIO, Exoscale, Scaleway, OVH, or any other S3-compatible provider.
Consult your provider’s documentation for help in setting up these options correctly.
S3_ENABLED
Must be set to true
to enable S3 storage.
Default: false
S3_BUCKET
The name of the S3 bucket at your provider.
Default: None
S3_REGION
The S3 region where your bucket was created.
Used to help construct S3_ENDPOINT
when using AWS, but not required by other providers.
Default: us-east-1
S3_ENDPOINT
The specific S3 target where Mastodon connects to perform API operations.
Used in conjuction with S3_REGION
when using AWS, but should be specifically set when using other providers.
Default: s3.<S3_REGION>.amazonaws.com
AWS_ACCESS_KEY_ID
Effectively this is the API username for the S3 provider. This is created/assigned to you by your S3 provider. Despite the name it is not AWS specific.
Default: None
AWS_SECRET_ACCESS_KEY
Effectively this is the API password for the S3 provider. This is created/assigned to you by your S3 provider. Despite the name it is not AWS specific.
Default: None
Client Access Variables
Once S3 file storage is enabled, Mastodon will provide new URLs for all media ‘read’ operations. These URLs can be accessed using plain HTTP GET methods, without requiring authentication. This means that they can be routed and/or cached through reverse proxies and CDNs.
Access-Control-Allow-Origin: *
, to ensure media visibility in the user’s browser and proper functioning of Mastodon’s web UI.
It is highly recommended to use a domain (or subdomain) that you control for delivering S3 stored media. This provides flexibility in case you decide to change S3 providers in the future. By properly configuring the URLs, you can hide the usage of the storage provider and use caching to reduce egress bandwidth costs. It also ensures that the address for your file storage, which may have already federated to other servers for older posts, remains accessible even if you need to change the storage provider’s address.
Some S3 providers, such as DigitalOcean Spaces, provide integrated CDN/caching services as part of the S3 service. For others, you will need to configure this manually or partner with another provider.
Proxying object storage through nginx
S3_ALIAS_HOST
Instead of using an address like https://s3-us-east-1.amazonaws.com/example-mastodon-bucket/image.jpg
, you can configure it to be delivered from something like https://files.example.com/image.jpg
.
In this example, S3_ALIAS_HOST
would be set to files.example.com
and constructed as shown:
- If
S3_ALIAS_HOST
is not set, then the media access URL will be<S3_PROTOCOL>://<S3_HOSTNAME>/<S3_BUCKET>/<object path>
- If
S3_ALIAS_HOST
is set, then the media access URL will be<S3_PROTOCOL>://<S3_ALIAS_HOST>/<object path>
Default: None
S3_PROTOCOL
Generally should not be changed from the default of HTTPS.
Default: https
S3_HOSTNAME
Required if not using AWS S3 and S3_ALIAS_HOST
is not set.
Default: s3-<S3_REGION>.amazonaws.com
Additional Variables
Due to the large number of S3 provider options, but inconsistencies in how they implement the S3 API, there may be some tuning required specific to your implemention.
S3_SIGNATURE_VERSION
The signature version used to authenticate and authorize requests to the S3 provider.
Default: v4
S3_OVERRIDE_PATH_STYLE
Set this to true
if the storage provider requires API operations to be sent to <S3_BUCKET>.<S3_ENDPOINT>
(domain-style).
Only used if S3_ENDPOINT
is also configured.
Default: false
S3_OPEN_TIMEOUT
The number of seconds before the HTTP handler should timeout while trying to open a new HTTP session.
Default: 5
S3_READ_TIMEOUT
The number of seconds before the HTTP handler should timeout while waiting for an HTTP response.
Default: 5
S3_FORCE_SINGLE_REQUEST
Set this to true
if you run into trouble processing large files.
Default: false
S3_ENABLE_CHECKSUM_MODE
Enables verification of object checksums when Mastodon is retrieving an object from the storage provider. This feature is available in AWS S3 but may not be available in other S3-compatible implementations.
Default: false
S3_STORAGE_CLASS
When using AWS S3, this variable can be set to one of the storage class options which influence the storage selected for uploaded objects (and thus their access times and costs).
If no storage class is specified then AWS S3 will use the STANDARD
class, but options include REDUCED_REDUNDANCY
, GLACIER
, and others.
Default: STANDARD
S3_MULTIPART_THRESHOLD
The maximum size (in megabytes) of objects that will be uploaded in a single operation. Objects above this threshold will be uploaded using the multipart chunking mechanism, which can improve transfer speeds and reliability.
Default: 15
S3_PERMISSION
Defines the S3 object ACL when uploading new files. When using an S3-compatible object storage backend, it is recommended to use a backend with ACL support, as it allows Mastodon to quickly improve the security of private data.
Default: public-read
BlockPublicAcls
option, as uploading objects with ACL public-read
will fail (403).
In that configuration you should set S3_PERMISSION
to private
.
S3_BATCH_DELETE_LIMIT
The official Amazon S3 API can handle deleting 1,000 objects in one batch job, but some providers may have issues handling this many in one request, or offer lower limits.
Default: 1000
S3_BATCH_DELETE_RETRY
During batch delete operations, S3 providers may perodically fail or timeout while processing deletion requests. Mastodon will back off and retry the request up to this maximum number of times.
Default: 3
Provider Specific Configurations
MinIO
MinIO is an open-source implementation of an S3 object provider.
You need to set a policy for anonymous access that allows read-only access to objects contained by the bucket without allowing listing them.
To do this, you need to set a custom policy (replace mastodata
with the actual name of your S3 bucket):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::mastodata/*"
}
]
}
Mastodon itself needs to be able to write to the bucket, so either use your admin MinIO account (discouraged) or an account specific to Mastodon (recommended) with the following policy attached (replace mastodata
with the actual name of your S3 bucket):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "arn:aws:s3:::mastodata/*"
}
]
}
You can set these policies from the MinIO Console (web-based user interface) or the command-line client (mcli
/ mc
).
Using the MinIO Console
Connect to the MinIO Console web interface and create a new bucket (or navigate to your existing bucket):
Then, configure the βAccess Policyβ to a custom one that allows read access (s3:GetObject
) without write access or the ability to list objects (see above):
RELEASE.2022-10-24T18-35-07Z
should be a safe version to update to that does not require an involved migration procedure.
Create a new mastodon-readwrite
policy (see above):
Finally, create a new mastodon
user with the mastodon-readwrite
policy:
Using the command-line utility
The same can be achieved using the MinIO Client command-line utility (which can be called mc
or mcli
depending on where it is installed from).
Create a new bucket:
mc mb myminio/mastodata
Save the anonymous access policy from above as anonymous-readonly-policy.json
and the Mastodon user access policy as mastodon-readwrite.json
(make sure to replace mastodata
with the name of your newly-created bucket).
Set the anonymous access policy for your bucket:
mc anonymous set-json anonymous-readonly-policy.json myminio/mastodata
Add a mastodon-readwrite
policy:
mc admin policy add myminio mastodon-readwrite mastodon-readwrite.json
Add the mastodon
user (replace the password):
mc admin user add myminio mastodon SECRET_PASSWORD
Apply the mastodon-readwrite
policy to the mastodon
user:
mc admin policy set myminio mastodon-readwrite user=mastodon
Wasabi Object Storage
Create a new bucket and define its policy to allow objects to be anonymously readable but not listable:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::mastodata/*"
}
]
}
Then, create a mastodon-readwrite
policy to grant read and write access to your bucket:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "arn:aws:s3:::mastodata/*"
}
]
}
Finally, create a new mastodon
user and don’t forget to enable the mastodon-readwrite
policy:
On Mastodon’s side, you need to set S3_FORCE_SINGLE_REQUEST=true
to properly handle large uploads.
DigitalOcean Spaces
In your DigitalOcean Spaces Bucket, make sure that βFile Listingβ is βRestrictedβ to users with access keys.
Scaleway
If you want to use Scaleway Object Storage, we strongly recommend you create a Scaleway project dedicated to your Mastodon instance assets and use a custom IAM policy.
First, create a new Scaleway project, in which you create your object storage bucket. You need to set your bucket visibility to “Private” to not allow objects to be listed.
Now that your bucket is created, you need to create API keys to be used in your Mastodon instance configuration.
Head to the IAM settings (in your organization menu, top right of the screen), and create a new IAM policy (eg mastodon-media-access
)
This policy needs to have one rule, allowing it to read, write and delete objects in the Scaleway project you created above (the scope).
Then head to the IAM Applications page, and create a new one (eg my-mastodon-instance
) and select the policy you created above.
Finally, click on the application you just created, then “API Keys”, and create a new API key to use in your instance configuration. You should use the “Yes, set up preferred Project” option and select the project you created above as the default project for this key.
Copy the Access Key ID and Secret, and use them for your AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
Mastodon config variables.
Exoscale
In Exoscale, your bucket should not have any read ACLs (Mastodon will set the ACLs on the object themselves as appropriate).
You need to create an API Key for the Mastodon app, restricted to the Object Storage (sos
) service, restricted to your bucket, and with unrestricted operations.
On Mastodon’s side, you need to set S3_FORCE_SINGLE_REQUEST=true
to properly handle large uploads.
Cloudflare R2
Cloudflare R2 does not support ACLs, so Mastodon needs to be instructed not to try setting them.
To do that, set the S3_PERMISSION
environment variable to an empty string.
To get credentials for use in Mastodon, select βManage R2 API Tokensβ and create a new API token with βEditβ permissions.
Last updated June 17, 2024 Β· Improve this page