Configuring extra file submission

Loculus supports the handling of arbitrary files associated with sequence entries. You can configure Loculus to support the submission of extra files for sequences, as well as providing extra files along with the sequence data and metadata for download. A typical use case would be for raw reads. The files are stored in S3.

To enable this feature you need to configure an S3 bucket for Loculus to use, and then configure the file categories per organism.

Conceptual overview

Extra files submitted alongside sequence and metadata are treated differently. Loculus uses S3 - a generic object storage service - to store these files. Files are uploaded directly to S3 using presigned URLs.

Unlike files that contain sequence data, the file sharing files are not inherently coupled to any particular sequence. Files are uploaded first, and then associated to a sequence entry; at the time of uploading, only an owning group needs to be specified. Because of this, the same file can also be attached to multiple sequence entries as well.

The files will not be publicly accessible, until an associated sequence entry is released.

When configuring this feature for an organism, you can configure file categories for which users can submit files, as well as file “output” categories, which will then be visible alongside other sequence data and metadata in the sequence detail view. You can also configure only submit files (which can then be used by the preprocessing pipeline in some way) or only “output” files, which the preprocessing pipeline can generate on its own. The preprocessing pipeline gets access to submitted files before they are released, and the pipeline can also upload its own new files.

Loculus configuration

Configuring an S3 bucket to use

You need admin access to an S3 bucket, and have the credentials at hand.

The credentials consist of an accessKey and a secretKey (an Access Key ID/Secret Access Key pair) that together authenticate as a single S3 identity - much like a username and password. This identity must be able to read, write, tag, and delete objects in the bucket (see IAM permissions below).

For AWS S3, you get such a pair by creating an IAM user and then generating an access key for that user.

Enable S3 and configure the location of the bucket:

s3:
  enabled: true
  bucket:
    region: us-east-1
    endpoint: https://my-s3.net
    bucket: loculus-data

For AWS S3 specifically:

s3:
  enabled: true
  bucket:
    region: eu-central-1
    endpoint: https://s3.eu-central-1.amazonaws.com
    bucket: my-loculus-bucket

Note

endpoint must include the https:// protocol and be the regional S3 endpoint, not a bucket-specific virtual-hosted one (e.g. not https://my-loculus-bucket.s3.eu-central-1.amazonaws.com) - the backend addresses objects path-style (endpoint/bucket/key) and supplies the bucket name separately via bucket. bucket is the bare bucket name, not the ARN (arn:aws:s3:::my-loculus-bucket).

Note

Have a look at the Helm Chart S3 reference for more information on these configuration settings.

Configure the credentials using sealed secrets:

secrets:
  s3-bucket:
    type: sealedsecret
    clusterWide: 'true'
    encryptedData:
      accessKey: AgCm73j1g21Dn....
      secretKey: AgAS8a/ldl....

To create the encryptedData above, seal your accessKey/secretKey with kubeseal - see Adding a sealed secret.

Note

Alternatively, you can also use the raw secret type. If you do, ensure the configuration file is properly access-protected, since it will contain credentials in plain text.

secrets:
  s3-bucket:
    type: raw
    clusterWide: 'true'
    data:
      accessKey: K93AK....
      secretKey: DK37sAKJ....

IAM permissions

The accessKey/secretKey only need enough permissions for the backend to read, write, tag, and delete objects under the bucket.

For AWS, attach a policy like this to the IAM user (replace my-loculus-bucket with your bucket name):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:PutObjectTagging", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::my-loculus-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-loculus-bucket"
    }
  ]
}

Configuring file submission

Users can submit files along with sequence metadata and sequences (or also instead of sequences). For this, you need to enable the files submission type, and configure at least one file category that users can submit:

fileSharing:
  maxFileSizeBytes: 5368709120 # 5 GiB
my-organism:
  schema:
    submissionDataTypes:
      files:
        enabled: true # enable the feature
        categories:
          - name: rawReads # configure a submission category

The example above configures the rawReads file category.

If a user submits these files, they will be passed along to the processing pipeline as well, and the pipeline can read them, pass them through as output files, or generate additional fields or process them in any other way.

You can also set a maximum accepted file size for your loculus instance in fileSharing.maxFileSizeBytes. Files exceeding this size are rejected by the website and by the backend for multipart uploads (note for single part uploads the default maximum size limit of 5GB cannot be modified and even if a lower maxFileSizeBytes threshold is configured it cannot be enforced for single part uploads). If unset, the website does not enforce a limit and multi-part uploads use the configured S3 service limit.

Note

Files are submitted by adding a files.<category> column to the metadata file, e.g. files.rawReads. The files. prefix is therefore reserved: no metadata field may have a name starting with it, as such columns are treated as file categories during submission rather than as metadata.

Configuring output files

By default, files are not shown in the sequence detail view as well. You need to configure output files as well, and the pipeline needs to be configured to pass them through (or alternatively create and upload new files). You can additionally configure whether the backend should supply file URLs as links to the website or the backend itself using fileSharing.outputFileUrlType.

To configure:

fileSharing:
  outputFileUrlType: website
my-organism:
  schema:
    files:
      - name: rawReads

Note

The name (rawReads in the example above) must not be a name that is also used by a metadata field!

S3 Configuration

To ensure Loculus can access the bucket and allow reading files directly from S3, certain configuration settings are required. The default settings may depend on your S3 provider, meaning that some steps may be unnecessary or different.

s3cmd is a good tool to configure S3. Once installed, you can configure it to access your bucket with s3cmd --configure.

Set CORS bucket policy

Depending on your S3 provider, you might have to configure a CORS policy for your bucket which allows access from the Loculus website to your bucket. Without it, your browser will raise an error when trying to upload files to S3.

You can set a permissive CORS policy on your bucket with s3cmd setcors cors.xml s3://<bucketname>, where <bucketname> is the name of your bucket, and cors.xml is a file with this content:

<CORSConfiguration>
  <CORSRule>
    <AllowedHeaders>*</AllowedHeaders>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedOrigin>*</AllowedOrigin>
    <ExposeHeader>ETag</ExposeHeader>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
  </CORSRule>
</CORSConfiguration>

Note

ExposeHeader: ETag is required for multipart uploads. MaxAgeSeconds is optional; it just lets the browser cache the CORS preflight response to reduce the number of preflight requests.

Also consult the documentation for you S3 provider to find out about CORS policy configuration. For example: AWS and Hetzner.

Set S3 policy

Note

This step is only required if you wish to be able to have the files be set to public on your S3 instance, accessible without the need to request pre-signed URLs from the backend.

There is currently a bug that may lead to a file not being set to public in rare cases (due to race conditioning, see GitHub issue).

The backend makes files in the bucket public, by tagging them with public=true. For this to work, you need to configure a bucket policy like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": ["arn:aws:s3:::$bucket/*"],
      "Condition": {
        "StringEquals": {
          "s3:ExistingObjectTag/public": "true"
        }
      }
    }
  ]
}

You can do this with s3cmd. Save the policy above in a file, policy.json, and then call s3cmd setpolicy policy.json s3://<bucket-name> (replace <bucket-name> with your bucket name).

Set a lifecycle policy to abort incomplete multipart uploads

Files are uploaded to S3 as multipart uploads directly from the browser. If an upload is interrupted (closed tab, network drop, etc.) before it completes, the parts already uploaded are not automatically deleted and continue to incur storage costs.

To avoid accumulating these orphaned parts, configure a bucket lifecycle rule with the AbortIncompleteMultipartUpload action, which tells S3 to abort (and delete the parts of) any multipart upload that hasn’t completed within a given number of days:

{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart-uploads",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 1
      }
    }
  ]
}

Additionally, if you choose to use a versioned s3 bucket you will need to fully remove deleted files (as the garbage collector will only add a delete marker), this can be done with an updated lifecycle policy:

{
  "Rules": [
    {
      "ID": "permanently-delete-after-30-days",
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 30
      },
      "Expiration": {
        "ExpiredObjectDeleteMarker": true
      }
    }
  ]
}