Loading

Direct Uploads to S3

Direct uploads let the browser send a large file straight to your S3 bucket instead of passing it through the web server. The server signs a one-off permission for that exact file, the browser posts the file to a holding area in the bucket, and the server then checks it and moves it into place. Use it on S3 installations where people upload large files, so those files stop tying up the server's disk and bandwidth.

Where to find it

Architect Panel → Data:

  • Large Uploads — each upload session, with Transport showing "direct" or "chunked"

Architect Panel → Automation:

  • Tasks — Upload Housekeeping, which also retries clean-up after direct uploads

Architect Panel → Integration & Connections:

  • Deployments — the File Storage card, to confirm the installation is on Amazon S3

The switch itself and the bucket rules are set by your hosting administrator. There is no screen setting.

When it is used

Direct uploads only happen when all of these are true:

  • The installation's storage backend is Amazon S3. Local and Azure installations always use the chunked route.
  • Your hosting administrator has switched direct uploads on. It ships off.
  • The file is at least 8 MB. Below that, posting it to the server is quicker.
  • The upload comes from a File Upload field on a record, or from a record's documents pane.

Even then, the platform quietly uses the chunked route instead for files it needs to handle on the server: images (which are resized), Web Video fields, files going into a File Store, fields with a custom file processor, and every file when antivirus scanning with ClamAV is switched on. People see no difference either way.

What your hosting administrator sets up

  1. A CORS rule on the bucket allowing POST from your site's address (and any custom tenant domains). Without it, browsers refuse to send the file.
  2. A lifecycle rule that expires the holding area (the _amincoming/ prefix) after a day, so files abandoned mid-upload do not accumulate. Expiring incomplete multipart uploads is also sensible.
  3. The direct upload switch, and if wanted a higher maximum file size than the standard 1 GB. Direct uploads can handle up to 4 GB per file.

What is checked

The permission the browser receives is for one exact file name, one exact size and one content type chosen by the server, and it lasts only as long as the file needs to upload, up to six hours. When the upload finishes, the server checks the stored size matches exactly, reads the start of the file to confirm its real type, moves it to its proper place and deletes the holding copy. A file that fails a check never becomes a file in the platform. The same file-type rules apply as for any other upload.

Checking it works

  1. Confirm with your hosting administrator that the bucket rules are in place before the switch is turned on.
  2. Upload a non-image file over 8 MB, such as a large PDF or ZIP, to a File Upload field.
  3. Open Large Uploads and find the session. Transport should read "direct", and Staged Object should be empty once it completes.
  4. Download the file and confirm it opens.

What goes wrong

  • Transport always reads "chunked". Direct uploads are off, the file is under 8 MB, or it is one of the kinds that always takes the chunked route. Check with an ordinary large PDF.
  • The upload falls back after a pause. If the browser's post to S3 fails, usually because of a missing CORS rule, the upload is abandoned and retried once through the server. It still succeeds, but slowly; ask for the CORS rule to be checked.
  • Staged Object stays filled in on completed sessions. The holding copy could not be deleted, so the file is stored twice. Upload Housekeeping retries the delete, so make sure it is enabled.
  • The Error Log says the server fell back to the chunked transport. The server could not sign the upload, often because of an S3 credential problem.

Worked example

An engineering firm's surveyors attach 300 MB drawing archives to site records. On chunked uploads each one passed through the server twice, and three at once slowed the site for everybody. The hosting administrator adds the CORS and lifecycle rules and turns direct uploads on. The next archive's session shows Transport "direct" with an empty Staged Object, and the server's load during uploads drops away. Photos are still resized on the server, as before.

Recommendations

  • Put the bucket rules in place first, then turn the switch on.
  • Enable Upload Housekeeping so leftover holding copies are cleared.
  • Test with a large non-image file, because images always go through the server.
  • Use Large Uploads to confirm which route each file actually took.