Loading

Large Uploads

Large files are not uploaded in one request. The browser splits them into chunks, the server reassembles them, and the whole transfer is tracked as an upload session you can inspect. On S3 installations a second route lets the browser send large files straight to the bucket instead. This article covers both, and how to diagnose an upload that will not finish.

Where to find it

Architect Panel → Data:

  • Large Uploads — upload sessions in progress and recently finished

Architect Panel → Automation:

  • Tasks — the Upload Housekeeping task that clears expired sessions

Architect Panel → Activity:

  • Error Log — why a session failed, and storage errors

The limits

  • Ordinary uploads through a File Upload field are sent in one request, up to the web server's limit (64 MB on a standard installation).
  • Chunked uploads are sent in 5 MB parts, up to 1 GB per file.
  • Direct uploads to S3 skip the server entirely, up to the same 1 GB unless your hosting administrator raises it. See Direct Uploads to S3.

All three limits, and whether direct uploads are on, are set by your hosting administrator. None is on a screen.

What a session records

Each row in Large Uploads records the upload key and the person who started it; the file name, extension, type and declared size against how much has been received; the chunk size and how many chunks are expected and received; where the file is going (datastore, field, record and storage kind); its state, the resulting file and any error; and when it started, when the last chunk arrived, and when it completed or expires.

Three fields describe how the file arrived: Transport is "chunked" when the parts came through the server and "direct" when the browser sent the file straight to S3. Staged Object and Staged ETag are used by direct uploads only. A Staged Object still filled in after a direct upload completed means the temporary copy could not be deleted, so the file is briefly stored twice; housekeeping retries the delete.

Each chunk is recorded

Chunks record their number, size, a checksum, where they were staged and when they arrived. A session that is missing chunks cannot be finalised, so a broken connection produces an incomplete session, never a silently truncated file.

Sessions expire, and housekeeping clears them

A session that is not finished within 24 hours expires. Expiry stops it being resumed, but its staged parts are only removed when the Upload Housekeeping task runs. That task ships switched off and in preview mode, so on a new installation abandoned uploads accumulate until you turn it on:

  1. Open Tasks and find Upload Housekeeping. It runs hourly once enabled.
  2. Use its Preview row action. Preview describes what the task would clear and changes nothing.
  3. Enable it and switch preview mode off on the task's row. The Task Log row action then shows each run.

Diagnosing a failing upload

  1. Ask for the file name and roughly when they tried.
  2. Find the session in Large Uploads. Compare the received size with the declared size, and note when the last chunk arrived.
  3. Read the error on the session, then the Error Log at the same time.
  4. If several people stall at a similar point, look for a size limit somewhere in the path, such as a proxy or load balancer, rather than at the platform.

What goes wrong

  • A large file fails and no session appears in Large Uploads. The file went as an ordinary single request and hit the web server's limit, or a proxy in front of the server refused it first.
  • Staging storage fills up. Upload Housekeeping is not enabled.
  • On Azure, a large upload reaches 100% and then fails. The chunks arrived; the server's transfer to Azure failed. The Error Log has the Azure error.

Worked example

A team investigating failed video uploads finds sessions stalling at the same received size, which points to a proxy limit rather than the platform; the proxy is fixed. Looking at Large Uploads, they also notice hundreds of expired sessions from the past year. Upload Housekeeping had never been enabled. Its Preview lists them, and once it is enabled the staged parts are cleared within the hour.

Recommendations

  • Enable Upload Housekeeping after checking its preview.
  • Use the session to diagnose, not the user's account.
  • A common stall size means a limit in the path.
  • Test from a phone on mobile data, and interrupt the upload part way.