Limit the size of fetched remote files

Merged
PU
1d ago
+118-5

Today

  • RemoteImage, RemoteDocument, RemoteAudio and RemoteVideo download the full file before they inline or upload it.
  • The download has no size limit, and a compressed body expands with no limit.

Problem

  • A very large file fills worker memory and temporary storage.
  • The request then fails late, with a memory error instead of a clear message.

Change

  • A download stops at 100 MB and throws an InvalidArgumentException that names the limit.
  • A file whose Content-Length is over the limit fails before the body downloads.
  • The limit counts the decoded bytes, so compressed bodies are limited too.

[!NOTE]
Remote files over 100 MB now throw. Set ai.remote_files.max_size (in kilobytes) to change the limit.