Let AI agents read files attached to records

Merged
NE
3d ago
+1346-149

AI agents can find a file on a record but cannot read it, an existing gap tracked in twentyhq/core-team-issues#3188. A model takes a file inline in one of two ways: a URL or the raw bytes. This follows the approach the AI SDK and the model providers document, which is to send a URL where the model accepts it and bytes otherwise.

File delivery

  • A new read_file tool takes the signed file URL returned with a record and adds the PDF or image to the conversation. code_interpreter now accepts the same URL for other file types.
  • Each file goes through the AI SDK experimental_download hook. It stays a URL when the server is public over https, AI_SIGNED_FILE_URLS_ENABLED is on and the model accepts that URL. Otherwise the bytes are read through FileService.
  • Type, size and file count limits are checked before a file is attached. A file that does not fit becomes a short text note to the model, so the run continues.

Permissions

  • No new permission rules. Signed links still come only from the existing permission-checked record queries.
  • The link check moved from FileByIdGuard into FileUrlService.verifyFileToken. The guard and every AI file read now use that one check, and the file endpoint keeps its behavior.

Provider refusals

When a provider refuses a request containing files, the run steps down in order: URL, then bytes, then one small billed check per file so only the refused files are replaced with a note, then no files. A prompt that is too long skips straight to no files. A provider may also refuse to fetch a URL, for example because of robots rules, which costs one failed request per run before bytes are used.

References

Sending a URL where the model accepts it, sending bytes otherwise, and signing the links all follow these docs:

  • Anthropic: images and PDFs can be sent as a URL or as base64. The PDF guide uses base64 "when a URL isn't available", and on Amazon Bedrock and Google Cloud "only base64-encoded sources are available".
  • OpenAI: images and files can be sent as a URL or as base64. Chat Completions does not accept file URLs, so support depends on the endpoint.
  • Google Gemini: file input methods lists inline data and "publicly accessible HTTPS URLs or pre-signed URLs". For private data it says to "create a signed URL with the correct access permissions and expiry".
  • AI SDK: a file part takes a URL or binary data. "The default download implementation automatically downloads files in parallel when they are not supported by the model", and experimental_download is the documented way to customize that, including for authentication and caching.
  • Amazon S3: a presigned URL grants "time-limited permission to download objects", the same model as our signed file links.

Providers also offer an upload API that returns a file ID. It keeps a copy of the file at each provider, so this change does not use it.

Testing

Tested manually in Chrome against Anthropic on a local server, so files were delivered as bytes:

  • Images, PDFs and other file types were read and described.
  • An invalid PDF refused by the provider was skipped and the other files still went through.
  • On the file endpoint, a valid link returned 200. A missing token, a tampered signature, the wrong file and an unsupported folder were refused.

Not covered: delivery by URL on the final code, a suspended workspace, providers other than Anthropic, and new automated tests. Long tool calls can hit the save failure fixed separately in #30066.

Screenshots

AI reads a PDF attached to a person

AI reads an image attached to a note

AI reads a PDF attached to an email