Connect your own agent

If your app has its own agent running on your server, the agent can use the person’s camera roll through your app. The agent asks for photos, your app searches on the iPhone and sends back descriptions, and a photo leaves the iPhone only after the person approves it.

How it works

Your agent asks for photos

It sends your app a search, such as “videos from the concert last Friday”, for example as a tool call.

Your app searches on the iPhone

search runs against the index on the device and sends nothing to Rollup.

Your app sends back descriptions

For each result, send the identifier and the few fields the agent needs to choose, such as the date, place and labels. Send no images.

The person approves

The agent picks the photos it wants. Your app shows them, loaded with PhotoKit from their identifiers, and asks the person to approve.

Your app uploads the approved photos

Load each approved photo or video with PhotoKit, or expo-media-library in Expo, and upload it the way your app already does.

Answer the agent

Keep what you send small. The agent needs enough to choose, not everything the index holds.

Swift
Expo and React Native

struct PhotoForAgent: Codable {
    let id: String
    let date: Date?
    let mediaType: SearchResult.MediaType
    let place: String?
    let location: String?
    let labels: [String]
}

func findPhotos(_ text: String) async throws -> [PhotoForAgent] {
    let results = try await Rollup.shared.search(text)
    return results.results
        .filter { $0.screenshot?.sensitivity.isEmpty ?? true }
        .prefix(20)
        .map { result in
            PhotoForAgent(
                id: result.id,
                date: result.date,
                mediaType: result.mediaType,
                place: result.places.first?.name,
                location: result.location?.description,
                labels: result.labels.prefix(5).map(\.name)
            )
        }
}

async function findPhotos(text: string) {
  const { results } = await Rollup.search(text);
  return results
    .filter((result) => !result.screenshot?.sensitivity.length)
    .slice(0, 20)
    .map((result) => ({
      id: result.id,
      date: result.date,
      mediaType: result.mediaType,
      place: result.places[0]?.name,
      city: result.location?.city,
      labels: result.labels.slice(0, 5).map((label) => label.name),
    }));
}

Leave out sensitive screenshots. search returns every match, including screenshots that hold credentials, financial, health or personal data. Their screenshot.sensitivity is not empty. Drop them before anything leaves the iPhone, as above.
Send text only when the agent needs it. text (what was read in the image) and screenshot can hold names, addresses and messages.
Identifiers belong to this iPhone. A PhotoKit local identifier means nothing on another device. Your agent sends it back to your app to show or upload that photo.

Search or query

Use search when the agent needs photos: it returns them with what the index holds, and the question stays on the iPhone. Use query when the agent needs an answer about the person’s life, such as “Where do they eat most?”. query sends text derived from the photos to Rollup and its model provider, so it needs the person’s permission first (see Privacy and App Review).

Privacy

What your app sends to your server is your app sharing the person’s data, not Rollup’s, and Rollup’s privacy page does not cover it.

Before your app first sends search results to your server, tell the person what it sends and why, and get their permission. If your agent runs on a third-party AI model, App Review guideline 5.1.2(i) requires this.
Upload a photo or video only after the person approves it.
Include what your server receives in your privacy policy and your App Store privacy details.

Get started

Guides

Other platforms

Connect your own agent

How it works

Answer the agent

Search or query

Privacy

​How it works

​Answer the agent

​Search or query

​Privacy

How it works

Answer the agent

Search or query

Privacy