Upload bundles

An upload bundle is a group of related uploads from a single device, such as every file from one recording or run.

Properties

Device

The device whose uploads the bundle groups.

Upload Collection

The upload collection whose uploads the bundle groups.

string[]

The capture keys from upload.bundle of the file rule that created the bundle.Capture keys do not affect membership, only captured values do. See Same capture values.Example: ["mission", "run"]

string[]

The captured values that identify the bundle.Together with the device and upload collection, these values are the bundle’s identity: no two bundles share all three. See Matching conditions.Example: ["mission_123", "run_03"]

Single file rule

Bundles are specified in a file rule’s upload.bundle field. Below is an example of a file rule that bundles files in the same {recording} folder:

name: Recordings

source:
  glob: "/data/recordings/{recording}/*"

upload:
  collection_slug: recordings
  bucket: my-uploads-bucket
  path: "{device_name}/{bundle_id}/{upload_id}/{file_name}"
  bundle: [recording]

Using this rule, all files in the run_42 folder are bundled together:

data

recordings

run_42

camera.mcap

metadata.yaml

run_43

camera.mcap

metadata.yaml

Files in another folder, such as run_43, land in a different bundle.

Multiple file rules

Sometimes you want to bundle files sourced from different file rules. In such cases, bundling is still possible. Consider the following two rules:

name: Camera Calibration

source:
  glob: "/data/calib/{run}/{camera}.yaml"

upload:
  collection_slug: calibration
  bucket: my-uploads-bucket
  path: "calib/{bundle_id}/{upload_id}/{file_name}"
  bundle: [run, camera]

name: Calibration Images

source:
  glob: "/data/images/{run}/{camera}.png"
  stability_window_secs: 5

upload:
  collection_slug: calibration
  bucket: my-uploads-bucket
  path: "calib/{bundle_id}/{upload_id}/{file_name}"
  bundle: [run, camera]

Using these rules, /data/calib/run_18/front.yaml and /data/images/run_18/front.png are bundled because they have the same capture values of run_18 and front.

data

calib

run_18

front.yaml

back.yaml

run_19

front.yaml

back.yaml

images

run_18

front.png

back.png

run_19

front.png

back.png

A different camera (run_18/back.yaml) or a later run (run_19/front.yaml) is placed into a different bundle.

Common patterns

Below are common patterns for specifying which files to bundle together.

Folder
Filename
Nested

Use when each group is already a directory. Capture the folder name and match the files inside it.

name: Bags

source:
  glob: "/data/bags/{bag}/*"

upload:
  collection_slug: bags
  bucket: my-uploads-bucket
  path: "{bundle_id}/{upload_id}/{file_name}"
  bundle: [bag]

Every file in drive_07/ shares a bundle. drive_08/ is a different bundle:

data

bags

drive_07

camera.mcap

metadata.yaml

drive_08

camera.mcap

metadata.yaml

Use when related files share an identifier in the filename instead of a folder. Capture the id, and keep the rest of the segment constant.

name: Run Recordings

source:
  glob: "/data/runs/{run}.mcap"

upload:
  collection_slug: runs
  bucket: my-uploads-bucket
  path: "{bundle_id}/{upload_id}/{file_name}"
  bundle: [run]

name: Run Metadata

source:
  glob: "/data/runs/{run}.yaml"
  stability_window_secs: 5

upload:
  collection_slug: runs
  bucket: my-uploads-bucket
  path: "{bundle_id}/{upload_id}/{file_name}"
  bundle: [run]

run_18.mcap and run_18.yaml join one bundle. run_19 is a different bundle:

data

runs

run_18.mcap

run_18.yaml

run_19.mcap

run_19.yaml

Use when the group has more than one part, such as a run and a camera. Capture each part in its own path segment and list them in upload.bundle.

name: Camera Calibration

source:
  glob: "/data/calib/{run}/{camera}.yaml"

upload:
  collection_slug: calibration
  bucket: my-uploads-bucket
  path: "{bundle_id}/{upload_id}/{file_name}"
  bundle: [run, camera]

run_18/front.yaml does not join run_18/back.yaml (different camera) or run_19/front.yaml (different run).How many captures you list is the grouping granularity: [run] puts every camera in run_18 in one bundle; [run, camera] keeps them separate.

Matching conditions

Specifying captures to bundle files is a powerful way to group related files together. Below is the full list of conditions that determine if two uploads are placed into the same bundle.

Same device

Bundles never span devices. Identical capture values from two devices are always placed into different bundles.

Same upload collection

Bundles never span upload collections. Identical capture values from the same device but specifying a different upload collection are always placed into different bundles.

Same capture values

Bundles are matched by their capture values; the order of capture values does matter. Notably, capture keys are ignored.

Examples

To illustrate capture matching conditions, consider the following file rules:

name: Missions

source:
  glob: "/data/missions/{mission}/{run}/*"

upload:
  collection_slug: missions
  bucket: my-uploads-bucket
  path: "missions/{bundle_id}/{upload_id}/{file_name}"
  bundle: [mission, run]

name: Sessions

source:
  glob: "/data/missions/{session}/{id}/*"

upload:
  collection_slug: missions
  bucket: my-uploads-bucket
  path: "missions/{bundle_id}/{upload_id}/{file_name}"
  bundle: [session, id]

name: Missions

source:
  glob: "/data/missions/{mission}.json"

upload:
  collection_slug: missions
  bucket: my-uploads-bucket
  path: "missions/{bundle_id}/{upload_id}/{file_name}"
  bundle: [mission]

Below are some examples of captured values and whether they would join the same bundle: Bundled

Same keys, values, and order

	Capture keys	Capture values
Upload A	`[mission, run]`	`["01", "02"]`
Upload B	`[mission, run]`	`["01", "02"]`

Explanation: Both uploads captured ["01", "02"] in that order.

Keys differ, values and order match

	Capture keys	Capture values
Upload A	`[mission, run]`	`["01", "02"]`
Upload B	`[session, id]`	`["01", "02"]`

Explanation: Capture keys are ignored. Both uploads captured ["01", "02"] in that order.

Not bundled

Order differs

	Capture keys	Capture values
Upload A	`[mission, run]`	`["02", "01"]`
Upload B	`[mission, run]`	`["01", "02"]`

Explanation: Both uploads captured 01 and 02, but in opposite order.

A value differs

	Capture keys	Capture values
Upload A	`[mission, run]`	`["03", "02"]`
Upload B	`[mission, run]`	`["01", "02"]`

Explanation: The first capture value is 03 vs 01.

Fewer values

	Capture keys	Capture values
Upload A	`[mission, run]`	`["01", "02"]`
Upload B	`[mission]`	`["01"]`

Explanation: One upload captured two values and the other captured one.

Welcome

Fleet

Config Management

Data Uploads

Developers

Administration

Upload bundles

Properties

Single file rule

Multiple file rules

Common patterns

Matching conditions

Same device

Same upload collection

Same capture values

Examples

​Properties

​Single file rule

​Multiple file rules

​Common patterns

​Matching conditions

​Same device

​Same upload collection

​Same capture values

​Examples

Properties

Single file rule

Multiple file rules

Common patterns

Matching conditions

Same device

Same upload collection

Same capture values

Examples