S3 Inventory: configure reports and check object metadata

S3 Inventory creates scheduled reports of objects and selected metadata in an Amazon S3 bucket. It is useful when you need to review storage classes, encryption status, or replication status across a defined set of keys. A report contains metadata; it does not contain the objects’ file contents or replace a recovery copy.

This guide configures a daily CSV report for a small prefix in a general-purpose S3 bucket, then shows how to inspect a manifest and one compressed data file. The configuration and parser were checked locally with synthetic data. No configuration was deployed to AWS, and no real report delivery or permissions were tested.

Choose the question and report scope

Start with a question such as “Which current objects under exports/ report failed replication?” A prefix and a small set of fields make that question easier to inspect. Inventory runs daily or weekly and supports CSV, ORC, and Parquet. AWS says the first delivery can take up to 48 hours; it is not an immediate listing. See the Inventory overview.

Decide whether you need current objects or all versions. The configuration reference defines Current and All. All-version reports add version and delete-marker fields. Counting their rows as current files can inflate totals or include delete markers, so the parser below deliberately accepts only a Current-only report.

Use CSV for a small inspection exercise. For repeated analytical queries, consider ORC or Parquet with Athena. Keep each report date distinct when querying; combining daily reports can count the same object repeatedly.

Define source, destination, and ownership

The example uses two existing general-purpose buckets in eu-west-1, owned by the example account 111122223333. Replace the account, bucket names, Region, and CLI profiles with values from your test environment. Use a new configuration ID rather than overwriting a report someone already depends on.

The source bucket holds the objects and the Inventory configuration. The destination receives reports and must be in the same Region. Here, exports/ selects source objects, while inventory places report output under a destination prefix. These are separate settings.

Save the following as inventory-config.json. The destination reference explains the owner account check. SSES3 configures encryption of the report files; the optional EncryptionStatus field describes the source objects. Neither setting changes source-object encryption.

{
  "Destination": {
    "S3BucketDestination": {
      "AccountId": "111122223333",
      "Bucket": "arn:aws:s3:::amzn-s3-demo-inventory-reports",
      "Format": "CSV",
      "Prefix": "inventory",
      "Encryption": {"SSES3": {}}
    }
  },
  "IsEnabled": true,
  "Filter": {"Prefix": "exports/"},
  "Id": "exports-daily",
  "IncludedObjectVersions": "Current",
  "OptionalFields": ["Size", "LastModifiedDate", "StorageClass", "ReplicationStatus", "EncryptionStatus"],
  "Schedule": {"Frequency": "Daily"}
}

For report encryption with SSE-KMS instead, use a customer-managed key and the required key-policy permissions. AWS does not support the aws/s3 managed key for Inventory’s SSE-KMS report encryption. Follow the current setup instructions before making that change.

Give report delivery and readers separate permissions

When using the CLI, an authorized administrator must add a delivery permission to the destination bucket policy. This example is a statement to merge into the existing policy’s Statement array, not a replacement for the entire policy:

{
  "Sid": "AllowInventoryFromDemoSource",
  "Effect": "Allow",
  "Principal": {"Service": "s3.amazonaws.com"},
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::amzn-s3-demo-inventory-reports/inventory/*",
  "Condition": {
    "ArnLike": {"aws:SourceArn": "arn:aws:s3:::amzn-s3-demo-source"},
    "StringEquals": {
      "aws:SourceAccount": "111122223333",
      "s3:x-amz-acl": "bucket-owner-full-control"
    }
  }
}

The source account and bucket conditions limit which source can use this delivery permission. The resource limits writes to the report prefix. Review existing Deny statements too. Separately, the configuring identity needs s3:PutInventoryConfiguration on the source bucket; reading the configuration requires s3:GetInventoryConfiguration.

A report reader needs access to the destination files and, when relevant, permission to decrypt them. Reading a report is separate from reading the original objects. Names and metadata can expose information about stored data, so keep report access scoped to its intended readers. This policy example is for general-purpose buckets; directory buckets use different Inventory permissions and service principals.

Save the configuration and read it back

With the destination policy reviewed and the example values replaced, this command saves the enabled configuration and starts scheduled reporting. It changes AWS configuration; it is not a preview:

aws s3api put-bucket-inventory-configuration --bucket amzn-s3-demo-source --id exports-daily --inventory-configuration file://inventory-config.json --expected-bucket-owner 111122223333 --profile inventory-admin --region eu-west-1

The put command reference documents the inputs. Check the exit status, then retrieve the saved configuration:

aws s3api get-bucket-inventory-configuration --bucket amzn-s3-demo-source --id exports-daily --expected-bucket-owner 111122223333 --profile inventory-admin --region eu-west-1

Compare the returned configuration with the intended source filter, destination, account ID, schedule, version selection, fields, and encryption. A successful readback proves that the configuration exists. Delivery still depends on the service’s schedule and its access to the destination.

Use the manifest to select the data files

After delivery, locate the report under the configured destination prefix, source bucket name, configuration ID, and report timestamp. Download that report’s manifest.json and manifest.checksum. The manifest lists the exact data-file keys, sizes, checksums, and schema. The timestamp represents the start of report generation. AWS documents this layout in Locating your inventory list.

Use a fresh local working directory. In the following commands, replace REPORT_TIMESTAMP with the actual delivered timestamp and DATA_FILE_KEY with one complete key from the manifest’s files array. The key is relative to the destination bucket and already includes its prefix. Use the same exact key in the parser invocation.

aws s3 cp "s3://amzn-s3-demo-inventory-reports/inventory/amzn-s3-demo-source/exports-daily/REPORT_TIMESTAMP/manifest.json" ./manifest.json --profile inventory-reader --region eu-west-1
aws s3 cp "s3://amzn-s3-demo-inventory-reports/inventory/amzn-s3-demo-source/exports-daily/REPORT_TIMESTAMP/manifest.checksum" ./manifest.checksum --profile inventory-reader --region eu-west-1
aws s3 cp "s3://amzn-s3-demo-inventory-reports/DATA_FILE_KEY" ./report-part.csv.gz --profile inventory-reader --region eu-west-1

A report may contain multiple data files. Reading one gives a partial result. Process every file listed by the selected manifest before reporting totals for that report; do not indiscriminately combine all files from the destination’s data directory across report dates.

Inspect one CSV file with its actual schema

Save the following Python 3 code as inspect-inventory.py. It verifies the downloaded manifest against its checksum, verifies the selected compressed file against its manifest entry, and reads the CSV using the field names in fileSchema. It does not assume that the first data row is a header or that optional fields always occupy the same positions.

import csv, gzip, hashlib, json, sys
from collections import Counter
from pathlib import Path
from urllib.parse import unquote

manifest_path, checksum_path, data_key, local_file = sys.argv[1:]
manifest_bytes = Path(manifest_path).read_bytes()
expected = Path(checksum_path).read_text(encoding='utf-8').strip().lower()
if hashlib.md5(manifest_bytes, usedforsecurity=False).hexdigest() != expected:
    raise SystemExit('Manifest checksum does not match.')
manifest = json.loads(manifest_bytes)
fields = [name.strip() for name in manifest['fileSchema'].split(',')]
required = {'Bucket', 'Key', 'Size', 'StorageClass', 'ReplicationStatus', 'EncryptionStatus'}
if manifest['fileFormat'] != 'CSV' or not required.issubset(fields):
    raise SystemExit('Expected CSV with the configured metadata fields.')
if len(fields) != len(set(fields)) or {'VersionId', 'IsLatest', 'IsDeleteMarker'} & set(fields):
    raise SystemExit('Use a Current-only report with unique field names.')
matches = [item for item in manifest['files'] if item['key'] == data_key]
if len(matches) != 1:
    raise SystemExit('Data key must match exactly one manifest entry.')
entry = matches[0]
digest = hashlib.md5(usedforsecurity=False)
with open(local_file, 'rb') as stream:
    for block in iter(lambda: stream.read(1024 * 1024), b''):
        digest.update(block)
if Path(local_file).stat().st_size != entry['size'] or digest.hexdigest() != entry['MD5checksum'].lower():
    raise SystemExit('Downloaded data file does not match its manifest.')
rows = total_bytes = review_count = 0
encryption, replication = Counter(), Counter()
with gzip.open(local_file, 'rt', encoding='utf-8', newline='') as stream:
    for values in csv.reader(stream, strict=True):
        if len(values) != len(fields):
            raise SystemExit('Unexpected CSV column count.')
        row = dict(zip(fields, values))
        if row['Bucket'] != manifest['sourceBucket']:
            raise SystemExit('Unexpected source bucket in data file.')
        size = int(row['Size'])
        if size < 0:
            raise SystemExit('Invalid negative object size.')
        rows += 1
        total_bytes += size
        encryption[row['EncryptionStatus'] or 'UNKNOWN'] += 1
        replication[row['ReplicationStatus'] or 'UNKNOWN'] += 1
        if row['EncryptionStatus'] == 'NOT-SSE' or row['ReplicationStatus'] == 'FAILED':
            review_count += 1
            print(json.dumps({'review_key': unquote(row['Key'], errors='strict'),
                              'encryption': row['EncryptionStatus'],
                              'replication': row['ReplicationStatus']}))
print(json.dumps({'scope': 'one data file', 'manifest_files': len(manifest['files']),
                  'rows': rows, 'object_bytes': total_bytes, 'review_count': review_count,
                  'encryption': dict(encryption), 'replication': dict(replication)}))

python inspect-inventory.py manifest.json manifest.checksum "DATA_FILE_KEY" report-part.csv.gz

Use python3 if that is your Python command. Run against trusted report files downloaded through your authenticated AWS connection. MD5 here checks agreement with the supplied manifest; it is not an authenticity guarantee. Check the process exit status before using its output.

The output lists review candidates and summarizes only the selected data file. Blank status values are counted as UNKNOWN, not success. The example identifies NOT-SSE and FAILED; it is not a complete compliance policy. For example, an object encrypted with a different KMS key than required still needs review.

Our local test file had seven rows totaling 1,510 object bytes and three review candidates. Tests confirmed that changed files and manifests, missing columns, incorrect row widths, unmatched file keys, and all-version schemas were rejected. Reordered columns and encoded keys were handled correctly, including preserving a literal plus sign. These are synthetic test results, not measurements of an AWS bucket.

Recheck an object before taking action

Inventory is eventually consistent for additions, overwrites, and deletions. A report can lag behind current state. Use a fresh metadata request before acting on a finding, with the decoded key from the report:

aws s3api head-object --bucket amzn-s3-demo-source --key "exports/review.csv" --expected-bucket-owner 111122223333 --profile inventory-reader --region eu-west-1

This HeadObject example reads metadata for the current object and requires the relevant source-object read permission. Replace exports/review.csv with the reviewed key. An all-version investigation also needs the intended version ID. A later write can still change the object after this check, so an automated remediation process needs its own concurrency controls.

For encryption findings, use our S3 encryption guide to check the actual method and key. For a replication failure or pending state, follow the replication guide. An Inventory status alone does not prove that a replica is usable for recovery.

Keep reporting useful over time

If the first report has not arrived, allow for the documented initial delivery window, then check the enabled setting, source filter, destination Region and policy, and any KMS permissions. A valid JSON document or saved configuration cannot verify these runtime conditions.

Plan for Inventory and S3 charges, report storage, and any query costs. Retain enough reports for comparison, then apply a narrowly scoped lifecycle policy to old report output. Keep that retention decision separate from deletion of source objects.

This workflow uses Amazon S3 Inventory. An S3-compatible object API does not by itself establish support for this AWS reporting feature, and no Hivenet compatibility was tested here.