S3

Objects, multipart, versioning, lifecycle, notifications, replication, website hosting.

fakecloud implements 107 of 107 S3 operations at 100% Smithy conformance.

Supported features

  • Objects — GET/PUT/DELETE/HEAD, versioning, delete markers, metadata, tags, ACLs
  • POST Object (browser POST Policy) — the multipart/form-data upload targeted by boto3's generate_presigned_post / the JS SDK's createPresignedPost / HTML form uploads. The signed base64 policy is verified (SigV4 over the policy document, checked against the caller's IAM secret; test* root-bypass keys skip the crypto check), its expiration and conditions (eq, starts-with, content-length-range) are enforced, and any form field not authorized by a condition is rejected with AccessDenied. ${filename} in the key is substituted, and the response honors success_action_status (204 default, 200/201 with PostResponse XML) and success_action_redirect (303 to the given URL with bucket/key/etag appended).
  • Presigned URLs with headers in the query: a SigV4 presigned URL can carry request headers as query parameters (the AWS SDK for JavaScript v3 presigner moves every x-amz-* header there by default). S3 applies them as if they were headers, so x-amz-meta-*, x-amz-tagging, x-amz-storage-class, x-amz-server-side-encryption, x-amz-checksum-* and the rest land on the object. The SigV4 auth parameters themselves are not treated as headers, and a header sent directly wins over the same query parameter. Under --verify-sigv4 the signature covers those parameters, so altering one after signing is SignatureDoesNotMatch.
  • Non-ASCII user metadata: handled as S3 documents it: a value made of RFC 2047 encoded-words is decoded before storing, raw header bytes are read as ISO-8859-1, and a stored value that is not pure US-ASCII is returned as =?UTF-8?B?...?=. Non-ASCII values from a presigned URL query or a browser POST form field are UTF-8.
  • Multipart uploads: full lifecycle; resumable across restarts in persistent mode. CompleteMultipartUpload emits s3:ObjectCreated:CompleteMultipartUpload. Encryption is chosen at CreateMultipartUpload the way PutObject chooses it (the request's x-amz-server-side-encryption* headers, else the bucket's default encryption) and reported on CreateMultipartUpload, UploadPart, UploadPartCopy and CompleteMultipartUpload; under SSE-KMS the completed object is stored as one envelope over the assembled plaintext, exactly like a PutObject body, so GetObject (ranges included), CopyObject and UploadPartCopy read the plaintext and nothing is kept unencrypted at rest. UploadPartCopy from an SSE-KMS source copies its plaintext. Checksum types are covered under Flexible checksums below.
  • Flexible checksums: all ten ChecksumAlgorithm values are computed for real: CRC32, CRC32C, CRC64NVME, SHA1, SHA256, SHA512, MD5, XXHASH64, XXHASH3 (XXH3-64) and XXHASH128 (XXH3-128), the xxHash family with seed 0 as the AWS Common Runtime computes them, each as base64 of the big-endian digest. On PutObject, UploadPart and PutObjectAnnotation an x-amz-checksum-<algorithm> value, a trailing checksum (x-amz-trailer on an aws-chunked body, the form the SDKs use for streaming uploads) and Content-MD5 are verified against the bytes received, in the same pass that stores them: a mismatch is BadDigest, a malformed value InvalidRequest (InvalidDigest for Content-MD5), more than one checksum header InvalidRequest, a declared trailer that never arrives MalformedTrailerError, and an x-amz-sdk-checksum-algorithm outside the enum InvalidRequest. Algorithm names are case-insensitive. Small request bodies (PutBucketCors, DeleteObjects, PutObjectTagging, ...) are checked against Content-MD5 / x-amz-checksum-* as well, an aws-chunked one de-framed and its trailing checksum verified. An empty integrity header counts as not sent. A request to a bucket that does not exist answers NoSuchBucket, and a failed precondition 412, before any digest is compared. An upload that names no algorithm gets the CRC64NVME full-object checksum S3 adds by default. GetObject / HeadObject with x-amz-checksum-mode: ENABLED return the stored checksum and x-amz-checksum-type; a partNumber request, or a range covering exactly one part, returns that part's checksum. ListObjects* / ListObjectVersions report ChecksumAlgorithm and ChecksumType. Multipart uploads take COMPOSITE or FULL_OBJECT with the user guide's per-algorithm rules (FULL_OBJECT only for the CRCs, no COMPOSITE for CRC64NVME, CRC64NVME defaulting to FULL_OBJECT and the rest to COMPOSITE); a part checksummed in another algorithm than the upload is rejected; UploadPart / UploadPartCopy return the part checksum, ListParts, ListMultipartUploads and GetObjectAttributes (ObjectParts, paged by x-amz-max-parts / x-amz-part-number-marker) report checksums; CompleteMultipartUpload checks every listed part checksum (one in another algorithm, or for a part stored without one, is rejected), requires them for a COMPOSITE upload, and verifies a whole-object value (a bare digest for FULL_OBJECT, <digest>-<part count> for COMPOSITE), x-amz-checksum-type and x-amz-mp-object-size. A FULL_OBJECT CRC is combined from the part CRCs rather than recomputed. An upload created without an algorithm completes with a CRC64NVME FULL_OBJECT checksum; a CRC32/CRC32C/SHA1/SHA256 value sent only at completion is accepted without being validated or stored, and one of the newer algorithms there is InvalidRequest, as the user guide describes. CopyObject stores a FULL_OBJECT checksum of the copy in the requested algorithm, else the source's, else CRC64NVME. PutObjectAnnotation and GetObjectAnnotation store and return the payload's checksum the same way. Objects other services write into a bucket (Firehose delivery, access logs, inventory reports) carry the default CRC64NVME full-object checksum of their content, like a PutObject that names no algorithm.
  • Object annotations: PutObjectAnnotation / GetObjectAnnotation / DeleteObjectAnnotation / ListObjectAnnotations attach named UTF-8 payloads (1 byte to 1 MiB, up to 1,000 per object version, names up to 512 bytes) to an object version without touching its body or ETag. x-amz-object-if-match makes the write conditional on the object's ETag, responses name the version in x-amz-object-version-id, and annotations persist with their object in persistent mode, each stored on its own so writing one never rewrites the others.
  • Lifecycle — expiration and storage class transitions via /_fakecloud/s3/lifecycle-processor/tick
  • Notifications — delivery to SNS, SQS, Lambda, and EventBridge on object create/delete, including batch DeleteObjects and version-specific deletes. Records follow the AWS notification content structure: s3SchemaVersion, the matching rule's configurationId, the bucket ownerIdentity, a URL-encoded object key, a monotonic sequencer, and versionId on versioning-enabled buckets (ObjectRemoved records carry no size/eTag, as on AWS). EventBridge events use the AWS detail types (Object Created, Object Deleted, ...) with reason/deletion-type, go to the bucket owner's default bus, and carry the owner's account, the bucket's region and the bucket ARN in resources, so rules in the owning account (in any region) match them. A browser form upload fires s3:ObjectCreated:Post, not :Put.
  • Versioning: enable/suspend, list object versions, delete specific versions. Suspended buckets follow the AWS null-version rules: a write takes over the null version (replacing whatever held it), a delete stacks a null-id delete marker, earlier versions are kept either way, and an object written before versioning was enabled survives later writes as the null version. Suspending is rejected with InvalidBucketState on an Object Lock bucket and on a bucket with a replication configuration (remove the configuration first), as on AWS.
  • Encryption: SSE-S3, SSE-KMS (real envelope encryption through the KMS hook with the aws:s3:arn encryption context), SSE-C. An object written with no SSE headers takes the bucket's default encryption (PutBucketEncryption and CloudFormation's BucketEncryption accept an aws:kms default without a KMSMasterKeyID, stored and returned as sent, whose objects use the aws/s3 key); an SSE-KMS object records its key's ARN, the account's AWS-managed aws/s3 key when none is named. aws:kms:dsse is recorded and reported but its body is not enveloped. Other services read objects the way GetObject returns them (DynamoDB imports, Athena queries, CloudFormation TemplateURLs and nested stacks, Route 53 Resolver domain lists, the RDS S3 bridge), so an SSE-KMS object reads as its plaintext everywhere.
  • Bucket subresources — policy, CORS, lifecycle, logging, website, public access block, object lock, replication, ownership, inventory, encryption, accelerate, request payment, tagging
  • Cross-account buckets: bucket names are one global namespace, so every bucket-addressed request (object reads and writes, multipart uploads, tagging, ACLs, versioning, subresources, presigned URLs, path-style or virtual-hosted) is served against the bucket in the account that owns it, whichever account sends it; objects another account writes land in the owner's bucket. Under --iam strict a principal from another account needs both its own identity policy and the bucket policy to allow the action (logged only under --iam soft; with IAM off the request is served unchecked). CopyObject and UploadPartCopy also need s3:GetObject (s3:GetObjectVersion for a versioned source) on the source object, evaluated against the source bucket's owner and policy. SSE-KMS on such a request resolves a key or alias ARN in the account it names (so a requester can encrypt under its own key) and a bare key id or alias in the bucket owner's account; using a key another account owns needs that key's policy to grant kms:GenerateDataKey (writes) or kms:Decrypt (reads) under --iam strict, so the owner's AWS-managed aws/s3 key refuses other accounts, and an unknown key is KMS.NotFoundException. CreateBucket with a name another account owns returns BucketAlreadyExists (409), and ListBuckets lists only the caller's own buckets.
  • Replication: every new object version in a replication source is replicated the way S3 does it, whichever path wrote it: PutObject, POST Object, CopyObject, CompleteMultipartUpload (the replica keeps the multipart ETag, part layout and checksum) and objects other services deliver into the bucket. A rule selects objects by its Filter (key prefix, tags, or both under And; an empty filter selects everything) or, for a V1 rule without a Filter, by its rule-level Prefix, which GetBucketReplication returns as sent. When several enabled rules select one object for the same destination bucket, only the one with the highest Priority applies, so a write produces one replica per destination, never duplicates; rules to different destinations all apply. The winning rule is chosen by filter and priority first, and its own settings (the SSE-KMS opt-in, delete marker replication) then decide whether the object or marker goes to that destination. A replica that cannot be stored marks the source FAILED without failing the source write. PutBucketReplication rejects two rules with the same Priority, a tag-based rule with delete marker replication enabled, and a rule opting in to SseKmsEncryptedObjects without a ReplicaKmsKeyID (InvalidRequest); a V2 rule sent without Priority or DeleteMarkerReplication gets an unused priority and delete marker replication disabled. A replica keeps the source version's id, metadata, tags, checksum and Object Lock retention, takes the destination's StorageClass (else the source's), and reports REPLICA; the source version reports COMPLETED, or FAILED when a destination could not take it (missing, not versioning-enabled, or refusing another account's replication role). As on AWS, a version without a real id is never replicated: replication copies only versions written after it was configured, and versioning cannot be suspended on a source (an object that predates versioning stays in the source as the null version). Replicas are not replicated again, S3 Glacier Flexible Retrieval and Deep Archive objects are skipped, and SSE-KMS objects are replicated only by rules that opt in with SourceSelectionCriteria SseKmsEncryptedObjects, the replica re-encrypted under the destination's ReplicaKmsKeyID. A delete marker created by DeleteObject / DeleteObjects is replicated, with the same version id, by a rule with DeleteMarkerReplication enabled (or a V1 rule) whose prefix matches; tag-based rules never replicate delete markers, and deleting a specific version is never replicated. Replicating into another account's bucket needs that bucket's policy to allow the replication role (s3:ReplicateObject, plus s3:ObjectOwnerOverrideToBucketOwner with ownership translation, s3:ReplicateDelete for delete markers); like other cross-account checks this is logged under --iam soft and enforced under --iam strict. A Destination.Account that does not own the destination bucket fails the replication, and AccessControlTranslation Owner=Destination (or a destination enforcing BucketOwnerEnforced) hands the replica to the destination bucket owner. Replication finishes with the write, so the source never shows PENDING.
  • Permissions a configuring create needs — CreateBucket is authorized as s3:CreateBucket plus one action per setting the request configures, as on AWS: s3:TagResource for a CreateBucketConfiguration tag set, s3:PutBucketAcl for an ACL that grants anything beyond the bucket owner (private, and a create with no ACL, need only s3:CreateBucket), s3:PutBucketOwnershipControls for x-amz-object-ownership, and s3:PutBucketObjectLockConfiguration plus s3:PutBucketVersioning for x-amz-bucket-object-lock-enabled: true. A principal holding only s3:CreateBucket can still create a plain bucket. The s3:x-amz-acl and s3:x-amz-grant-* condition keys are populated from those headers, so a policy can pin down which ACL a create may ask for.
  • Bucket tags at create time — CreateBucket honors the Tags tag set inside CreateBucketConfiguration, so a bucket comes out of CreateBucket already tagged and GetBucketTagging returns the set without a follow-up PutBucketTagging. This is the path the AWS Terraform provider (6.x) uses for aws_s3_bucket's tags. Duplicate keys and aws:-prefixed keys are rejected with InvalidTag and no bucket is created, and the tags persist across a restart like any other bucket subresource. Under --iam, a tagged create is authorized as s3:CreateBucket and s3:TagResource, as on AWS — a grant of s3:CreateBucket alone still creates untagged buckets — and the tag set populates aws:RequestTag/<key> / aws:TagKeys so policies can condition on it.
  • CORS — PutBucketCors rules drive real browser CORS: the OPTIONS preflight is matched on origin, Access-Control-Request-Method and every header in Access-Control-Request-Headers before returning Access-Control-Allow-Origin/-Methods/-Headers and Access-Control-Max-Age, and an actual request carrying Origin gets Access-Control-Allow-Origin plus Access-Control-Expose-Headers. Actual requests are matched on method as well as origin, so a rule allowing only GET hands no allow-origin to a DELETE. Multipart operations are CORS-evaluated like any other request, so browser multipart upload works. Errors from S3 itself are evaluated too, so a 404 NoSuchKey fetched by an allowed origin carries Access-Control-Allow-Origin and reaches the caller as a real 404 rather than an opaque CORS failure. Denials raised before the request reaches S3 (SigV4 or IAM, under --verify-sigv4 / --iam) are not CORS-decorated, so under auth enforcement a rejected browser request still surfaces as an opaque failure. Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method keeps a cache from reusing one origin's response for a different origin. Every preflight response carries it, since a preflight's outcome always turns on Origin: 200 when the rules allow it, 403 AccessForbidden when they do not, and 400 BadRequest ("Insufficient information. Origin request header needed.") when the request omits Origin altogether. On the actual-request path it goes on every response evaluated against a bucket's CORS config, successes and errors alike, while a request without Origin is not CORS-evaluated and is served normally with neither header, as on S3. A concrete allowed origin also gets Access-Control-Allow-Credentials: true, so credentials: 'include' works. Note that an actual request declares no headers, so it can match an earlier, broader rule than the preflight did: if a * rule precedes the concrete one, the actual response carries Access-Control-Allow-Origin: * and no credentials, and the browser blocks a credentialed request that just passed preflight. That is S3's behavior too — order the concrete rule first. AllowedOrigin and AllowedHeader each take one * anywhere in the value (https://*.example.com, x-amz-*). ExposeHeader takes no wildcard at all, as on S3. A rule missing AllowedMethods or AllowedOrigins, carrying more than one wildcard in a value, or setting a non-numeric MaxAgeSeconds, is rejected at write time rather than silently matching nothing. CloudFormation's CorsConfiguration goes through the same validation, so a template cannot deploy a bucket whose preflights all fail.
  • Bucket and object ACLs — a canned x-amz-acl, the x-amz-grant-* headers and an AccessControlPolicy body are mutually exclusive, as on S3: naming an ACL two ways is InvalidRequest rather than one of them silently winning. Canned values are checked against the set S3 accepts for the target (bucket ACLs take private, public-read, public-read-write, authenticated-read, aws-exec-read and log-delivery-write; bucket-owner-read and bucket-owner-full-control are object-only), and a grant naming no resolvable grantee is rejected instead of being stored as an entry that matches nobody. emailAddress= grantees are kept as AmazonCustomerByEmail so they round-trip, but they are not resolved to a canonical user and so convey no access. A bucket whose ObjectOwnership is BucketOwnerEnforced rejects every ACL-setting call, CreateMultipartUpload and CopyObject included. The object-write paths (PutObject, CopyObject, CreateMultipartUpload) make one exception, as AWS does: a bucket-owner-full-control ACL is accepted, whether spelled x-amz-acl: bucket-owner-full-control or as the equivalent explicit grant (x-amz-grant-full-control naming the owner and nobody else). The exception is that ACL specifically, not any ACL that happens to resolve to owner-only grants: --acl private is still AccessControlListNotSupported there, since AWS accepts only a request that specifies no ACL at all or one that specifies bucket-owner-full-control. PutObjectAcl and PutBucketAcl reject every ACL on such a bucket. CopyObject honors x-amz-acl and the x-amz-grant-* headers for the destination object under the same rules; a copy is a new object, so an unspecified ACL is the default owner grant rather than whatever the source carried.
  • Object Lock — legal hold, retention modes. Lock headers apply to objects created by PutObject, CopyObject (a copy never inherits the source's lock) and multipart upload, and the bucket's default retention applies to every new version that carries no retention of its own -- those, replicas without a source lock, and objects other services deliver -- measured from when the version is written; Years defaults are calendar years. Lock headers are validated as on AWS (mode and retain-until date together, a future ISO 8601 date, GOVERNANCE/COMPLIANCE, legal hold ON/OFF), and PutObject with lock headers needs Content-MD5 or an x-amz-checksum-* header. PutObjectRetention cannot shorten, change the mode of, or clear an unexpired retention: always refused under COMPLIANCE, and under GOVERNANCE unless x-amz-bypass-governance-retention: true is sent. With IAM enforcement on, the bypass header needs s3:BypassGovernanceRetention, and lock headers on a write need s3:PutObjectRetention / s3:PutObjectLegalHold. A bucket created with x-amz-bucket-object-lock-enabled keeps its lock configuration (and its canned ACL and object-ownership rule) across a restart in persistent mode, so retention stays enforced.
  • Website hosting — index/error documents, redirect rules
  • Access Points — full control plane (CreateAccessPoint, GetAccessPoint, DeleteAccessPoint, ListAccessPoints) via the s3-control host prefix; data plane traffic to s3-accesspoint.<region> resolves the alias to its underlying bucket so standard S3 operations work unchanged.
  • S3 Control resource tagging: TagResource, UntagResource and ListTagsForResource on /v20180820/tags/{resourceArn} accept general purpose bucket ARNs (arn:aws:s3:::<bucket>, sharing the tag set with PutBucketTagging / GetBucketTagging) and access point ARNs (their own tag set). Terraform's aws_s3_bucket reads bucket tags through this API.
  • S3 Select: real SelectObjectContent over CSV/JSON via EventStream framing (Records, Stats, End messages). InputSerialization.CompressionType GZIP and BZIP2 objects are decompressed before evaluation; an object that does not match the declared compression fails with InvalidCompressionFormat.
  • Object Lambda — WriteGetObjectResponse actually stores the transformed body + metadata against the original request token; the next GetObject on the access point returns the transformed payload.
  • Public Access Block — IgnorePublicAcls is enforced on GetObject; public-read ACL grants are ignored when the bucket-level block is set. BlockPublicAcls is enforced when the ACL is written, on the object-write paths as well as PutObjectAcl / PutBucketAcl: put-object --acl public-read against a bucket with BlockPublicAcls is AccessDenied, as on AWS, rather than quietly storing the AllUsers grant the block exists to prevent.
  • Anonymous access — unsigned GET/HEAD requests (GET /bucket/key with no SigV4) reach S3 directly, so you can serve static assets / CDN origins straight from a bucket. In the default (no-IAM) mode they're served permissively; under --iam soft|strict an anonymous request is authorized only when a bucket policy grants the action to Principal:"*" or a public-read ACL (AllUsers) is set, denied otherwise.
  • ACL ownership modes — BucketOwnerEnforced disables ACL WRITES: PutBucketAcl and PutObjectAcl are rejected outright. Reads are not reinterpreted: GetBucketAcl / GetObjectAcl return whatever grants are stored, so an ACL set before the mode was turned on is still reported (AWS reports owner-only there). The object-write paths keep the one exception AWS documents, described under Bucket and object ACLs above: a request specifying no ACL, or bucket-owner-full-control (canned or as the equivalent explicit grant to the bucket owner), is accepted.
  • Service Quotas enforcement (opt-in): with "General purpose buckets" (L-DC2B2D3D) enforced in Service Quotas, CreateBucket is refused with TooManyBuckets once the account owns that many buckets across all regions (a global quota: one applied value per account, whichever region it is read or raised in). Off by default

Protocol

REST. Path-based routing (/bucket/key), HTTP method + query string for actions, XML responses.

Introspection

  • GET /_fakecloud/s3/notifications — list all S3 notification events recorded during the test
  • POST /_fakecloud/s3/lifecycle-processor/tick — run one lifecycle processing tick
  • GET /_fakecloud/s3/access-points — list every S3 access point registered across all accounts (name, alias, bucket, network origin, VPC config, creation timestamp)
  • GET /_fakecloud/s3/object-lambda-responses — list bodies stored by WriteGetObjectResponse calls (S3 Object Lambda). Body is returned base64-encoded under bodyBase64, with requestToken, requestRoute, statusCode, contentType, errorMessage, and metadata

Cross-service delivery

  • S3 -> SNS / SQS / Lambda / EventBridge — Bucket notifications on object create/delete
  • Other services -> S3 -- objects written into a bucket by another service (CloudWatch Logs export tasks and deliveries, Firehose, ELB access logs, Athena query and notebook results, DynamoDB exports, ElastiCache snapshot exports, ACM PCA audit reports, the SES receipt-rule S3 action, the RDS S3 export bridge, CodeBuild artifacts, server access logs, inventory reports and replicas) are ordinary new object versions, as on AWS, and go through the same write path as PutObject: a versioning-enabled bucket gives each one a version id and keeps the history, a suspended bucket stores it as the null version, an Object Lock bucket applies its default retention (a replica keeps its source version's id and any retention that version carries; one without gets the destination's default), and the object is written through to disk in persistent mode. An object the producer did not encrypt itself takes the bucket's default encryption (a producer that names its own, such as a DynamoDB export's S3SseAlgorithm, overrides it): SSE-S3 is recorded as AES256, and SSE-KMS encrypts the body under the configured KMSMasterKeyID (the AWS-managed aws/s3 key when none is set), so HeadObject reports aws:kms with the key ARN and GetObject decrypts it; a delivery that needs SSE-KMS with no KMS available fails rather than storing plaintext labelled aws:kms. The bucket's event notifications fire s3:ObjectCreated:Put (the PUT the producing service makes), with the receiving account as the requester, and a replication source replicates the object (AWS: "If Amazon S3 delivers logs to a bucket that has replication enabled, it replicates the log objects"). A replica written by replication itself is not replicated further. A write the disk store rejects is reported to the producing service (the export job or calculation fails) rather than kept only in memory.

Gotchas

  • In persistent mode, object bodies stream to disk with a bounded LRU cache (--s3-cache-size, default 256 MiB). Objects larger than cache-size / 2 bypass the cache.
  • The /_fakecloud/s3/notifications introspection buffer is intentionally not persisted across restarts.
  • In persistent mode, CreateBucket clears whatever the store still holds under that name, so a re-created bucket is genuinely empty rather than inheriting the previous incarnation's objects on the next load. That matters after /_fakecloud/reset, which clears in-memory state and deliberately leaves the store alone. The exception is a bucket the loader could not read -- either its objects (a corrupt object sidecar, a missing multipart part body) or its own stored configuration (an unparseable tags.toml, acl.toml or inventory.toml): it is skipped at load with a warning, so it is absent from ListBuckets while its data sits intact on disk, and CreateBucket refuses the name with BucketAlreadyExists rather than becoming the thing that destroys it. Both ways out are in the data path, where an operator with an unreadable store already is: repair the one bad file and restart to get the bucket back, or remove the directory to free the name -- that frees it immediately, with no restart needed, and a later CreateBucket for it succeeds. There is deliberately no API call that discards it: DeleteBucket could check neither emptiness (the objects are exactly what could not be read) nor ownership (the metadata carrying it is what failed), so it would be a verb that destroys retained data for any caller. It answers NoSuchBucket like any other name it cannot see. Every read agrees with that -- HeadBucket 404s and ListBuckets omits the name -- while the create refuses it, so a head-then-create script fails on the create rather than silently overwriting the data.
  • SigV4 signatures are parsed for request routing and only validated when --verify-sigv4 is set.

Source