Setup
This page discusses customer-managed encryption keys and how they are used inCloud Storage. For other encryption options, see Data Encryption Options.
Overview
If you need more control over key operations than what thestandard Cloud Storage encryption allows, you can usecustomer-managed encryption keys (CMEKs). These keys are created and managedusing Cloud Key Management Service (Cloud KMS), and you store the keys as softwarekeys, in an HSM cluster, or externally. You can use CMEKs onindividual objects, or configure your bucket to use a key by default on allnew objects added to a bucket.
When using a CMEK, an object is encrypted with the key by Cloud Storageat the time it's stored in a bucket, and the object is automatically decryptedby Cloud Storage when the object is served to requesters.
You can create CMEKs directly, or you can use Cloud KMS Autokey(Preview) to create these keys on yourbehalf. For more information, seeAutokey overview.
When is the key used?
When you apply a CMEK to an object, Cloud Storage uses the key whenencrypting:
- The object's data.
- The object's CRC32C checksum.
- The object's MD5 hash.
Cloud Storage uses standard server-side keys to encrypt theremaining metadata for the object, including the object's name. Thus, ifyou have sufficient permission, you can perform actions such as readingmost metadata, listing objects, and deleting objects even after you've disabledor destroyed the associated CMEK.
Service agents
Each project has a special Cloud Storage service account called aservice agent that performs encryption and decryption with CMEKs. Once yougive the service agent access to an encryption key, that service agentencrypts:
- Objects added to a bucket that uses the key as the default key.
- Specific objects that you indicate should be encrypted with that key.
When adding or rewriting an object in Cloud Storage, if you have both adefault key set on your bucket and a specific key included in your request,Cloud Storage uses the specific key to encrypt the object.
When a requester wants to read an object encrypted with a CMEK, they simplyaccess the object as they normally would. During such a request, the serviceagent automatically decrypts the requested object as long as:
- The service agent still has permission to decrypt using the key.
- You have not disabled or destroyed the key.
If one of these conditions is not met, the service agent does not decryptthe data, and the request fails.
Restrictions
The following restrictions apply when using CMEKs:
You cannot encrypt an object with a CMEK by updating the object's metadata.Include the key as part of a rewrite of the object instead.
gcloud storageuses theobjects updatecommand to set encryption keyson objects, but the command rewrites the object as part of the request.
You must create the Cloud KMS key ring in the same location as thedata you intend to encrypt. For example, if your bucket is located in
US-EAST1, any key ring used for encrypting objects in that bucket mustalso be created inUS-EAST1.For most dual-regions, you must create the Cloud KMS keyring in the associated multi-region. For example, if your bucket islocated in the pair
US-EAST1,US-WEST1, any key ring used forencrypting objects in that bucket must be created in theUSmulti-region.For the
ASIA1,EUR4, andNAM4predefined dual-regions, you mustcreate the key ring in the same predefined dual-region.For available Cloud KMS locations, seeCloud KMS locations.
Cloud KMS encryption and decryption rates are subject to aquota.
The CRC32C checksum and MD5 hash of objects encrypted with CMEKs are notreturned when listing objects with the JSON API.
- When appropriate, some tools, such as
gcloud storage, perform anadditional metadataGETrequest on each object encrypted with aCMEK in order to retrieve the CRC32C and MD5 information. These additionalrequests can make listing substantially slower than listing objectsencrypted with standard Cloud Storage encryption.
- When appropriate, some tools, such as
Only symmetric encryption keys can be used as CMEKs.
Relation to customer-supplied encryption keys
In addition to customer-managed encryption, Cloud Storage offersCustomer-Supplied Encryption Keys as a way of controlling your dataencryption. You can encrypt different objects in a single bucket with differentencryption methods, but note that:
A single object can only be encrypted by one of these methods at a time.
If you have a default CMEK set for your bucket and specify acustomer-supplied key in a request, Cloud Storage uses thecustomer-supplied key to encrypt the object.
Key management
This section discusses considerations when rotating keys, replacing keys, anddisabling or destroying key versions.
Key rotation
Cloud KMS supports both automatic and manual key rotation to anew version. After rotating a key, Cloud Storage uses the newversion for all operations that encrypt using the key, such as:
Object uploads when the destination bucket uses the key as its defaultencryption key.
Object upload, copy, and rewrite operations that specifically use thekey in the operation.
Previous versions of the key are not disabled or destroyed, soCloud Storage can still decrypt existing objects that were previouslyencrypted using those versions.
Key replacement
Use the following guidelines when replacing the key you use to encryptCloud Storage objects with a new key:
Check your buckets to see which use the key as their defaultencryption key. For these buckets, replace the old key with a new key.
This ensures that all objects written to the bucket use the new keygoing forward.
Inspect your source code to understand which requests use the key in ongoingoperations, such as setting bucket configurations and uploading,copying, or rewriting objects. Update these instances to use the new key.
Check for objects, in all of your buckets, encrypted with the old key.Use the Rewrite Object method to re-encrypt each object with the newkey.
Disable all versions of the old key. After disabling old key versions,monitor client and service logs for operations that fail due to a versionbecoming unavailable.
Disabling or destroying a key version
When you disable or destroy a specific key version, you cannotdecrypt any object that is currently encrypted with that key version.
For example, you cannot download, copy, or rewrite the object, andattempting to do so results in an error.
If you disable a key version, you can re-enable it. Once re-enabled,you can access objects that were encrypted by that key version.
If you destroy a key version, downloads of objects encrypted with thatversion are never possible again.
Before disabling or destroying a key version, you should identify allobjects, in all buckets, that were encrypted using the specific key version.Once identified, use the Rewrite Object method to re-encrypt eachobject using a new key version, a whole new key, or server-side keys.
When you disable or destroy the primary version of a key, you cannot usethe key for encryption until you have a new primary version. For example,without a primary version:
You cannot specify the key as part of an object upload, copy, or rewrite.
You cannot upload, copy, or rewrite objects to a bucket that has the keyset as its default encryption key unless you specify a different,valid key as part of the operation.
Once you have a primary version for your key, operations that use the key toencrypt objects succeed.
Before disabling or destroying a key version that is the primary version ofthe key, you should first stop using it as the primary version. You can doso by either:
- Replacing it with a new primary version, typically byperforming a key rotation.
- Removing instances where you use the key for encryption. When you do so,Cloud Storage uses server-side keys for encryptioninstead.
Key versions and locked objects
If a key version encrypts an object that is locked, either because the object isstored in a bucket with a locked retention policy or because the objecthas its own locked retention configuration, the key version can only bedestroyed if the following conditions are met:
- The encrypted object's retention expiration time must be in the past.
- The encrypted object must not have any object holds placed on it.
Once all relevant objects have met these conditions, it's possible to destroythe key version, even without deleting the objects. If you do so, affectedobject data becomes permanently inaccessible.
What's next
- Set CMEKs on your Cloud Storage buckets and objects.
- Learn more about encryption in Cloud Storage.
- Learn more about Cloud KMS.
