How to transfer Data from AWS S3 bucket to GCP Google cloud storage

2 min readMar 1, 2024

Assumptions

You know what you are doing !

All we can advise is — DO NOT copy data that you shouldn’t be for — legal, commercial, regulatory or compliance reasons.

2. You know AWS S3 and Google Cloud Storage. If you dont yet, this blog if not for you … yet. Its good to get some understanding of how these work, before proceeding further.

Prerequisites

Create AWS access keys and secret with s3 bucket access
Create a bucket Google cloud storage to store s3 data

Step#1 — Install GCloud CLI

Install GCloud CLI on a OS of your choice (which works, refer internet). We would be going with Ubuntu 20.04 for the purpose of this blog.

References: https://cloud.google.com/sdk/docs/install

# Debian / Ubuntu
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates gnupg curl sudo -y

Import the Google Cloud public key.

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o
/usr/share/keyrings/cloud.google.gpg

Add the gcloud CLI distribution URI as a package source.

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt
cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

Update and install the gcloud CLI

sudo apt-get update && sudo apt-get install google-cloud-cli -y

Finally — Init the gcloud CLI

# Update the "projectid", "region" of Google Cloud when requested
gcloud init

Step#2 — Setup AWS S3

Edit ~/.boto file

nano ~/.boto

Fill in the aws_access_key_id and aws_secret_access_key settings with your S3 credentials and uncomment the variables.

# To add aws credentials ("s3://" URIs), edit and uncomment the # following two lines:
aws_access_key_id = <ACCESS_KEY>
aws_secret_access_key = <SECRET_ACCESS_KEY>

You are ready to copy !!

Step#3 — Initiate, Launch

#Syntax - Logic: <gcloudCLI> <storage svc> copy <source> <destination>
gcloud storage cp -R s3://s3bucketname gs://googlecloudbucketname
gcloud storage cp -R s3://server-backup gs://server-backup

All is well !!

Author: Mubin Sheikh, SRE, BridgeApps Ltd. | March 01 2024