AWS tool boto3 usage notes

Get started with AWS quickly with [boto3 , the] [AWS] SDK for Python . Boto3 enables you to easily integrate your Python application, library, or script with AWS services, including Amazon S3, Amazon EC2, Amazon DynamoDB, and more.

Upload and download data

class CephBOTO3:
    def __init__(self):
        self.access_key = [your access key]
        self.secret_access_key = [your secret access key]
        self.bucket = [your bucket]
        self.endpoint = [your endpoint]
        self.service_name = [your service_name]

        self.session = boto3.Session(aws_access_key_id=self.access_key, aws_secret_access_key=self.secret_access_key)
        self.client = self.session.client(service_name=self.service_name, use_ssl=False, endpoint_url=self.endpoint)
        self.resource = boto3.resource(self.service_name, endpoint_url=self.endpoint,
                                       aws_access_key_id=self.access_key,
                                       aws_secret_access_key=self.secret_access_key)

    def upload(self, directory):
        file_paths = []
        for root, dirs, filenames in os.walk(directory):
            for name in filenames:
                file_path = os.path.join(root, name)
                file_paths.append((name, file_path))

        for file_name, file_path in file_paths:
            print("Uploading {}".format(file_name))
            self.resource.meta.client.upload_file(file_path, self.bucket, file_name)

    def download(self, directory):
        bucket = self.resource.Bucket(self.bucket)
        keys = self.get_buckets()
        for key in keys:
            file_name = directory + '/' + key
            bucket.download_file(Filename=file_name, Key=key)

Progress bar for upload and download

Sometimes the uploaded or downloaded data is large, the user needs a progress bar, and only needs to define a callback class

class ProgressPercentage(object):
    def __init__(self, file_name, file_path):
        self.file_name = file_name
        self.file_path = file_path
        self._size = float(os.path.getsize(file_path))
        self._seen_so_far = 0
        self._lock = threading.Lock()

    def __call__(self, bytes_amount):
        # To simplify, assume this is hooked up to a single filename
        with self._lock:
            self._seen_so_far += bytes_amount
            percentage = (self._seen_so_far / self._size) * 100
            sys.stdout.flush()

Then add it to the upload and download function as a callback

key = "A/B/C/{}".format(file_name)
bucket.upload_file(Filename=file_path, Key=key, Callback=ProgressPercentage(file_name, file_path))

Store data in a remote specified directory

Use key to specify the directory when passing parameters

key = "A/B/C/{}".format(file_name)
bucket.upload_file(Filename=file_path, Key=key, Callback=ProgressPercentage(file_name, file_path))

After the above code runs, the file will be stored in the remote A/B/C directory

references

Leave a Comment

Your email address will not be published. Required fields are marked *