In modern cloud-based applications, accessing resources like files from Object Storage Services (OSS) is a common task. Many applications use temporary credentials for accessing such services, such as Security Token Service (STS) tokens. These tokens, however, have a limited lifespan. In our specific use case, we are working with an application that needs to download files from an OSS bucket. The application uses STS tokens to authenticate and access the bucket. However, these tokens are short-lived, and once the token expires, any further attempts to download files from the OSS bucket fail.

The challenge arises when the application still needs to access files after the STS token has expired. This situation creates an operational roadblock, as the application must explicitly request a new set of STS credentials to continue interacting with the OSS bucket. While this behavior is expected, it can be cumbersome, especially if the application needs to perform continuous file downloads over an extended period.

Ideally, we wanted a seamless solution that would allow the application to continue functioning smoothly beyond the expiration of the STS token without requiring users to manually handle the token renewal process. The goal was to automate the renewal process transparently without altering the application's core logic or requiring extra code from the user.

Before diving into the solution, it's important to understand the foundational concepts that led to the problem and ultimately paved the way for the solution.

Security Token Service (STS): STS is a service provided by cloud providers (such as AWS, Azure, etc.) to issue temporary security credentials that are used to access resources. These temporary credentials typically consist of an access key, secret key, and a session token, and they come with an expiration time. When an application requests these credentials, it receives a short-lived token, and once the token expires, the application can no longer access the resources unless it requests new credentials.

Object Storage Services (OSS): Object Storage Services are scalable storage solutions that allow applications to store and retrieve data in the cloud. Examples include AWS S3, Google Cloud Storage, and Azure Blob Storage. These services often require authentication mechanisms to securely control access to the stored data. When using temporary credentials, the STS token is used to authenticate and authorize the application's requests to these services.

The Challenge of Expiration: Given the short-lived nature of STS credentials, one of the inherent challenges is token expiration. When a token expires, the application loses access to the resources it was using, leading to failures in subsequent operations (such as downloading files). If the application needs to perform multiple download operations over an extended period, it becomes necessary to handle token renewal.

While token renewal is possible, the process often requires manually requesting a new set of credentials, which can introduce friction in user experience and result in additional operational complexity.

The usual process of accessing a file in an OSS bucket is the following:

1. Get STS token by authenticating with the right role
2. Use the STS token to instantiate an OSS Bucket instance as follows:
   

                                       
                                           bucket = oss2.Bucket(credentials, endpoint, "my_favourite_bucket")
                                           for object in oss2.ObjectIterator(bucket):
                                               # Download the file 
                                               file_path = "my/favourite/path"
                                               bucket.get_object_to_file(object.key, file_path)
                                       
                                       

   Simple, is it not 😅? But there are problems! You see, the bucket instance held by the variable bucket in the snippet above is like a gate pass that we have access to the object remote storage(OSS in this case).We can access the remote storage only as long as the gate pass is valid, post that if we try to use the gate pass to try to download a file from the remote storage, we face permission issues! Now that womps! Ideally, as a consumer, I would want this code to work and not fail for reasons like expired STS credentials!
   

We have seen the problems of the above style of accessing objects from a remote store. Now, the above code snippet in terms of expressability is quite concise. All we need is a bucket instance, and a for loop to iterate through the objects in the remote storage, sweeet! Only if we had a way to renew the gate pass without letting the consumer take the burden of the same.

So as a consumer, the way I express to read through all the objects in a remote storage remains the same, plus I don't need to worry about an invalid bucket instance because of expired STS credentials! Isn't that convenient? This is the intent of the solution I will be presenting below

Here is a brief overview of the solution:

• We plan to extend the functionality of the ObjectIterator that oss2 library provides us
• We would call a special class called OSSClient, which is tasked with providing a bucket instance with a valid gate pass at all times