Spring Cloud AWS S3: File Storage and Download with Amazon S3

File storage is a fundamental requirement for nearly every application. Whether you're handling user uploads in a SaaS platform, storing logs for analytics, or serving static assets for a content management system, you need a reliable, scalable, and cost-effective solution. Amazon S3 is the gold standard for object storage, and Spring Cloud AWS provides first-class integration to make it painless from your Java applications.

But let's be honest: the official documentation glosses over the real-world gotchas. I've seen production outages caused by misconfigured credentials, silent failures from region mismatches, and memory blow-ups from streaming large files incorrectly. In this article, I'll share battle-tested patterns for uploading and downloading files with S3 using Spring Cloud AWS. We'll cover synchronous operations, multipart uploads for large files, and secure access via pre-signed URLs. I'll also walk through a real production incident where a simple timeout configuration saved a fintech startup from a costly outage.

By the end, you'll know not just how to use the API, but how to do it right in production. Let's get to it.

Setting Up Spring Cloud AWS with S3

First, add the necessary dependencies. I recommend using the latest stable version – as of 2024, that's spring-cloud-starter-aws 2.4.4. Don't use the old 1.x versions; they lack critical fixes.

In pom.xml:

``xml <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-aws</artifactId> <version>2.4.4</version> </dependency> ``

Then configure your AWS credentials and region. Never hardcode credentials! Use IAM roles in production, environment variables for local dev, or the default credential chain.

application.yml:

``yaml cloud: aws: region: static: us-east-1 credentials: access-key: ${AWS_ACCESS_KEY_ID} secret-key: ${AWS_SECRET_ACCESS_KEY} ``

Now you can inject the AmazonS3 client:

``java @Autowired private AmazonS3 amazonS3; ``

But wait – there's a trap. The default AmazonS3 client uses the AWS SDK v1. For new projects, consider using the AWS SDK v2 with the s3 module. Spring Cloud AWS doesn't yet support v2 natively, but you can manually configure it. I'll show both approaches.

import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class S3Config {

    @Bean
    public AmazonS3 amazonS3() {
        return AmazonS3ClientBuilder.standard()
                .withCredentials(new DefaultAWSCredentialsProviderChain())
                .withRegion(Regions.US_EAST_1)
                .build();
    }
}

Uploading Files to S3

Uploading a file is straightforward with putObject. But the devil is in the details. Here's a robust method that handles streams properly:

```java public String uploadFile(String bucketName, String key, InputStream inputStream, long contentLength, String contentType) { ObjectMetadata metadata = new ObjectMetadata(); metadata.setContentLength(contentLength); metadata.setContentType(contentType);

PutObjectRequest request = new PutObjectRequest(bucketName, key, inputStream, metadata); PutObjectResult result = amazonS3.putObject(request); return result.getETag(); } ```

Important: Always set Content-Length if you know it. If you don't, the SDK buffers the entire stream into memory, which can cause OOM for large files. For unknown length streams, use multipart upload.

For larger files (say > 100 MB), use multipart upload:

``java public String uploadLargeFile(String bucketName, String key, File file) { TransferManager transferManager = TransferManagerBuilder.standard() .withS3Client(amazonS3) .build(); Upload upload = transferManager.upload(bucketName, key, file); try { upload.waitForCompletion(); } catch (InterruptedException e) { Thread.currentThread().interrupt(); throw new RuntimeException("Upload interrupted", e); } return upload.getETag(); } ``

TransferManager handles multipart uploads automatically based on a size threshold (default 16 MB). This avoids memory issues and provides progress tracking.

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.PutObjectResult;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;
import com.amazonaws.services.s3.transfer.Upload;
import org.springframework.stereotype.Service;
import java.io.File;
import java.io.InputStream;

@Service
public class FileUploadService {

    private final AmazonS3 amazonS3;

    public FileUploadService(AmazonS3 amazonS3) {
        this.amazonS3 = amazonS3;
    }

    public String uploadFile(String bucketName, String key, InputStream inputStream, long contentLength, String contentType) {
        ObjectMetadata metadata = new ObjectMetadata();
        metadata.setContentLength(contentLength);
        metadata.setContentType(contentType);

        PutObjectRequest request = new PutObjectRequest(bucketName, key, inputStream, metadata);
        PutObjectResult result = amazonS3.putObject(request);
        return result.getETag();
    }

    public String uploadLargeFile(String bucketName, String key, File file) {
        TransferManager transferManager = TransferManagerBuilder.standard()
                .withS3Client(amazonS3)
                .build();
        Upload upload = transferManager.upload(bucketName, key, file);
        try {
            upload.waitForCompletion();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException("Upload interrupted", e);
        }
        return upload.getETag();
    }
}

Downloading Files from S3

Downloading is similarly straightforward. Use getObject to get an S3Object and read the content stream:

``java public S3ObjectInputStream downloadFile(String bucketName, String key) { S3Object s3Object = amazonS3.getObject(bucketName, key); return s3Object.getObjectContent(); } ``

But again, be careful with streams. The S3ObjectInputStream must be closed to avoid connection leaks. I always wrap it in a try-with-resources or ensure the caller closes it.

For large files, consider using GetObjectRequest with range headers to download in chunks:

``java public InputStream downloadFileRange(String bucketName, String key, long start, long end) { GetObjectRequest request = new GetObjectRequest(bucketName, key); request.setRange(start, end); S3Object s3Object = amazonS3.getObject(request); return s3Object.getObjectContent(); } ``

This is useful for resumable downloads or streaming large files to clients without buffering the entire object in memory.

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import org.springframework.stereotype.Service;
import java.io.InputStream;

@Service
public class FileDownloadService {

    private final AmazonS3 amazonS3;

    public FileDownloadService(AmazonS3 amazonS3) {
        this.amazonS3 = amazonS3;
    }

    public S3ObjectInputStream downloadFile(String bucketName, String key) {
        S3Object s3Object = amazonS3.getObject(bucketName, key);
        return s3Object.getObjectContent();
    }

    public InputStream downloadFileRange(String bucketName, String key, long start, long end) {
        GetObjectRequest request = new GetObjectRequest(bucketName, key);
        request.setRange(start, end);
        S3Object s3Object = amazonS3.getObject(request);
        return s3Object.getObjectContent();
    }
}

Generating Pre-Signed URLs for Secure Access

Pre-signed URLs are a game-changer. They allow you to grant temporary access to a specific S3 object without exposing your AWS credentials. This is perfect for file downloads in web applications.

Here's how to generate one:

``java public URL generatePresignedUrl(String bucketName, String key, int expirationInMinutes) { Date expiration = new Date(System.currentTimeMillis() + TimeUnit.MINUTES.toMillis(expirationInMinutes)); GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(bucketName, key) .withMethod(HttpMethod.GET) .withExpiration(expiration); return amazonS3.generatePresignedUrl(request); } ``

This URL can be used by anyone to download the object within the expiration period. You can also generate PUT pre-signed URLs to allow clients to upload directly to S3, bypassing your server entirely – great for large uploads.

Security Note: The URL includes your secret access key in a signed signature. Never log it or expose it in error messages. Also, set a short expiration (e.g., 5-15 minutes) to minimize risk.

import com.amazonaws.HttpMethod;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import org.springframework.stereotype.Service;
import java.net.URL;
import java.util.Date;
import java.util.concurrent.TimeUnit;

@Service
public class PresignedUrlService {

    private final AmazonS3 amazonS3;

    public PresignedUrlService(AmazonS3 amazonS3) {
        this.amazonS3 = amazonS3;
    }

    public URL generatePresignedUrl(String bucketName, String key, int expirationInMinutes) {
        Date expiration = new Date(System.currentTimeMillis() + TimeUnit.MINUTES.toMillis(expirationInMinutes));
        GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(bucketName, key)
                .withMethod(HttpMethod.GET)
                .withExpiration(expiration);
        return amazonS3.generatePresignedUrl(request);
    }
}

What the Official Docs Won't Tell You

The official Spring Cloud AWS documentation is decent, but it leaves out several hard-earned lessons. Here are the ones that have bitten me and my teams:

1. The Default Credential Chain is Your Friend, But Test It

The DefaultAWSCredentialsProviderChain tries environment variables, system properties, profile config, and instance profile. In production, use instance profiles (EC2) or task roles (ECS). But I've seen cases where the chain picks up an expired profile from ~/.aws/credentials and causes intermittent failures. Always test your credential resolution in the target environment.

2. Multipart Upload Cleanup

If a multipart upload fails, S3 retains the parts. They accumulate and cost money. Use TransferManager which automatically aborts incomplete uploads, but if you use the low-level API, you must handle AbortMultipartUpload yourself. Set a lifecycle policy on your bucket to expire incomplete multipart uploads after a few days.

3. S3 Consistency Model

S3 offers read-after-write consistency for PUTs of new objects, but eventual consistency for overwrites and DELETEs. If you update an object and immediately read it, you might get the old version. This is rare but can cause bugs in distributed systems. Use versioning or avoid overwrites if consistency is critical.

4. Pre-signed URLs and KMS Encryption

If your bucket uses AWS KMS for encryption, pre-signed URLs require the generator to have kms:GenerateDataKey permission. Also, the URL might not work if the requester doesn't have KMS decrypt permissions. Test this early.

5. Client Configuration Tuning

The default ClientConfiguration uses a connection timeout of 10 seconds and socket timeout of 50 seconds. For large file uploads, you'll need to increase these. Set them explicitly:

``java ClientConfiguration clientConfig = new ClientConfiguration(); clientConfig.setConnectionTimeout(30_000); clientConfig.setSocketTimeout(120_000); AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .withClientConfiguration(clientConfig) .build(); ``

Ignoring this caused a production incident I'll never forget.

import com.amazonaws.ClientConfiguration;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;

public class S3ClientWithTimeout {

    public static AmazonS3 createClient() {
        ClientConfiguration clientConfig = new ClientConfiguration();
        clientConfig.setConnectionTimeout(30_000); // 30 seconds
        clientConfig.setSocketTimeout(120_000); // 2 minutes
        return AmazonS3ClientBuilder.standard()
                .withClientConfiguration(clientConfig)
                .build();
    }
}

Error Handling and Retries

S3 operations can fail for many reasons: network issues, throttling, permissions, etc. The AWS SDK has a built-in retry mechanism, but it's not always sufficient. Here's my approach:

1. Use Exponential Backoff

The SDK retries up to 3 times by default with exponential backoff. That's usually enough, but for critical operations, consider implementing your own retry with a circuit breaker pattern using Spring Retry or Resilience4j.

2. Handle Specific Exceptions

• AmazonS3Exception: General S3 error. Check the error code.
• AmazonServiceException: Service-side error (e.g., 503). Retry.
• AmazonClientException: Client-side error (e.g., connection timeout). Retry.
• SdkClientException: SDK-level error.

3. Log and Monitor

Always log S3 errors with enough context (bucket, key, error code). Set up CloudWatch alarms for high error rates.

Here's a retry wrapper using Spring Retry:

``java @Retryable(value = {AmazonServiceException.class}, maxAttempts = 3, backoff = @Backoff(delay = 1000, multiplier = 2)) public String uploadFileWithRetry(String bucketName, String key, InputStream inputStream, long contentLength, String contentType) { return uploadFile(bucketName, key, inputStream, contentLength, contentType); } ``

But be careful: retrying idempotent operations (like GET) is safe, but for PUTs, ensure your upload is idempotent or handle duplicates.

import com.amazonaws.AmazonServiceException;
import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Retryable;
import org.springframework.stereotype.Service;
import java.io.InputStream;

@Service
public class RetryableUploadService {

    private final FileUploadService fileUploadService;

    public RetryableUploadService(FileUploadService fileUploadService) {
        this.fileUploadService = fileUploadService;
    }

    @Retryable(value = {AmazonServiceException.class}, maxAttempts = 3, backoff = @Backoff(delay = 1000, multiplier = 2))
    public String uploadFileWithRetry(String bucketName, String key, InputStream inputStream, long contentLength, String contentType) {
        return fileUploadService.uploadFile(bucketName, key, inputStream, contentLength, contentType);
    }
}