前言
为了节约时间，可以直接看文章最后
四、最佳实践部分，前面内容是方便已经进入泥潭正面对各种问题的人能搜到本文，找到解决的办法。
官网文档例子是上传后可下载到服务端成一个图片，然后自己组装存放在自己服务器文件夹下图片的访问URL，给用户展现，而不是一个可以供嵌入浏览器页面的图片URL，这样会浪费我们服务器空间并且不能充分利用AWS的带宽。多数情况是：AWS的S3做云存储，把文件上传上去，在数据库中记录对应的URL，HTML页面直接使用这个URL显示图片。
上传图片后，发现直接访问URL，提示没有权限预览，错误信息You are not authorized to perform this operation。这个问题通过我们的AWS后端支持沟通了10个来回，最后发现可以上传成功并且在AWS控制台可以正常预览图片，但我们代码生成的URL不可以浏览的原因是：在中国区需要ICP备案。所以要使用S3和EC2先联系AWS售后支持进行ICP备案。
<Error>
<Code>UnauthorizedAccess</Code>
<Message>You are not authorized to perform this operation</Message>
<RequestId>FF22D40A1C1F2376</RequestId>
<HostId>
NfNHEkB/e7cC6BkeeESk3Wuh8sBAafXWQYPUWVBFKkaL9yXG+Oc09Kj/j5yGVdVaA8YzTw/kuWw=
</HostId>
</Error>
当时尝试的解决方案：
一开始以为是权限不够，但是，这个开发环境我们已经把S3的权限放的很大。配置位置如下：
存储桶策略
但是在控制台发现一个可以预览的URL，发现参数有签名信息，这正是解决上面错误提示
UnauthorizedAccess的钥匙，于是找预签名URL的方法。由于官方文档没提及此功能是在github的示例代码翻到的此功能，假如他有说明是不是大家可以节约一天的研究时间？差评！
一、官方下载图片的代码
效果是调用此功能，在浏览器提示文件下载成功，可以在下载后的查看图片。
package com.example.s3;
// snippet-start:[s3.java2.getobjectdata.import]
import software.amazon.awssdk.core.ResponseBytes;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
// snippet-end:[s3.java2.getobjectdata.import]
public class GetObjectData {
public static void main(String[] args) {
if (args.length < 3) {
System.out.println("Please specify a bucket name, a key name that represents a PDF file (ie, book.pdf), and a path (ie, C:\\AWS\\AdobePDF.pdf)");
System.exit(1);
}
String bucketName = args[0];
String keyName = args[1];
String path = args[2];
Region region = Region.US_WEST_2;
S3Client s3 = S3Client.builder()
.region(region)
.build();
getObjectBytes(s3,bucketName,keyName, path);
}
// snippet-start:[s3.java2.getobjectdata.main]
public static void getObjectBytes (S3Client s3, String bucketName, String keyName, String path ) {
try {
// create a GetObjectRequest instance
GetObjectRequest objectRequest = GetObjectRequest
.builder()
.key(keyName)
.bucket(bucketName)
.build();
// get the byte[] this AWS S3 object
ResponseBytes<GetObjectResponse> objectBytes = s3.getObjectAsBytes(objectRequest);
byte[] data = objectBytes.asByteArray();
//Write the data to a local file
File myFile = new File(path );
OutputStream os = new FileOutputStream(myFile);
os.write(data);
System.out.println("Successfully obtained bytes from an S3 object");
// Close the file
os.close();
} catch (IOException ex) {
ex.printStackTrace();
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
// snippet-end:[s3.java2.getobjectdata.main]
}
}
二、坊间的一个解决方案
此方案代码结构符合我们预期，但是2014年的代码，已经略显陈旧，最新的aws-java-sdk已经不支持这种写法。
public static String uploadToS3(File tempFile, String remoteFileName) throws IOException {
PropertiesUtil propertiesUtil = new PropertiesUtil("s3.properties");
//首先创建一个s3的客户端操作对象（需要amazon提供的密钥）
AmazonS3 s3 = new AmazonS3Client(
new BasicAWSCredentials(propertiesUtil.getKeyValue(Consts.S3_ACCESS_KEY),
propertiesUtil.getKeyValue(Consts.S3_SCERET_KEY)));
Region usWest2 = Region.getRegion(Regions.US_WEST_2);
s3.setRegion(usWest2);
//设置bucket,key
String bucketName = Consts.S3_BUCKET_NAME;
String key = UUID.randomUUID() + ".apk";
try {
//验证名称为bucketName的bucket是否存在，不存在则创建
if (!checkBucketExists(s3, bucketName)) {
s3.createBucket(bucketName);
}
//上传文件
s3.putObject(new PutObjectRequest(bucketName, key, tempFile));
S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));
//获取一个request
GeneratePresignedUrlRequest urlRequest = new GeneratePresignedUrlRequest(
bucketName, key);
Date expirationDate = null;
try {
expirationDate = new SimpleDateFormat("yyyy-MM-dd").parse("2020-12-31");
} catch (Exception e) {
e.printStackTrace();
}
//设置过期时间
urlRequest.setExpiration(expirationDate);
//生成公用的url
URL url = s3.generatePresignedUrl(urlRequest);
System.out.println("=========URL=================" + url + "============URL=============");
if (url == null) {
throw new OperateFailureException("can't get s3 file url!");
}
return url.toString();
} catch (AmazonServiceException ase) {
ase.printStackTrace();
logger.info("====================================AWS S3 UPLOAD ERROR START======================================");
logger.info("Caught an AmazonServiceException, which means your request made it "
+ "to Amazon S3, but was rejected with an error response for some reason.");
logger.info("Caught an AmazonServiceException, which means your request made it "
+ "to Amazon S3, but was rejected with an error response for some reason.");
logger.info("Error Message: " + ase.getMessage());
logger.info("HTTP Status Code: " + ase.getStatusCode());
logger.info("AWS Error Code: " + ase.getErrorCode());
logger.info("Error Type: " + ase.getErrorType());
logger.info("Request ID: " + ase.getRequestId());
logger.info(ase.getMessage(), ase);
logger.info("====================================AWS S3 UPLOAD ERROR END======================================");
throw new OperateFailureException("error occurs during upload to s3!");
} catch (AmazonClientException ace) {
logger.info("====================================AWS S3 UPLOAD ERROR START======================================");
logger.info("Caught an AmazonClientException, which means the client encountered "
+ "a serious internal problem while trying to communicate with S3, "
+ "such as not being able to access the network.");
logger.info("Error Message: " + ace.getMessage());
logger.info("====================================AWS S3 UPLOAD ERROR END======================================");
throw new OperateFailureException("error occurs during upload to s3!");
}
}
/**
* 验证s3上是否存在名称为bucketName的Bucket
* @param s3
* @param bucketName
* @return
*/
public static boolean checkBucketExists (AmazonS3 s3, String bucketName) {
List<Bucket> buckets = s3.listBuckets();
for (Bucket bucket : buckets) {
if (Objects.equals(bucket.getName(), bucketName)) {
return true;
}
}
return false;
}
三、最新版本的上传图片AWS官方示例
不是完全符合我们的要求，没有预览图片的代码逻辑。
package com.example.s3;
// snippet-start:[presigned.java2.generatepresignedurl.import]
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.time.Duration;
import software.amazon.awssdk.services.s3.model.ObjectCannedACL;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.PutObjectPresignRequest;
// snippet-end:[presigned.java2.generatepresignedurl.import]
public class GeneratePresignedUrlAndUploadObject {
public static void main(String[] args) {
if (args.length < 2) {
System.out.println("Please specify a bucket name and a key name that represents a text file");
System.exit(1);
}
String bucketName = args[0];
String keyName = args[1];
// Create a S3Presigner by using the default AWS Region and credentials
S3Presigner presigner = S3Presigner.create();
signBucket(presigner, bucketName, keyName);
}
// snippet-start:[presigned.java2.generatepresignedurl.main]
public static void signBucket(S3Presigner presigner, String bucketName, String keyName) {
try {
// Use a PutObjectRequest to set additional values
PutObjectRequest objectRequest = PutObjectRequest.builder()
.bucket(bucketName)
.key(keyName)
.contentType("text/plain")
.build();
PutObjectPresignRequest presignRequest = PutObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(10))
.putObjectRequest(objectRequest)
.build();
PresignedPutObjectRequest presignedRequest = presigner.presignPutObject(presignRequest);
System.out.println("Pre-signed URL to upload a file to: " +
presignedRequest.url());
System.out.println("Which HTTP method needs to be used when uploading a file: " +
presignedRequest.httpRequest().method());
// Upload content to the bucket by using this URL
URL url = presignedRequest.url();
// Create the connection and use it to upload the new object by using the pre-signed URL
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type","text/plain");
connection.setRequestMethod("PUT");
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
out.write("This text uploaded as an object via presigned URL.");
out.close();
connection.getResponseCode();
System.out.println("HTTP response code: " + connection.getResponseCode());
/*
* It's recommended that you close the S3Presigner when it is done being used, because some credential
* providers (e.g. if your AWS profile is configured to assume an STS role) require system resources
* that need to be freed. If you are using one S3Presigner per application (as recommended), this
* usually isn't needed
*/
presigner.close();
} catch (S3Exception e) {
e.getStackTrace();
} catch (IOException e) {
e.getStackTrace();
}
// snippet-end:[presigned.java2.generatepresignedurl.main]
}
}
四、最佳实践
前提：
In China,要使用S3和EC2先联系AWS售后支持进行ICP备案。
- 依据他官方文档进行桶各种权限的设置，其实默认创建个S3存储桶bucket，不进行任何权限设定即可。
- 新建IAM的S3用户，生成Access key和seceret Key供后续代码调用。新建用户
创建访问的密钥，点击创建访问密钥按钮：image.png
- 引入依赖：
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>aws-sdk-java</artifactId>
<version>2.14.26</version>
</dependency>
- 核心方法
以下方法是跑通的一个逻辑，有2个方法，一个是上传图片。另外getPresignedUrl是生成预览图片的带有签名的URL的方法，使用输出的这个URL即可直接预览图片。
package com.erbadagang.aws;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.model.PutObjectPresignRequest;
import software.amazon.awssdk.utils.IoUtils;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.time.Duration;
public class PresignerAndUploader {
public static void main(String[] args) {
String bucketName = "dev-npay-s3";
String objectKey = "head_pic.jpg";
Region region = Region.CN_NORTHWEST_1;
String accessKeyId = "替换成你的access key";
String secretAccessKey = "替换成你的secret access key";
String presignUrlDurationMinutes = "5000";
long presignUrlDurationMinutesLong = Long.valueOf(presignUrlDurationMinutes);
String objectLocalPath = "D:\\backup\\weixin_file\\WeChat Files\\All Users\\116f8f343203cef0107f2ca0f4fc0b03.jpg";
String contentType = "image/jpg";
/*
* S3Presigner and credentials.
*/
AwsBasicCredentials awsBasicCredentials = AwsBasicCredentials.create(accessKeyId, secretAccessKey);
S3Presigner s3Presigner = S3Presigner.builder().region(region)
.credentialsProvider(StaticCredentialsProvider.create(awsBasicCredentials)).build();
/*
* Generate presigned URL.
*/
URL presignedUrl = generatePresignedUrl(s3Presigner, bucketName, objectKey, region, accessKeyId, secretAccessKey,
presignUrlDurationMinutesLong);
/*
* Upload object to S3.
*/
uploadObject(presignedUrl, objectLocalPath, contentType);
getPresignedUrl(s3Presigner, bucketName, objectKey);
}
private static URL generatePresignedUrl(S3Presigner s3Presigner, String bucketName, String objectKey, Region region, String accessKeyId,
String secretAccessKey, long presignUrlDurationMinutesLong) {
URL presignedUrl = null;
// PutObjectRequest
PutObjectRequest putObjectRequest = PutObjectRequest.builder().bucket(bucketName).key(objectKey).build();
// PutObjectPresignRequest
PutObjectPresignRequest putObjectPresignRequest = PutObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(presignUrlDurationMinutesLong)).putObjectRequest(putObjectRequest)
.build();
// PresignedPutObjectRequest
PresignedPutObjectRequest presignedPutObjectRequest = s3Presigner.presignPutObject(putObjectPresignRequest);
presignedUrl = presignedPutObjectRequest.url();
System.out.println("Presigned URL: " + presignedUrl);
System.out.println("Method needed: " + presignedPutObjectRequest.httpRequest().method());
s3Presigner.close();
return presignedUrl;
}
public static void getPresignedUrl(S3Presigner presigner, String bucketName, String keyName) {
try {
// Create a GetObjectRequest to be pre-signed
GetObjectRequest getObjectRequest =
GetObjectRequest.builder()
.bucket(bucketName)
.key(keyName)
.build();
// Create a GetObjectPresignRequest to specify the signature duration
GetObjectPresignRequest getObjectPresignRequest =
GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(1000))
.getObjectRequest(getObjectRequest)
.build();
// Generate the presigned request
PresignedGetObjectRequest presignedGetObjectRequest =
presigner.presignGetObject(getObjectPresignRequest);
// Log the presigned URL，这个URL是我们要预览图片的URL
System.out.println("Presigned URL: " + presignedGetObjectRequest.url());
// Create a JDK HttpURLConnection for communicating with S3
HttpURLConnection connection = (HttpURLConnection) presignedGetObjectRequest.url().openConnection();
// Specify any headers that the service needs (not needed when isBrowserExecutable is true)
presignedGetObjectRequest.httpRequest().headers().forEach((header, values) -> {
values.forEach(value -> {
connection.addRequestProperty(header, value);
});
});
// Send any request payload that the service needs (not needed when isBrowserExecutable is true)
if (presignedGetObjectRequest.signedPayload().isPresent()) {
connection.setDoOutput(true);
try (InputStream signedPayload = presignedGetObjectRequest.signedPayload().get().asInputStream();
OutputStream httpOutputStream = connection.getOutputStream()) {
IoUtils.copy(signedPayload, httpOutputStream);
}
}
// Download the result of executing the request
try (InputStream content = connection.getInputStream()) {
System.out.println("Service returned response: ");
IoUtils.copy(content, System.out);
}
/*
* It's recommended that you close the S3Presigner when it is done being used, because some credential
* providers (e.g. if your AWS profile is configured to assume an STS role) require system resources
* that need to be freed. If you are using one S3Presigner per application (as recommended), this
* usually isn't needed
*/
presigner.close();
} catch (S3Exception e) {
e.getStackTrace();
} catch (IOException e) {
e.getStackTrace();
}
}
private static void uploadObject(URL presignedUrl, String objectLocalPath, String contentType) {
BufferedOutputStream bufferedOutputStream = null;
BufferedInputStream bufferedInputStream = null;
try {
HttpURLConnection httpURLConnection = (HttpURLConnection) presignedUrl.openConnection();
httpURLConnection.setDoOutput(true);
httpURLConnection.setRequestProperty("Content-Type", contentType);
httpURLConnection.setRequestMethod("PUT");
bufferedOutputStream = new BufferedOutputStream(httpURLConnection.getOutputStream());
/*
* Read from local and write to S3.
*/
bufferedInputStream = new BufferedInputStream(new FileInputStream(new File(objectLocalPath)));
byte[] buffer = new byte[1024];
int length = bufferedInputStream.read(buffer);
while (length > 0) {
// Write.
bufferedOutputStream.write(buffer);
// Read next.
length = bufferedInputStream.read(buffer);
}
bufferedOutputStream.flush();
bufferedOutputStream.close();
System.out.println("Response Code: " + httpURLConnection.getResponseCode());
System.out.println("Response Message: " + httpURLConnection.getResponseMessage());
} catch (IOException e) {
e.printStackTrace();
} finally {
if (bufferedOutputStream != null) {
try {
bufferedOutputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (bufferedInputStream != null) {
try {
bufferedInputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
使用
// Log the presigned URL，这个URL是我们要预览图片的URL
System.out.println("Presigned URL: " + presignedGetObjectRequest.url());
打印出来的URL即可正常访问图片。注意：代码里这个URL是有有效期的，为了安全，如果下次再需要使用，需要使用同样的方法生成新的签名url来使用。
