概述：
官方文档介绍了如何直接使用pre-signed url上传小文件, 但没有介绍如何使用pre-signed url来multipart 上传大文件，本文将详述。另， 本文将介绍如何分别使用OKHttp和resttemplate来将inputstream直接作为request body体发送给pre-signed url，这种方式省去了将上传内容保存到本地临时文件或者读入内存中的步骤。也会介绍我遇到的一些坑。
直接上传
服务端（生成pre-signed url）：
服务端我是用java SDK签发pre-signed url，然后通过接口返回给调用端。
amazonS3.generatePresignedUrl(new GeneratePresignedUrlRequest(BUCKET_NAME,informationObject.getObjectKey(),HttpMethod.PUT)
.withContentMd5(informationObject.getMd5DigestValue())//如果在签发pre-signed url时附带了content-md5，那么当调用pre-signed url上传文件时，需要将文件的content-md5放到header中。
.withExpiration(new Date(System.currentTimeMillis() +EXPIRE_INTERVAL))
)
客户端（需要上传文件到S3，但没有AWS credential）：
没有什么特别之处，与multipart upload共用同一上传inputstream的方法即可。
Multipart上传
采用multipart上传文件时，总共需要四步，解下来分别介绍服务端与客户端：
1. Initiate multipart upload.
2. Generate presigned url for every part.
3. Upload part through invoking pre-signed url.
4. Complete the multipart upload.
服务端：
作为拥有credentials、可通过SDK直接请求AWS的服务端，我们需要对client端提供1，2，4步所需的接口（为减少服务器承受的压力，这里建议每个接口做成批量处理，即一个请求体中包含所有要上传的文件的所有part）。
Interfaces in server:
1. Initiate multipart uploadS.
amazonS3.initiateMultipartUpload(new InitiateMultipartUploadRequest(BUCKET_NAME,informationObject.getObjectKey()))
2. Generate presigned urls for every part of all fileS.
给每个part签发pre-signed url（如果要上传多个文件，建议放到同一次请求中签发url）。并且request parameter中需要携带uploadID和partNumber。
GeneratePresignedUrlRequest generatePresignedUrlRequest =new GeneratePresignedUrlRequest(
BUCKET_NAME,objectKey).withContentMd5(uploadPart.getMd5DigestValue())
.withExpiration(new Date(System.currentTimeMillis() +EXPIRE_INTERVAL))
.withMethod(HttpMethod.PUT);
generatePresignedUrlRequest.addRequestParameter("uploadId",uploadId);
generatePresignedUrlRequest.addRequestParameter("partNumber",Integer.toString(uploadPart.getPartNumber()));
amazonS3.generatePresignedUrl(generatePresignedUrlRequest);
3. Complete the multipart uploadS.
ListpartETags =new ArrayList<>();
int partNumberMarker =0;
boolean isTruncated =true;
while (isTruncated) {
PartListing partListing =amazonS3.listParts(
new ListPartsRequest(BUCKET_NAME, objectKey, uploadId).withPartNumberMarker(partNumberMarker));
partETags.addAll(partListing.getParts().stream()
.map(part ->new PartETag(part.getPartNumber(), part.getETag())).collect(Collectors.toList()));
isTruncated =partListing.isTruncated();
if (isTruncated) {
partNumberMarker =partListing.getNextPartNumberMarker();
}
}
CompleteMultipartUploadResult completeMultipartUploadResult =amazonS3
.completeMultipartUpload(new CompleteMultipartUploadRequest(BUCKET_NAME,
objectKey, uploadId,partETags));
客户端：
1. 调用服务端提供的init接口
2.调用服务端提供的presign接口
3. 调用pre-signed url上传文件流
两种调用pre-signed url的方式：
1. OKHttp client(如果上传的是inputstream，请求的header中需要包含"content-type: application/octet-stream")
package com.osthus.ais.file.monitor.service.common;
import static com.osthus.ais.file.monitor.util.FileMonitorConfigurationConstants.OKHTTP_TIMEOUT;
import static com.osthus.ais.file.monitor.util.FileMonitorConfigurationConstants.OKHTTP_TIMEOUT_DEFAULT;
import java.io.IOException;
import java.io.InputStream;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
import org.apache.commons.lang3.StringUtils;
import org.springframework.http.ResponseEntity;
import de.osthus.ambeth.config.Property;
import de.osthus.ambeth.ioc.IStartingBean;
import de.osthus.ambeth.log.ILogger;
import de.osthus.ambeth.log.LogInstance;
import okhttp3.MediaType;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.RequestBody;
import okhttp3.Response;
import okhttp3.ResponseBody;
import okhttp3.internal.Util;
import okio.BufferedSink;
import okio.Okio;
import okio.Source;
public class OKHttpService implements IStartingBean {
private static OkHttpClient client;
@LogInstance
private ILogger log;
@Property(name =OKHTTP_TIMEOUT, defaultValue =OKHTTP_TIMEOUT_DEFAULT)
private long okHttpTimeOut;
public ResponseEntityput(String url,InputStream inputStream,long partLength,String contentType,
Map headers) {
log.debug(String.format("PUT request to %s", url));
MediaType mediaType =MediaType.parse(contentType);
RequestBody body =this.createRequestBody(mediaType, inputStream, partLength);
Request.Builder builder =new Request.Builder().url(url).method("PUT",body);
builder.addHeader("Content-Type", contentType);
if (headers !=null) {
headers.forEach(builder::addHeader);
}
Request request =builder.build();
try (Response response =client.newCall(request).execute()) {
ResponseBody responseBody =response.body();
String returnBody ="";
if (Optional.ofNullable(responseBody).isPresent()) {
String result =responseBody.string();
returnBody =StringUtils.isNotEmpty(result) ?result : returnBody;
}
return ResponseEntity.status(response.code()).body(returnBody);
}catch (IOException e) {
throw new RuntimeException(e);
}
}
private RequestBody createRequestBody(MediaType contentType,final InputStream inputStream,long length) {
return new RequestBody() {
@Override
public MediaType contentType() {
return contentType;
}
@Override
public long contentLength() {
return length;
}
@Override
public void writeTo(BufferedSink bufferedSink)throws IOException {
Source source =null;
try {
source =Okio.source(inputStream);
bufferedSink.writeAll(source);
}finally {
Util.closeQuietly(source);
}
}
};
}
@Override
public void afterStarted()throws Throwable {
client =new OkHttpClient().newBuilder()
.readTimeout(okHttpTimeOut,TimeUnit.MILLISECONDS)
.writeTimeout(okHttpTimeOut,TimeUnit.MILLISECONDS)
.connectTimeout(okHttpTimeOut,TimeUnit.MILLISECONDS)
.build();
}
}
2. Rest template
public void uploadInputStream(String url, InputStream inputStream, long contentLength, String md5DigestValue){
Map<String, String> headers = new HashMap<>();
headers.put(CONTENT_MD5, md5DigestValue);
headers.put(CONTENT_TYPE, MediaType.APPLICATION_OCTET_STREAM_VALUE);
ResponseEntity<?> response = restInvoker.put(URI.create(url), new CommonInputStreamResource(inputStream, contentLength), ResponseEntity.class, headers);String result = (String)response.getBody();
加粗的代码URI.create(url)有一个坑，这里不可以将url以字符串的形式直接传进去。因为aws所签发的pre-signed url是经过encode的，里面会包含%。而restTemplate会再一次encode传入的url字符串，这就导致url变得无效。所以用加粗位置的代码，自己直接传入URI。如果我们对aws返回的per-signed url进行decode，然后传入，也是不行的。
因为restemplate会对url中的%进行encode，这就导致经过decode的pre-signed url无法成功（aws返回的默认是encode过的，应该被decode后使用）；另外, restemplate不会对url中的+进行encode（实际上应该encode才会成功），这就导致手动decode后的pre-signed url也会报错（里面包含+）。详细分析见：https://blog.csdn.net/xs_challenge/article/details/109451263
public class CommonInputStreamResource extends InputStreamResource {
private long length;
public CommonInputStreamResource(InputStream inputStream) {
super(inputStream);
}
public CommonInputStreamResource(InputStream inputStream,long length) {
super(inputStream);
this.length = length;
}
@Override
public String getFilename() {
return "temp";
}
@Override
public long contentLength() {
long estimate =length;
return estimate ==0 ?1 :estimate;
}
}
4. 调用服务端的complete接口
当enable KMS时，generate和consume pre-signed url需要做哪些修改在下面这篇文章里说的很详细。
https://aws.amazon.com/blogs/developer/generating-amazon-s3-pre-signed-urls-with-sse-kms-part-2/
解决方案：应用服务OkHttpClient创建大量对外连接时内存溢出
https://zhuanlan.zhihu.com/p/266363937
