Volley源码分析之流程和缓存
前言
Android一开始提供了HttpURLConnection和HttpClient两种方式请求网络,但是这两种控制粒度太细,需要自己做很多操作,同时也导致了需要写很多的重复代码。为此,2013年Google推出Volley网络请求框架,基于此框架,开发者只需设置请求url,传递参数和请求成功或失败的回调的方法即可,大大简化了网络请求;虽然现在Okhttp很流行,但Volley的源码非常值得学习;
使用
可分为三个步骤,简称三部走:
- 创建RequestQueue实例
- 创建StringRequest实例
- 调用RequestQueue实例方法add将request作为参数传入
StringRequest中并没有提供设置POST参数的方法,但是当发出POST请求的时候,Volley会尝试调StringRequest的父类——Request中的getParams()方法来获取POST参数,那么解决方法自然也就有了,我们只需要在StringRequest的匿名类中重写getParams()方法,在这里设置POST参数就可以了。示范代码如下:
String requestUrl = "http://yourapi";//建立请求的api的URL; Response.Listener successListener = new Response.Listener<String>() {//成功回调 @Override public void onResponse(String response) { Log.d("TAG", response); } }; Response.ErrorListener errorListener = new Response.ErrorListener() {//失败回调 @Override public void onErrorResponse(VolleyError error) { Log.e("TAG", error.getMessage(), error); } }; StringRequest stringRequest = new StringRequest(Request.Method.POST, requestUrl, successListener, errorListener) { @Override protected Map<String, String> getParams() throws AuthFailureError { //重写getParams方法传POST请求参数 Map<String, String> map = new HashMap<String, String>(); map.put("phone", "phone num"); map.put("passwd", "passwd"); return map; } }; Volley.newRequestQueue(mContext).add(stringRequest);//创建newRequestQueue并传入参数
请求成功后,Logcat中会打印出服务器返回的信息;
源码分析
Volley适合通信频繁及数据量少的应用,大多展示类的App都适用;不适合繁重的下载或者流的操作,因为Volley会把解析到的响应数据保持在内存中;
首先从我们一开始的三部走的第一步开始,创建RequestQueue实例,代码语句
Volley.newRequestQueue(mContext)
跟进代码查看
public static RequestQueue newRequestQueue(Context context) { return newRequestQueue(context, null); }
然后调用了两个参数的newRequestQueue方法,默认第二个参数HttpStack为null;
/** * Creates a default instance of the worker pool and calls {@link RequestQueue#start()} on it. * * @param context A {@link Context} to use for creating the cache dir. * @param stack An {@link HttpStack} to use for the network, or null for default. * @return A started {@link RequestQueue} instance. */ public static RequestQueue newRequestQueue(Context context, HttpStack stack) { File cacheDir = new File(context.getCacheDir(), DEFAULT_CACHE_DIR);//缓存目录 String userAgent = "volley/0"; try { String packageName = context.getPackageName(); PackageInfo info = context.getPackageManager().getPackageInfo(packageName, 0); userAgent = packageName + "/" + info.versionCode; } catch (NameNotFoundException e) { } if (stack == null) { if (Build.VERSION.SDK_INT >= 9) {//根据系统版本号判断适用哪个请求方式 stack = new HurlStack(); } else { // Prior to Gingerbread, HttpUrlConnection was unreliable. // See: http://android-developers.blogspot.com/2011/09/androids-http-clients.html stack = new HttpClientStack(AndroidHttpClient.newInstance(userAgent)); } } Network network = new BasicNetwork(stack); RequestQueue queue = new RequestQueue(new DiskBasedCache(cacheDir), network);//进入DiskBasedCache构造方法可知,此处设置缓存目录,以及设定了一个默认5M的缓存大小; queue.start(); return queue; }
HttpStack有实现两个类:HttpClientStack和HurlStack,都实现了performRequest方法,主要的作用是执行网络请求操作并获取返回结果;HttpClientStack使用了HttpClient进行网络通信,而HttpStack是使用HttpURLConnection;当系统版本小于9时使用HttpClientStack,反之使用HurlStack;这个判断主要是因为系统版本小于9时HttpURLConnection有bug,调用 close() 函数会影响连接池,导致连接复用失效;稳定性不如HttpClient稳定;而9之后系统作了修改,默认开启了 gzip 压缩,提高了 HTTPS 的性能,HttpURLConnection成了最佳选择;郭霖文章给出的原因
最后创建了RequestQueue,调用了start方法并返回RequestQueue实例;注意,RequestQueue的创建方法中我们可以看到,自己可以自定义RequestQueue,配置缓存目录,及缓存的大小;然后调用了RequestQueue的start方法
/** * Starts the dispatchers in this queue. */ public void start() { stop(); // Make sure any currently running dispatchers are stopped. // Create the cache dispatcher and start it. mCacheDispatcher = new CacheDispatcher(mCacheQueue, mNetworkQueue, mCache, mDelivery); mCacheDispatcher.start(); // Create network dispatchers (and corresponding threads) up to the pool size. for (int i = 0; i < mDispatchers.length; i++) { NetworkDispatcher networkDispatcher = new NetworkDispatcher(mNetworkQueue, mNetwork, mCache, mDelivery); mDispatchers[i] = networkDispatcher; networkDispatcher.start(); } }
start方法主要创建了CacheDispatcher和NetworkDispatcher,并调用了对应的start方法;CacheDispatcher负责处理调度走缓存逻辑的请求,而NetworkDispatcher则是处理调度走网络的请求;一看start方法可以推测这两个类为Thread的子类,进入看果然是继承了Thread
public class CacheDispatcher extends Thread{ }
for循环默认创建4个NetworkDispatcher线程并开启,也就是说当Volley.newRequstQueue这句代码执行时,就创建了5个线程在不断运行,不断阻塞内部的请求队列知道请求的添加才开始下面的处理逻辑;
Volley.newRequstQueue创建完RequestQueue后,到了三步走的第三部,调用add方法把Request插入RequestQueue中
public Request add(Request request) { /* ... ...隐藏部分代码 ...*/ // If the request is uncacheable, skip the cache queue and go straight to the network. //判断request是否需要缓存,若不需要缓存则加入mNetworkQueue中,默认Request都需要缓存, //setShouldCache可改变设置 if (!request.shouldCache()) { mNetworkQueue.add(request); return request; } /* ... ...隐藏部分代码 ...*/ mWaitingRequests.put(cacheKey, null); mCacheQueue.add(request);//默认的request都加入到mCacheQueue return request; } }
默认添加Request都会添加到mCacheQueue,缓存调度会不断轮询此队列;上面说到创建了RequestQueue后会默认开启5个线程,其中包括了缓存线程CacheDispatcher,现在看下CacheDispatcher的run方法
@Override public void run() { /* ...隐藏部分代码 */ final Request request = mCacheQueue.take();//获取队列头部元素,若为空则一直阻塞,采用了BlockingQueue request.addMarker("cache-queue-take"); // If the request has been canceled, don't bother dispatching it. if (request.isCanceled()) { request.finish("cache-discard-canceled"); continue; } // Attempt to retrieve this item from cache. Cache.Entry entry = mCache.get(request.getCacheKey()); if (entry == null) { request.addMarker("cache-miss"); // Cache miss; send off to the network dispatcher. mNetworkQueue.put(request); continue; } // If it is completely expired, just send it to the network. //若过期,则添加到NetworkQueue中执行网络请求 if (entry.isExpired()) { request.addMarker("cache-hit-expired"); request.setCacheEntry(entry); mNetworkQueue.put(request); continue; } // We have a cache hit; parse its data for delivery back to the request. request.addMarker("cache-hit"); Response<?> response = request.parseNetworkResponse( new NetworkResponse(entry.data, entry.responseHeaders)); request.addMarker("cache-hit-parsed"); if (!entry.refreshNeeded()) { // Completely unexpired cache hit. Just deliver the response. mDelivery.postResponse(request, response);//缓存可用则进行回调 } else { // Post the intermediate response back to the user and have // the delivery then forward the request along to the network. mDelivery.postResponse(request, response, new Runnable() { @Override public void run() { try { mNetworkQueue.put(request); } catch (InterruptedException e) { // Not much we can do about this. } } }); } } catch (InterruptedException e) { // We may have been interrupted because it was time to quit. if (mQuit) { return; } continue; } } }
可以看到开启线程后会一直阻塞直到获取缓存队列中插入的数据,阻塞队列使用了BlockingQueue,可另外了解;获取后查找缓存中是否有response,若response过期或需要刷新则会向mNetworkQueue中插入,走网络请求,反之则使用缓存中的response;若缓存可用便调用request的parseNetworkResponse解析数据,然后就是调用mDelivery.postResponse回调;mDelivery默认的实现是创建RequestQueue时候创建的
public RequestQueue(Cache cache, Network network, int threadPoolSize) { this(cache, network, threadPoolSize, new ExecutorDelivery(new Handler(Looper.getMainLooper()))); }
public ExecutorDelivery(final Handler handler) { // Make an Executor that just wraps the handler. mResponsePoster = new Executor() { @Override public void execute(Runnable command) { handler.post(command); } }; }
mDelivery.postResponse最终会调用ExecutorDelivery的execute方法,其传入的参数ResponseDeliveryRunnable,最终回调用handler的post方法把线程切换到主线程;ResponseDeliveryRunnable的run方法执行在主线程,如下:
public void run() { /* 隐藏部分代码 */ // Deliver a normal response or error, depending. if (mResponse.isSuccess()) { mRequest.deliverResponse(mResponse.result);//回调到我们一开始创建的Listenner回调中 } else { mRequest.deliverError(mResponse.error); } // If this is an intermediate response, add a marker, otherwise we're done // and the request can be finished. if (mResponse.intermediate) {//响应需要刷新的时候为true mRequest.addMarker("intermediate-response"); } else { mRequest.finish("done"); } // If we have been provided a post-delivery runnable, run it. if (mRunnable != null) { mRunnable.run(); }
可以看到若成功了则调用mRequest.deliverResponse(mResponse.result);这句代码,而deliverResponse则调用了我们一开始创建的StringRequest代码的Response.Listener的onResponse,我们便可以在这里处理返回来的响应并更新View;
现在看下当没有缓存或缓存实效时,使用网络获取响应的NetworkDispatcher,看下run方法:
@Override public void run() { Process.setThreadPriority(Process.THREAD_PRIORITY_BACKGROUND); Request request; while (true) { try { // Take a request from the queue. request = mQueue.take();//从Volley中创建RequestQueue后调用start方法中,同用一个队列传进构造方法可知:mQueue和刚才CacheDispatcher的mNetworkQueue是同一个队列, } catch (InterruptedException e) { // We may have been interrupted because it was time to quit. if (mQuit) { return; } continue; } // Perform the network request. NetworkResponse networkResponse = mNetwork.performRequest(request);//网络请求,实际上mNetwork的实现是BasicNetwork request.addMarker("network-http-complete"); /* 隐藏部分代码 */ // Parse the response here on the worker thread. Response<?> response = request.parseNetworkResponse(networkResponse); request.addMarker("network-parse-complete"); // Write to cache if applicable. // TODO: Only update cache metadata instead of entire record for 304s. if (request.shouldCache() && response.cacheEntry != null) { mCache.put(request.getCacheKey(), response.cacheEntry); request.addMarker("network-cache-written"); } // Post the response back. request.markDelivered(); mDelivery.postResponse(request, response); /* 隐藏部分代码 */ } }
mNetwork.performRequest这句代码实现了请求代码,mNetwork的唯一实现为BasicNetwork,其performRequest方法主要根据HttpStack来执行网络请求并构造NetworkResponse返回;然后就是调用mDelivery.postResponse(request, response)方法回调,跟CacheDispatcher一样的流程了;
Volley.newRequestQueue(mContext).add(stringRequest)
到此,这句代码的流程完全走了一遍;总结一下:调用newRequestQueue创建了RequestQueue,同时开启了CacheDispatcher和NetworkDispatcher 5个线程不断阻塞轮询,当调用add方法时,会先在缓存队列mCacheQueue中插入request,此时缓存线程CacheDispatcher先查找缓存中是否有该request的响应缓存,若有且没有过期,且不需要刷新操作,则request自身的parseNetworkResponse方法解析成response且回调给主线程自己创建request的Response.Listener中的onResponse,若过期,没有缓存或需要刷新,则添加到mNetworkQueue中,NetworkDispatcher轮询获取request后,根据系统版本使用HttpClient或HttpURLConnection执行网络请求,得到响应后解析并回调给主线程,若request需要缓存且响应不为null,则存入响应缓存中;
流程图总结如下:
缓存策略
-
request的缓存
现在我们从缓存的角度分析下源码,当调用Volley.newRequestQueue(mContext).add(stringRequest)时,我们进入RequestQueue的add方法
public Request add(Request request) { // Tag the request as belonging to this queue and add it to the set of current requests. request.setRequestQueue(this);//把request与当前RequestQueue建立联系 synchronized (mCurrentRequests) { mCurrentRequests.add(request);//存放所有的request } // Process requests in the order they are added. request.setSequence(getSequenceNumber()); request.addMarker("add-to-queue"); // If the request is uncacheable, skip the cache queue and go straight to the network. if (!request.shouldCache()) {//是否应该缓存,默认所有request都缓存 mNetworkQueue.add(request); return request; } // Insert request into stage if there's already a request with the same cache key in flight. synchronized (mWaitingRequests) { String cacheKey = request.getCacheKey(); if (mWaitingRequests.containsKey(cacheKey)) {//当前add的request是否已经有相同的请求,若请求没有完成(成功或者失败),则会一直存在mWaitingRequest中 // There is already a request in flight. Queue up. Queue<Request> stagedRequests = mWaitingRequests.get(cacheKey); if (stagedRequests == null) { stagedRequests = new LinkedList<Request>(); } stagedRequests.add(request);//创建一个队列并添加request mWaitingRequests.put(cacheKey, stagedRequests);//缓存含request的队列 if (VolleyLog.DEBUG) { VolleyLog.v("Request for cacheKey=%s is in flight, putting on hold.", cacheKey); } } else { // Insert 'null' queue for this cacheKey, indicating there is now a request in // flight. mWaitingRequests.put(cacheKey, null);//如果request没有则存入map中 mCacheQueue.add(request); } return request; } }
当add方法调用时,会判断mWaitingRequests是否包含cacheKey,假设第一个request请求mWaitingRequests不包含,则缓存cacheKey,键值为null,同时添加入缓存队列。当第一个request还在请求中,第二个和第三个相同的request这时添加进来,则走进入逻辑新建一个队列并加入两个request,和cacheKey缓存进mWaitingRequests的map中,注意此时缓存的都是request,当第一个request请求完成后会回调request的finish方法
void finish(final String tag) { if (mRequestQueue != null) { mRequestQueue.finish(this);//调用RequestQueue的finish方法 } /* 隐藏部分代码 */ }
而request的finish方法会调用RequestQueue的finish方法
void finish(Request request) { // Remove from the set of requests currently being processed. synchronized (mCurrentRequests) { mCurrentRequests.remove(request); } if (request.shouldCache()) { synchronized (mWaitingRequests) { String cacheKey = request.getCacheKey(); Queue<Request> waitingRequests = mWaitingRequests.remove(cacheKey);//移除该request的cacheKey,并得到缓存时创建的request队列 if (waitingRequests != null) { if (VolleyLog.DEBUG) { VolleyLog.v("Releasing %d waiting requests for cacheKey=%s.", waitingRequests.size(), cacheKey); } // Process all queued up requests. They won't be considered as in flight, but // that's not a problem as the cache has been primed by 'request'. mCacheQueue.addAll(waitingRequests);//若缓存起来的队列不为空,把里面的request全部添加到缓存队列; } } } }
当waitingRequests中的request添加到缓存队列后,又进入到CacheDispatcher的调度线程中了
- CacheDispatcher里的缓存
又从CacheDispatcher的run分析,这次是从缓存角度;主要有三点
- 没有缓存,添加到网络请求队列
- 缓存过期,添加到网络请求队列
- 缓存需要刷新,先回调数据给用户,再添加请求到网络请求队列
public void run() { if (DEBUG) VolleyLog.v("start new dispatcher"); Process.setThreadPriority(Process.THREAD_PRIORITY_BACKGROUND); // Make a blocking call to initialize the cache. mCache.initialize(); while (true) { try { // Get a request from the cache triage queue, blocking until // at least one is available. final Request request = mCacheQueue.take(); request.addMarker("cache-queue-take"); // If the request has been canceled, don't bother dispatching it. if (request.isCanceled()) { request.finish("cache-discard-canceled"); continue; } // Attempt to retrieve this item from cache. Cache.Entry entry = mCache.get(request.getCacheKey()); if (entry == null) {//没有缓存,添加到网络请求队列 request.addMarker("cache-miss"); // Cache miss; send off to the network dispatcher. mNetworkQueue.put(request); continue; } // If it is completely expired, just send it to the network. if (entry.isExpired()) {//缓存过期,添加到网络请求队列 request.addMarker("cache-hit-expired"); request.setCacheEntry(entry); mNetworkQueue.put(request); continue; } // We have a cache hit; parse its data for delivery back to the request. request.addMarker("cache-hit"); Response<?> response = request.parseNetworkResponse( new NetworkResponse(entry.data, entry.responseHeaders)); request.addMarker("cache-hit-parsed"); if (!entry.refreshNeeded()) {//不需要刷新,回调给用户设置的listener // Completely unexpired cache hit. Just deliver the response. mDelivery.postResponse(request, response); } else {//缓存需要刷新,先回调数据给用户,再添加请求到网络请求队列 // Soft-expired cache hit. We can deliver the cached response, // but we need to also send the request to the network for // refreshing. request.addMarker("cache-hit-refresh-needed"); request.setCacheEntry(entry); // Mark the response as intermediate. response.intermediate = true;//在需要刷新时置为true,在mDelivery.postResponse后,会调用ResponseDeliveryRunnable的run方法,其中中有个判断决定是添加信息或者调用request的finish方法 // Post the intermediate response back to the user and have // the delivery then forward the request along to the network. mDelivery.postResponse(request, response, new Runnable() { @Override public void run() { try { mNetworkQueue.put(request); } catch (InterruptedException e) { // Not much we can do about this. } } }); } } catch (InterruptedException e) { // We may have been interrupted because it was time to quit. if (mQuit) { return; } continue; } } }
注意这里当需要刷新数据的时候,会先回调用户设置的listener,然后再去发起请求,若数据有变化会再次回调,所以这种情况会回调用户设置的listener两次
- NetworkDispatcher的缓存
请求前添加和cache相关的headers
private void addCacheHeaders(Map<String, String> headers, Cache.Entry entry) { // If there's no cache entry, we're done. if (entry == null) { return; } if (entry.etag != null) { headers.put("If-None-Match", entry.etag); } if (entry.serverDate > 0) { Date refTime = new Date(entry.serverDate); headers.put("If-Modified-Since", DateUtils.formatDate(refTime)); } }
request的字段含义:
If-None-Match:存文件的Etag(Hash)值,与服务器回应的Etag比较判断是否改变
If-Modified-Since:缓存文件的最后修改时间
当网络请求完成后,调用了request中的parseNetworkResponse方法,用我们一开始的StringRequest来分析下。
protected Response<String> parseNetworkResponse(NetworkResponse response) { String parsed; try { parsed = new String(response.data, HttpHeaderParser.parseCharset(response.headers)); } catch (UnsupportedEncodingException e) { parsed = new String(response.data); } return Response.success(parsed, HttpHeaderParser.parseCacheHeaders(response)); }
此时又调用了HttpHeaderParser.parseCacheHeaders(response)来构建Cache.Entry对象,代码就不放了,主要是用response.headers中的Date,Cache-Control,Expires,ETag来构建对象,然后若request需要缓存及响应不为空则放进响应缓存中
response字段含义:
Cache-Control:告诉所有的缓存机制是否可以缓存及哪种类型
Expires:响应过期的日期和时间
Last-Modified:请求资源的最后修改时间
ETag:请求变量的实体标签的当前值
NetworkDispatcher
Response<?> response = request.parseNetworkResponse(networkResponse); request.addMarker("network-parse-complete"); // Write to cache if applicable. // TODO: Only update cache metadata instead of entire record for 304s. if (request.shouldCache() && response.cacheEntry != null) { mCache.put(request.getCacheKey(), response.cacheEntry); request.addMarker("network-cache-written"); } // Post the response back. request.markDelivered(); mDelivery.postResponse(request, response);
存入缓存的mCache的实现是DiskBasedCache,是不是有点熟悉,没错,就是我们一开始Volley.newRequestQueue中创建RequestQueue中的一个参数,我们看下DiskBasedCache的put方法
public synchronized void put(String key, Entry entry) { pruneIfNeeded(entry.data.length);//put之前先修正大小,避免超出限定的缓存大小 File file = getFileForKey(key); try { FileOutputStream fos = new FileOutputStream(file); CacheHeader e = new CacheHeader(key, entry); e.writeHeader(fos); fos.write(entry.data); fos.close(); putEntry(key, e); return; } catch (IOException e) { } boolean deleted = file.delete();//try出错时会把file给delete if (!deleted) { VolleyLog.d("Could not clean up file %s", file.getAbsolutePath()); } }
进入pruneIfNeeded方法
private void pruneIfNeeded(int neededSpace) { if ((mTotalSize + neededSpace) < mMaxCacheSizeInBytes) { return; } if (VolleyLog.DEBUG) { VolleyLog.v("Pruning old cache entries."); } long before = mTotalSize; int prunedFiles = 0; long startTime = SystemClock.elapsedRealtime(); Iterator<Map.Entry<String, CacheHeader>> iterator = mEntries.entrySet().iterator(); while (iterator.hasNext()) { Map.Entry<String, CacheHeader> entry = iterator.next(); CacheHeader e = entry.getValue(); boolean deleted = getFileForKey(e.key).delete(); if (deleted) { mTotalSize -= e.size; } else { VolleyLog.d("Could not delete cache entry for key=%s, filename=%s", e.key, getFilenameForKey(e.key)); } iterator.remove(); prunedFiles++; if ((mTotalSize + neededSpace) < mMaxCacheSizeInBytes * HYSTERESIS_FACTOR) { break; } } if (VolleyLog.DEBUG) { VolleyLog.v("pruned %d files, %d bytes, %d ms", prunedFiles, (mTotalSize - before), SystemClock.elapsedRealtime() - startTime); } }
传入需要插入数据的大小参数neededSpace和当前缓存文件总大小的记录mTotalSize计算出是否超出了设定的缓存大小,若超出则获取CacheHeader列表;CacheHeader里包含缓存文件大小等信息,遍历CacheHeader列表删除缓存,直到小于设定缓存的0.9倍,为什么是这个数?猜测是个缓存过大的标记,同时可以留有一定可缓存空间;
-
如果让你去设计Volley的缓存功能,你要如何增大它的命中率?
可以知道,在pruneIfNeeded中删除缓存时,并没有判断缓存是否过期,只是遍历来删除,很大程度上会误删刚缓存下来的数据,而过期数据却仍然存在;所以可以在删除前进行一次过期缓存的清除,然后再使用Lru算法清除最近最久未使用的缓存;
总结一下缓存方面的流程:当从RequestQueue中add的时候会判断request是否可以缓存,若不可以则添加到网络队列;否则添加到缓存队列;进入缓存队列后,CacheDispatcher中不断轮询取出缓存队列中的request,然后判断该request是否已经缓存了响应,响应是否过期,响应是否需要刷新;若request取消、没有缓存响应、过期或需要刷新,则添加request到网络队列;NetworkDispatcher也不断的轮询,取出添加到网络请求队列的request,用BasicNetwork请求网络,请求前添加headers相关的信息,请求返回的响应会被HttpHeaderParser的parseCacheHeaders解析成Cache.Entry对象,若此时的request能缓存且响应不为null,则添加到响应缓存中;
注:才疏学浅,如有有写错,理解错的地方麻烦指出更正下,非常感谢!