Using Redis in Real-time Statistics Application

Background

In our application, there is a producer produce cid(a unique string id)s into a message queue, knowing as concurrent user.

Provide real-time statistics of user.

1. Extract the cid from message queue and save it into Redis's Sorted-Set data structure, with timestamp as score.

Time complexity: O(log(N)) where N is the number of elements in the sorted set.

2. Run an backend thread to remove the expired cids(which is not actived in the last five minutes) every one second.

Time complexity: O(log(N)+M) with N being the number of elements in the sorted set and M the number of elements removed by the operation.

3. Provide data api to count the number of elements in the sorted set.

Time complexity: O(1)

4. UI refreshes every five seconds to call data api to provide real-time concurrent user data.

Instead of saving cid into sorted-set directly, holding a mapping of cid to unique int id can save memory since the cid will be reused.

So change #1 step as follows:

1.1 Extract the cid from message queue, save it into Redis's Hash data structure, using cid as key, an auto-increase unique int id as vaule.

Time complexity: O(1)

1.2 Save the int id into Redis's Sorted-Set data structure, with timestamp sa score.

Time complexity: O(log(N)) where N is the number of elements in the sorted set.

By this optimization, there is also a good way to calculate AU and NU in real-time. See http://blog.csdn.net/gaoyingju/article/details/9671283

For #2 step, record the number of elements every second and update when the next second has more elements, PCU is also available.