Redis6 new data type - hyperloglog

1. brief introduction

At work , We often encounter functional requirements related to statistics , For example, statistical websites PV （PageView Page visits ）, have access to Redis Of incr、incrby Make it easy . But like UV（UniqueVisitor, Independent visitor ）、 Independent IP Count 、 The number of search records needs to be de duplicated and How to solve the problem of counting ？ The problem of finding the number of non repeating elements in a set is called the cardinality problem . There are many solutions to the cardinality problem ：
（1） The data is stored in MySQL In the table , Use distinct count Calculate the number of non duplicates

（2） Use Redis Provided hash、set、bitmaps And other data structures

The results of the above scheme are accurate , But as the data increases , Resulting in more and more occupied space , For non Very large data sets are impractical .

Whether it can reduce a certain accuracy to balance the storage space ？Redis Launched HyperLogLog

Redis HyperLogLog It's an algorithm for cardinality statistics ,HyperLogLog The advantages of , When the number or volume of input elements is very, very large , The space needed to calculate the cardinality is always fixed 、 And it's very small Of .

stay Redis Inside , Every HyperLogLog Keys only cost 12 KB Memory , You can calculate the connection near 2^64 Cardinality of different elements . This is the same as calculating the cardinality , The more elements consume memory, the more collections there are .

however , because HyperLogLog Only the input elements will be used to calculate the cardinality , Instead of storing input elements In itself , therefore HyperLogLog It can't be like a collection , Return the various elements of the input .

What is the cardinality ?
Like data sets {1, 3, 5, 7, 5, 7, 8}, So the cardinality set of this dataset is {1, 3, 5 ,7, 8}, base ( Don't repeat elements ) by 5. Cardinality estimation is within the range of acceptable error , Fast base calculation .

2. command

1、pfadd
（1） Format pfadd <key>< element> [element ...] Add specified elements to HyperLogLog in

（2） example

Adds all elements to the specified HyperLogLog In the data structure . If after executing the command HLL The approximate cardinality of the estimate changes , Then return to 1, Otherwise return to 0.

2、pfcount
（1） Format
pfcount<key> [key ...] Calculation HLL The approximate cardinality of , Multiple can be calculated HLL, For example, use HLL Store daily UV, Calculate a week's UV have access to 7 Days of UV Consolidation calculation is enough

（2） example
3、pfmerge
（1） Format
pfmerge<destkey><sourcekey> [sourcekey ...] Put one or more HLL The merged results are stored in another HLL in , For example, monthly active users can use daily active users to consolidate and calculate the available

（2） example