KwaiRec: A Fully-observed Dataset for Recommender Systems (Density: Almost 100%)

Overview

KuaiRec: A Fully-observed Dataset for Recommender Systems (Density: Almost 100%)

LICENSE

KuaiRec is a real-world dataset collected from the recommendation logs of the video-sharing mobile app Kuaishou. For now, it is the first dataset that contains a fully observed user-item interaction matrix. For the term "fully observed", we mean there are almost no missing values in the user-item matrix, i.e., each user has viewed each video and then left feedback.

The following figure illustrates the user-item matrices in traditional datasets and KuaiRec.

kuaidata

With all user's preference known, KuaiRec can used in offline evaluation (i.e., offline A/B test) for recommendation models. It can benefit lots of research directions, such as unbiased recommendation, interactive/conversational recommendation, or reinforcement learning (RL) and off-policy evaluation (OPE) for recommendation.

If you use it in your work, please cite our paper: LINK PDF

@article{gao2022kuairec,
  title={KuaiRec: A Fully-observed Dataset for Recommender Systems}, 
  author={Chongming Gao and Shijun Li and Wenqiang Lei and Biao Li and Peng Jiang and Jiawei Chen and Xiangnan He and Jiaxin Mao and Tat-Seng Chua},
  journal={arXiv preprint arXiv:2202.10842},
  year={2022}
}

This repository lists the example codes in evaluating conversational recommendation as described in the paper.

We provide some simple statistics of this dataset here . It is generated by Statistics_KuaiRec.ipynb. You can do it online at Google Colab colab.


News ! ! ! ! !

2022.05.16: We update the dataset to version 2.0. We made the following changes:

  • We removed the unused video ID=1225 from all tables having the field video_id and reindex the rest videos, i.e., ID = ID - 1 if ID > 1225.
  • We added two tables to enhance the side information for users and videos, respectively. See 4.item_daily_feet.csv and 5. user_feat.csv under the data description section for details.

Download the data

We provides several options to download this dataset:

Option 1. Download via the "wget" command.

 wget https://chongming.myds.me:61364/data/KuaiRec.zip --no-check-certificate
 unzip KuaiRec.zip

Option 2. Download manually throughs the following links:

The script loaddata.py provides a simple way to load the data via Pandas in Python.


Data Descriptions

KuaiRec contains millions of user-item interactions as well as the side information include the item categorires and social network. Four files are included in the download data:

KuaiRec
├── data
│   ├── big_matrix.csv          
│   ├── small_matrix.csv
│   ├── social_network.csv
│   └── item_categories.csv

The statistics of the small matrix and big matrix in KuaiRec.

#Users #Items #Interactions Density
small matrix 1,411 3,327 4,676,570 99.6%
big matrix 7,176 10,728 12,530,806 16.3%

Note that the density of small matrix is 99.6% instead of 100% because some users have explicitly indicated that they would not be willing to receive recommendations from certain authors. I.e., They blocked these videos.

1. Descriptions of the fields in big_matrix.csv and small_matrix.csv.

Field Name: Description Type Example
user_id The ID of the user. int64 0
video_id The ID of the viewed video. int64 3650
play_duration Time of video viewing of this interaction (millisecond). int64 13838
video_duration Time of this video (millisecond). int64 10867
time Human-readable date for this interaction str "2020-07-05 00:08:23.438"
date Date of this interaction int64 20200705
timestamp Unix timestamp float64 1593878903.438
watch_ratio The video watching ratio (=play_duration/video_duration) float64 1.273397

The "watch_ratio" can be deemed as the label of the interaction. Note: there is no "like" signal for this dataset. If you need this binary signal in your scenarios, you can create it yourself. E.g., like = 1 if watch_ratio > 2.0.

2. Descriptions of the fields in social_network.csv

Field Name: Description Type Example
user_id The ID of the user. int64 5352
friend_list The list of ID of the friends of this user. list [4202,7126]

3. Descriptions of the fields in item_categories.csv.

Field Name: Description Type Example
video_id The ID of the video. int64 1
feat The list of tags of this video. list [27,9]

4. Descriptions of the fields in item_daily_feet.csv. (Added on 2022.05.16)

Field Name: Description Type Example
video_id The ID of the video. int64 3784
date Date of the statistics of this video. int64 20200730
author_id The ID of the author of this video. int64 441
video_type Type of this video (NORMAL or AD). str "NORMAL"
upload_dt Upload date of this video. str "2020-07-08"
upload_type The upload type of this video. str "ShortImport"
visible_status The visible state of this video on the APP now. str "public"
video_duration The time duration of this duration (in millisecond). float64 17200.0
video_width The width of this video on the server. int64 720
video_height The height of this video on the server. int64 1280
music_id Background music ID of this video. int64 989206467
video_tag_id The ID of tag of this video. int64 2522
video_tag_name The name of tag of this video. string "祝福"
show_cnt The number of shows of this video within this day (the same with all following fields) int64 7716
show_user_num The number of users who received the recommendation of this video. int64 5256
play_cnt The number of plays. int64 7701
play_user_num The number of users who plays this video. int64 5034
play_duration The total time duration of playing this video (in millisecond). int64 138333346
complete_play_cnt The number of complete plays. complete play: finishing playing the whole video, i.e., #(play_duration >= video_duration). int64 3446
complete_play_user_num The number of users who perform the complete play. int64 2033
valid_play_cnt valid play: play_duration >= video_duration if video_duration <= 7s, or play_duration > 7 if video_duration > 7s. int64 5099
valid_play_user_num The number of users who perform the complete play. int64 3195
long_time_play_cnt long time play: play_duration >= video_duration if video_duration <= 18s, or play_duration >=18 if video_duration > 18s. int64 3299
long_time_play_user_num The number of users who perform the long time play. int64 1940
short_time_play_cnt short time play: play_duration < min(3s, video_duration). int64 1538
short_time_play_user_num The number of users who perform the short time play. int64 1190
play_progress The average video playing ratio (=play_duration/video_duration) int64 0.579695
comment_stay_duration Total time of staying in the comments section int64 467865
like_cnt Total likes int64 659
like_user_num The number of users who hit the "like" button. int64 657
click_like_cnt The number of the "like" resulted from double click int64 496
double_click_cnt The number of users who double click the video. int64 163
cancel_like_cnt The number of likes that are cancelled by users. int64 15
cancel_like_user_num The number of users who cancel their like. int64 15
comment_cnt The number of comments within this day. int64 13
comment_user_num The number of users who comment this video. int64 12
direct_comment_cnt The number of direct comments (depth=1). int64 13
reply_comment_cnt The number of reply comments (depth>1). int64 0
delete_comment_cnt The number of deleted comments. int64 0
delete_comment_user_num The number of users who delete their comments. int64 0
comment_like_cnt The number of comment likes. int64 2
comment_like_user_num The number of users who like the comments. int64 2
follow_cnt The number of increased follows from this video. int64 151
follow_user_num The number of users who follow the author of this video due to this video. int64 151
cancel_follow_cnt The number of decreased follows from this video. int64 0
cancel_follow_user_num The number of users who cancel their following of the author of this video due to this video. int64 0
share_cnt The times of sharing this video. int64 1
share_user_num The number of users who share this video. int64 1
download_cnt The times of downloading this video. int64 2
download_user_num The number of users who download this video. int64 2
report_cnt The times of reporting this video. int64 0
report_user_num The number of users who report this video. int64 0
reduce_similar_cnt The times of reducing similar content of this video. int64 2
reduce_similar_user_num The number of users who choose to reduce similar content of this video. int64 2
collect_cnt The times of adding this video to favorite videos. int64 0
collect_user_num The number of users who add this video to their favorite videos. int64 0
cancel_collect_cnt The times of removing this video from favorite videos. int64 0
cancel_collect_user_num The number of users who remove this video from their favorite videos int64 0

5. Descriptions of the fields in user_feat.csv (Added on 2022.05.16)

Field Name: Description Type Example
user_id The ID of the user. int64 0
user_active_degree In the set of {'high_active', 'full_active', 'middle_active', 'UNKNOWN'}. str "high_active"
is_lowactive_period Is this user in its low active period int64 0
is_live_streamer Is this user a live streamer? int64 0
is_video_author Has this user uploaded any video? int64 0
follow_user_num The number of users that this user follows. int64 5
follow_user_num_range The range of the number of users that this user follows. In the set of {'0', '(0,10]', '(10,50]', '(100,150]', '(150,250]', '(250,500]', '(50,100]', '500+'} str "(0,10]"
fans_user_num The number of the fans of this user. int64 0
fans_user_num_range The range of the number of fans of this user. In the set of {'0', '[1,10)', '[10,100)', '[100,1k)', '[1k,5k)', '[5k,1w)', '[1w,10w)'} str "0"
friend_user_num The number of friends that this user has. int64 0
friend_user_num_range The range of the number of friends that this user has. In the set of {'0', '[1,5)', '[5,30)', '[30,60)', '[60,120)', '[120,250)', '250+'} str "0"
register_days The days since this user has registered. int64 107
register_days_range The range of the registered days. In the set of {'15-30', '31-60', '61-90', '91-180', '181-365', '366-730', '730+'}. str "61-90"
onehot_feat0 An encrypted feature of the user. Each value indicate the position of "1" in the one-hot vector. Range: {0,1} int64 0
onehot_feat1 An encrypted feature. Range: {0, 1, ..., 7} int64 1
onehot_feat2 An encrypted feature. Range: {0, 1, ..., 29} int64 17
onehot_feat3 An encrypted feature. Range: {0, 1, ..., 1075} int64 638
onehot_feat4 An encrypted feature. Range: {0, 1, ..., 11} int64 2
onehot_feat5 An encrypted feature. Range: {0, 1, ..., 9} int64 0
onehot_feat6 An encrypted feature. Range: {0, 1, 2} int64 1
onehot_feat7 An encrypted feature. Range: {0, 1, ..., 46} int64 6
onehot_feat8 An encrypted feature. Range: {0, 1, ..., 339} int64 184
onehot_feat9 An encrypted feature. Range: {0, 1, ..., 6} int64 6
onehot_feat10 An encrypted feature. Range: {0, 1, ..., 4} int64 3
onehot_feat11 An encrypted feature. Range: {0, 1, ..., 2} int64 0
onehot_feat12 An encrypted feature. Range: {0, 1} int64 0
onehot_feat13 An encrypted feature. Range: {0, 1} int64 0
onehot_feat14 An encrypted feature. Range: {0, 1} int64 0
onehot_feat15 An encrypted feature. Range: {0, 1} int64 0
onehot_feat16 An encrypted feature. Range: {0, 1} int64 0
onehot_feat17 An encrypted feature. Range: {0, 1} int64 0
Owner
Chongming GAO (高崇铭)
A Ph.D. student at Lab for Data Science, USTC. Research Interests: Recommender Systems.
Chongming GAO (高崇铭)
Unofficial implementation of Proxy Anchor Loss for Deep Metric Learning

Proxy Anchor Loss for Deep Metric Learning Unofficial pytorch, tensorflow and mxnet implementations of Proxy Anchor Loss for Deep Metric Learning. Not

Geonmo Gu 3 Jun 09, 2021
Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models

Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models Abstract Many applications of generative models rely on the marginali

Stanford Intelligent Systems Laboratory 9 Jun 06, 2022
First-Order Probabilistic Programming Language

FOPPL: A First-Order Probabilistic Programming Language This is an implementation of FOPPL, an S-expression based probabilistic programming language d

Renato Costa 23 Dec 20, 2022
AI Flow is an open source framework that bridges big data and artificial intelligence.

Flink AI Flow Introduction Flink AI Flow is an open source framework that bridges big data and artificial intelligence. It manages the entire machine

144 Dec 30, 2022
A modern pure-Python library for reading PDF files

pdf A modern pure-Python library for reading PDF files. The goal is to have a modern interface to handle PDF files which is consistent with itself and

6 Apr 06, 2022
Spatial Contrastive Learning for Few-Shot Classification (SCL)

This repo contains the official implementation of Spatial Contrastive Learning for Few-Shot Classification (SCL), which presents of a novel contrastive learning method applied to few-shot image class

Yassine 34 Dec 25, 2022
113 Nov 28, 2022
Image Segmentation with U-Net Algorithm on Carvana Dataset using AWS Sagemaker

Image Segmentation with U-Net Algorithm on Carvana Dataset using AWS Sagemaker This is a full project of image segmentation using the model built with

Htin Aung Lu 1 Jan 04, 2022
Spatiotemporal resampling methods for mlr3

mlr3spatiotempcv Package website: release | dev Spatiotemporal resampling methods for mlr3. This package extends the mlr3 package framework with spati

45 Nov 21, 2022
Contextual Attention Network: Transformer Meets U-Net

Contextual Attention Network: Transformer Meets U-Net Contexual attention network for medical image segmentation with state of the art results on skin

Reza Azad 67 Nov 28, 2022
The first machine learning framework that encourages learning ML concepts instead of memorizing class functions.

SeaLion is designed to teach today's aspiring ml-engineers the popular machine learning concepts of today in a way that gives both intuition and ways of application. We do this through concise algori

Anish 324 Dec 27, 2022
《Improving Unsupervised Image Clustering With Robust Learning》(2020)

Improving Unsupervised Image Clustering With Robust Learning This repo is the PyTorch codes for "Improving Unsupervised Image Clustering With Robust L

Sungwon Park 129 Dec 27, 2022
A quick recipe to learn all about Transformers

Transformers have accelerated the development of new techniques and models for natural language processing (NLP) tasks.

DAIR.AI 772 Dec 31, 2022
A pytorch-based real-time segmentation model for autonomous driving

CFPNet: Channel-Wise Feature Pyramid for Real-Time Semantic Segmentation This project contains the Pytorch implementation for the proposed CFPNet: pap

342 Dec 22, 2022
QuickAI is a Python library that makes it extremely easy to experiment with state-of-the-art Machine Learning models.

QuickAI is a Python library that makes it extremely easy to experiment with state-of-the-art Machine Learning models.

152 Jan 02, 2023
RetinaFace: Deep Face Detection Library in TensorFlow for Python

RetinaFace is a deep learning based cutting-edge facial detector for Python coming with facial landmarks.

Sefik Ilkin Serengil 512 Dec 29, 2022
MADE (Masked Autoencoder Density Estimation) implementation in PyTorch

pytorch-made This code is an implementation of "Masked AutoEncoder for Density Estimation" by Germain et al., 2015. The core idea is that you can turn

Andrej 498 Dec 30, 2022
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in Tensorflow Lite.

TFLite-msg_chn_wacv20-depth-completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model

Ibai Gorordo 2 Oct 04, 2021
SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation

SE3 Pose Interpolation Pose estimated from SLAM system are always discrete, and

Ran Cheng 4 Dec 15, 2022
Implementation of Kronecker Attention in Pytorch

Kronecker Attention Pytorch Implementation of Kronecker Attention in Pytorch. Results look less than stellar, but if someone found some context where

Phil Wang 16 May 06, 2022