概
考虑流行度偏差的知识蒸馏, 应用于推荐系统.
UnKD
-
Motivation 就不讲了, 感觉不是很强烈.
-
方法很简单, 就是将按照流行度给 items 进行分组 (实际上, 实验中仅仅分了两个组).
-
然后, 对每个用户, 每个 group 进行组内的排序, 然后按照排名采样正负样本 (话说负样本的概率分布是啥?):
\[p_i \propto e^{-rank(i) / \mu}. \] -
接着, 计算如下的蒸馏损失:
\[\mathcal{L}_{G} = -\sum_{u} \frac{1}{|\mathcal{U}|} \sum_{g \in \mathcal{G}} \sum_{(i^+, i^-) \in \mathcal{S}_{ug}} \log \sigma (\mathbf{e}_u^T \mathbf{e}_{i^+} - \mathbf{e}_u^T \mathbf{e}_{i^-}). \] -
最后的损失为:
\[\mathcal{L} = \mathcal{L}_R + \lambda \mathcal{L}_G. \]
代码
[official]
- Recommendation Distillation Knowledge Unbiased forrecommendation distillation knowledge unbiased understanding distillation knowledge empirical distillation relational knowledge distillation recommender knowledge framework distillation target-aware transformer knowledge distillation innovative knowledge landscape distillation knowledge stronger teacher distillation unsupervised adaptation knowledge distillation decoupled knowledge recommendation time-aware knowledge reasoning