当前位置：网站首页>Tools to measure the gap between two distributions: cross entropy and KL divergence

Tools to measure the gap between two distributions: cross entropy and KL divergence

2022-07-18 13:30:00 【Wsyoneself】

Cross entropy ：
1. entropy = Best average coding
2. Entropy reflects the uncertainty of events in information theory , The greater the entropy , The more uncertain the event is , The more dispersed the possibilities , Then the information must be longer
3. Put it from a distribution q Of messages using another distribution p The average message length conveyed by the best code becomes cross entropy .
4. p and q The more different the length of ,p be relative to q The cross entropy of will be greater than p The entropy of
5. Cross entropy is often used as the loss function of classifiers
KL The divergence ： The difference between entropy and cross entropy , If the two distributions are the same , The difference is 0, This difference is KL The divergence ：
Likelihood function is our objective function . If you put a minus sign in front of it , It turns into a loss function .

版权声明
本文为[Wsyoneself]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/199/202207160325260969.html