“英特尔创新大师杯”深度学习挑战赛 赛道3：CCKS2021中文NLP地址相关性任务

Related tags

Deep Learningccks2021-track3-top1
Overview

ccks2021-track3

CCKS2021中文NLP地址相关性任务-赛道三-冠军方案

团队：我的加菲鱼- wodejiafeiyu

初赛第二/复赛第一/决赛第一

前言

19年开始，陆陆续续参加了一些比赛，拿到过一些top，比较懒一直都没分享过，这次比较幸运又拿了top1，打算分享下

分类的任务用这个框架基本都能top10，之前个人参加的《全球人工智能大赛-赛道一》和这个也差不多，只是有些具体任务的trick不同

比赛主页

https://tianchi.aliyun.com/competition/entrance/531901/introduction

环境

  • torch==1.6.0
  • transformers=3.0.2

预训练模型

- nezha-base 
- nezha-wwm
- macbert
  • 下载预训练模型，放到文件夹user_data/model_param/pretrain_model_param/下，文件夹和模型名字一一对应

全流程脚本

sh run.sh

核心思路

  • 预训练mlm中的mask策略使用ngram-mask，相比原始的动态mask，提升了预训练难度
  • 标签也包含语义信息，预训练部分融入标签信息，提升预训练效果
  • 混合精度预训练，损失一部分精度，提升整体的训练速度，实际测试结果精度损失不大，速度提升明显
  • 对抗训练，已经是一个比较常用的trick了
  • 后12层加上embedding的cls动态加权平均
  • multi-sample dropout
  • 推理的时候，阈值搜索
  • 三折交叉验证，每折使用不同的随机数种子使用dynamic pad
  • 加权平均，模型融合

总结

模块 提分点分析 提升
ngram-mask 相比于单个字的遮蔽，ngram-mask加大了预训练任务的难度，从而提升效果 2.6个千分点
融入标签信息预训练 标签也含有语义信息，模型学习的更多 1个千分点
后12层加上embedding的cls动态加权平均+multi-sample dropout 加权平均能增强向量的语义表征能力从而提升效果。multi-sample dropout能加速训练，增强泛化能力 3.3个千分点
对抗训练fgm 生成对抗样本，对抗样本的训练，增加模型泛化能力 2.2个千分点
多分类阈值搜索 缩放系数使得f1最优 0.7个千分点
模型融合 提升模型的鲁棒性和泛化能力 1.5个千分点
动态pad和预训练混合精度训练和每折使用不同的随机数种子 提升模型训练速度，不同的随机数种子加大了模型的差异，提升融合的效果
Owner
shaochenjie
shaochenjie
GitHub Repository
A hybrid framework (neural mass model + ML) for SC-to-FC prediction

The current workflow simulates brain functional connectivity (FC) from structural connectivity (SC) with a neural mass model. Gradient descent is applied to optimize the parameters in the neural mass

Yilin Liu 1 Jan 26, 2022
CM building dataset Timisoara

CM_building_dataset_Timisoara Date created: Febr-2020 The Timi\c{s}oara Building Dataset - TMBuD - is composed of 160 images with the resolution of 76

Orhei Ciprian 5 Sep 07, 2022
HAR-stacked-residual-bidir-LSTMs - Deep stacked residual bidirectional LSTMs for HAR

HAR-stacked-residual-bidir-LSTM The project is based on this repository which is presented as a tutorial. It consists of Human Activity Recognition (H

Guillaume Chevalier 287 Dec 27, 2022
This repository contains the code for TACL2021 paper: SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

SummaC: Summary Consistency Detection This repository contains the code for TACL2021 paper: SummaC: Re-Visiting NLI-based Models for Inconsistency Det

Philippe Laban 24 Jan 03, 2023
A clean and scalable template to kickstart your deep learning project 🚀 ⚡ 🔥

Lightning-Hydra-Template A clean and scalable template to kickstart your deep learning project 🚀 ⚡ 🔥 Click on Use this template to initialize new re

Hyunsoo Cho 1 Dec 20, 2021
Transformers are Graph Neural Networks!

🚀 Gated Graph Transformers Gated Graph Transformers for graph-level property prediction, i.e. graph classification and regression. Associated article

Chaitanya Joshi 46 Jun 30, 2022
Dynamic Bottleneck for Robust Self-Supervised Exploration

Dynamic Bottleneck Introduction This is a TensorFlow based implementation for our paper on "Dynamic Bottleneck for Robust Self-Supervised Exploration"

Bai Chenjia 4 Nov 14, 2022
Improving Contrastive Learning by Visualizing Feature Transformation, ICCV 2021 Oral

Improving Contrastive Learning by Visualizing Feature Transformation This project hosts the codes, models and visualization tools for the paper: Impro

Bingchen Zhao 83 Dec 15, 2022
EquiBind: Geometric Deep Learning for Drug Binding Structure Prediction

EquiBind: geometric deep learning for fast predictions of the 3D structure in which a small molecule binds to a protein

Hannes Stärk 355 Jan 03, 2023
Self Driving RC Car Code

Derp Learning Derp Learning is a Python package that collects data, trains models, and then controls an RC car for track racing. Hardware You will nee

Not Karol 39 Dec 07, 2022
Self-Supervised Learning for Domain Adaptation on Point-Clouds

Self-Supervised Learning for Domain Adaptation on Point-Clouds Introduction Self-supervised learning (SSL) allows to learn useful representations from

Idan Achituve 66 Dec 20, 2022
Reproduced Code for Image Forgery Detection papers.

Image Forgery Detection With over 4.5 billion active internet users, the amount of multimedia content being shared every day has surpassed everyone’s

Umar Masud 15 Dec 06, 2022
Unofficial PyTorch implementation of Neural Additive Models (NAM) by Agarwal, et al.

nam-pytorch Unofficial PyTorch implementation of Neural Additive Models (NAM) by Agarwal, et al. [abs, pdf] Installation You can access nam-pytorch vi

Rishabh Anand 11 Mar 14, 2022
GLANet - The code for Global and Local Alignment Networks for Unpaired Image-to-Image Translation arxiv

GLANet The code for Global and Local Alignment Networks for Unpaired Image-to-Image Translation arxiv Framework: visualization results: Getting Starte

stanley 29 Dec 14, 2022
Baseline inference Algorithm for the STOIC2021 challenge.

STOIC2021 Baseline Algorithm This codebase contains an example submission for the STOIC2021 COVID-19 AI Challenge. As a baseline algorithm, it impleme

Luuk Boulogne 10 Aug 08, 2022
Audio Source Separation is the process of separating a mixture into isolated sounds from individual sources

Audio Source Separation is the process of separating a mixture into isolated sounds from individual sources (e.g. just the lead vocals).

Victor Basu 14 Nov 07, 2022
Self-Supervised Monocular DepthEstimation with Internal Feature Fusion(arXiv), BMVC2021

DIFFNet This repo is for Self-Supervised Monocular DepthEstimation with Internal Feature Fusion(arXiv), BMVC2021 A new backbone for self-supervised de

Hang 94 Dec 25, 2022
MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Driving

MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Driving Code will be available soon. Motivation Architecture

Kai Chen 24 Apr 19, 2022
The implementation of the lifelong infinite mixture model

Lifelong infinite mixture model 📋 This is the implementation of the Lifelong infinite mixture model 📋 Accepted by ICCV 2021 Title : Lifelong Infinit

Fei Ye 5 Oct 20, 2022
[ICME 2021 Oral] CORE-Text: Improving Scene Text Detection with Contrastive Relational Reasoning

CORE-Text: Improving Scene Text Detection with Contrastive Relational Reasoning This repository is the official PyTorch implementation of CORE-Text, a

Jingyang Lin 18 Aug 11, 2022