A Repository of Community-Driven Natural Instructions
TLDR; this repository maintains a community effort to create a large collection of tasks and their natural language definitions/instructions. We're looking for more contributions to make this data bigger!
Background
Why define tasks in natural language?
While the current dominant paradigm (supervised learning with task-specific labeled examples) has been successful in building task-specific models, such models can't generalize to unseen tasks; for example, a model that is supervised to solve questions cannot solve a classification task. We hypothesize that a model equipped with understanding and reasoning with natural language instructions should be able to generalize to any task that can be defined in terms of natural language.
Any empirical evidence that this might be true?
In our earlier effort, we built a smaller data (61 tasks) and observed that language models benefit from language instructions, i.e., their generalization to unseen tasks when they were provided with more instructions.
Also, generalization to unseen tasks improves as the model is trained on more tasks.
Why build this dataset?
We believe that our earlier work is just scratching the surface and there is probably so much that be studied in this setup. We hope to put together a much larger dataset that cover a wider range of reasoning abilities. We believe that this expanded dataset will serve as a useful playground for the community to study and build the next generation of AI/NLP models.
Task definitions
Each consists of input/output. For example, think of the task of sentiment classification:
- Input:
I thought the Spiderman animation was good, but the movie disappointed me. - Output:
Mixed
Here is another example from the same task:
- Input:
The pumpkin was one of the worst that I've had in my life. - Output:
Negative
Additionally, each ask contains a task definition:
Given a tweet, classify it into one of 4 categories: Positive, Negative, Neutral, or Mixed.
Overall, each tasks follows this schema:
Or if you're comfortable with json files, here is how it would look like:
{
"Definition": "",
"Positive Examples": [ { "input": "", "output": "", "explanation": ""} ],
"Negative Examples": [ { "input": "", "output": "", "explanation": ""} ],
"Instances": [ { "input": "", "output": [""]} ],
}
How to contribute
We would appreciate any external contributions!
- All submissions must be submitted via Github pull requests. These submissions will undergo a review before being merged.
- Each task must contain contain a
.jsonfile that contains the task content. You can look inside thetasks/directory for several examples.- Make sure that your json is human readable (use proper indentation; e.g., in Python:
json.dumps(your_json_string, indent=4)) - Make sure that you json file is not bigger than 50MB.
- Make sure your task has no more 6.5k instances (input/output pairs).
- Make sure to number your task json correctly (Look at the task number in the latest pull request, task number in your submission should be the next number). Make sure to include the source dataset name and the task category name while creating the json file name. You can use this format: taskabc__.json E.g. in task001_quoref_question_generation.json, the source dataset is 'quoref' and the task category is 'question generation. Note that, source need not necesserily be a dataset and can be a website e.g. leetcode. Finally, if you have created the json without any reference, use 'synthetic' in place of source.
- Make sure to create a pull request after creating all possible tasks from a dataset. You should have one pull request per dataset.
- If you're building your tasks based existing datasets and their crowdsourcing templates, see these guidelines.
- Make sure that your json is human readable (use proper indentation; e.g., in Python:
- Add your task to our list of tasks.
- To make sure that your addition is formatted correctly, run the tests:
> python src/test_all.py
If you have any questions or suggestions, please use the issues feature.
Frequently Asked Questions
Can I submit non-English tasks?
Yes! We welcome submission from any of the languages.
Can I create tasks without using crowdsourcing templates?
Yes! just make sure that the quality of instructions is good enough for a human to understand the task just based on instructions. You can take a different route than the guidelines.
What is the minimun number of instances I can have in my task?
Anything north of 100 is a safe number. The more, the merrier! Also, you should not have more than 6500 instances.
What do you mean by "meaningful contribution"?
If you're among top k contributors (say, k=25), or if you have contributed at least l tasks (say, l=10). Depending on the overall contributions, we will adjust these constants so that the number of authors don't exeed m (say, m=35).
My commits are not tied to my Github profie; Github does not show all my contributions.
Make sure that your email is set in your git environment and it is also mentioned in your github profile. See this and this.
1 Jun 12, 2022
40 Dec 14, 2022
64 Oct 11, 2022
2 Mar 19, 2022
79 Sep 01, 2022
115 Dec 23, 2022
9 Nov 14, 2022
14 Nov 22, 2022
2 Nov 05, 2021
82 Oct 22, 2022
182 Jan 06, 2023
53 Oct 18, 2022
349 Aug 06, 2022
10 Dec 06, 2022
215 Jan 02, 2023
33 Nov 01, 2022
31 Oct 10, 2022
52 Dec 22, 2022
3 Aug 04, 2022
61 Nov 12, 2022