The purpose of getting out media core is to allow the file management to be independent of the annotation UI, and make all the annotations UI's to only receive a single file without consciousness of how that file was fetched.

This is a requirement to have before starting to implement the actual text interface: https://github.com/diffgram/diffgram/pull/45

Hi all,

I have an issue when I trying to connect with s3 to upload my dataset. I already have all permissions (fullS3Access). But still facing this error:

{'status_text': 'Exceeded retry limit, no valid response'}.

Would you please help me with this problem?

Seemed that the real problem was not so much with hot keys but with the selected prop of the instances. When we changed frames the watcher would always reset to false the instance select and that would cause some of the hotkeys for related to selection to stop working (like copy and paste).

This was cause due to not updating the this.previous_current_frame value in the watcher. Now the select is working properly and the hotkeys are following

I also tested the w,a,s,d the space bar, the escape keys, numkeys 1-9 and they all seem to be working fine.

Please let me know if you are aware of any other hotkeys malfunctioning so I can dive deeper into them here.

In this PR:

• [x] Add new Label Schema Model

• [x] Refactor frontend services to centralize api calls.

• [x] Add new schema_id prop to label selectors.

• [x] Add new UI to allow for schema creation, deletion and edition.

• [x] Add support for selecting schemas instead of label files in task templates.

• [x] Change Task template detail section so that users can edit/schemas instead of labels.

• [x] Change Schema Editor Section to Support Schema Concepts

• [x] Added label schema selector on Annotation UI Images

• [x] Added label schema selector on Text Annotation

• [x] Added Label Schema Selector on 3D Annotation

• [x] Preload Label Schema on Task Context for all Interfaces Types

• [x] Change API Route names on the SDK /refresh to /labels

• [x] Update job card list view to show schema instead of labels

• [x] Update Job detail tab to be able to update schema instead of labels.

• [x] Add Unit tests

• [x] Add e2e tests

**Edit this is added https://github.com/diffgram/diffgram/pull/857 https://github.com/diffgram/diffgram/releases/tag/0.15.0 **End Edit

Diffgram only support clould storage?

This new feature aims to provide an easier way to upload files with prelabeled data without requiring any coding at all.

• Creating new components for upload wizard
• Create a new schema mapping concept to map any JSON file to diffgram's format.
• Added stronger bindings to the input_batch concept.\
• Added new alembic migrations to support prelabeled data at the batch level.
• Pending to modify some E2E tests as the old upload flow will break with this new code.

Adding this PR as a draft and WIP to give visibility of work progress.

Adds Custom UI Schema Controls

https://www.loom.com/share/78e71d30405c4331a186b3da77ee4c8f

V1 Has some rough edges Still WIP for the HTTP end points to save the schema DB And to wire that to tasks etc

Directionally this will be a 'great what you see is what you get' way to further customize the UI. It will also help support other UI customizations like enable and disable features, alternate styles etc.

Also fixes image sizing in task mode

Add Annotator Focused Screen

A lot of this is UI centric. In terms of new functions, the "Annotate" button now

1. Crosscuts jobs. Gets next across project. Means the Annotator does not need to know which job they are working on. Also a starting point to build more around QA centric tasks UI.
2. Uses SkipLock to strengthen guarantees. Concept that we may build this into the "on complete" / get next during normal annotation. So if the tasks happen to be small, and there are many annotators, there's a collision chance.
3. https://github.com/diffgram/diffgram/commit/9e8e49df5c85e2533ae57b4c76108c7151cf9e8c#diff-e0558f300c30e0d7e278609a6fa9363326ff3dc3871d962f7ceb28cc4f7e809eR316 Checks job ids are either in jobs assigned to user or open to all.

The nuance here is that it's project wide, but if the job is assigned to me, or is set to to invite only, then it's not eligible.

Tangent: Note that for status, we are checking that the status == available. The assumption here is the status is the more key thing then the user assigned. We could also check there is no user assigned, but then if an admin wants to assign it to someone else, we would have to expressly clear that or something.

1. Show assigned users on task view
2. Show in progress

Cleaner and User Customizable Task Home

1. Now uses regular table
2. Defaults to show more typical user features
3. Shows annotation count from preview

Better task assignment

Improves task assignment concept.

1. From Task Detail Page
2. From Complete/Defer in Annotation

Now those actions will by default assign the task the user gets to themselves. Task assignment etc still needs some more work. It's a step in the direction of making that more visible and hardening that

After #1070,now i can upload.During uploading,comes another exception. this is my console error: so,i go to check the docker-walrus,it gives me this:

There is no minimum limitation of multipart upload on S3 API of minio server.

How can i change the code inside shared/data_tools_core_s3.py line 153? @PJEstrada

Here is my temporary solution,i comment out the MultipartUpload inside shared/data_tools_core_s3.py line 148-154 related. When i start to run the upload script,minio server connection error occurs.This is werid,i dont know why.Is this about message queue jam issue?Sometimes when i start upload script,the message queue dont handle it.MQ always do the heartbeat check,just miss the real job!

After several tries,finally i get upload worked.I dont do anything except restarting all diffgram docker services.

Saddly there is always a but,here comes an error. docker hosted diffgram view: I go to check the minio buckets page,there is no uploaded file.

Does it mean my file uploading fail?I dont know how to edit code inside shared/data_tools_core_s3.py,it seems that my temporary solution will lead to unexpected errors.

Context

I've tried to use Minio (as a local storage provider) with Diffgram but i've encurred in an issue when trying to create a new connection from Diffgram to Minio.

Setup:

1- Install Minio (I've also tried with the docker version, but got the same error) wget https://dl.min.io/server/minio/release/linux-amd64/minio chmod +x minio MINIO_ROOT_USER=admin MINIO_ROOT_PASSWORD=password ./minio server /home/ubuntu/minio_test_data --console-address ":9001"

I have then created a bucket called 'images' and uploaded some data in it.

2- Install Diffgram 1.1.11 git clone https://github.com/diffgram/diffgram.git cd diffgram pip install -r requirements.txt python3 install.py

Selected Minio Web Server Services Storage (which is running in the same machine), inserted the required parameters and the tests: Connection, Write permission and Read permission are OK.

Error

I logged into Diffgram (from the webpage), tried to create a new connection to Minio, so: project -> connections -> new connection I used the same parameters as i've inserted in the install.py but i got this error:

I think the original file got uploaded instead of the thumbnail. Result is shown below (file 14 + thumb is old code, file 15 + thumb is the new code)

This will fix #592

A Task was created related to a Dataset but when more files were imported to the Dataset, the number of files in task did not increased.

Task was created with long term sync (see screenshot attached). Dataset currently has 17, 919 but task has 17, 617. I will appreciate your kind support on this issue.

Description

...

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.

I have...

• [ ] added ! to the type prefix if Breaking Changes.
• [ ] considered the impact to the SDK.
• [ ] included the correct type prefix in the PR title
• [ ] provided a link to the relevant issue in JIRA or Github Link in PR.
• [ ] included the unit and integration tests
• [ ] included comments & docs for new API Endpoints
• [ ] updated the relevant documentation or specification in readme.io
• [ ] reviewed "Files changed" and left comments if necessary
• [ ] confirmed all CI checks have passed

Breaking Changes including adding required params.

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.

I have...

• [ ] confirmed the Author checklist was followed
• [ ] reviewed API design and naming
• [ ] reviewed documentation is accurate
• [ ] reviewed tests and test coverage
• [ ] manually tested (if applicable)

Description

...

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.

I have...

• [ ] added ! to the type prefix if Breaking Changes.
• [ ] considered the impact to the SDK.
• [ ] included the correct type prefix in the PR title
• [ ] provided a link to the relevant issue in JIRA or Github Link in PR.
• [ ] included the unit and integration tests
• [ ] included comments & docs for new API Endpoints
• [ ] updated the relevant documentation or specification in readme.io
• [ ] reviewed "Files changed" and left comments if necessary
• [ ] confirmed all CI checks have passed

Breaking Changes including adding required params.

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.

I have...

• [ ] confirmed the Author checklist was followed
• [ ] reviewed API design and naming
• [ ] reviewed documentation is accurate
• [ ] reviewed tests and test coverage
• [ ] manually tested (if applicable)

Need reports like https://docs.google.com/spreadsheets/d/1-AK64UpC6EQfT-_Vdj-OWwdrcvxrjHSFD7N-WKh1pJc/edit#gid=0

The current state is that you can drill down per user, however it only shows one user then at a time. Or you can see per user, but it losses over time.

There was a related one that was Task / By Task Status, that exported well (and maybe did both?), however it appears to have regressed.

Some of this may not graph very well, (since for every date n number of users) so may want to have it more with the intent of being able to download to a spreadsheet or be accessible via API for further filtering by analysts etc.

Strategically, the shift is to limit expectations for what kind of reporting is available natively in Diffgram. And better support accessing it from the API, third party reporting tools, etc. etc. There still needs to be some level of abstraction between the "raw" concepts and something usable for reporting.

Examples are the tasks per person count and average time

Workaround

Click "save and refresh" Then can edit and use new report template like normal.

Note this is only for new templates, existing reports are unaffected

Issue

When clicking "New report" throws 500

{error: "index.html"}
error
: 
"index.html"