The Open edX platform, the software that powers edX!

Sometimes we were seeing a connection timeout when booting up ecs containers. We should be catching this error (like we are with ClientError and retrying.

@cpennington Could you check this one out? @rocha Note the analytics issue below @jinpa How about you do testing, it is your story I believe!

Just FYI: @jaericson @shnayder

Adds studio and lms support for an option to shuffle the displayed order of multiple choice questions. Works by shuffling the xml tree nodes in the problem during the get_html process. The shuffling uses the problem's seed. The added syntax for the xml and markdown is documented in the Changelog.

One concern was: will this mess up analytics? My experiments suggest that analytics will work fine:

The question is -- when the question choices are displayed shuffled, is enough logged so that analytics still works?

I have a 4-option problem with shuffling enabled. In this case, the options are presented in the order b a d c (i.e. 1 0 3 2). I selected option c, the last one displayed (which happens to be correct), and then looking in courseware_studentmodulehistory I have the following row

2013-10-22 22:23:40.658495|1.0|1.0|{"correct_map": {"i4x-Stanford-CS99-problem-408c0fcffb8c41ec87675d8c3f7a3b5b_2_1": {"hint": "", "hintmode": null, "correctness": "correct", "npoints": null, "msg": "", "queuestate": null}}, "input_state": {"i4x-Stanford-CS99-problem-408c0fcffb8c41ec87675d8c3f7a3b5b_2_1": {}}, "attempts": 2, "seed": 282, "done": true, "student_answers": {"i4x-Stanford-CS99-problem-408c0fcffb8c41ec87675d8c3f7a3b5b_2_1": "choice_2"}}||4|52

In student_answers I see choice_2 which is "c" in the 0-based numbering, so that's right. It looks to me that it has successfully recorded which choice I made, not getting confused by the fact that the options where displayed in a different order. The rest of the stuff in the rows looks reasonable, but mostly it's greek to me.

Because we're using standard python shuffle, knowing the seed, you can recreate the shuffle order. Maybe you never need to do this since the problem will just do it for you. Still. Noting that the logged seed above is 282:

r = random.Random(282) a = [0, 1, 2, 3] r.shuffle(a) a [1, 0, 3, 2]

Cute!

This PR adds new "extended hint" features for many question types.

For example per-choice feedback..

  >>What is your favorite color?<<
  () Red   {{A lot of people think it's Red, but those people are wrong.}}
  () Green
  (x) Blue

When the user tries Red as an answer, the extended hint appears. For a sample course .tar.gz with examples of all the extended hint features see https://drive.google.com/file/d/0BzXwXoUU5YRpdVJhbUphT19Hb1E

Here I will outline the implementation side of things.

The implementation has roughly three parts: -The markdown parser, adding support for {{ ... }} as above -The capa xml layer, to store and retrieve extended hints per question -Problem html generation, to show the extended hints

Thomas Brennan-Marquez wrote an original draft of many of these features, and I, Nick Parlante, then did some heavy revision and extension.

1. Markdown Parsing

See edit.coffee and companion edit_spec_hint.coffee

I made a high level decision to not change the old markdown parsing at all, staying quirk for quirk compatible. So all the old tests pass unchanged. The extended hint parsing is layered on top. The original parsing code is written in a rather functional style, so the additions are written that way too.

2. responsetypes.py - get_extended_hints

Many extended hints associate a hint with particular choices in the xml, e.g.

  <choicegroup label="What is your favorite color?" type="MultipleChoice">
    <choice correct="false">Red <choicehint>No, Blue!</choicehint> </choice>

Therefore, for each response type - multiple choice, checkbox, dropdown, text input, numeric input - there's a get_extended_hints() method that digs the appropriate hint out of the xml and puts it in the map 'msg' to go back to the client. In some cases, there is nontrivial logic to pick out the right extended hint depending on the specific student choices.

3. capa_base.py - get_problem_html

Many extended hints are sent to the client through the existing cmap['msg'] mechanism.

The demand hints are added in their own div in the problem.html template.

As an additional wrinkle, with the addition of extended hints, the question xml often has tags that are not just to be echoed to the user, e.g. in the above. Therefore, the get-html path needs logic to strip out these tags. Previously, the xml was almost always just stuff to show the user, so there was nothing to remove.

4. Other XML changes:

The PR changes additional_answer to use an attribute, as below, to be consistent with the other text-input cases. Compatibility code is provided.

  <additional_answer answer="Blue"> <correcthint>hint2</correcthint> </additional_answer>

The PR provides a longer xml form, as below, that also supports per-option hints. Compatibility is provided.

<optioninput>
  <option correct="False">dog <optionhint>No, too friendly</optionhint> </option>
  <option correct="True">cat</option>
  <option correct="False">parrot</option>
</optioninput>

The various studio question templates have been updated to show off the new extended-hint features in markdown.

Add: With this PR, the xml has many tags, such as which are in the problem, but which are not supposed to be echoed in the client html. These are stripped out explicitly in get_problem_html(). Before this PR, the <additional_answer> node was kept out of the html by deleting it from the XML tree during parsing. This tag-stripping, now with many more types of tag to keep out, is now done just once in get_problem_html.

Add: added 2 markdown syntax cases, not expanding any underlying capability, just what cases can be done in the markdown shorthand:

1. not= -- express the stringresponse <stringequalhint> case, a feedback matching a specific incorrect answer
2. s= -- force a text-entry question to be stringresponse, instead of the usual behavior of guessing numericalresponse vs. stringresponse by looking at the syntax of the answer

For a cohorted course, we needed a way to set inline discussions non-cohorted by default and select manually the discussions that need to be cohorted. 2 course settings added:

• always_cohort_inline_discussions: Set it to False to get the inline discussions NON-cohorted by default for a cohorted course. Then, you can add the inline discussions id in the cohorted_discussions array:

{
    "always_cohort_inline_discussions": false,
    "cohorted_discussions": [
        "<inline discussion id obtained from an inline discussion block>"
    ],
    "cohorted": true
}

The implementation is simple and doesn't affect much code.

Sandbox URL:

• LMS, with course configured to use the feature: http://sandbox2.opencraft.com/courses/TestX/TST-COHORT-1/now/about
• Studio: http://sandbox2.opencraft.com:18010/

cc @antoviaque

Background: The edX platform includes a number of features that use S3 directly. These include:

• Instructor CSV data exports
• Video storage
• xqueue
• Student Identity Verification (photo uploads)
• Analytics event dumps (via a logrotate script)

For deployments outside of AWS it would make sense to use a different cloud storage provider, or even the local filesystem.

In particular, OpenStack deployments will want to use an OpenStack Swift Object Store for user-uploaded content. Although Swift offers S3 API compatibility as a plugin (called "Swift3") in theory, in practice the compatibility is limited, often not installed on the OpenStack instance, and hard to get working.

This pull request is an attempt to solve this problem by replacing all S3-specific code with the django storage API.

Studio updates: None yet, although video uploads will need to be modified. I am currently holding off on this as it will require changes to VEDA and I don't have access to that repo. Once Greg Martin has made the necessary changes to VEDA I will update Studio in a separate PR.

LMS updates: Instructor CSV exports and student photo uploads modified to use the django storage API. Added an openstack.py settings file.

Sandbox: http://pr11286.sandbox.opencraft.com/

Related PRs:

• https://github.com/edx/configuration/pull/2723
• https://github.com/edx/xqueue/pull/103
• https://github.com/edx/edx-platform/pull/11374

Testing: Deploy a sandbox using https://github.com/edx/configuration/pull/2723. Use these ansible vars, completed with your openstack credentials.

To test the instructor CSV reports:

• Go to the instructor dashboard for a course
• Open the 'data download' tab
• Click 'generate grade report'
• An entry for the report should appear in the list below
• Check that the report CSV has been uploaded to the swift container
• Click the link to download the report directly from swift

To test student identity verification:

• In the django admin, create a new CourseMode for an existing course. Set the mode to verified, and make sure the price is > 0 (https://github.com/edx/edx-platform/pull/11374)
• In the LMS course list, there should now be an 'upgrade' button if you are already enrolled in the course. Click it
• You will be presented with a payment page. We need to get past this to get to the ID verification step. Run ./manage.py lms --settings=openstack shell, then:

from django.contrib.auth.models import User
from opaque_keys.edx.keys import CourseKey
from student.models import CourseEnrollment

user = User.objects.get(username='<your_username>')
course_key = CourseKey.from_string('course-v1:edX+DemoX+Demo_Course')
enrollment = CourseEnrollment.get_or_create_enrollment(user, course_key)
enrollment.update_enrollment(mode='verified')

Alternatively, you can simply update the CourseEnrollment in the django admin:

This fools the LMS into thinking that the course has already been paid for. Go back to the payment page, and you should see the webcam verification step. Note that Chrome will not allow access to the webcam over plain http unless you are running on localhost. Firefox doesn't seem to mind.

To test xqueue:

• In studio, add a blank problem and set its xml to:

<problem display_name="XQueue Test">
   <text>
       <p>Upload a program that prints "hello world".</p>
   </text>
   <coderesponse queuename="edX-Open_DemoX">
       <filesubmission/>
       <codeparam>
           <initial_display>
             # students please write your program here
             print ""
           </initial_display>
           <answer_display>
             print "hello world"
           </answer_display>
           <grader_payload>
             {"output": "hello world"}
           </grader_payload>
       </codeparam>
   </coderesponse>
</problem>

• Go to that problem in LMS (xqueue does not work in studio), select a file and click the 'Check' button
• You should see this:
• The submission should now be in the configured swift bucket

To test log sync to swift:

SSH into the sandbox, and run

sudo service supervisor stop

This should trigger a log sync, after which the tracker logs can be found in a swift container.

Future tasks: write some tests for django-swift-storage. It doesn't have any yet.

Settings

EDXAPP_SETTINGS: 'openstack'
XQUEUE_SETTINGS: 'openstack_settings'
COMMON_VHOST_ROLE_NAME: 'openstack'
edx_ansible_source_repo: 'https://github.com/open-craft/configuration.git'
configuration_version: 'omar/openstack'
xqueue_source_repo: 'https://github.com/open-craft/xqueue.git'
xqueue_version: 'omar/django-storage-api'

This PR implements "Personal Online Courses" or POCs (working title). POCs are a simplified approach to SPOCs that allow for a course (or portions of a course) to be reused with a small groups of students. Course instructors assign a "POC Coach" who can create a POC, choose a subset of course modules for the POC (with start and end dates) and administer students. Importantly, POC Coaches cannot change the content and do not have the same permissions as course staff.

POCs were originally proposed on the edx code list last year and have been discussed with edX Product (including @explorerleslie ) and with edX Engineering ( @cpennington and @nedbat ).

This feature is intended to be used on edX.org. In particular, we are fielding requests to use this feature with 15.390.1x and 15.390.2x. This PR only affects LMS.

For manual testing, please view the demo screencast. The edX Demo course can be used for testing.

We are under constant demand for this feature. We would like to have it merged by the end of January.

For this WIP PR we have a couple of issues that we could use some help with:

1. This PR has a dependency on PR https://github.com/edx/edx-platform/pull/5802 which provides the settings override feature that allows POC start and end dates to be applied just for POC students.
2. Since POCs may omit some content from the MOOC, we need to override grading policy. We would like feedback from @ormsbee on our approach.
3. We have introduced a new role -- POC Coach (working title) -- and a new course tab to go with it. @chrissrossi inquired about the best approach to handling tab permissions on the mailing list but we would like a confirmation that the approach suggested by @andyarmstrong is OK. We have not yet implemented this.

All the authors of this PR are covered under the edX-MIT agreement.

I was seeing a flaky failure once in a while inside CrossStoreXmlRoundtrip, specifically from the test_round_trip test. Here's an example: https://build.testeng.edx.org/view/edx-platform-pipeline-pr-tests/job/edx-platform-python-pipeline-pr/1288/

I know the odds of 2 processes interfering because of a randomly generated number in such a large range is small, but keep in mind thanks to ddt, this test is run 36 times per build, and runs for a total of about 15 minutes. Mix in the fact that this was only failing maybe 10% of the time, and it seems like it actually could be related. Regardless, sticking with the theme of tying the process id to the db name seems consistent and better than a random number.

This is a public, shared, versioned, RESTful API for importing and exporting full course content. The code was initially ripped from the existing import/export API in the CMS contentstore djangoapp and wrapped in Django Rest Framework view classes. It's simply a new djangoapp in the openedx directory with 4 views:

• GET /api/v1/courses - List courses (course keys) for which the user has full author access
• GET /api/v1/courses/{course_key} - Export a full course in the form of a tarball
• POST /api/v1/courses/{course_key} - Import a full course tarball
• GET /api/v1/courses/{course_key}/import_status/{filename} - View the status of a course import (see https://github.com/edx/edx-platform/pull/6190/files#diff-4c6e0b7531e770ee217a370f8d990ac8R108)

This PR includes configuration changes. Most notably, Studio is configured to serve the OAuth2 provider alongside the LMS.

Here's the relevant thread on the code list: https://groups.google.com/forum/#!msg/edx-code/DmnHWmly25A/ZqjD1zb4o7oJ

There are 28 non-covered lines, all of which are missing coverage in the existing CMS API. They're mostly error conditions, such as handling of multipart file upload errors.

TNL-4759

Description

Redesign inline discussions to have "two-level" UI.

Sandbox

• [x] Inline discussions
• [x] Teams
• [x] Discussion board (this didn't technically change, but good to check)

Testing

• [x] i18n
• [x] RTL
• [x] safecommit violation code review process
• [x] Unit, integration, acceptance tests as appropriate
• [x] Analytics
• [x] Performance
• [x] Database migrations are backwards-compatible

Reviewers

If you've been tagged for review, please check your corresponding box once you've given the :+1:.

• [x] Code review: @andy-armstrong
• [x] Code review: @alisan617
• [x] Doc Review: @catong
• [ ] UX review: @chris-mike
• [ ] Accessibility review: @cptvitamin
• [x] Product review: @marcotuts

FYI: @dianakhuang

Post-review

• [ ] Rebase and squash commits

This adds ability to refresh the list of uploaded videos without refreshing the whole page.

~~Added a refresh button that when clicked:~~ An event is triggered on each successful upload that:

• fetches a fresh list of uploaded files from the server
• updates PreviousVideoUploadListView
• removes the successfully completed uploads from ActiveVideoUploadListView
• retains the ongoing or failed uploads in ActiveVideoUploadListView so that they can be monitored/retried

~~The view can also be refreshed without user action, but I felt it may be less surprising to have a button instead.~~

Sandbox

• Sandbox

Reviewers

If you've been tagged for review, please check your corresponding box once you've given the :+1:.

• [x] Code review: @muzaffaryousaf
• [x] Code review: @mushtaqak
• [x] Accessibility review: @cptvitamin
• [x] Product review: @marcotuts or @sstack22

cc: @antoviaque

https://openedx.atlassian.net/browse/TNL-1499 https://openedx.atlassian.net/browse/TNL-1534

The "Connected Accounts" functionality will be implemented in a separate PR.

Description

Should be merged after: https://github.com/openedx/edx-platform/pull/31466 Fourth part of preparing tests for Remove support for children in Old Mongo task (https://github.com/openedx/edx-platform/pull/31134).

• updated test_import.py and test_course_index.py for split modulestore
• fixed 400 error for cms old_style assets

Useful information to include: First part: https://github.com/openedx/edx-platform/pull/31422 Second part: https://github.com/openedx/edx-platform/pull/31427 Third part: https://github.com/openedx/edx-platform/pull/31466 Remove support for children in Old Mongo PR: https://github.com/openedx/edx-platform/pull/31134 https://github.com/openedx/public-engineering/issues/80

Description

This PR fetches the user's country code from the IP address and adds it to learner home.

Ticket: https://2u-internal.atlassian.net/browse/VAN-1120

Python requirements update. Please review the changelogs for the upgraded packages.

Deleted obsolete pull_requests: https://github.com/openedx/edx-platform/pull/31457

Description

Work in progress PR to unify ModuleSystem and DescriptorSystem

Supporting information

OpenCraft internal ticket BB-5967

Testing instructions

Please provide detailed step-by-step instructions for testing this change.

Deadline

"None" if there's no rush, or provide a specific date or event (and reason) if there is one.

Other information

Include anything else that will help reviewers and consumers understand the change.

• Does this change depend on other changes elsewhere?
• Any special concerns or limitations? For example: deprecations, migrations, security, or accessibility.
• If your database migration can't be rolled back easily.

Description

Should be merged after: https://github.com/openedx/edx-platform/pull/31427 Third part of preparing tests for Remove support for children in Old Mongo task (https://github.com/openedx/edx-platform/pull/31134).

• updated test_authoring_mixin.py
• updated test_xblock_utils.py
• updated TestOnboardingView tests (updated default course key)
• updated UsersDefaultRole tests

Useful information to include: First part: https://github.com/openedx/edx-platform/pull/31422 Second part: https://github.com/openedx/edx-platform/pull/31427 Remove support for children in Old Mongo PR: https://github.com/openedx/edx-platform/pull/31134 https://github.com/openedx/public-engineering/issues/80