GitHub repository for the SecureDrop whistleblower platform. Do not submit tips here!

Status

Ready for Review

Description of Changes

Fixes #1195

A cron job runs daily on the app server and updates the

/var/lib/securedrop/submissions_today.txt

file which contains the number of submissions sent in the past 24h, as created by the manage.py how_many_submissions_today command.

The OSSEC agent on the app server runs a command daily, displaying the content of /var/lib/securedrop/submissions_today.txt. The output of the command is sent to the OSSEC server.

A new rule is defined on the OSSEC server to send a mail to when the output is received from the OSSEC agent running on the app server.

A new procmail rule is definied on the OSSEC server to catch mails encrypt mails containing the /var/lib/securedrop/submissions_today.txt string and send them to the email defined by the journalist_alert_email ansible variable.

A new set of (optional) ansible variables, similar to ossec_alert_gpg_public_key, ossec_gpg_fpr, ossec_alert_email are defined: journalist_alert_gpg_public_key, journalist_gpg_fpr, journalist_alert_email. They are used to upload a journalist public key to the OSSEC server and inserted into the send_encrypted_alarm.sh script which handles mails received by procmail.

The modified send_encrypted_alarm.sh script takes one argument (journalist or ossec) and dispatches the mail read from stdin to the corresponding recipient.

Integration tests are implemented to verify the following:

• manage.py how_many_submissions_today
• the app OSSEC agent sends a mail to the journalist address
• cover all branches of send_encrypted_alarm.sh

Testing

• make build-debs
• vagrant up /staging/
• ./testinfra/test.py staging

Running Testinfra suite against 'staging'...
Target roles:
    - testinfra/ossec
==================================================================== test session starts =====================================================================
platform linux2 -- Python 2.7.13, pytest-3.3.1, py-1.5.2, pluggy-0.6.0 -- /home/loic/software/securedrop/virtualenv/bin/python2
cachedir: .cache
rootdir: /home/loic/software/securedrop/securedrop, inifile: setup.cfg
plugins: testinfra-1.7.1, xdist-1.21.0, forked-0.2
collected 10 items                                                                                                                                           

testinfra/ossec/test_journalist_mail.py::TestJournalistMail::test_procmail[ansible://app-staging] SKIPPED                                              [ 10%]
testinfra/ossec/test_journalist_mail.py::TestJournalistMail::test_send_encrypted_alert[ansible://app-staging] SKIPPED                                  [ 20%]
testinfra/ossec/test_journalist_mail.py::TestJournalistMail::test_missing_journalist_alert[ansible://app-staging] SKIPPED                              [ 30%]
testinfra/ossec/test_journalist_mail.py::TestJournalistMail::test_ossec_rule_journalist[ansible://app-staging] SKIPPED                                 [ 40%]
testinfra/ossec/test_journalist_mail.py::TestJournalistMail::test_journalist_mail_notification[ansible://app-staging] SKIPPED                          [ 50%]
testinfra/ossec/test_journalist_mail.py::TestJournalistMail::test_procmail[ansible://mon-staging] PASSED                                               [ 60%]
testinfra/ossec/test_journalist_mail.py::TestJournalistMail::test_send_encrypted_alert[ansible://mon-staging] PASSED                                   [ 70%]
testinfra/ossec/test_journalist_mail.py::TestJournalistMail::test_missing_journalist_alert[ansible://mon-staging] PASSED                               [ 80%]
testinfra/ossec/test_journalist_mail.py::TestJournalistMail::test_ossec_rule_journalist[ansible://mon-staging] PASSED                                  [ 90%]
testinfra/ossec/test_journalist_mail.py::TestJournalistMail::test_journalist_mail_notification[ansible://mon-staging] PASSED                           [100%]

====================================================================== warnings summary ======================================================================
None
  Module already imported so cannot be rewritten: testinfra

-- Docs: http://doc.pytest.org/en/latest/warnings.html
===================================================== 5 passed, 5 skipped, 1 warnings in 106.35 seconds ======================================================

Comments

See the forum discussion for the rationale behind the testinfra: remove some XXX tests duplicating Ansible commits.

Deployment

• The journalist emails is not set for existing installations, nothing will happen
• During sdconfig the admin will be prompted for the email of the journalist and the encryption key. If set, the daily notification mails will be sent.

Checklist

If you made changes to the app code:

• [x] Unit and functional tests pass on the development VM

If you made changes to the system configuration:

• [x] Configuration tests pass

If you made changes to documentation:

• [x] Doc linting passed locally

Status

Work in progress (tests required)

Description of Changes

Fixes #6297. Fixes #6298.

adds options in the JI instance config, and in the SI, to:

• set a minimum message length for source's initial messages, as per #6297
• reject initial messages that consist of the source's codename, as per #6298

Testing

• check out this branch and run make dev

minimum message length

• in the JI, navigate to the Instance Config page via the Admin page
  • [x] verify that the Submission Preferences section includes a "Prevent sources from sending initial messages shorter than the minimum required length" option, currently unchecked and with the length field set to 0.
• check the "Prevent sources from sending initial messages shorter than..." checkbox but do not set a length. Click Update Submission Preferences
  • [x] verify that the checkbox is unchecked and an error message is displayed asking you to set a length.
• check the "Prevent sources from sending initial messages shorter than..." checkbox and set a length of between 5 and 20 chars. Click Update Submission Preferences
  • [x] verify that the checkbox is checked, the length you chose is set, and a success message is displayed.
• Create a new source account on the SI, navigating through to the /lookup page
  • [x] verify that a notice is displayed under the message textfield like "If you are only sending a message, it must be at least N characters long" where N is the length you chose above
  • [x] Verify that if you try to submit too short a message, the submission fails with a flashed error like "Your first message must be at least N characters long"
  • [x] Verify that a message N characters long can be submitted successfully
  • [x] after a successful first submission, verify that the notice under the text field is no longer displayed and you can submit message shorter than N chars
  • [x] in the JI, uncheck the "Prevent sources from sending initial messages shorter than..." checkbox,, click Update Submission Prefs and verify that the length is set to zero and a success message is displayed
  • [x] Create a new source account in the SI and verify that no message length limit is mentioned or set

codename messages

• in the JI, navigate to the Instance Config page via the Admin page
  • [x] verify that the Submission Preferences section includes a "Prevent sources from submitting their codename as an initial message." option, currently unchecked.
• check the "Prevent sources from submitting their codename as an initial message" checkbox. Click Update Submission Preferences
  • [x] verify that the checkbox is checked and a success message is displayed.
• Create a new source account on the SI, navigating through to the /lookup page
  • [x] Enter the codename as a message and click Submit - verify that the message is not submitted and an error is flashed like "Please do not submit your codename.."
  • [x] Enter anything else as a message and click Submit - verify that the message was submitted correctly
  • [x] Enter your codename again and click Submit - verify that the message was submitted correctly.

Deployment

DB migration script required, but new functionality is disabled by default.

Checklist

If you made changes to the server application code:

• [ ] Linting (make lint) and tests (make test) pass in the development container

If you made non-trivial code changes:

• [ ] I have written a test plan and validated it for this PR

Choose one of the following:

• [ ] I have opened a PR in the docs repo for these changes, or will do so later
• [x] I would appreciate help with the documentation
• [ ] These changes do not require documentation

Status

Ready for review

Description of Changes

Fixes https://github.com/freedomofpress/securedrop-client/issues/1080

Adds many different test strings to test securedrop-client text formatting.

Testing

• [ ] make dev should have 2 sources
• [ ] NUM_SOURCES=10 make dev should create 10 sources
• [ ] NUM_SOURCES=ALL make dev shoudl create 43 sources
• [ ] Use the securedrop-client against the local server and look many strange looking texts :)

Deployment

Any special considerations for deployment? Consider both:

1. Upgrading existing production instances.
2. New installs.

Checklist

If you made changes to the server application code:

• [ ] Linting (make lint) and tests (make test) pass in the development container

If you made changes to securedrop-admin:

• [ ] Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

• [ ] Configuration tests pass

If you made non-trivial code changes:

• [ ] I have written a test plan and validated it for this PR

If you made changes to documentation:

• [ ] Doc linting (make docs-lint) passed locally

If you added or updated a code dependency:

Choose one of the following:

• [ ] I have performed a diff review and pasted the contents to the packaging wiki
• [ ] I would like someone else to do the diff review

We currently use the default number of rounds (edit: cost factor, see below comments) for bcrypt to hash the codenames - 12. We could easily make brute force attacks more difficult by increasing this value. Given our threat model, 12 might not be enough.

Thanks to John Adams for reporting this issue.

Status

Ready for review

Description of Changes

Fixes #4628 Fixes #4667

Started adding related Ansible changes to have v3 onion services (including the authenticated ones).

Testing

v3 only

• [ ] Fresh install
• [ ] Rerun ./securedrop-admin install on a already installed system.

For both the options above, make the following two changes in install_files/ansible-base/group_vars/all/site-specific

v2_onion_services: false
v3_onion_services: true

For this case:

• [ ] No v2 address working and or in the system
• [ ] v3 source address works
• [ ] v3 journalist interface works (follow https://kushaldas.in/posts/setting-up-authorized-v3-onion-services.html about setting up Torrc)

v2/v3 alongside

v2_onion_services: true
v3_onion_services: true

For this:

• [ ] All old v2 address working and are in the app and mon servers
• [ ] v3 source address works
• [ ] v3 journalist interface works (follow https://kushaldas.in/posts/setting-up-authorized-v3-onion-services.html about setting up Torrc)

v2 only

There is also the third point to test which is not adding v2_onion_services or v3_onion_services to site-specific and instead the default values (v2 on, v3 off).

This should keep the current v2 systems as it is, and no v3 details should be generated in the servers.

Deployment

Any special considerations for deployment? Consider both:

1. Upgrading existing production instances.
2. New installs.

Checklist

If you made changes to the server application code:

• [ ] Linting (make lint) and tests (make -C securedrop test) pass in the development container

If you made changes to securedrop-admin:

• [ ] Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

• [ ] Configuration tests pass

If you made non-trivial code changes:

• [ ] I have written a test plan and validated it for this PR

If you made changes to documentation:

• [ ] Doc linting (make docs-lint) passed locally

Status

Ready for review

Description of Changes

This PR-

1. Fixes #2807 by resizing the uploaded logo to a maximum width of 500px, or a maximum height of 450px (whichever is higher) while maintaining the original aspect ratio. Maximum limit is chosen in accordance with the recommended size by SecureDrop docs. Images with dimensions below this constraint are not resized.

2. Fixes #3043 by converting JPG/JPEG image files to PNG before saving them. Thus no matter which format the Admin chooses to upload, the final image is always a PNG file.

3. Possibly fixes #2801 by adding a message at the logo upload screen informing the user about the recommended image size as per the docs.

4. Fixes #3058 by only accepting valid image files.

Dependencies Introduced

1. Python Pillow, installed via pip

2. libjpeg-dev, installed via apt

Testing

1. After successfully uploading, image size can be verified by checking the dimensions of securedrop/securedrop/static/i/custom_logo.png.

2. After successfully uploading a JPG/JPEG image, it can be verified that securedrop/securedrop/static/i/ always contains custom_logo.png

3. Journalist app should display the recommended size on the upload page.

4. Try to upload a non-image file with an extension like JPG/JPEG/PNG. A warning message should appear that image cannot be processed.

Checklist

If you made changes to the app code:

• [x] Unit and functional tests pass on the development VM

Status

Ready for Review

Description of Changes

Fixes #2522

Refactor ./securedrop-admin sdconfig from Ansible to python.

Act I

The commits up to securedrop-admin: refactor sdconfig as a python script included implement the basic features securedrop-admin needs to replace the YAML implementation. It produces the site-specific file will all know and love. With the nice addition that it can also be used to update it. The current value is displayed for the user to edit (with emacs & vi bindings) and is validated as soon as the user hits enter. Verifications involving multiple variables (i.e. fingerprint and public key) are done at the very end.

Act II

Each securedrop-admin: refactor sdconfig ABC commit is a replacement for the implementation of the corresponding YAML variable found in site-specific. Validation classes are implemented when a variable needs it and not in a different commit although they can be re-used later. Each variable is implemented in the order in which they originally showed in the Ansible implemenation. With the notable exception of the securedrop-admin: define ValidatePath commit which is a reminder of the ancient time when the logo was updated from securedrop-admin instead of the journalist admin interface.

Act III

Over time securedrop-admin grew from a tiny standalone script isolated at the root of the git repository to a handsome young module who moved with its relatives down in the securedrop directory. Alas, its users were different and it became clear securedrop-admin had to move to its own directory: admin. From ansible: trim validate role comment now obsolete up it got a setup.py, requirements and tox to run the tests, a Dockerfile and a Makefile to help the CI. It also was split in two: the setup action is now a standlone bootstrap.py script instead of being included in a rather contorted way into a single standalone script.

Epilogue

One day securedrop-admin could be uploaded to pypi and have its own repository even. But that's a story for another time.

Testing

In both Tails and the development environment, try to enter invalid (non existent domain name) or inconsistent (fingerprint not matching the public key) values to check the resulting interaction.

In the Debian GNU/Linux stretch development environment

• ./securedrop-admin setup
• ./securedrop-admin sdconfig
• fill all values
• verify install_files/ansible-base/group_vars/all/site-specific has the desired variables and values

In Tails

• setup tails with persistence
• git clone -b wip-dachary-2522-sdconfig https://github.com/freedomofpress/securedrop
• cd securedrop
• ./securedrop-admin setup
• ./securedrop-admin sdconfig
• verify install_files/ansible-base/group_vars/all/site-specific has the desired variables and values
• ./securedrop-admin sdconfig # and change one value
• vierify install_files/ansible-base/group_vars/all/site-specific has the desired variables and values

In CI

• verify the new .circleci/config.yml target admin-tests actually runs

Deployment

It is supposed to be a backward compatible replacement of `./securedrop-admin sdconfig``, reason why the documentation is not modified.

Checklist

If you made changes to the app code:

• [x] Unit and functional tests pass on the development VM

If you made changes to documentation:

• [x] Doc linting passed locally

Changed name due to debate over whether CAPTCHA is right method to use. Consensus is on developing a responsive approach that combines several methods, including CAPTCHAs.

Status

Ready for review

• See next section for tests outstanding

Status of requested changes (since b5d493c)

1. [x] fix: "Error strings [...], and also in the inline comment securedrop/source_templates/lookup.html, which could be confusing to sources." (1, 2)

2. [x] fix: "Bug: textarea width on Source Interface"

3. [x] refactor: "Proposed alternative language" for "Instance Configuration" page

   1. [x] fix: "UX issue: 'Update' button on Instance Configuration page"
   2. [x] fix: "UX issue: 'Document Uploads', 'Allow' language on Instance Configuration page"

4. [x] refactor: versioned instance_config

   1. [x] "one configuration option per column"
   2. [x] migration to add column sets default value
   3. [x] "use the configuration option that has null valid_until. When we update a config, we store the historical configuration entry by setting valid_until=datetime.datetime.utcnow(), then store the new configuration with null valid_until."
   • [x] test: migrations

5. [x] test: integration test

Description of Changes

1. [x] InstanceConfig versioned key-value store as outlined

   • [x] load_instance_config() sets each app's app.instance_config (via @before_request)

2. [x] Source interface checks app.instance_config.allow_document_uploads; if False:

   1. /lookup hides the file input and
   2. changes its heading to "Submit Messages" (rather than "Submit Files or Messages") and other strings accordingly; and
   3. /submit skips processing of request.files.
   4. /metadata returns this setting as allow_document_uploads.

3. [x] Journalist interface:

   1. /admin/config adds a section "Submission Preferences"; and
   2. /admin/update-submission-preferences updates InstanceConfig.allow_document_uploads.

Testing

Test by toggling the Prevent sources from uploading documents checkbox in the "Submission Preferences" section of the "Instance Config" view.

1. If unchecked (default): Observe no changes from current behavior.

2. If checked: Observe the changes described in (2) and (3) above.

Deployment

No action required. After upgrading, administrators may use the "Submission Preferences" section of the "Instance Config" view.

Checklist

If you made changes to the server application code:

• [x] Linting (make lint) and tests (make test) pass in the development container
  • [x] Fix tests/test_i18n.py::test_verify_default_locale_en_us_if_not_defined_in_config (expected by end of day Monday, October 21)

Description

The SD Source Interface should support file uploads of up to 500MB, with larger uploads failing immediately. Instead, large file uploads <500MB are frequently failing after a long timeout.

(This was initially reported as happening with files of 50MB or more, but it isn't consistently reproducible at that size.)

Steps to Reproduce

Using VMs or a hardware instance:

1. Visit the Source Interface using the Tor Browser
2. Click through to the /lookup page
3. Upload a large file (50MB+)

Expected Behavior

File upload completes with success message on page.

Actual Behavior

Upload has a pretty good chance of failing with message below:

Forking this from #249.

@aegis said:

Would be great to also present a graphic visualization of the SecureDrop system, like the 'Cryptocat Threat Model Connections Overview' image https://github.com/cryptocat/cryptocat/wiki/Threat-Model

Description

We could use https://github.com/shellcheck-py/shellcheck-py to install shellcheck instead of relying on a docker image or OS-installed version.

This would be more consistent with the rest of our developer tooling, making it easier to use.

Learned about this via https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/de712d13fb2e7e3426cac11ee6dca533c4e973ce/tox.ini#30

Splitting from https://github.com/freedomofpress/securedrop/issues/6514

We now have a decent set of steps to run when testing new kernel builds documented at https://developers.securedrop.org/en/latest/kernel.html. In brief, it's to 1) run paxtest, 2) then meltdown-checker, and 3) then running a basic SecureDrop smoke test.

At least 1 and 2 are easily automated, 3 could be with a bit more effort, but I think we can ignore it for now.

My proposal for now is:

• we have a script run on a timer, daily. It checks the latest kernel version, and if it's different from the latest stored version, it starts running the checks
• the script installs the necessary apt dependencies, runs paxtest, capturing the output.
• it downloads the latest meltdown-checker, runs it and captures the output.
• it sends an email to securedrop@ (or some other address), with the log files for human review.

Using email mostly avoids us needing to have some central actor/collector that logs into each node to run the tests or see the output.

Presumably there would be some ansible role that enables the apt-test repo and installs the script.

Description

After a download from a SecureDrop fails when accessed from a Journalist Workstation, if you click the 'Retry' button, it "completes" the download by saving a 4kb empty text file, rather than the original submission.

Steps to Reproduce

1. Fill the local filesystem with enough data to nearly be out of storage space.
2. Download a relatively large submission from a SecureDrop instance.
3. The download should fail once the filesystem runs out of space.
4. Make some more room on the disk.
5. Click "Retry" on the download from inside Tor browser.
6. An empty file will be downloaded.

Expected Behavior

Clicking "Retry" downloads the original submission.

Actual Behavior

Clicking "Retry" downloads an empty file.

Status

Work in progress

Description of Changes

Fixes # 6531

Changes proposed in this pull request:

This adds a GNOME Shell extension to the Journalist and Admin Workstations. The goal of the extension is to provide a more consistent and better-supported method for linking a user to their SecureDrop interfaces, and to make it easier to access the management features of SecureDrop.

As of now, the extension looks like this:

The Shell Extension gets saved into the appropriate .dotfiles directory as part of the tailsconfig Ansible task, during install or upgrade.

Although the code here is functional and can be included in the main project as-is, the intent of this PR is to provide a firmer foundation for testing and discussion. Every menu entry in the extension is functional, and it is ready for wider testing.

There are some additional concerns that need to be ironed out:

• Do we want to keep the icon in the titlebar where the extension is accessed? If so, do we use the generic icon, or do we create/install a symbolic SD icon?
• Do we want to dynamically hide the administrative options when it is being used only on a journalist workstation? If yes, what is the most reliable method of detecting that difference?
• Do we want to remove the current desktop icons altogether, or have this in addition to those icons (either temporarily or indefinitely until that support is removed from Tails)?

Testing

This needs to be widely tested across multiple installs.

At a minimum, we should:

• [x] Test that the extension gets added and activated with a clean install
• [ ] Test that the extension gets added and activated while upgrading an existing install
• [x] Test that each menu item in the extension reliably triggers the labeled action

Deployment

Deployment should be straight-forward, although as mentioned above, we need to test thoroughly for both clean install and upgrade paths.

I have already tested with a clean install, and it should work with an upgrade, but would appreciate other confirmations.

Checklist

If you made changes to the server application code:

• [x] Linting (make lint) and tests (make test) pass in the development container

If you made changes to the system configuration:

• [x] Configuration tests pass

If you added or removed a file deployed with the application:

• [ ] I have updated AppArmor rules to include the change

If you made non-trivial code changes:

• [ ] I have written a test plan and validated it for this PR

Choose one of the following:

• [x] I have opened a PR in the docs repo for these changes, or will do so later
• [ ] I would appreciate help with the documentation
• [ ] These changes do not require documentation

Description

We build our packages in a container using a custom docker image (see molecule/builder-focal). The way the image is currently used is that it's frozen/pinned, pushed to quay.io, and then the person building the packages will pull it down and use it, as is. But if there are any missing security updates, the build will fail after the fact and then you need to go through the hurdle of updating the builder, rebuilding packages, etc.

Also during the last builder update, I noticed that the logic to detect pending security updates itself was buggy, because it simulates a dist-upgrade and sees if any packages will be pulled in from focal-security, except we don't intend to do a dist-upgrade, so it complaining that a package not in the container was a pending security update!

The theory is that pinning the builder helps with reproducible builds, except it doesn't because the build isn't reproducible for other reasons.

Proposal

We just build the image on the fly. At runtime we see if there are any pending updates and if so, bail. The deployer can rebuild the image locally and restart the package building. For reproducibility, we will implement #6356, but that shouldn't block this.

Description

Port our 2 cron jobs into systemd timers

Rationale

• systemd timers are nicer to deal with, they have more configuration knobs and provide more diagnostic output
• timers are defined in files, so manipulating them is much easier than needing to use something like ansible comments for crontabs
  • consider that efb2c766fe388c9cb22ac8c7bb5c89c214b7ddce would've just been adjusting the User=www-data line instead of needing to write and test a hacky bash migration step