Honcho: a python clone of Foreman. For managing Procfile-based applications.

If I set PYTHONUNBUFFERED I can kind of use pdb, but it's less than ideal because the (Pdb) prompt isn't displayed until after I enter a command, and all output is prefixed with the timestamp/process indicator.

This adds the ability to use a custom template when exporting to supervisord.

Fixes: GH-88

Example:

$ honcho export supervisord . -t my_supervisord_template.conf.jinja2

$ cat export_test.conf
# *****************************************************
# This is a custom supervisord template for honcho
# blah blah blah
# *****************************************************

[program:export_test-ansvc]
MONKEY=YES
command=/bin/sh -c 'bin/ansvc start'
autostart=true
autorestart=true
stopsignal=QUIT
stdout_logfile=/var/log/export_test/ansvc-0.log
stderr_logfile=/var/log/export_test/ansvc-0.error.log
user=marca
directory=/Users/marca/dev/git-repos/honcho/export_test
environment=PORT="5000"

[group:export_test]
programs=export_test-ansvc

This makes honcho run primarily about running a Procfile-specified process in the foreground (the first proposal at https://github.com/nickstenning/honcho/issues/56#issuecomment-34546939). Previously it was about running an arbitrary shell command, which still works, but now as the fall-through case if a single arg to honcho run doesn't match a Procfile process.

The reason this change is desirable is to be able to use debugging tools such as pdb along with Procfile process aliases (#56). Furthermore, this brings Honcho back into harmony with Foreman's behavior, but in a way that is documented more sensibly (Foreman presents the process case as the exception rather than the command case).

This change is backwards-compatible except for edge cases where a shell command name was used as a Procfile process name. So, for example, if mv was a process name, then calling honcho run mv would now invoke the Procfile process rather than the shell command. Since we also check for the number of arguments when trying to interpret as a process name, this edge case is especially edgy. I think this can safely be considered a backwards-compatible change for all practical purposes.

P.S. For Gittip folks: I'm introducing this fork on Gittip in https://github.com/gittip/www.gittip.com/pull/2384. If/when you change it here's how to revendor it:

• Go into a local working copy of the fork.
• echo __version__ = "'$(git rev-parse --short=8 HEAD)'" > honcho/__init__.py
• python setup.py sdist
• mv dist/honcho-*.tar.gz ../www.gittip.com/vendor/

Then go back over to www.gittip.com, remove the old version and update requirements.txt.

This is to fix a problem that when supervisord terminated honcho, the child processes kept running.

• Catch SIGTERM so we know when honcho is being terminated, and run terminate().
• In terminate(), signal the entire process group, not just the shell process that was used to start the process. Without this, the shell process exited but not its children.

This PR:

• Moves responsibility for I/O back into honcho.command rather than honcho.export.
• Changes exporters so that #export() takes a list of honcho.environ.ProcessParams objects and optional template context. This means that concurrency, process naming, and port assignment is now much more likely to be consistent between honcho start and honcho export.
• Improves test coverage of the honcho.export package.
• Removes a couple more fragile export integration tests.

Here is a working version of support for Windows for #28 All tests are passing. Most of the adjustments were on the quirks of MSFT of course, such as command line arguments quoting, no support for Signals, adjusting tests expectations in particular line endings. This is working acceptably enough in that state, though it may not be bullet proof as it is as there are too many process handling quirks in Windows compared to POSIX.

That said, this could be a great addition for now. I could then work out integrating something like mozprocess that provides enhanced handling of Windows processes and should make the Windows side more solid.

(honcho) c:\w421\honcho>nosetests
............................
----------------------------------------------------------------------
Ran 28 tests in 18.568s

OK

This PR:

• Adds integration tests for the export functionality (4b33c2c)
• Tweaks a few things so that tests pass consistently. E.g.:
  • Sort dictionary keys to get consistent ordering from run to run (e32be90)
  • Use items in place of iteritems in upstart template for Python 3 compatibility (3e9c2d9)
  • Apply fix from https://github.com/nickstenning/honcho/pull/102 so that we don't get spurious test failures that confuse things. (5148677)

$ tox -e py26,py27,py32,py33,py34,lint
...
  py26: commands succeeded
  py27: commands succeeded
  py32: commands succeeded
  py33: commands succeeded
  py34: commands succeeded
  lint: commands succeeded
  congratulations :)

I'm experiencing crashes when Honcho tries to print to console with special characters:

Traceback (most recent call last):
File "/app/.heroku/python/bin/honcho", line 9, in <module>
    load_entry_point('honcho==0.4.2', 'console_scripts', 'honcho')()
File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 292, in main
    app.parse()
File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 129, in parse
    options.func(self, options)
File "/app/.heroku/python/lib/python2.7/site-packages/honcho/command.py", line 190, in start
    sys.exit(process_manager.loop())
File "/app/.heroku/python/lib/python2.7/site-packages/honcho/process.py", line 114, in loop
File "/app/.heroku/python/lib/python2.7/site-packages/honcho/printer.py", line 22, in write

Here is a Stackoverflow question with more details:

http://stackoverflow.com/questions/19100116/looks-like-logs-are-crashing-my-django-app

currently in foreman if you run $ foreman run PROCNAME it will run from the procfile if it's available, and the exit code of the command will equal the exit code of the subprocess. This diff honors that change.

This approach stops child processing hanging with exported upstart process by leveraging setuid (http://upstart.ubuntu.com/cookbook/#setuid). service {app} restart now works correctly for me.

This is a honcho analogue of the corresponding options to foreman export -- See:

• http://ddollar.github.io/foreman/#EXPORTING
• https://github.com/ddollar/foreman/pull/46

This lets you specify a custom Jinja template to be used when exporting to a supervisord config file.

Fixes: GH-88

See: https://github.com/nickstenning/honcho/issues/88

OK, I just updated this so that --template is a directory. I'm not quite sure how to test this in an automated way, but I did do manual testing.

[[email protected] export_test]$ ls -l upstart_template.d
total 24
-rw-r--r--+ 1 marca  staff  353 Feb 23 06:01 master.conf
-rw-r--r--+ 1 marca  staff  380 Feb 23 06:02 process.conf
-rw-r--r--+ 1 marca  staff  119 Feb 23 06:02 process_master.conf

[[email protected] export_test]$ honcho export upstart upstart-export --template upstart_template.d
2015-02-23 07:59:43 [20837] [INFO] Writing 'upstart-export/export_test.conf'
2015-02-23 07:59:43 [20837] [INFO] Writing 'upstart-export/export_test-ansvc.conf'
2015-02-23 07:59:43 [20837] [INFO] Writing 'upstart-export/export_test-ansvc-1.conf'

[[email protected] export_test]$ honcho export upstart upstart-export
2015-02-23 08:00:16 [20841] [INFO] Writing 'upstart-export/export_test.conf'
2015-02-23 08:00:16 [20841] [INFO] Writing 'upstart-export/export_test-ansvc.conf'
2015-02-23 08:00:16 [20841] [INFO] Writing 'upstart-export/export_test-ansvc-1.conf'

[[email protected] export_test]$ ls -l supervisor-template.d
total 8
-rw-r--r--+ 1 marca  staff  776 Dec 23 08:09 supervisord.conf

[[email protected] export_test]$ honcho export supervisord out --template=supervisor-template.d
2015-02-23 08:00:39 [20847] [INFO] Writing 'out/export_test.conf'

[[email protected] export_test]$ honcho export supervisord out
2015-02-23 08:01:05 [20855] [INFO] Writing 'out/export_test.conf'

Specifically, the Python 3.10 images on CircleCI no longer include pkg_resrouces as it's deprecated:

$ honcho start
Traceback (most recent call last):
  File "/home/circleci/myproject/.venv/bin/honcho", line 5, in <module>
    from honcho.command import main
  File "/home/circleci/myproject/.venv/lib/python3.10/site-packages/honcho/command.py", line 10, in <module>
    from pkg_resources import iter_entry_points
ModuleNotFoundError: No module named 'pkg_resources'

Starting in Python 3.8, importlib.metdata is the preferred way to find entry points.

Hey @nickstenning thanks for this project!

I was wondering if there's a way to prevent a .env file from overwriting an existing env var. Consider the following example:

$ cat .env
FOO="using FOO from .env"

$ FOO="Using FOO from shell" honcho run -e .env env | grep FOO
FOO=Using .env.local

Ideally, if FOO is already set, I don't think a .env file should overwrite it, like if the environment variable gets set at runtime in a docker-compose.yml, k8s, or whatever orchestrator may be setting them.

Similarly this would be helpful when passing multiple .env files for different environments for example:

❯ honcho run -e .env.local,.env.dev,.env env | grep FOO
FOO=Using .env

It seems that right now the last file parsed will always take over an environment variable previously set, but it would be nice if it didn't override them.

Just leaving myself a note here that the systemd exporter seems broken. The process group targets don't correctly specify that they Want the service files.

Also probably worth fixing StandardOutput=syslog (use StandardOutput=journal instead) and pick the right KillMode (as process is unlikely to be the correct choice.

Also fixes #220.

Any falsey value for time_format passed to the printer disables it entirely. It could have been done before, but it ends up prefixing with an extra space.