Implement the pytest --cov-context option in an "add-on" pytest plugin

[ionelmc]

This is a port of pytest-dev/pytest-cov#342

Few notes about this:

• if develop installs are used (they appear to be in the tox configuration) then you get these pesky Module already imported so cannot be rewritten: __main__ warnings. They are caused by an unfortunate combination of problems: there's a __main__.py file in the root of the repo (which shouldn't exist IMO) and the fact that pytest will enumerate all the installed files (importlib_metadata will wrongly enumerate files that aren't normally installed when install is a develop install).
• the pytest-cov==2.7.1 dep should be replaced to something that is the pytest-cov master
• I've fixed the xdist support (the correct activation point is pytest_sessionstart - otherwise we'd bind to the wrong cov_controller)
• I've renamed --cov-contexts to --cov-context - I don't get why it was pluralized since there's no comma splitting or something like that in validate_context.

So my suggestion is to take this approach to get more alpha/beta releases of coveragepy out since this whole thing needs overall more work anyway (customization of context, showing something in the reporting about contexts and so on).

[nedbat]

I understand your point about the alpha/betas of coverage.py, but I would rather not have this plugin here. This code has far too much pytest knowledge in it: it should be part of pytest-cov. This code even has details of the internals of the pytest-cov plugin.

The overall work remaining ("customization of context, showing something in the reporting about contexts and so on") is mostly on the coverage.py side of things, not in the pytest integration.

The code should live in the repo that makes the most sense as its forever home. If there's a pytest-cov plugin, then this code should be part of it. If somewhere down the line, we completely fold pytest-cov into coverage.py, then this code will be part of that folding.

Can we apply your fixes to pytest-dev/pytest-cov#342 ?

[codecov-io]

Codecov Report

Merging #850 into master will decrease coverage by 0.04%.
The diff coverage is 69.76%.

@@            Coverage Diff             @@
##           master     #850      +/-   ##
==========================================
- Coverage   92.19%   92.15%   -0.05%     
==========================================
  Files          85       87       +2     
  Lines       11326    11369      +43     
  Branches     1146     1149       +3     
==========================================
+ Hits        10442    10477      +35     
- Misses        741      748       +7     
- Partials      143      144       +1

Impacted Files	Coverage Δ
tests/test_pytest.py	100% <100%> (ø)
coverage/pytest_plugin.py	56.66% <56.66%> (ø)
coverage/control.py	88.25% <0%> (+0.28%)	⬆️
coverage/inorout.py	55.55% <0%> (+0.82%)	⬆️
tests/conftest.py	96% <0%> (+8%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 69c519c...b534a45. Read the comment docs.

[ionelmc]

Oooof ... well what can I say.