Fix#12021.
Reopens#11706.
This reverts commit 12b9bd5801.
This change caused a bad regression in pytest-xdist:
https://github.com/pytest-dev/pytest-xdist/issues/1024
pytest-xdist necessarily has special handling of `--maxfail` and session
fixture teardown get executed multiple times with the change.
Since I'm not sure how to adapt pytest-xdist myself, revert for now.
I kept the sticky `shouldstop`/`shouldfail` changes as they are good
ideas regardless I think.
00d9640abc