Make xpass failure again warning #11467

[TanyaAgarwal28]

Problem:

Some years ago, a change was made that caused xpass not to be considered a failure. However, there have been multiple cases where it was missed that a test was fixed, and this is not an acceptable default behavior.

Solution:

This PR aims to address the issue by modifying the behavior of xpass in the following way: In order to ensure backward compatibility we cannot go straight from non-strict to strict instead we have to start by warning if strict was not set to true or false. The warning should indicate that a future major release of pytest will change the default from False to True and recommend to use strict=True as default and a plugin for actually flaky tests.

Checklist:

Include documentation for the proposed change.
Include new tests or update existing tests, where applicable.
Allow maintainers to push and squash when merging my commits.
This change closes issue #11467.

Changelog: Created a new changelog file named 11467.feature.rst and added the entry for this change.

Author: Tanya Agarwal

[bluetech]

Thanks for the work on this @TanyaAgarwal28.

I think many test suites don't know or have no use for/of xfail and so I think we shouldn't bother them with the deprecation. Rather we should do something more like the plan outlined by @nicoddemus here: #11467 (comment)

[Zac-HD]

Closing this because @TanyaAgarwal28 hasn't been seen on GitHub for several months. Happy to reopen if it updates!