Buildbot failure in test_importlib UTF-16 tests

[jaraco]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL9 Refleaks 3.x has failed when building commit ba687d9.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1589/builds/145) and take a look at the build logs.
4. Check if the failure is related to this commit (ba687d9) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1589/builds/145

Failed tests:

• test_importlib

Failed subtests:

• test_open_text - test.test_importlib.resources.test_functional.FunctionalAPITest_StringAnchor.test_open_text
• test_read_text - test.test_importlib.resources.test_functional.FunctionalAPITest_StringAnchor.test_read_text
• test_read_text - test.test_importlib.resources.test_functional.FunctionalAPITest_ModuleAnchor.test_read_text
• test_open_text - test.test_importlib.resources.test_functional.FunctionalAPITest_ModuleAnchor.test_open_text

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.refleak/build/Lib/test/test_importlib/resources/test_functional.py", line 144, in test_open_text
    self.assertEndsWith(  # ignore the BOM
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
        f.read(),
        ^^^^^^^^^
    ...<2 lines>...
        ),
        ^^
    )
    ^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.refleak/build/Lib/test/test_importlib/resources/test_functional.py", line 50, in assertEndsWith
    self.assertEqual(string[-len(suffix) :], suffix)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '\x00H\x00e\x00l\x00l\x00o\x00,\x00 \x00U\[61 chars]00\n' != 'H\x00e\x00l\x00l\x00o\x00,\x00 \x00U\x00T[61 chars]\x00'
- �H�e�l�l�o�,� �U�T�F�-�1�6� �w�o�r�l�d�!�
? -
+ H�e�l�l�o�,� �U�T�F�-�1�6� �w�o�r�l�d�!�
- 
+ �


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.refleak/build/Lib/test/test_importlib/resources/test_read.py", line 46, in test_read_text_with_errors
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        result,
        ^^^^^^^
    ...<2 lines>...
        '\x00w\x00o\x00r\x00l\x00d\x00!\x00\n\x00',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
AssertionError: '\x00H\x00e\x00l\x00l\x00o\x00,\x00 \x00U\[61 chars]00\n' != 'H\x00e\x00l\x00l\x00o\x00,\x00 \x00U\x00T[61 chars]\x00'
- �H�e�l�l�o�,� �U�T�F�-�1�6� �w�o�r�l�d�!�
? -
+ H�e�l�l�o�,� �U�T�F�-�1�6� �w�o�r�l�d�!�
- 
+ �


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.refleak/build/Lib/test/test_importlib/resources/test_functional.py", line 92, in test_read_text
    self.assertEndsWith(  # ignore the BOM
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
        resources.read_text(
        ^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
        ),
        ^^
    )
    ^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.refleak/build/Lib/test/test_importlib/resources/test_functional.py", line 50, in assertEndsWith
    self.assertEqual(string[-len(suffix) :], suffix)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '\x00H\x00e\x00l\x00l\x00o\x00,\x00 \x00U\[61 chars]00\n' != 'H\x00e\x00l\x00l\x00o\x00,\x00 \x00U\x00T[61 chars]\x00'
- �H�e�l�l�o�,� �U�T�F�-�1�6� �w�o�r�l�d�!�
? -
+ H�e�l�l�o�,� �U�T�F�-�1�6� �w�o�r�l�d�!�
- 
+ �


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.refleak/build/Lib/test/test_importlib/resources/test_open.py", line 49, in test_open_text_with_errors
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        result,
        ^^^^^^^
    ...<2 lines>...
        '\x00w\x00o\x00r\x00l\x00d\x00!\x00\n\x00',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
AssertionError: '\x00H\x00e\x00l\x00l\x00o\x00,\x00 \x00U\[61 chars]00\n' != 'H\x00e\x00l\x00l\x00o\x00,\x00 \x00U\x00T[61 chars]\x00'
- �H�e�l�l�o�,� �U�T�F�-�1�6� �w�o�r�l�d�!�
? -
+ H�e�l�l�o�,� �U�T�F�-�1�6� �w�o�r�l�d�!�
- 
+ �

Originally posted by @bedevere-bot in #123037 (comment)

Linked PRs

• gh-123994: Generate utf-16 file using little endian. #123995

[jaraco]

It looks like maybe an issue with byte ordering? Previously, these UTF-16 files were stored statically in the repo. Now they're dynamically generated. Perhaps all that's needed is to make sure the generation is byte-order agnostic.

[jaraco]

One of the failing tests mentions ignoring the architecture-specific BOM:

cpython/Lib/test/test_importlib/resources/test_functional.py

Line 49 in b9ce84f

Used to ignore an architecture-specific UTF-16 byte-order mark."""

[jaraco]

See #117569, where that was created.

Now the question is - should the test fixtures be made to restore the expectation that the resources use utf-16-le? My guess is probably so, given that other tests are opening this utf-16 file using utf-8 or latin-1 to perform assertions.