Update `check_assets` to also check in the R2 staging bucket

[targos]

https://github.com/nodejs/build/blob/main/ansible/www-standalone/tools/promote/check_assets.js

To avoid future issues like nodejs/node#55829

[richardlau]

Started looking at this but it opens a can of worms:

• check_assets.js has tests, rather unintuitively run with node check_assets.js test
  

  build/ansible/www-standalone/tools/promote/check_assets.js

  Lines 18 to 21 in c26ecff

  	// `check_assets.js test` kicks it into test mode
  	if (stagingDir === 'test') {
  	test()
  	} else {

  • These tests appear to fail when run😞.
  • Tests also leave several check_assets.js_* test fixture directories behind.
• We're still running Node.js 16.20.2(!) on the server.

I'll look at updating our version of Node.js first. We're using the Nodesource distribution and I'll need to point apt at the newer repository to pick up later versions of Node.js.

Plan:

• ansible: update www Node.js version #3967
• promote: update tests for check_assets.js #3971
• Add checking assets on R2 staging.

[richardlau]

• These tests appear to fail when run😞.

For posterity:

node check_assets.js test

$ node check_assets.js test

Test 6 Assertions ???
STDOUT------------------------------------------

... Checking assets
... Expecting a total of 41 assets for v10.x
... 2 assets waiting in staging
... 39 assets already promoted
 ✓  Complete set of expected assets in place for v10.x

------------------------------------------------
AssertionError [ERR_ASSERTION]: '... Checking assets\n' +
  '... Expecting a total of 41 assets for v10.x\n' +
  '... 2 assets waiting in staging\n' +
  '... 39 assets already promoted\n' +
  ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v10.x\n' == '... Checking assets\n' +
  '... Expecting a total of 40 assets for v10.x\n' +
  '... 2 assets waiting in staging\n' +
  '... 38 assets already promoted\n' +
  ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v10.x\n'
    at runTest (/home/rlau/sandbox/github/build/ansible/www-standalone/tools/promote/check_assets.js:290:12) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: '... Checking assets\n' +
    '... Expecting a total of 41 assets for v10.x\n' +
    '... 2 assets waiting in staging\n' +
    '... 39 assets already promoted\n' +
    ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v10.x\n',
  expected: '... Checking assets\n' +
    '... Expecting a total of 40 assets for v10.x\n' +
    '... 2 assets waiting in staging\n' +
    '... 38 assets already promoted\n' +
    ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v10.x\n',
  operator: '=='
}

Test 7 Assertions ???
STDOUT------------------------------------------

... Checking assets
... Expecting a total of 41 assets for v10.x
... 2 assets waiting in staging
... 41 assets already promoted
 ✓  Complete set of expected assets in place for v10.x
 ✖  The following assets were found in staging but are not expected for v10.x:
    • ooolaalaa.tar.gz
    • whatdis.tar.xz
    Does the expected assets list for v10.x need to be updated?
    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v10.x
    Promote if you are certain this is the the correct course of action

------------------------------------------------
AssertionError [ERR_ASSERTION]: '... Checking assets\n' +
  '... Expecting a total of 41 assets for v10.x\n' +
  '... 2 assets waiting in staging\n' +
  '... 41 assets already promoted\n' +
  ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v10.x\n' +
  ' \x1B[31m\x1B[1m✖\x1B[22m\x1B[39m  The following assets were found in staging but are not expected for v10.x:\n' +
  '    • ooolaalaa.tar.gz\n' +
  '    • whatdis.tar.xz\n' +
  '    Does the expected assets list for v10.x need to be updated?\n' +
  '    ... == '... Checking assets\n' +
  '... Expecting a total of 40 assets for v10.x\n' +
  '... 2 assets waiting in staging\n' +
  '... 40 assets already promoted\n' +
  ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v10.x\n' +
  ' \x1B[31m\x1B[1m✖\x1B[22m\x1B[39m  The following assets were found in staging but are not expected for v10.x:\n' +
  '    • ooolaalaa.tar.gz\n' +
  '    • whatdis.tar.xz\n' +
  '    Does the expected assets list for v10.x need to be updated?\n' +
  '    ...
    at runTest (/home/rlau/sandbox/github/build/ansible/www-standalone/tools/promote/check_assets.js:290:12) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: '... Checking assets\n' +
    '... Expecting a total of 41 assets for v10.x\n' +
    '... 2 assets waiting in staging\n' +
    '... 41 assets already promoted\n' +
    ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v10.x\n' +
    ' \x1B[31m\x1B[1m✖\x1B[22m\x1B[39m  The following assets were found in staging but are not expected for v10.x:\n' +
    '    • ooolaalaa.tar.gz\n' +
    '    • whatdis.tar.xz\n' +
    '    Does the expected assets list for v10.x need to be updated?\n' +
    '    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v10.x\n' +
    '...',
  expected: '... Checking assets\n' +
    '... Expecting a total of 40 assets for v10.x\n' +
    '... 2 assets waiting in staging\n' +
    '... 40 assets already promoted\n' +
    ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v10.x\n' +
    ' \x1B[31m\x1B[1m✖\x1B[22m\x1B[39m  The following assets were found in staging but are not expected for v10.x:\n' +
    '    • ooolaalaa.tar.gz\n' +
    '    • whatdis.tar.xz\n' +
    '    Does the expected assets list for v10.x need to be updated?\n' +
    '    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v10.x\n' +
    '...',
  operator: '=='
}

Test 8 Assertions ???
STDOUT------------------------------------------

... Checking assets
... Expecting a total of 41 assets for v11.x
... 43 assets waiting in staging
... 0 assets already promoted
 ✓  Complete set of expected assets in place for v11.x
 ✖  The following assets were found in staging but are not expected for v11.x:
    • docs/bar
    • docs/foo
    Does the expected assets list for v11.x need to be updated?
    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v11.x
    Promote if you are certain this is the the correct course of action

------------------------------------------------
AssertionError [ERR_ASSERTION]: '... Checking assets\n' +
  '... Expecting a total of 41 assets for v11.x\n' +
  '... 43 assets waiting in staging\n' +
  '... 0 assets already promoted\n' +
  ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v11.x\n' +
  ' \x1B[31m\x1B[1m✖\x1B[22m\x1B[39m  The following assets were found in staging but are not expected for v11.x:\n' +
  '    • docs/bar\n' +
  '    • docs/foo\n' +
  '    Does the expected assets list for v11.x need to be updated?\n' +
  '    https://github... == '... Checking assets\n' +
  '... Expecting a total of 40 assets for v11.x\n' +
  '... 42 assets waiting in staging\n' +
  '... 0 assets already promoted\n' +
  ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v11.x\n' +
  ' \x1B[31m\x1B[1m✖\x1B[22m\x1B[39m  The following assets were found in staging but are not expected for v11.x:\n' +
  '    • docs/bar\n' +
  '    • docs/foo\n' +
  '    Does the expected assets list for v11.x need to be updated?\n' +
  '    https://github...
    at runTest (/home/rlau/sandbox/github/build/ansible/www-standalone/tools/promote/check_assets.js:290:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: '... Checking assets\n' +
    '... Expecting a total of 41 assets for v11.x\n' +
    '... 43 assets waiting in staging\n' +
    '... 0 assets already promoted\n' +
    ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v11.x\n' +
    ' \x1B[31m\x1B[1m✖\x1B[22m\x1B[39m  The following assets were found in staging but are not expected for v11.x:\n' +
    '    • docs/bar\n' +
    '    • docs/foo\n' +
    '    Does the expected assets list for v11.x need to be updated?\n' +
    '    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v11.x\n' +
    '...',
  expected: '... Checking assets\n' +
    '... Expecting a total of 40 assets for v11.x\n' +
    '... 42 assets waiting in staging\n' +
    '... 0 assets already promoted\n' +
    ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v11.x\n' +
    ' \x1B[31m\x1B[1m✖\x1B[22m\x1B[39m  The following assets were found in staging but are not expected for v11.x:\n' +
    '    • docs/bar\n' +
    '    • docs/foo\n' +
    '    Does the expected assets list for v11.x need to be updated?\n' +
    '    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v11.x\n' +
    '...',
  operator: '=='
}

Test 2 Assertions ???
STDOUT------------------------------------------

... Checking assets
... Expecting a total of 41 assets for v10.x
... 38 assets waiting in staging
... 0 assets already promoted
 ⚠  The following assets are expected for v10.x but are currently missing from staging:
    • node-v10.1.0-linux-armv6l.tar.gz
    • node-v10.1.0-linux-armv6l.tar.xz
    • node-v10.1.0.pkg
    Promote if you are certain this is the the correct course of action

------------------------------------------------
AssertionError [ERR_ASSERTION]: '... Checking assets\n' +
  '... Expecting a total of 41 assets for v10.x\n' +
  '... 38 assets waiting in staging\n' +
  '... 0 assets already promoted\n' +
  ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  The following assets are expected for v10.x but are currently missing from staging:\n' +
  '    • node-v10.1.0-linux-armv6l.tar.gz\n' +
  '    • node-v10.1.0-linux-armv6l.tar.xz\n' +
  '    • node-v10.1.0.pkg\n' +
  '    \x1B[33mPromote if you are certain this is the the correct course of action\x1B[39m\n' == '... Checking assets\n' +
  '... Expecting a total of 40 assets for v10.x\n' +
  '... 37 assets waiting in staging\n' +
  '... 0 assets already promoted\n' +
  ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  The following assets are expected for v10.x but are currently missing from staging:\n' +
  '    • node-v10.1.0-linux-armv6l.tar.gz\n' +
  '    • node-v10.1.0-linux-armv6l.tar.xz\n' +
  '    • node-v10.1.0.pkg\n' +
  '    \x1B[33mPromote if you are certain this is the the correct course of action\x1B[39m\n'
    at runTest (/home/rlau/sandbox/github/build/ansible/www-standalone/tools/promote/check_assets.js:290:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: '... Checking assets\n' +
    '... Expecting a total of 41 assets for v10.x\n' +
    '... 38 assets waiting in staging\n' +
    '... 0 assets already promoted\n' +
    ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  The following assets are expected for v10.x but are currently missing from staging:\n' +
    '    • node-v10.1.0-linux-armv6l.tar.gz\n' +
    '    • node-v10.1.0-linux-armv6l.tar.xz\n' +
    '    • node-v10.1.0.pkg\n' +
    '    \x1B[33mPromote if you are certain this is the the correct course of action\x1B[39m\n',
  expected: '... Checking assets\n' +
    '... Expecting a total of 40 assets for v10.x\n' +
    '... 37 assets waiting in staging\n' +
    '... 0 assets already promoted\n' +
    ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  The following assets are expected for v10.x but are currently missing from staging:\n' +
    '    • node-v10.1.0-linux-armv6l.tar.gz\n' +
    '    • node-v10.1.0-linux-armv6l.tar.xz\n' +
    '    • node-v10.1.0.pkg\n' +
    '    \x1B[33mPromote if you are certain this is the the correct course of action\x1B[39m\n',
  operator: '=='
}

Test 1 Assertions ???
STDOUT------------------------------------------

... Checking assets
... Expecting a total of 44 assets for v8.x
... 44 assets waiting in staging
... 0 assets already promoted
 ✓  Complete set of expected assets in place for v8.x

------------------------------------------------
Test 1 Assertions ✓✓✓

Test 10 Assertions ???
STDOUT------------------------------------------

... Checking assets
... Expecting a total of 44 assets for v8.x
... 44 assets waiting in staging
 ⚠  4 assets already promoted will be overwritten, is this OK?
    • docs/
    • docs/api/
    • node-v8.0.0-linux-armv6l.tar.gz
    • node-v8.0.0-linux-armv6l.tar.xz
 ✓  Complete set of expected assets in place for v8.x
    Promote if you are certain this is the the correct course of action

------------------------------------------------
Test 10 Assertions ✓✓✓

Test 5 Assertions ???
STDOUT------------------------------------------

... Checking assets
 ⚠  No expected asset list is available for v9.x, using the list for v8.x instead
    Should a new list be created for v9.x?
    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v9.x
... Expecting a total of 44 assets for v9.x
... 41 assets waiting in staging
... 0 assets already promoted
 ⚠  The following assets are expected for v9.x but are currently missing from staging:
    • node-v9.9.9-linux-x86.tar.gz
    • node-v9.9.9-linux-x86.tar.xz
    • node-v9.9.9-sunos-x86.tar.gz
    • node-v9.9.9-sunos-x86.tar.xz
 ✖  The following assets were found in staging but are not expected for v9.x:
    • docs/apilinks.json
    Does the expected assets list for v9.x need to be updated?
    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v9.x
    Promote if you are certain this is the the correct course of action

------------------------------------------------
AssertionError [ERR_ASSERTION]: '... Checking assets\n' +
  ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  No expected asset list is available for v9.x, using the list for v8.x instead\n' +
  '    Should a new list be created for v9.x?\n' +
  '    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v9.x\n' +
  '... Expecting a total of 44 assets for v9.x\n' +
  '... 41 assets waiting in staging\n' +
  '... 0 assets already promoted\n' +
  ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  The following assets are expec... == '... Checking assets\n' +
  ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  No expected asset list is available for v9.x, using the list for v8.x instead\n' +
  '    Should a new list be created for v9.x?\n' +
  '    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v9.x\n' +
  '... Expecting a total of 44 assets for v9.x\n' +
  '... 40 assets waiting in staging\n' +
  '... 0 assets already promoted\n' +
  ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  The following assets are expec...
    at runTest (/home/rlau/sandbox/github/build/ansible/www-standalone/tools/promote/check_assets.js:290:12) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: '... Checking assets\n' +
    ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  No expected asset list is available for v9.x, using the list for v8.x instead\n' +
    '    Should a new list be created for v9.x?\n' +
    '    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v9.x\n' +
    '... Expecting a total of 44 assets for v9.x\n' +
    '... 41 assets waiting in staging\n' +
    '... 0 assets already promoted\n' +
    ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  The following assets are expected for v9.x but are currently missing from staging:\n' +
    '    • node-v9.9.9-linux-x86.tar.gz\n' +
    '    • node-v9.9.9-linux-x86.tar.xz\n' +
    '...',
  expected: '... Checking assets\n' +
    ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  No expected asset list is available for v9.x, using the list for v8.x instead\n' +
    '    Should a new list be created for v9.x?\n' +
    '    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v9.x\n' +
    '... Expecting a total of 44 assets for v9.x\n' +
    '... 40 assets waiting in staging\n' +
    '... 0 assets already promoted\n' +
    ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  The following assets are expected for v9.x but are currently missing from staging:\n' +
    '    • node-v9.9.9-linux-x86.tar.gz\n' +
    '    • node-v9.9.9-linux-x86.tar.xz\n' +
    '...',
  operator: '=='
}

Test 9 Assertions ???
STDOUT------------------------------------------

... Checking assets
... Expecting a total of 46 assets for v6.x
... 46 assets waiting in staging
... 2 assets already promoted
 ✓  Complete set of expected assets in place for v6.x
 ✖  The following assets were already promoted but are not expected for v6.x:
    • ooolaalaa.tar.gz
    • whatdis.tar.xz
    Does the expected assets list for v6.x need to be updated?
    https://github.com/nodejs/build/tree/main/ansible/www-standalone/tools/promote/expected_assets/v6.x
    Promote if you are certain this is the the correct course of action

------------------------------------------------
Test 9 Assertions ✓✓✓

Test 4 Assertions ???
STDOUT------------------------------------------

... Checking assets
... Expecting a total of 41 assets for v11.x
... 41 assets waiting in staging
 ⚠  41 assets already promoted will be overwritten, is this OK?
 ✓  Complete set of expected assets in place for v11.x
    Promote if you are certain this is the the correct course of action

------------------------------------------------
AssertionError [ERR_ASSERTION]: '... Checking assets\n' +
  '... Expecting a total of 41 assets for v11.x\n' +
  '... 41 assets waiting in staging\n' +
  ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  41 assets already promoted will be overwritten, is this OK?\n' +
  ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v11.x\n' +
  '    \x1B[33mPromote if you are certain this is the the correct course of action\x1B[39m\n' == '... Checking assets\n' +
  '... Expecting a total of 40 assets for v11.x\n' +
  '... 40 assets waiting in staging\n' +
  ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  40 assets already promoted will be overwritten, is this OK?\n' +
  ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v11.x\n' +
  '    \x1B[33mPromote if you are certain this is the the correct course of action\x1B[39m\n'
    at runTest (/home/rlau/sandbox/github/build/ansible/www-standalone/tools/promote/check_assets.js:290:12) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: '... Checking assets\n' +
    '... Expecting a total of 41 assets for v11.x\n' +
    '... 41 assets waiting in staging\n' +
    ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  41 assets already promoted will be overwritten, is this OK?\n' +
    ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v11.x\n' +
    '    \x1B[33mPromote if you are certain this is the the correct course of action\x1B[39m\n',
  expected: '... Checking assets\n' +
    '... Expecting a total of 40 assets for v11.x\n' +
    '... 40 assets waiting in staging\n' +
    ' \x1B[33m\x1B[1m⚠\x1B[22m\x1B[39m  40 assets already promoted will be overwritten, is this OK?\n' +
    ' \x1B[32m\x1B[1m✓\x1B[22m\x1B[39m  Complete set of expected assets in place for v11.x\n' +
    '    \x1B[33mPromote if you are certain this is the the correct course of action\x1B[39m\n',
  operator: '=='
}

Test 3 Assertions ???
STDOUT------------------------------------------

... Checking assets
... Expecting a total of 46 assets for v6.x
... 46 assets waiting in staging
 ⚠  3 assets already promoted will be overwritten, is this OK?
    • node-v6.0.1-headers.tar.gz
    • node-v6.0.1-x86.msi
    • node-v6.0.1.tar.gz
 ✓  Complete set of expected assets in place for v6.x
    Promote if you are certain this is the the correct course of action

------------------------------------------------
Test 3 Assertions ✓✓✓

There were 6 test failures
$

[richardlau]

• These tests appear to fail when run😞.

It looks like the tests broke when #2278 landed. Which was one of mine 😞.

[richardlau]

I've created a new check_r2_assets as modifying the existing check_assets started getting complicated because there are some differences between promotion on the DO droplet vs promotion in R2 (listed in the PR/commit message): #3977