Fix FastReducer.SubMod infinite loop on 32-bit platforms

[stephentoub]

PR #125799 rewrote BigInteger internals to use native-width (nuint) limbs. Its FastReducer.SubMod added a compensating loop:

while (CompareActual(left, right) < 0)
{
    AddSelf(left, modulus);
}

This hangs on 32-bit platforms. After truncating both operands to k limbs, the truncated right (q̂·m mod b^k) can appear much larger than left (x mod b^k). Each loop iteration adds modulus (~2^1025), but the gap can be ~2^1088, requiring ~2^64 iterations on 32-bit — effectively infinite.

Barrett reduction guarantees the true residual x − q̂·m is in [0, 2·modulus]. After k-limb truncation, unsigned subtraction underflow naturally wraps to the correct residual, which fits in k limbs since 2·modulus < b^k. The original (pre-#125799) code relied on this property — it just called SubtractSelf directly (and hit a debug assertion on borrow, which was separately suppressed in #98212).

This PR:

• Replaces the AddSelf loop with an inline subtraction that tolerates unsigned underflow
• Re-enables the FastReducer_AssertFailure_RegressionTest on 32-bit platforms (disabled in Disable FastReducer_AssertFailure_RegressionTest on 32-bit platforms #126216)

All 3,031 System.Runtime.Numerics tests pass on x64; the previously-hanging test completes in ~10 seconds.

Fixes #126212

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

Fixes a 32-bit hang in BigInteger Barrett reduction by removing an unbounded compensation loop in FastReducer.SubMod and restoring the previously-disabled 32-bit regression test to ensure coverage of the affected ModPow path.

Changes:

• Replace FastReducer.SubMod’s AddSelf compensation loop with an inline subtraction that tolerates unsigned underflow (wrap-around).
• Re-enable FastReducer_AssertFailure_RegressionTest on 32-bit by removing the [ActiveIssue] skip.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/libraries/System.Runtime.Numerics/src/System/Numerics/BigIntegerCalculator.FastReducer.cs	Removes potentially unbounded correction loop in Barrett reduction; performs wrap-safe subtraction and then canonicalizes by subtracting modulus as needed.
src/libraries/System.Runtime.Numerics/tests/BigInteger/modpow.cs	Re-enables the regression test that previously hung on 32-bit, ensuring the fix is exercised.

Comments suppressed due to low confidence (1)

src/libraries/System.Runtime.Numerics/tests/BigInteger/modpow.cs:376

• Now that this test runs on 32-bit again, its expected-value computation is potentially very expensive: the body uses an O(exponent) loop (and the input data includes exponent=65537). Consider switching the expected calculation to exponentiation-by-squaring (O(log exponent)) to keep the test runtime predictable on slower 32-bit/low-end machines and reduce the risk of future timeouts.

        public static void FastReducer_AssertFailure_RegressionTest(BigInteger value, int exponent, BigInteger modulus)
        {
            // Regression test exercising the code path that was resulting in
            // assertion failures per https://github.com/dotnet/runtime/issues/97780.
            // The failures seemingly have no impact on functional correctness,