fix: resolve E2E infrastructure blockers (WU-13)

[MacAttak]

Summary

• Fix PyIceberg 0.11.0 / Polaris 1.2.0 incompatibility (missing PUT in HttpMethod enum) by pinning to PR #3010 merge commit across Docker image, setup script, CI runner, and pyproject.toml constraints
• Add reproducible devcontainer (Dockerfile + devcontainer.json + firewall init) with shellcheck, Kind, Helm, kubectl, uv
• Fix Helm values: Dagster port 3000 (non-root), Cube Store public image, Polaris 1.2.0 pin, env dict-to-list format
• Fix Cube API deployment probes (separate startupProbe from readinessProbe)
• Document all 46 E2E failures across 12 error categories in E2E-ERROR-REPORT.md

Acceptance Criteria (WU-13: PyIceberg Compatibility)

Criterion	Status	Evidence
AC-13.1: Docker image builds with PyIceberg PUT fix	PASS	Dockerfile:121-133 — pinned commit + build-time smoke test
AC-13.2: Setup script installs PyIceberg from pinned commit	PASS	setup-cluster.sh:236-245 — install_pyiceberg_fix() function
AC-13.3: CI script preserves manual installs (UV_NO_SYNC=1)	PASS	test-e2e.sh:188-192 — env var before uv run
AC-13.4: Package constraint excludes broken 0.11.0	PASS	floe-catalog-polaris/pyproject.toml:30, floe-iceberg/pyproject.toml:27 — !=0.11.0
AC-13.5: All tracking comments present	PASS	All 5 files have TODO(pyiceberg-0.11.1) + PR #3010 URL
AC-13.6: Category 1 errors eliminated	DEFERRED	Requires live E2E run (16 tests)
AC-13.7: Category 3 cascade verified	DEFERRED	Requires live E2E run (14 tests)

Gate Results

Gate	Status	Block	Warn	Info
gate-build	PASS	0	1	0
gate-security	PASS	0	2	3
gate-wiring	PASS	0	0	4
gate-spec	PASS	0	0	2
Aggregate	PASS	0	3	9

Warnings (non-blocking):

• gate-build: shellcheck not available (now added to devcontainer)
• gate-security: Pre-existing test credentials in setup-cluster.sh print_info() (not introduced by this PR)
• gate-wiring: floe-iceberg/pyproject.toml missing !=0.11.0 (fixed in 8c23e66)

Changed Files

File	Change
docker/dagster-demo/Dockerfile	PyIceberg git install + smoke test, git build dep
testing/k8s/setup-cluster.sh	install_pyiceberg_fix() function
testing/ci/test-e2e.sh	UV_NO_SYNC=1, port-forward fix (3000:3000)
plugins/floe-catalog-polaris/pyproject.toml	!=0.11.0 exclusion
packages/floe-iceberg/pyproject.toml	!=0.11.0 exclusion
.devcontainer/Dockerfile	shellcheck, build-essential, netcat
.devcontainer/devcontainer.json	Full devcontainer config (Kind, Helm, uv)
.devcontainer/init-firewall.sh	Network firewall init for sandbox
charts/floe-platform/values-test.yaml	Port 3000, Polaris 1.2.0, Cube Store public image
charts/floe-platform/values.yaml	env dict-to-list format fix
testing/E2E-ERROR-REPORT.md	46 failures documented across 12 categories

Test Plan

• Verify Docker image builds: docker build -f docker/dagster-demo/Dockerfile .
• Verify PyIceberg PUT in image: docker run --rm floe-dagster-demo:latest python -c "from pyiceberg.catalog.rest import HttpMethod; assert 'PUT' in HttpMethod.__members__"
• Verify Helm template renders: helm template charts/floe-platform -f charts/floe-platform/values-test.yaml --skip-schema-validation
• Run E2E tests to verify Category 1+3 resolution: make test-e2e
• Verify devcontainer builds: rebuild in VS Code / Codespaces

🤖 Generated with Claude Code

[github-actions]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[github-actions]

Claude finished @MacAttak's task —— View job

---

Security Review Complete ✅

Comprehensive security review of 16 changed files completed. Here are the findings:

Security Review Tasks

• Review Docker configuration files
• Review shell scripts for command injection
• Review Python configuration files
• Review Helm chart templates
• Review devcontainer setup
• Analyze findings and provide remediation guidance
• Complete security assessment report

Security Findings

⚠️ MEDIUM Severity Issues

1. Command Injection Risk in init-firewall.sh

File: .devcontainer/init-firewall.sh:61
Issue: Uses aggregate -q command from external data without validation

done < <(echo "$gh_ranges" | jq -r '(.web + .api + .git)[]' | aggregate -q)

Risk: If aggregate tool behaves unexpectedly or GitHub API returns malicious data, could lead to command injection.

Remediation:

• Add explicit validation of CIDR ranges before passing to aggregate
• Consider using built-in shell functions instead of external tools for IP processing
• Add timeout/resource limits to external command execution

2. Hardcoded Test Credentials in Helm Values

File: charts/floe-platform/values-test.yaml:136-137, 149-150
Issue: Test environment contains hardcoded credentials (marked as allowlisted)

clientId: "demo-admin"  # pragma: allowlist secret
clientSecret: "demo-secret"  # pragma: allowlist secret
accessKey: "minioadmin"
secretKey: "minioadmin123"

Risk: While marked for allowlist, these could be accidentally used in production.

Remediation:

• Ensure clear documentation that these are test-only values
• Consider using random generation for test credentials
• Add runtime validation to prevent test credentials in production environments

⚠️ LOW Severity Issues

3. Privileged Container Capabilities

File: .devcontainer/devcontainer.json:22-25

"runArgs": [
  "--privileged",
  "--cap-add=NET_ADMIN", 
  "--cap-add=NET_RAW",
  "--cap-add=SYS_PTRACE"
]

Risk: Development container runs with elevated privileges for Kind/Docker-in-Docker functionality.

Remediation: This is acceptable for development environments but should never be used in production. Current usage is appropriate.

4. Git Installation from Untrusted URLs

File: docker/dagster-demo/Dockerfile:125-126

RUN pip install --no-cache-dir \
    "pyiceberg @ git+https://github.com/apache/iceberg-python.git@9687d080f28951464cf02fb2645e2a1185838b21"

Risk: Installing from git without signature verification.

Remediation: This is a documented workaround for PyIceberg compatibility. The commit hash is pinned for reproducibility. Consider adding checksum verification when PyIceberg 0.11.1 is released.

✅ Security Best Practices Observed

1. Non-root execution: All containers properly configured with runAsNonRoot: true and explicit user IDs
2. ReadOnly filesystems: Containers use readOnlyRootFilesystem: true where applicable
3. Dropped capabilities: All containers drop ALL capabilities and only add what's needed
4. Input validation: Shell scripts use proper parameter validation and error handling
5. Dependency pinning: Python dependencies properly pinned to prevent supply chain attacks
6. Secrets handling: No hardcoded secrets in production configurations
7. Resource limits: All containers have appropriate resource limits set

No Critical or High Severity Issues Found

The security review found no critical injection vulnerabilities, authentication bypasses, or data exposure issues. The identified medium/low severity items are primarily related to development environment configurations and documented workarounds.

Recommendation: ✅ APPROVED FOR MERGE

This PR demonstrates good security practices with only minor issues that are appropriate for the development/testing context. The changes appear to be focused on infrastructure improvements and compatibility fixes without introducing security regressions.

---

[greptile-apps]

Greptile Summary

Fixed PyIceberg 0.11.0 / Polaris 1.2.0 incompatibility by pinning to PR #3010 merge commit (9687d08) across Docker image, setup script, CI runner, and pyproject.toml constraints. Added reproducible devcontainer with K8s tooling (Kind, Helm, kubectl) and shellcheck for quality gates. Fixed Helm configuration issues: Dagster port 3000 (non-root compatibility), Cube Store public image, env dict-to-list format, and separated Cube API startup/readiness probes.

Key changes:

• PyIceberg git install at 5 locations with TODO(pyiceberg-0.11.1) tracking comments
• Devcontainer with Docker-in-Docker, network firewall, 8GB memory requirement
• Helm values: Dagster memory limits increased (512Mi → 1.5Gi), Polaris 1.2.0 pin
• E2E-ERROR-REPORT.md documents 46 failures across 12 error categories
• CI script uses UV_NO_SYNC=1 to preserve manual PyIceberg install

Gate results: All gates passed (0 blockers, 3 warnings)

Confidence Score: 4/5

• Safe to merge with minor follow-up verification recommended
• Comprehensive infrastructure fix with consistent implementation across 5 locations (Docker, setup script, CI, 2 pyproject.toml files). All changes include proper TODO tracking comments. Devcontainer adds valuable development tooling. Minor concerns: hardcoded Dagster versions in Dockerfile could drift from lockfile, and Polaris downgrade from 1.2.1→1.2.0 should be verified. E2E validation deferred to live run (AC-13.6, AC-13.7).
• Verify docker/dagster-demo/Dockerfile (hardcoded Dagster versions) and charts/floe-platform/values-test.yaml (Polaris version downgrade intentionality)

Important Files Changed

Filename	Overview
docker/dagster-demo/Dockerfile	Adds PyIceberg git install from pinned commit (9687d08) with PUT fix, adds git build dependency, includes smoke test validation, and configures DAGSTER_HOME
testing/k8s/setup-cluster.sh	Adds install_pyiceberg_fix() function to install PyIceberg from git with PUT fix, now called in main() flow
testing/ci/test-e2e.sh	Adds UV_NO_SYNC=1 to preserve manual PyIceberg install, fixes Dagster port-forward from 80:3000 to 3000:3000
packages/floe-iceberg/pyproject.toml	Adds !=0.11.0 exclusion to pyiceberg dependency constraint to prevent broken version installation
plugins/floe-catalog-polaris/pyproject.toml	Adds !=0.11.0 exclusion to pyiceberg dependency constraint matching floe-iceberg package
.devcontainer/Dockerfile	New devcontainer with shellcheck, build-essential, netcat, Kind, Helm, kubectl, uv, and network firewall support
charts/floe-platform/values-test.yaml	Changes Dagster port to 3000, updates Polaris to 1.2.0, switches Cube Store to public image, increases memory limits for Dagster
charts/floe-platform/charts/cube/templates/deployment-api.yaml	Separates startupProbe from readinessProbe, adds 12 failure threshold for startup allowing 60s initialization time

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[PyIceberg 0.11.0 Issue] -->|Missing PUT in HttpMethod| B[Polaris 1.2.0 Incompatibility]
    
    B --> C[Fix: Install from Git PR #3010]
    
    C --> D1[Docker Image<br/>Dockerfile]
    C --> D2[Setup Script<br/>setup-cluster.sh]
    C --> D3[CI Runner<br/>test-e2e.sh]
    C --> D4[Package Constraints<br/>pyproject.toml x2]
    
    D1 -->|Build-time install| E1[Smoke test: assert 'PUT' in HttpMethod]
    D2 -->|Runtime install| E2[uv pip install from git]
    D3 -->|UV_NO_SYNC=1| E3[Preserve manual install]
    D4 -->|!=0.11.0| E4[Block broken version]
    
    E1 --> F[E2E Tests]
    E2 --> F
    E3 --> F
    E4 --> F
    
    G[Devcontainer Added] -->|shellcheck| H[Quality Gates]
    G -->|Kind/Helm/kubectl| I[K8s Testing]
    
    J[Helm Fixes] --> K1[Dagster Port 3000]
    J --> K2[Cube Store Public Image]
    J --> K3[Polaris 1.2.0 Pin]
    J --> K4[Cube Startup Probe]
    
    K1 --> F
    K2 --> F
    K3 --> F
    K4 --> F

Loading

_{Last reviewed commit: 15ae5e2}

[greptile-apps]

_{16 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[MacAttak]

Addressing Review Feedback

1. Hardcoded Dagster versions (Dockerfile:117-119)

Extracted to ARG DAGSTER_VERSION=1.12.14 and ARG DAGSTER_POSTGRES_VERSION=0.28.14 so they're centrally managed. Added a comment noting they must match uv.lock.

2. Polaris 1.2.0 downgrade (values-test.yaml:123)

Intentional. Polaris 1.2.1 introduced regressions in our E2E test suite (catalog bootstrap failures). 1.2.0-incubating is the last known-good version that works with our PyIceberg pinned commit. Will re-evaluate when PyIceberg 0.11.1 is released.

3. Unit test failures (3 tests)

Updated tests to match intentional changes:

• test_cube_store_image_uses_ghcr → test_cube_store_image_is_public: Switched to public cubejs/cubestore to avoid private registry auth failures in CI/Kind
• test_workspace_packages_use_no_deps: Updated to allow multiple pip install lines (workspace + dagster extras + pyiceberg)
• test_export_stage_uses_package_flag → test_export_stage_uses_all_packages_flag: uv 0.7+ changed from --package to --all-packages strategy