dev: add make notebook (#2528)

<!--
Thanks for opening a pull request!
-->

<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->

# Rationale for this change
Add 2 new Make command (`make notebook`) to spin up a jupyter notebook;
`make notebook-infra` spins up jupyter notebook along with integration
test infrastructure.

### Pyiceberg Example Notebook
Pyiceberg example notebook (`notebooks/pyiceberg_example.ipynb`) is
based on the
https://py.iceberg.apache.org/#getting-started-with-pyiceberg page and
doesn't require additional test infra.

### Spark Example Notebook
Spark integration example notebook
(`notebooks/spark_integration_example.ipynb`) is based on
https://iceberg.apache.org/docs/nightly/spark-getting-started/ and
requires integration test infrastructure (Spark, IRC, S3)

With spark connect (#2491) and our testing setup, we can quickly spin up
a local env with `make test-integration-exec` which includes:
* spark
* iceberg rest catalog
* hive metastore
* minio

In the jupyter notebook, connect to spark easily
```
from pyspark.sql import SparkSession

# Create SparkSession against the remote Spark Connect server
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
spark.sql("SHOW CATALOGS").show()
```

## Are these changes tested?
Yes, run both `make notebook` and `make notebook-infra` locally and run
the example notebooks

## Are there any user-facing changes?

<!-- In the case of user-facing changes, please add the changelog label.
-->

Author: kevinjqliu

.gitignore

@@ -41,6 +41,9 @@ bin/
 .mypy_cache/
 htmlcov
 
+# Jupyter notebook checkpoints
+.ipynb_checkpoints/
+
 pyiceberg/avro/decoder_fast.c
 pyiceberg/avro/*.html
 pyiceberg/avro/*.so

.pre-commit-config.yaml

@@ -32,6 +32,11 @@ repos:
       - id: ruff
         args: [ --fix, --exit-non-zero-on-fix ]
       - id: ruff-format
+  - repo: https://github.com/nbQA-dev/nbQA
+    rev: 1.9.1
+    hooks:
+      - id: nbqa-ruff
+        args: [ --fix, --exit-non-zero-on-fix ]
   - repo: https://github.com/pre-commit/mirrors-mypy
     rev: v1.18.2
     hooks:

Makefile

@@ -97,7 +97,7 @@ test: ## Run all unit tests (excluding integration)
 
 test-integration: test-integration-setup test-integration-exec test-integration-cleanup ## Run integration tests
 
-test-integration-setup: ## Start Docker services for integration tests
+test-integration-setup: install ## Start Docker services for integration tests
 	docker compose -f dev/docker-compose-integration.yml kill
 	docker compose -f dev/docker-compose-integration.yml rm -f
 	docker compose -f dev/docker-compose-integration.yml up -d --build --wait
@@ -153,6 +153,21 @@ docs-serve: ## Serve local docs preview (hot reload)
 docs-build: ## Build the static documentation site
 	uv run $(PYTHON_ARG) mkdocs build -f mkdocs/mkdocs.yml --strict
 
+# ========================
+# Experimentation
+# ========================
+
+##@ Experimentation
+
+notebook-install: ## Install notebook dependencies
+	uv sync $(PYTHON_ARG) --all-extras --group notebook
+
+notebook: notebook-install ## Launch notebook for experimentation
+	uv run jupyter lab --notebook-dir=notebooks
+
+notebook-infra: notebook-install test-integration-setup ## Launch notebook with integration test infra (Spark, Iceberg Rest Catalog, object storage, etc.)
+	uv run jupyter lab --notebook-dir=notebooks
+
 # ===================
 # Project Maintenance
 # ===================
@@ -167,6 +182,8 @@ clean: ## Remove build artifacts and caches
 	@find . -name "__pycache__" -exec echo Deleting {} \; -exec rm -rf {} +
 	@find . -name "*.pyd" -exec echo Deleting {} \; -delete
 	@find . -name "*.pyo" -exec echo Deleting {} \; -delete
+	@echo "Cleaning up Jupyter notebook checkpoints..."
+	@find . -name ".ipynb_checkpoints" -exec echo Deleting {} \; -exec rm -rf {} +
 	@echo "Cleanup complete."
 
 uv-lock: ## Regenerate uv.lock file from pyproject.toml

dev/.rat-excludes

@@ -5,3 +5,4 @@ build
 .gitignore
 uv.lock
 mkdocs/*
+notebooks/*

mkdocs/docs/contributing.md

@@ -228,6 +228,41 @@ export PYICEBERG_CATALOG__TEST_CATALOG__ACCESS_KEY_ID=username
 export PYICEBERG_CATALOG__TEST_CATALOG__SECRET_ACCESS_KEY=password
 ```
 
+## Notebooks for Experimentation
+
+PyIceberg provides Jupyter notebooks for quick experimentation and learning. Two Make commands are available depending on your needs:
+
+### PyIceberg Examples (`make notebook`)
+
+For basic PyIceberg experimentation without additional infrastructure:
+
+```bash
+make notebook
+```
+
+This will install notebook dependencies and launch Jupyter Lab in the `notebooks/` directory.
+
+**PyIceberg Example Notebook** (`notebooks/pyiceberg_example.ipynb`) is based on the [Getting Started with PyIceberg](https://py.iceberg.apache.org/#getting-started-with-pyiceberg) page. It demonstrates basic PyIceberg operations like creating catalogs, schemas, and querying tables without requiring any external services.
+
+### Spark Integration Examples (`make notebook-infra`)
+
+For working with PyIceberg alongside Spark, use the infrastructure-enabled notebook environment:
+
+```bash
+make notebook-infra
+```
+
+This command spins up the full integration test infrastructure via Docker Compose, including:
+
+- **Spark** (with Spark Connect)
+- **Iceberg REST Catalog** (using the [`apache/iceberg-rest-fixture`](https://hub.docker.com/r/apache/iceberg-rest-fixture) image)
+- **Hive Metastore**
+- **S3-compatible object storage** (Minio)
+
+**Spark Example Notebook** (`notebooks/spark_integration_example.ipynb`) is based on the [Spark Getting Started](https://iceberg.apache.org/docs/nightly/spark-getting-started/) guide. This notebook demonstrates how to work with PyIceberg alongside Spark, leveraging the Docker-based testing setup for a complete local development environment.
+
+After running `make notebook-infra`, open `spark_integration_example.ipynb` in the Jupyter Lab interface to explore Spark integration capabilities.
+
 ## Code standards
 
 Below are the formalized conventions that we adhere to in the PyIceberg project. The goal of this is to have a common agreement on how to evolve the codebase, but also using it as guidelines for newcomers to the project.

mkdocs/docs/index.md

@@ -198,6 +198,10 @@ Since the catalog was configured to use the local filesystem, we can explore how
 find /tmp/warehouse/
 ```
 
+## Try it yourself with Jupyter Notebooks
+
+PyIceberg provides Jupyter notebooks for hands-on experimentation with the examples above and more. Check out the [Notebooks for Experimentation](contributing.md#notebooks-for-experimentation) guide.
+
 ## More details
 
 For the details, please check the [CLI](cli.md) or [Python API](api.md) page.