Jupyter Notebook is an interactive computing environment for exploring data, running Python-based Spark code, and combining lakehouse data access with iterative analysis. It gives practitioners programmable access rather than restricting them to predefined BI metrics. section 3.3.2 section 9.2.3
Effective notebook environments in a lakehouse provide:
- Low-latency access to raw and transformed data, supported by predicate pushdown, column pruning, and query planning. section 9.2.3
- Open interfaces such as JDBC, ODBC, and Arrow Flight, with integrations for pandas, PyArrow, DuckDB, and SQLAlchemy. section 9.2.3
- Interoperable compute through Spark, embedded SQL engines, and machine-learning pipelines, without unnecessary data movement. section 9.2.3
- A path from exploratory notebooks to scheduled workflows, reusable semantic definitions, and production operations. section 9.2.3
Running PySpark against Iceberg
In a Docker-based lakehouse setup, open Jupyter Notebook at localhost:8888, create a Python notebook, and run PySpark to read PostgreSQL data and write Iceberg tables to a MinIO warehouse bucket. When container DNS fails, use the MinIO container IP discovered from the Docker network. section 3.2.4 section 3.3.2
docker network ls
Inspect Docker networks and container addresses when connecting notebook code to MinIO. section 3.3.2
