Serverless cloud analytics for New Zealand electricity demand and real-time dispatch

Overview

Designed and implemented a serverless AWS lakehouse prototype for New Zealand electricity demand and near-real-time dispatch analytics, combining historical batch data with API-based ingestion for reporting and analysis.

My Role

Solution Design • Cloud Architecture • Data Engineering • Power BI

I designed the end-to-end architecture, configured the ingestion pipelines, structured the storage layers, worked with AWS analytics services, and connected the resulting data to Power BI for reporting with ODBC Amazon Athena.

The Challenge

Historical electricity-demand data and near-real-time dispatch data arrive in different formats and at different frequencies. The goal was to create one scalable analytics architecture capable of handling both batch and near-real-time ingestion while supporting reporting and future AI/ML use cases.

The Solution

The architecture uses two main pipelines:

Historical Demand
CSV → Amazon S3 Raw → AWS Glue → Cleaned Parquet → Apache Iceberg

Real-Time Dispatch
REST API → EventBridge Scheduler → AWS Lambda → Amazon S3 Raw JSON → AWS Glue → Apache Iceberg

The solution was implemented in AWS Asia Pacific (Sydney), ap-southeast-2, using managed/serverless services to reduce infrastructure administration.

Technology

AWS S3 • Lambda • EventBridge • Glue • Athena • Apache Iceberg • AWS Systems Manager Parameter Store • AWS KMS • HTTPS/TLS

Outcome

Built a working cloud analytics prototype capable of supporting:

• Historical electricity-demand analysis
• Near-real-time dispatch ingestion
• Centralised lakehouse storage
• Athena-based querying
• Power BI reporting
• Future AI/ML expansion

Visuals

1. AWS Architecture Diagram

2. Power BI dashboard screenshots