Backend & DataEngineer
Building production-grade data pipelines and backend systems — from ingestion to analytics — using Python, dbt, Airflow, Snowflake, and SQL, with a full-stack TypeScript background.
Daud Abdi
Backend & Data Engineer
15+
Projects
2 Yrs
Experience
10+
DE & Backend Tools
Core Expertise
End-to-end data pipeline development
dbt modelling, testing & Snowflake deployment
Airflow orchestration & Docker containerisation
Backend APIs & full-stack TypeScript systems
Engineering Data at Scale
Passionate about building scalable pipelines with clean architecture and production-grade reliability
Backend & Data Engineer
I'm a Backend & Data Engineer based in London, building end-to-end data pipelines and backend systems. My DE work spans Python ingestion scripts, dbt transformations (SCD Type 2, snapshots, tests), Airflow orchestration, and dual-target deployments to both PostgreSQL and Snowflake.
I bring a full-stack TypeScript/Node.js background to every project — meaning I understand the systems that produce the data, not just the pipelines that move it. My approach is hands-on and production-focused: real APIs, real databases, real orchestration, clean architecture.
15+
Projects
2 Yrs
Experience
Data Pipeline Development
End-to-end pipelines from ingestion to analytics with Python & dbt
dbt & Snowflake
SCD Type 2 snapshots, dbt models and tests, dual-target deployments
Airflow Orchestration
DAG-based pipeline orchestration with Docker Compose deployments
Backend APIs
Production-grade REST APIs with TypeScript, Node.js, and PostgreSQL
Technology Stack
Primary Languages
Frontend
Backend & APIs
Database & Cloud
Projects & Case Studies
Real-world data pipelines and backend systems built for production
NHS A&E Waiting List Pipeline
End-to-end pipeline ingesting NHS England A&E attendance and waiting time data across 11 months (2,197+ rows). Uses PySpark for transformation, loads into Snowflake, and models with dbt — all orchestrated via Airflow on Docker.
UK Land Registry Pipeline
Big data pipeline processing 10,057,373 property transactions from HM Land Registry (2016–2026). Python ingestion loads ~1.7 GB of CSV data into Snowflake, with DuckDB for local exploration. dbt models produce national price trends, property type breakdowns, and London borough-level analysis — 7/7 tests passing.
Fintech Fraud Detection Pipeline
Production-grade fraud detection pipeline ingesting 590,540 IEEE-CIS transactions. Features 12 engineered fraud signals, rule-based scoring (0–100), Apache Kafka streaming for real-time alerts, dbt models (6/6 tests passing), and end-to-end Airflow orchestration — all Dockerised.
UK Labour Stress Pipeline
Databricks-native pipeline ingesting ONS regional labour market data and producing a composite stress score index across UK regions. Uses a medallion architecture (Bronze → Silver → Gold) with PySpark, Delta Lake, and dbt models.
Crypto Stream Pipeline
Real-time streaming pipeline ingesting live BTC/ETH price ticks from the Binance WebSocket API through Apache Kafka into PostgreSQL. dbt models produce 1-minute OHLC candlesticks and rolling volatility metrics.
BoE Stats Pipeline
Data pipeline extracting macroeconomic series from the Bank of England IADB API — base rate, GBP/USD, GBP/EUR, mortgage rates, and M4 money supply. dbt project with 14/14 passing tests, staging and mart layers, orchestrated via Airflow.
FCA Register Pipeline
End-to-end data pipeline ingesting the UK FCA financial services register. Features SCD Type 2 snapshots, dbt models and tests (12/12 passing), and dual-target deployment to both local PostgreSQL and Snowflake.
UK Crime Pipeline
Batch data pipeline extracting 17,000+ crime records from the UK Police API. Stores data as Parquet, loads to PostgreSQL, and extends to AWS with S3, Glue, and Athena for cloud analytics.
UK Trade Pipeline
Airflow-orchestrated pipeline ingesting HMRC trade data via REST API. Dual PostgreSQL instances via Docker Compose, dbt project with 6 models and 10 passing tests.
TFL Disruptions Pipeline
Real-time pipeline polling the TFL API every 60 seconds for live disruption data. Fully Dockerised with dbt models and pushed to GitHub.
Fintech Microservices
Production-style fintech backend in Go with three independently deployable microservices — accounts, transactions, and notifications — each with its own PostgreSQL database. Built using Go's standard library only (no frameworks), with Docker Compose orchestration.
Job Scraper API
FastAPI backend that scrapes London Data Engineering and AI job listings from Reed.co.uk, stores them in PostgreSQL, and exposes a queryable REST API with filters, pagination, and aggregated stats (top companies, locations, salary ranges).
Payments API with Stripe Integration
Production-ready RESTful API with complete Stripe payment processing, secure webhook handling with SHA-256 signature verification, and comprehensive test coverage (38+ passing tests). Features Payment Intents, customer management, refunds, and CSRF protection.
OAuth Login API
Production-ready OAuth authentication API deployed on AWS (EC2 + RDS). Features Google and GitHub login, session management with HttpOnly cookies, CSRF protection, rate limiting, and 24/7 uptime with PM2. Includes 38 passing tests.
Advanced Authentication & Authorization API
Production-ready REST API deployed on AWS (EC2 + RDS PostgreSQL). Features JWT authentication, refresh tokens, email verification, password reset, and 24/7 uptime with PM2. Includes comprehensive test coverage with Jest.
JWT Authentication API (TypeScript)
Secure authentication API with JWT tokens, user registration, login, and protected routes. Built with TypeScript for type safety.
JWT Authentication API (JavaScript)
RESTful authentication API featuring JWT token-based authentication, bcrypt password hashing, and PostgreSQL database integration.
Budgeting API
RESTful API for budget management with full CRUD operations for budgets, income, and expenses tracking.
Expense Tracker API
Backend API for tracking expenses with category management and full CRUD functionality.
Tech Stack & Expertise
Modern data and backend technologies for building scalable, production-ready pipelines and systems
Languages
5 technologies
- Python
- TypeScript
- Go
- SQL
- JavaScript
Data Engineering
7 technologies
- dbt
- Apache Airflow
- Apache Kafka
- PySpark
- Pandas
- Parquet
- Delta Lake
Cloud & Warehousing
5 technologies
- Snowflake
- Databricks
- AWS S3
- AWS Glue
- AWS Athena
Databases
4 technologies
- PostgreSQL
- MySQL
- Database Design
- Query Optimisation
Backend & APIs
6 technologies
- Node.js
- Express
- FastAPI
- REST APIs
- JWT
- OAuth 2.0
DevOps & Tools
5 technologies
- Docker
- Git
- Linux
- Postman
- CI/CD
Shipping value across teams & clients
Production-focused delivery with clean architecture, documentation, and reliability top of mind.
2024 — Present
Software Engineer (Freelance)
Upwork & Indie Clients
- Delivered Go-based APIs, dashboards, and deployment-ready infrastructure guidance for startups.
- Collaborated closely with founders to scope MVPs, iterate quickly, and ship production features.
- Owned delivery end-to-end: architecture, implementation, QA, and documentation.
2025 — now
Backend Developer
Community & Open Source
- Maintained open-source Go services, emphasizing clean architecture and test coverage.
- Mentored peers on Git, Docker, and API best practices through workshops and pair sessions.
- Explored cloud-native tooling to keep deployment pipelines lean and discoverable.
Contact
Let's build something together
Whether you're looking for a data engineer or backend engineer for your next product, or just want to discuss an idea, feel free to reach out. I usually reply within 24 hours.