IT subsidiary of a retail group — Senior Systems Analyst (AI Application Engineer)
Cloud architecture, batch and data platforms, AI application development and operations, spanning cloud architecture, DevOps, data engineering and AI products.
Retail CDP: batch platform evolution (ADF polling → queue → job scheduler)
ADF 15-minute polling → Storage Queue → single-table job scheduler; led the production cutover, hardened after three incidents, gen-2 with 1,190 unit tests
- Replaced a 15-minute ADF polling job with a .NET BackgroundService + Azure Storage Queue pipeline: global sp_getapplock claiming, heartbeat lease renewal, poison queue + failure notifications, recovery scan as a safety net; claim latency 15 min → seconds; led the production cutover after a rehearsal in test that surfaced trigger-branch and storage-firewall issues in advance
- Hardened it after three incidents: Run From Package to end deployment races, streamed download lists to fix a 6.64M-member OOM, second-level graceful handoff on restart instead of a 60-minute zombie reclaim, transient-error retry on SQL connections
- Designed and implemented a second-generation job scheduler (dev complete, production pending): one job table for six task types, one applock per resource, every state transition a conditional update checked by affected rows, DB-clock timeouts, scale-out and concurrency by table values without redeploys; retires three ADF chains and the Storage Queue; 710-line spec, 1,190 unit tests, 68/68 dev integration tests; fixed cross-instance races and a cross-thread connection deadlock
- Working method: design phases produce documents only (spec plus ~40 briefs/reviews); after decisions, subagents implement (sonnet for simple tasks, opus for SQL/concurrency) with a review per task; 88 commits in one day
Retail CDP: SQL builder rewrite and performance root-causing
Razor SQL assembly → parameterized builder; three measurement-first root-causes: 559s → 0.4s, full run 24m45s → 47s, zero builder changes
- Rewrote Razor-template SQL assembly as a parameterized C# SqlBuilder (four box types) and fixed three known bugs; ran a Synapse parameterization PoC first (multi-statement CTAS, temp-table visibility, TOP(@var), string parameter typing) which decided a DECLARE fallback; implemented by Copilot over 15 TDD tasks with Claude reviewing in batches
- Three measurement-first root-cause investigations with zero builder changes: NVARCHAR implicit conversion killing an index seek on 82.7M rows → CONVERT on the parameter side, 559s → 0.4s; stale production view definition in dev → one box 1237s → 12.5s and a full smoke run 24m45s → 47s; appsettings.json never deployed due to a csproj setting → three 3600s timeouts became a 5.7-minute run, with the default fixed to prevent recurrence
- Dashboard contract revision in production (three-parameter SPs, dashboard computed in the same message after packaging, replacing ADF dispatch), SP bucket completion so zero-purchase packages pass validation; materialized reused CTEs as CTAS temp tables: largest package 12.6M members 425/364s → 141/131s with row-identical output across four sizes
Retail CDP: Text-to-SQL (PoC → production)
Natural-language queries over 920M rows, PoC → production FastAPI service; gold SQL 10/10, behavior tests 23/23, zero fabrication
- PoC v1 on synthetic data + DuckDB; PoC v2 against a production replica (920M transaction rows): understand → value-linking guards → option-based clarification (≤3 rounds) → echo understanding → generate SQL → sql_guard validation and auto-repair → EXPLAIN gate → execute → code computes statistics, the LLM only interprets
- Production: standalone FastAPI service; sqlglot AST dialect packs for both T-SQL and PostgreSQL; semantic-model YAML where every fact carries confidence / evidence / review; 10 ADRs; data profiling 650s → 338s
- Evaluation suite: gold-SQL numeric comparison (0.5% tolerance) 10/10, behavior tests (should ask / should not ask / cannot do) 23/23 × 2, insight fabrication check at zero, actionability 3.3 → 4.17; Claude vs GPT comparison (deeper conclusions vs ~100× cost)
- Product-positioning ADR: once two more systems were confirmed to onboard, pivoted from a single-system tool to "execution engine + 13-step onboarding process"; guard rule: any condition that cannot become SQL must be asked back, never silently dropped
SRE-agent: Azure Data Factory failure-triage system (designed and built solo, ~5 weeks)
Read-only ADF failure triage, built solo in ~5 weeks; evals rejected a model upgrade, precedent override took accuracy 12% → 80%
- Read-only across 3 ADF factories / 44 pipelines: every 5 minutes detect failed runs → de-duplicate into incidents (failure chains, soft-failure detection) → classify with rules first, LLM as fallback, unclassified on failure → Teams Adaptive Card to the on-call engineer (conclusion / what to do now / what to check first); ~9,900 lines Python, 208 tests, 140 incidents
- Rule engine with patterns, an actions allowlist and playbooks all in YAML, hot-reloaded every cycle; the LLM (Azure OpenAI, switchable to Claude) is called only when no rule matches (~3% of cases)
- Used evals to reject a model upgrade: on 65 cases rules 12% / gpt-mini 46% / Opus 83%, but after removing duplicates the two models tied → the bottleneck was rule actions, not the model → built human-precedent override instead; action accuracy 12% → 80%, act/don't-act 25% → 92%; daily digest cut noise 53% with zero misses
- Learning agent (Claude Opus on Microsoft Foundry) is proposal-only: a human clicks apply on the site, pytest runs automatically and reverts on failure, passing changes auto-commit; tools limited to reading the DB and knowledge files, commenting and proposing — zero shell, zero Azure permissions
- Streamlit on-call site (with /mobile), supervisor guarding three processes plus a watchdog, SQLite / Azure Table dual storage, archive of 73,292 historical ADF runs
Retail allocation platform
Enterprise allocation system (1,100+ files) delivered solo in 2.5 months; FastAPI + React 19, contract-first, Azure Pipelines CI/CD
- Independently delivered an enterprise retail allocation platform (1,100+ files) in 2.5 months with AI collaboration; heavy requirements interviews, allocation logic designed from per-store sales data and special cases
- Backend Python 3.12 / FastAPI / SQLAlchemy 2 / Alembic / PostgreSQL in a layered architecture (api / core / engine / etl / repositories / services); frontend React 19 + TypeScript / Vite / TanStack Query·Table·Virtual / react-hook-form + zod / Tailwind 4
- Contract-first: openapi.json → openapi-typescript generates frontend types, keeping both sides in sync
- CI/CD on Azure Pipelines: PRs run backend pytest and frontend lint/test/build in parallel; 8 backend test modules, 56 frontend test files, UAT smoke tests
- Requirement alignment via single-file HTML prototypes that stakeholders preview before real development, cutting the spec → build → acceptance cycle; disciplined docs (decisions / lessons / runbooks / roadmap)
Demand-allocation ML models
Store-level allocation forecasting with LightGBM / CatBoost, Optuna tuning, systematic experiments (baseline → ablation → blending)
- Store-level allocation forecasting with LightGBM, CatBoost, Optuna tuning and isotonic calibration
- Systematic pipeline: baseline → feature engineering → ablation → blending → rules overlay → misallocation diagnosis; delivered ML allocation plan draft, ER model and table schema
Enterprise Customer Data Platform (6 repos / 2,200+ files): AI API and foundations
A 6-repo CDP: React + .NET + Synapse; AI API on a Google ADK pipeline for product tagging and marketing copy
- React 19 + TS + TanStack Router/Query frontend, .NET Web API, .NET batch system with unit tests; SQL Server multi-schema design + Azure Synapse warehouse (CTAS / HASH-distributed tuning)
- AI API: Python 3.13 + FastAPI + Google ADK agent pipeline + LiteLLM → Azure OpenAI; automated product tagging (migrated from a legacy service) and a 3-stage marketing-copy pipeline; strict API contract (four status codes, per-user rate limiting, fail-honestly-no-fallback)
- Three environments (dev/test/prod) per repo on Azure DevOps pipelines; incident records and architecture runbooks; legacy-service retirement and a production misconfiguration incident
MCP Gateway (self-built AI infrastructure)
Live in 3 days: internal APIs wrapped as MCP tools with per-client auth, SSRF protection, encrypted credentials, input/output audit
- Wraps internal HTTP APIs as MCP tools so Claude Code / Codex / Copilot reach internal systems through one secure entry point; tool catalog in DB, approved tools go live without code changes or restarts
- Security: per-client Bearer auth, SSRF protection (blocks localhost/cloud metadata + allowlist), Fernet-encrypted upstream credentials, audit logs with sensitive-parameter masking
- Python 3.12 / MCP SDK (Streamable HTTP) / Starlette / SQLAlchemy 2 / Streamlit admin; 20+ test files incl. acceptance tests (pytest-asyncio + respx); v1 live in 3 days
- Extensions: response bodies added to the audit log (UTF-8-safe truncation with original byte count), a connect page generating configs for eight AI clients, a per-client → per-user permission draft
Convenience-store auto-ordering system: operations documentation
Onboarding plan and daily-dispatch runbook for an auto-ordering system; inventoried 183 pipelines and a 1,324-line core SP
- Wrote a 10-section onboarding plan for engineers new to the system: inventory of 183 ADF pipelines (39 without triggers), 66 triggers, a 1,324-line core SP; risk register (single maintainer, plaintext SAS, no test environment, global temp tables) and a four-week learning path
- Daily-dispatch runbook: timing of five main stages with latest start times, a parallel emergency path, a decision tree; key insight that ML timeout plus retry can stall 8 hours without an error, so decisions must be time-based (09:30 checkpoint) rather than alert-driven; BlobFuse upgrade investigation, ML pipeline retry / webhook design proposal
Log anomaly analysis
Gradio two-stage filtering → Azure OpenAI analysis → SQLite; token and cost estimate before every AI call
- Gradio UI with two-stage filtering of large logs → Azure OpenAI analysis → SQLite; strict three-layer decoupling (filter / ai / db); token and cost estimation before each AI call with threshold warnings
Enterprise chatbot product deployment
Dockerized a Java enterprise chatbot (700+ files) with a deployment SOP; production and UAT upgrades, incident handling
- Source governance and dockerization of an enterprise chatbot product (Java/JSP/Kotlin, 700+ files): Compose with MSSQL 2022, Redis Stack, Solr 9.5, custom parser; DB init, health checks, fresh-VM deployment SOP; git filter-repo history cleanup, Git LFS; production and UAT upgrades (including training a colleague), OOM / WSL incident handling
Azure IaC, security & data engineering
Modular Bicep IaC, reconciled with portal drift, Dify on Container Apps, a post-quantum crypto asset scanner
- Modular Bicep IaC (network / keyvault / storage / postgres / appservice): IP allowlist + Private Endpoint dual path, Key Vault secrets, VNet Integration + RBAC, rollback docs
- Reconciled IaC with portal drift: what-if reports "no change" for resources with listKeys() and runtime references → az show diffs per resource plus activity-log tracing; kept existing RBAC assignment GUIDs to avoid duplicate guid() resources; crontab lost on App Service Linux restarts → startup script re-registers schedules
- Dify (open-source LLM platform) on Azure Container Apps (PostgreSQL + pgvector / Redis / Storage, with cost estimate); Terraform for LLM resource groups
- Post-quantum crypto asset scanner M0: inventory across TLS / Key Vault / App Service / App Gateway with algorithm, key size, expiry and quantum-risk grade (NIST FIPS 203/204/205)
AI engineering methodology (institutionalized)
Dual-agent AGENTS.md, AI code-review loop, Claude Skills with evals, dispatch skill and context-bar mod, internal talks
- One shared AGENTS.md driving Claude Code and Copilot CLI on enterprise projects; git worktree for parallel branches
- Adopted and operated the agent-skill toolchain and built an automated AI code-review loop (AI fixes → commit → replies to review thread → re-review until it converges)
- Packaged team domain knowledge as Claude Skills with evals (Azure Synapse tuning, ADF standards) plus project-specific skills (e.g. an Azure DevOps PR flow)
- A self-made dispatch skill routes each task to the cheapest model that is good enough (haiku to find / sonnet for well-specified work / opus for cross-file and concurrency); a self-made Claude Code mod, context-bar, shows context usage, token cost and rate limits live
- Internal tech talk on the requirement → AI prototype → user alignment → build workflow (14-page outline, script, 4 diagrams) with 5 reusable de-projectized skills