Streaming 2 million football records a month from many data vendors

In short

Problem: Football data arriving from many vendors at scale: every event of every match, for every player, across leagues like the EPL and the World Cup
My role: CTO: architecture, data, deployment, DevOps, cost and the AI roadmap
Key decisions: A multi-vendor streaming data pipeline; hybrid cloud across DigitalOcean, Hetzner, Cloudflare and AWS; Fastify and MongoDB; RAG and LLMs on Neo4j for AI
Result: 2 million records a month streamed from multiple vendors, automated CI/CD, faster deployments and a faster Node.js backend

The story

Sportinerd is a sports platform in the UAE: live scores, sports data analytics and news across football and beyond, with fan challenges, player insights and personalised updates. Since 2025, as CTO, I have designed and architected the platform behind it, and I own its engineering end to end.

The problem

Football data is huge. Every match produces events, every event involves players, and Sportinerd covers leagues like the EPL and the World Cup. That data arrives from several vendors at once, and all of it has to come together in one platform.

What I did

Built a data pipeline that streams data from multiple vendors for leagues like the EPL and the World Cup: about 2 million records a month.
Automated the CI/CD pipelines and streamlined the DevOps process.
Made the change request system frictionless, so deployments go out faster.
Optimised the cloud setup with a hybrid infrastructure across DigitalOcean, Hetzner, Cloudflare and AWS, with Nginx and Docker.
Improved backend performance by optimising the Node.js codebase with Fastify and MongoDB.

What I’m building now

A fantasy recommendation engine, using RAG and large language models over the match and player data.
Score prediction with AI.

What the role covers

The role is hands-on and architect-level. It covers:

Deployment, end to end. From bare-metal provisioning and CI/CD to zero-downtime canary rollouts, with GitOps as the single source of truth.
Agentic workflows. Multi-agent systems where specialised AI workers handle everything from real-time data ingestion to decisions driven by inference.
Graph RAG on Neo4j. Large language models combined with Neo4j’s property graph: entity-rich knowledge graphs, vector similarity search over graph projections, and reasoning with more context than a flat vector store can give.
Neo4j as the core data layer. Cypher-optimised schemas, clustering for resilience, and graph data science to find patterns across athletes, teams and events.
AI in DevOps. CI/CD pipelines that fix themselves, infrastructure that rightsizes itself, and runbooks that run from plain-language instructions.
Agents that generate ML services. Agents that read API contracts and generate complete ML microservice modules: model endpoints, feature stores, drift monitors and documentation.
Team and task distribution. Turning the roadmap into work packages and spreading them across developers, ML engineers and domain experts.
Cost (FinOps). Watching every compute cycle: spot instances, scaling schedules that follow when fans are online, and AI-powered cost anomaly detection.
Scaling with the user base. Predictive autoscaling from real-time telemetry, horizontal and vertical, whether the platform serves a thousand concurrent fans or a million.
Moving off managed databases. A phased move to self-hosted, Kubernetes-native clusters for Neo4j, Postgres and Redis, with backup and restore playbooks and multi-region replication.
Sport-specific agents. Match-day storytellers, injury-risk analysers and personal coaching companions, built from fine-tuned SLMs, tools and persistent memory on top of the graph.