A personal project in applied machine learning, data pipelines, market data ingestion, and decision systems.

Overview

This is an applied ML and market-systems project I built to test a simple question:

Can a custom NLP system identify pricing inefficiencies in live Kalshi prediction markets before the market fully incorporates current context?

The project focuses on Trump-related Kalshi word markets, where each contract asks whether a specific word will be said. I built a pipeline that:

• scrapes and structures historical speech data,
• aligns speech samples with historical news context,
• generates training examples around target words,
• fine-tunes a transformer-based classifier,
• calibrates model probabilities,
• pulls live Kalshi market data, and
• compares model belief against market-implied probability to surface potential edge.

This was not just a modeling exercise. I treated it like a small production system: data collection, feature generation, training, calibration, inference, live API integration, and recommendation logic all had to work together coherently.

I also used agentic development patterns while building it. In addition to writing the core pipeline, I generated repository documentation and system schematics that made the codebase easier for coding agents to navigate safely. That reduced ambiguity around component boundaries and blast radius, which made it easier to iterate on training, inference, and orchestration code without introducing unnecessary regressions.

What I Built

At a high level, the system has two halves.

1 - Offline training pipeline

I built a dataset-generation pipeline that combines several inputs:

• historical Trump transcripts,
• historical news context tied to the date of each speech,
• Kalshi market vocabulary,
• semantic target expansion for related terms.

From that, the pipeline creates chunked training samples that teach a model to estimate whether a target word is likely to appear given the surrounding political/news context.

The training stack includes:

• data preparation in dataset_builder.py,
• model training in trainer.py / fasttrainer.py,
• a transformer classifier based on DeBERTa,
• post-training probability calibration using logistic regression.

That last step mattered. For a trading-style decision system, raw classifier confidence is not enough. I added calibration so the model output behaves more like a usable probability estimate instead of a generic score.

2 - Online inference and decision layer

For the runtime path, I built a scanner that can take a live Kalshi market URL, discover all related sub-contracts, fetch order book data, estimate market-implied probabilities, and run model inference against each candidate word.

The live system pulls together:

• public Kalshi order book data,
• current news context,
• local model inference,
• edge computation: model_probability - market_probability.

The final output is not just a score. It produces a directional recommendation such as:

• BUY YES
• BUY NO
• BUY NEITHER

That framing forced the project to cross the line from “interesting notebook” into an actual decision engine.

Technical Highlights

End-to-end system ownership

I built the full pipeline rather than stopping at one layer:

• data acquisition,
• preprocessing,
• training set design,
• model fine-tuning,
• calibration,
• live market ingestion,
• inference orchestration,
• recommendation logic.

That gave me experience with the real engineering constraint that matters most in ML systems: the model is only one component, and often not the hardest one.

Agent-assisted development with explicit system context

One useful meta-layer in this project was structuring the repository so coding agents could contribute effectively. I created autogenerated documentation and architecture notes that described how data moved through the system, which modules owned which responsibilities, and what downstream behavior each component could affect.

That matters because agentic coding is only as useful as the context you provide it. By making system boundaries explicit, I improved the quality of code assistance and reduced the risk of broad, low-confidence edits. In practice, that meant faster iteration on a multi-stage ML codebase with better awareness of inference paths, training dependencies, and operational blast radius.

Probability calibration for decision quality

A useful lesson from this project was that model confidence and decision usefulness are different problems. I added a calibration layer because market comparisons require well-behaved probabilities, not just relative ranking.

That design choice reflects how I think about applied ML: optimize for the downstream decision, not just the training metric.

Working with weakly structured real-world data

The project required stitching together several messy inputs:

• transcripts from public sources,
• historical event context,
• live market contracts,
• target vocabularies that do not map cleanly to natural language.

A large part of the work was building reasonable abstractions around noisy data rather than assuming ideal inputs.

Translating ML output into an operational signal

I designed the runtime system around a simple but explicit decision rule:

if the model probability differs meaningfully from the market-implied probability, there may be tradeable edge.

Even when a project is exploratory, I prefer making the decision logic concrete. It exposes failure modes early and makes the system easier to evaluate.

Why This Project Matters

This project represents the kind of work I want to do professionally: building systems where ML, product thinking, and engineering discipline meet.

It demonstrates that I can:

• scope and build an end-to-end technical system independently,
• work across data engineering, applied NLP, and inference infrastructure,
• choose modeling approaches based on the operational use case,
• turn ambiguous ideas into a working product with clear logic,
• use agentic tooling pragmatically to accelerate development without losing architectural control.

It also reflects how I approach problems:

• start from a real use case,
• build the pipeline instead of only the model,
• force outputs into a decision-ready form,
• learn from the mismatch between theory and messy inputs.

Challenges I Worked Through

Some of the harder parts were not purely algorithmic.

Building a usable training set

There is no clean off-the-shelf dataset for “will this political speaker say this word in this context?” I had to define the problem as a learnable dataset, decide how to chunk speeches, pair them with historical context, and generate positive/negative examples.

Making probabilities meaningful

If a model is going to be compared directly against a market price, miscalibrated confidence becomes a real system problem. Adding calibration was important because the downstream use case depends on trustworthy probability estimates.

Closing the loop from data to recommendation

Many ML projects stop at evaluation output. I pushed this one into a usable runtime loop that could ingest live market data and produce a recommendation table in real time.

What I’d Improve Next

If I continue the project, the next steps would be:

• improve evaluation with out-of-time validation rather than only random splits,
• track paper trading results to measure real decision quality,
• add feature comparisons between pure text models and hybrid market/context models,
• harden the runtime path into a more production-ready service,
• extend the framework beyond a single political domain.

Those are the kinds of follow-on steps I think about naturally: not just “can the model run,” but “what would it take to trust this in a more realistic setting?”

Stack

• Python
• Hugging Face Transformers
• DeBERTa
• scikit-learn
• REST API integration
• custom scraping / data pipelines
• local inference and CLI-based orchestration

---