Prediction Market Thoughts part 3: The Holy Trinity of Bots

I previously posted my thoughts on backend and frontend features I'd like to see in prediction markets. This post will cover my thoughts on bots.

The three types of bot participation in prediction markets

This post is partially inspired by Ozzie Gooen's post here on using bots to resolve questions. This is a cool post, but I feel like it's only one part of the puzzle. In my mind, there are three main types of bots that could be useful in prediction markets:

1. Question Creation Bots. These bots define resolution criteria for markets.

2. Trading Bots. Bots that make trades.

3. Question Resolution Bots. Bots that resolve questions, as Gooen discusses.

I feel strongly not just that each of these avenues for bots is under-explored, but that having all three types together in a single ecosystem realizes some important synergies:

• Scale: A theme of this blog is that prediction markets are information-theoretical constructs with the ability to learn. But learning requires lots of data to learn from, which means lots of markets and predictions. If we can have a prediction market platform that functions end-to-end without human intervention, we can potentially get to a new level of scale.

• Frictionlessness: Bot-generated markets can be uninteresting or ambiguous, and bot resolutions can be incorrect, which makes these markets unappealing to humans. But bot traders don't care about this type of friction and can potentially learn to adjust to it better than humans can.

• Retrodiction: If an LLM has a data-cutoff date, we have good reason to believe a bot based on that LLM would not have any special advantage in predicting events after that date. Similarly, any markets proposed by such a bot would not, by their existence on the platform, offer predictive power. This leads to the intriguing possibility of letting a variety of such bots compete in a retrospective simulation of a prediction market over a period of years before the present, where the bots in the market are gradually fed news from news sources one date at a time.

To paint with a broad brush: Systems optimized for machines are different from systems optimized for humans, so we should explore what a reorientation of the prediction market ecosystem would look like as a radical change of design space.

Question Creation Bot Ideas

Manifold already offers AI assistance in creating markets, but this usually starts with human prompting. What could a more autonomous market creation engine look like?

RAG from rags

Most prediction markets pertain to events covered in the news. Consider a bot that works on something like the following prompts:

You are a master prediction market creator. It is your job to create prediction markets that are interesting, have clear resolution criteria, and which are relevant to events in the news. Today, you will be creating a title for a prediction market based on the following news article.

Insert news article here.

Please brainstorm a future event that might or might not come to pass relevant to the content of this article. Here are some examples of the types of events you might track, based on previous articles

Insert zero-shot examples

All the information relevant to decide if the market has resolved yes or no should be publicly available by Date.

And then:

Here is a title and date for a prospective prediction market.

Insert title and date generated by previous pass

Now please write resolution criteria for the market you have just titled. The resolution should be clear and unambiguous: Anyone who is familiar with the events in question after the resolution date should be able to agree on how the market should resolve. Please also include a list of URL sources that could be accessed on the resolution date that could be used to resolve the market.

It seems likely to me that these prompts¹ could give us as many passable markets as we had news articles to feed it.

Variants on existing markets

… and in fact, we could probably get even more markets than that, because essentially any prediction market admits modification of details. Details like:

• Names

  • Example prompt: Here is a prediction market titled <“Will JD Vance win the 2028 US Presidential Election?”>. Does this prediction market reference a particular name? If so, please come up with a similar market but with a different name.

• Dates

  • Example prompt: Here is a prediction market titled <“Will GPT-5 be released before August 2025”>. Does this prediction market reference a particular date? If so, please come up with a similar market but with a different date.

• Places

  • Example prompt: Here is a prediction market titled <“Will Japan experience a large disaster during July 2025?”>. Does this prediction market reference a particular place? If so, please come up with a similar market but with a different place.

• Prices

  • Example prompt: Here is a prediction market titled <“Bitcoin $130K in July?”>. Does this prediction market reference a particular price? If so, please come up with a similar market but with a different price.

& cetera.

Trading Bot Ideas

There is a nice post here that criticizes some prior literature on AI forecasting. As far as I can tell, the forecasting bots in these articles all work by a straightforward RAG pipeline where they request news articles from a search engine and then are prompted to base a prediction off of the articles.

The post highlights three things that the AIs have problems with²:

• The AI fails to find up-to-date information on the questions

• The AI finds only low-quality information on the questions

• The AI lacks high-quality quantitative reasoning

Given the architecture of the bots in question, these problems seem pretty unsurprising. But perhaps this leaves the door open for more sophisticated bots that work better.

In particular, I notice that many of the strategies I personally use to trade are amenable to replication by a bot, but probably not by a bot of the type described above. So consider this section my alpha dump.

Using price data from other markets to inform trades

I very frequently use prices of other markets to help me determine trades. Sometimes this involves determining a price for a thinly traded market by taking averages of similar big markets. Other times it involves directly arbitraging markets against each other. Strategies of this type clearly fight back against the out-of-date-info and no-quantitative-info problems.

I could imagine both a "soft" form which identifies markets which are similar-but-not-the-same and analyzes correlations to keep the price vector close to its principal components, and a "hard" version which locates and rigorously vets identical markets or markets where an outcome on one implies an outcome on the other and arbitrages these directly.

Identify markets that experience time decay

Many prediction markets can be expressed with titles of the form "Will X happen by Y date?". It does not seem too hard to get an LLM/regex to consistently identify markets of this form. We can then look at the price: If a market like this is unresolved and not at an extremely high price, it is usually a safe bet that "X" has not happened yet. The prices of such markets should then decay over time, but one often finds (at least on Manifold) markets that haven't been traded in a while. This suggests that a bot could trade on the information of the previous price.

A typical way to model this might be to assume that "X" events happen according to a Poisson process. We can then infer the rate of this process from the time and price of the last trade, and extrapolate what the price should be now. Of course, there is the risk that the event is more likely to happen on certain dates than others - perhaps this could also be dealt with by asking the LLM a question like, "which of these dates is the most likely for X to happen?" and using that to inform the model.

Calibration Bots

In an earlier post I advocated avoiding calibration analysis in favor of analysis of particular trading strategies. Perhaps the synthesis of this is: We should have more calibration bots - bots which specifically analyze calibration and make bets to correct miscalibration. There are a few possible flavors:

• Sitewide calibration

• Topic-by-topic calibration

• User-by-user calibration (i.e. identify fish)

  • The Manifold house bot once did this, which is in retrospect somewhat ironic, since this seems like one of the worst ways to make a profit in terms of keeping new users interested. But this is less of a problem in a bot-centric platform.

Market Resolution Bot Ideas

Much is already covered in the post I mentioned above. I'll try to keep to ideas not discussed there.

Combinatorial Markets

I have already written about these here and here. But it's important to realize that this is a type of market resolution strategy that is totally automatic, yet relatively rare on most platforms today.

Combinatorial choice of LLMs and Resolution Sources

To extend this idea: In designing an LLM/RAG based resolution bot, one might consider questions of which LLM or data source is best, or if there are effective ways of aggregating multiple choices into a single resolver. But another approach is to simply create multiple different markets, each with a different combination of LLM or data source choice. From here, we can already implement simple aggregation schemes like majority vote using combinatorial market over the individual resolutions. Ideally, traders would have the chance to learn more about the nuances of the different systems, and the "best" resolvers (in terms of the usefulness of their resolution decisions in predicting further outcomes) would win market share.

Lazy resolution

Already on Manifold there are many examples of questions which are "decidable" in the computational sense. And one could imagine much more technologically useful questions than these (for example, about whether certain training regimes for ML models will produce certain levels of performance).

The downside of such markets is their uncertain resolution date. But as time goes on and compute becomes cheaper, markets like these will only become easier to resolve. One could imagine that joint markets over the time of resolution and the answer could be insightful. One can even think of integrating formal proof into the resolution procedure to allow for the avoidance of "brute force"³.

Concluding Thoughts

This is turning into something of a series of posts in which I argue that there are a lot of potential frontiers to be pushed in prediction markets. AI is a particularly interesting direction because it’s still an evolving technology, and there is already a fundamental connection between AI and prediction markets in that they both have to do with “knowledge creation”. Hopefully we will see this space explored more!

1. Perhaps after several rounds of tweaking, elaboration, and CoT-instruction provision.

2. and a fourth way which just creates noise, namely data leakage

3. but discussion of that is best left to a future post.