Intelligence, at runtime | Levanto

The premise

Models changed how software is written: you prompt a goal, the model writes the code. At runtime, that code is still mostly deterministic. It may call an LLM where waiting seconds is fine or text is the point (chatbots, analysis, research).

So the way code runs has not changed.

The question

What if runtime logic could be written in English too?

Application logic is flows: conditions, loops and branches. Most conditions end in True or False.

What if models become intelligent, fast, cheap, and honest enough to:

  1. iReplace these deterministic conditions
  2. iiRun the branching
  3. iiiand/or generate code at runtime

Imagine the LLM that writes your code also writing runtime logic that blends English and code:

A runtime filter, in English

with decisions():
    outages = [\
        t for t in tickets\
        if likely("this is a production outage", t) > 0.7\
    ]

Written by your LLM. Reads as N calls. The comparison is what fires the model.

Thesis 1

As they get smarter, faster, cheaper, and more honest, models will eat more and more of your code. In some cases, only models may remain.

What this unlocks

Five things you couldn’t do before.

1 / 5

Expressing the inexpressible

Before: wasn't doable

# works only if titles exactly match
if new_ticket.title == existing.title:
    merge_into(existing, new_ticket)

After

if likely("same underlying bug", pair) > 0.85:
    merge_into(existing, new_ticket)

2 / 5

One sentence instead of a subsystem

Before

if SSN.search(doc) or CC.search(doc):
    if dest in PUBLIC_BUCKETS:
        block(doc)
    elif "passport" in doc.lower():
        classify("restricted")
elif HEALTHCARE.search(doc) and region == "EU":
    classify("restricted")
# ... 120 more lines of regexes and lists

After

safe = (judge("contains no PII", doc)
        & judge("contains no profanity", doc)
        & judge("is on topic for this channel", doc))

match safe:
    case Yes():    publish(doc)
    case No():     block(doc)
    case Unsure(): queue_for_review(doc)

3 / 5

Real-world input without a parser

The ticket is English. So is the refund policy. Both go into the decision as-is:

Before

fields = parse_ticket(raw)
# breaks when the template changes
if tenant == "acme" and fields["amount"] < 50:
    approve(req)
elif tenant == "globex" and fields["vip"]:
    approve(req)

After

policy = tenant.policy
# "auto-approve small duplicate refunds"
if likely(policy, raw) > 0.9:
    approve(req)

4 / 5

Dynamic application logic

The software adapts without a deploy, because it writes part of its code at runtime:

Before

if vol_30d > 0.4 and rsi_14 < 30:
    grid.run(portfolio)
elif trend_7d > 0:
    dca.run(portfolio)
# thresholds hand-tuned, logic frozen at deploy

After

@adaptive("swing between a grid and a DCA strategy
           based on market conditions")
def trade(portfolio, market): ...

Rewritten as the market moves. Not a condition replaced: a block of code.

5 / 5

Train/evals on the entire software

CI/CD stops checking that the application still works and starts measuring how well it performs. When a number moves the wrong way, the pipeline has two levers: change the confidence thresholds in the ifs, or retrain the models those likely() calls hit.

The same train/eval rigor you apply to a model, applied to the whole application.

tests/test_welcome.py · runs in CI on every PR

def test_welcome_email():
    email = render_welcome_email(user)
    assert likely(
        "addresses the user by name and contains exactly one call to action",
        email,
    ) > 0.95
    assert not likely(
        "contains placeholder text like TODO or lorem ipsum", email
    ) > 0.3

Then the suite scores the whole app against last week's traffic

$ eval --suite app --against traces/week-32.jsonl
| Controller | Knob | Now | Proposed | Metric |
| --- | --- | --- | --- | --- |
| refunds.auto_approve | p> | 0.90 | **0.74** | escalate 41% → 12% |
| refunds.external_bank | p> | 0.50 | **0.50** | hold |
| feed.spam | p> | 0.80 | **0.80** | hold |
| feed.scam_or_fraud | p> | 0.50 | **finetune** | new pattern in traces |
| agent.block_tool_call | p> | 0.85 | **0.88** | 3 false-allows this week |
| app score0.91 → **0.94** |
| actionwrite knobs.json, queue finetune on feed.scam_or_fraud |

You stop asking “do the tests pass” and start asking “did it get better”.

Are we ready?

Models have to clear four bars. Today:

Intelligent

Yes

Fast

Nope

Used as software controllers, they need tens of milliseconds, not seconds.

Cheap

Nope

Frontier models are still too expensive for this pervasive, high-frequency use.

Honest

Nope

Today's models sound equally sure when right and when wrong. Fine for chat; a disaster for software that branches on confidence.

Which is fine: they were built for chatbots.

Thesis 2

While today’s LLMs are designed above all to talk to humans, a new category of LLM-like architectures, decision models, will emerge to be used at runtime inside software.

A different kind of model

A decision model has to be:

1. Smart

As capable as the models we built for people.

2. Machine-fluent

It speaks software's vocabulary: yes/no, probabilities, estimates, choices, rankings, classes, and code generated at runtime.

3. Fast

Milliseconds per call.

4. Cheap

Cheap enough to call hundreds of times per run.

5. Honest

Every answer carries a confidence calibrated to how often it is right, so the controller can act, escalate or ask a human. Not a number a chat model happens to print.

Objections

The reasonable objections.

Determinism works.

“Models” means anything fit to data, a deterministic decision tree included. Read it as both probabilistic and deterministic controllers.

Models will never be fast enough to generate code at runtime.

Look at how fast tokens per second are rising, and project the curve.

Isn’t this the same as small models? Why a new category?

Decision models will gain from small-model progress. But:

  1. The objective is different. Small models are still optimized for chat. Here you optimize for runtime software. Architecture and inference both change: that’s a different stack. E.g. if you are classifying, you don’t need generation at inference time.
  2. They still have no real confidence, only next-token-prediction theatre.
  3. Why small? In many cases you need large-model-like intelligence.

“One model for everything” is a slogan, not an architecture.

Generating code at runtime is reckless.

  1. Look at how much coding quality improved in 18 months, and project the line.
  2. Generate at runtime only the pieces that benefit from inputs you don’t have at coding time.

English from the outside is an attack surface.

Mitigate it. External input is already dangerous, and software already takes it in.

Smart, fast, cheap, and honest may never arrive together.

Possible. Read this as a direction, not a date.

Intelligence, at runtime