This text is about Deterministic and Probabilistic Software Development using monkeys as an example
A lot of people believe: the future of software development is going towards autonomous development, and the Future IT Revenues are not coming from hiring 100's of Indian devs and upselling them to the USA with locking margin on your Seichelles account, but from paying for electricity and running self-hosted coders, which will build features for another nicely pivoted AI-first startup.
As for us, peasants, who own <800GB of VRAM - we can only gossip about bigger open-source models like DeepSeek V3.2 (which is actually worse in SWE-Bench Pro, but a bit better in Terminal-Bench 2.0) and try to run smaller models on available compute with a power of statistical probability, build the features not from the 3rd turn with šOpusš (only for a dramatic note, I like Opus), but with 4th turn based on āļøQwen Coder Nextāļø
Why am I so happy about it? You see... If you believe that one generative model is better than others, you're not fully understanding the stakes we're playing in the World of Software now.
Deterministic Software Development
For the last 70 years, as people use and deploy applications on digital computers, we have always followed the unhindered guidelines:
In school/uni/illegal concentration camp, for the first time, they explain to you the logical constructions OR & AND, they were telling:
You can write your code on paper and try to compile it in your head. See the logic, and if it works there, you can try to run it on a machine, and it will work (if you're smart enough)
But times are changing, nothing is permanent (except Kobol developers). Now we're already facing a transition in the way digital interfaces are being created.
Probabilistic Software Development
What happens if you give a monkey a computer and three bananas? Itāll spend two bananas on a Claude Pro subscription and wait until the limits reset at 2 PM.
What happens if you give a monkey a computer and an email account? Itāll come up with KPIs and shareholder value. And then will wait for the GPT Plus limit update.
In any case, the code will be written/generated based on the business hypothesis or the research teamās findings. Each of the monkeys tries to understand the business requirements and produce some amount of code.
In such a system, the only bottleneck is the monkey that cannot parallelize its tasks using multi-level agentic codingāit is precisely this monkey and its attempts to understand business and figure out the correct (always) code that constitute the greatest flaw in modern software development pipelines. Still, any monkey will always be smarter than any management system...
Letās solve a simple problem: 1 monkey can complete 1 task in 6 hours (or it might not)
Letās assume the probability of success is 0.90 (a very smart monkey)
Yes, 3 monkeys will give us a 0.999 success rate. Probability is only theoretical on small amounts of observations, but in the case of software development, we have ... 20kk Python devs? Sounds about right.
But what is the success rate? Depends on the scale and perspective.
From the monkey's perspective, "any task is successful if I get a banana". For business - "application is delivered, and users give bananas."
There is no truth but relatively to the software - we could state the object of deliverables as an application which a) works b) satisfies the expected criteria (which were translated from stakeholders to devs without any direct communication)
And here we goooooooooooooo! Truth is - dev monkey shouldn't care about business goals and focus on code. Otherwise, the code is never going to be shipped. Cause business is turbulent, and software development takes time. And to save some time, our monkey decided to...
Get a digital monkey to help with some coding. And there are two possible outcomes here.
P1 - Many monkeys will each use one digital monkey, which will level the playing field and ensure all monkeys have equal capabilities.
P2 - One monkey (the top performer) will use more digital monkeys, thereby increasing the amount of code produced (stakeholders will be satisfied) and achieving a certain probability of obtaining a ready-made solution directly from the model on the first try.
To simplify the calculations, we will use the results from the Terminal Hard bench, as it is best suited for describing the digital monkey.
Great! If our little monkey uses 5 instances of Opus, it will get a ready-made solution on the first try with almost the same probability as if it were working on its own!
However, it will save a significant amount of time. Currently, according to various estimates (mostly random), Opus can complete tasks that would take several hours in just a few minutes.
Therefore, by sacrificing 2% accuracy, weāve saved a potential two days of development (16 hours), which saves us the number of bananas we owe the developer monkey for the hours of work.
Alternatively, this benefits us by allowing more working apps per unit of time.
What's the downside? The amount of bananas we should give to the Model Provider based on usage.
And if we have downslides and upsides...
I don't want to spend your time on math. Ask an agent to do that for you. In most cases (after 5 instances), you'll have almost constant efficient results (theoretically) since our error will be in ^5. But we're not optimising error - we're optimising Price/App
āProbability of Success for Models under Budget Constraints with Correlated Errorsā
| Budget | Opus | Groq | Sobaka | Split |
|---|---|---|---|---|
| 5 | 66.6% | 72.6% | 97.8% | 99.3% |
| 10 | 83.5% | 89.7% | 99.96% | 99.97% |
| 20 | 92.6% | 97.0% | 99.9999% | 99.9999% |
| 50 | 98.6% | 99.95% | 99.999999% | 99.999999% |
Success per cost šø
That's what I call Probabilistic Software Development.
As you could see... if you have a limited budget but unlimited monkeys - never go for the biggest model.
If you have a limited budget and limited monkeys: buy interference from Sobaka and save bananas šŗ
