Back to insights

The Code Took Two Days. Everything After That Was the Job.

Roy LevyRoy Levy14 min readProduct & delivery
The build used to be the project. Now it's the first step of a loop.

The code took two days. A real merchant found what it was missing in one.

On November 2 I made the first commit on a Shopify app for butchers and fish shops. It lets them charge for the actual weight of a cut instead of the estimate. On November 3 the App Store listing was drafted and submitted.

Two weeks later, a butcher shop found it on the App Store and installed it on a free trial.

The next afternoon, my app refunded $20.10 on one of their orders.

At that moment, Shopify showed $20.07 still due on the same order.

The merchant had edited the order after checkout. My app looked at the weights, worked out the difference, and refunded it. It never asked Shopify whether the order already had a balance of its own.

Here's the uncomfortable part: I still don't know whether $20.10 was the right number.

If the edit had nothing to do with weight, the refund was fine and the customer still owes the $20.07. If the edit already covered the weights, that $20.10 refund shouldn't have happened at all. Only the merchant knows what the edit changed. My app moved money without knowing.

So what was I supposed to have done differently?

Not more documentation. I'd read the refund docs.

The part AI made cheap

A few years ago, a v1 like this was weeks of work. Read the API docs. Write the boilerplate. Wire up auth, billing and webhooks. Most of a solo build went there.

That part now takes days. I wrote this app with AI coding agents, and the build stopped being the bottleneck.

What didn't get cheaper is everything the docs can't tell you.

Docs tell you what fields an API accepts. They don't tell you that a butcher will edit an order, merge a fulfillment and weigh three cuts before collecting the edit's balance. No amount of reading in advance finds that. A real shop on a busy day does.

So the job moved. It's less "write the code" and more "run the loops around the code".

The job now

Four loops around a two-day build.

  • 01

    Before code

    Is anyone feeling this?

  • 02

    After launch

    What are they actually doing?

  • 03

    Correctness

    What does production know that I don't?

  • 04

    Memory

    Does the next app start ahead?

Loop 1: Before code, is anyone feeling this?

My app ideas have come from three places. The third one surprised me.

Someone tells you. My first marketplace app started with a conversation in late 2025, when AI was just getting good at code. The founder of a well-known Shopify upsell app told me someone at Wix had asked him to build a checkout-links app, and he didn't have the capacity. So I built it. Then I mostly forgot about it for a year, until Wix updated its platform and the app stopped working. I rebuilt it from scratch, and that's when I noticed how fast building had become.

The market tells you. I remembered Wix had a page listing the features and apps merchants were asking for. I went back to it and built three apps from that list: address checks for delivery, deposits, and checkout fees. That's the fast kind of research. Read what people already asked for: forums, community threads, feature-request pages, reviews of the apps that already exist. A day or two of reading, then the smallest thing that answers it.

Your own data tells you. Every one of my apps reports to one gateway: installs, setups, usage, orders. So I asked a coding agent to look through that data and suggest a new app. It came back with Package Tiers, an idea I wouldn't have had, and wrote the email to the store that fit it best. That store installed it, reviewed it and my checkout-links app, and paid for a yearly plan right away.

The weight app was an AI's suggestion too. By then the tools I asked could see everything we'd collected about these marketplaces, their merchants and what they pay for.

That's when the loops started feeding each other. The data and lessons from the last app became the research for the next one.

Talking to people is still the best signal, and the slowest. For the weight app, I emailed 25 butchers, fish shops and cheese shops, each time quoting their own product page ("final price based on actual weight"). Most never answered. The kind ones said they already had a solution.

That told me something too. There was already a paid app for it at $19 a month, with no free plan. The problem was real and people paid to solve it. The gap was a free tier for small shops.

Where you launch decides how fast you learn

This is the part I underrated.

In a marketplace (Shopify, Wix, monday.com), people are already searching for the job your app does. A listing can bring real installs within days.

None of those 25 emails led to an install. The first real install came from the App Store listing, the day after I sent them.

That's one launch, so I won't call it a law. But it matches how I see it: a standalone SaaS usually starts with nobody searching for it, and you either wait for SEO or pay for visitors before there's usage to learn from.

If you want to learn fast, a marketplace is the shortest path I've found.

Loop 2: After launch, what are they actually doing?

Shipping isn't the finish line. It's where you start getting data you can't get any other way.

I also run a few Wix apps. One of them creates shareable checkout links. Seven weeks after a big release, the funnel looked like this:

Checkout links, real installs since the last big release

The leak was install → first link.

  • Installed

    22

  • Still installed after 2 days

    14

  • Made a link

    5

  • Got orders through a link

    1

The obvious plan was more features and some cross-promotion between my apps.

The data said nobody was reaching the first feature.

Everyone who made a link did it within about an hour of installing. Stores that stayed but never made a link mostly had big catalogs. My first guess was "service businesses with no products". Wrong. They all had products.

The funnel told me where to look. It couldn't tell me why.

So I seeded a test site with a thousand products and tried it. On stores with 37 or more products on Wix's newer catalog, the app couldn't load products at all. 34 worked. 37 failed with "Couldn't load your products".

No market research would have told me that. No feature would have fixed it. It shipped the same day.

Pricing works the same way. On another app, one store clicked upgrade on day one and didn't buy. About 22 hours after a 30%-off sale email, they bought the yearly plan. One store proves nothing, but I didn't have to guess where they stopped. I could see it.

Loop 3: What does production know that I don't?

Back to the $20.10.

When I rebuilt the order on my test store, Shopify showed me three things I'd never have found in the docs:

A refund for an amount doesn't touch Shopify's balance. On my test order, Shopify showed $16.34 due before my weight refund and $16.34 due after it. The status flipped to "Partially refunded", which made the open balance easy to miss.

An edit that raises a quantity repeats that line's tax lines. My code added up the rates, so the tax on the weight difference came out doubled. On one test order: $7.07 instead of $6.53. I can't tell you whether that was part of the $20.10. I only know it was wrong.

Tax-exempt items can still carry a tax line. In some states, exempt groceries come with a rate and a $0 amount. My code saw the rate and added tax to the weight difference. The merchant reported that one on day one.

The first version of the money logic was roughly this (simplified):

// v1: the weights were the whole story
const difference = weighedTotal - listedTotal;
if (difference < 0) await refund(order, -difference);
if (difference > 0) await sendPaymentLink(order, difference);

It's correct in isolation. That's the problem. It treats the weights as the only thing that ever changes an order.

The fix was to stop deciding money from the weights alone. Now the app asks Shopify for the order's state first and refuses to move money when it's in a state it doesn't fully understand:

// Simplified from the real check. Any reason found means no money moves.
function blocker(order: OrderSnapshot): string | null {
  if (order.outstanding !== 0 && !order.coveredByAuthorization)
    return `Shopify shows ${order.outstanding} still due. Collect it first, then weigh.`;
  if (order.editAddedCustom)
    return "An edit added a custom item or charge. It may already cover the weight.";
  if (order.amountOnlyRefund)
    return "There is already a refund for an amount, not for items.";
  return null;
}

The real version also blocks a balance owed back to the customer, edits Shopify can't explain, cancelled orders and discounted lines.

Every blocker says what to do next in plain shop language. On my test order, the packer sees "Shopify shows $16.34 still due on this order after an edit. Collect it first with Collect payment on the order, then weigh the items."

When the app doesn't know, it stops and says why.

Six releases in one day

The day the merchant started using it for real looked like this:

One day, six versions

Every release came from something real.

  • 01

    v6

    The merchant's day-one reports: tax on exempt items, no weights on payment links, duplicate drafts

  • 02

    v7

    Outside audit: one claim per order, so two windows can't refund twice

  • 03

    v8

    Second-model review: recheck every line right before money moves

  • 04

    v9

    After the $20.10: block edited or unsettled orders

  • 05

    v10

    Weigh settled edits again, fix the doubled tax, 3 gaps from review

  • 06

    v11

    Log every weighing and every block with its reason

Two habits made that pace safe enough.

One model writes, another hunts. The v9 guard was written by one coding model and reviewed by another. The v10 rework went the other way, and the reviewer found three gaps I'd missed: a refund someone had already made by hand, an edited order with no edit history, and an edited order covered only by a card authorization. Each one could have moved the wrong amount.

The test drives the real admin. I can't click through ten scenarios by hand every release, across several apps. So a Playwright suite opens the real order page on a test store, runs the weighing screen, and checks Shopify's books afterwards. The v10 release passed 71 unit tests and 9 of 9 of those end-to-end scenarios, then passed again against the live release itself.

Safe, and still wrong

Look at v9 again.

After the $20.10, I pushed a quick guard that blocked every edited order. No more wrong refunds. Also no more weighing for this merchant, because this merchant edits orders. Every one of them went back to being worked out by hand in Shopify, which is the exact job they'd installed the app to stop doing.

They uninstalled.

My email to them started with "I saw you uninstalled, and I don't blame you."

The money was safe. The customer was gone.

Failing closed was right. Failing closed on everything was just a different bug. A blocker that's too broad never moves the wrong money. It quietly stops the app from doing its job, and the merchant does the work by hand until they decide they don't need you.

v10 narrowed it. Settled edits are weighed again, and only the cases I genuinely can't verify are blocked. v11 logs every weighing and every block with its reason, numbers only, no customer data. Next time a blocker fires too often, I want to see it in the logs before a merchant feels it in their workday.

That's the loop. Ship lean, block only what you can't verify, log every block, and harden the paths real users hit.

Loop 4: The next app starts ahead

Every trap goes into a skill file that my coding agents read before they touch these apps. This is from the real one:

## Money-moving patterns (both apps follow these; keep them)

- Tax: an order edit that raises a quantity **repeats the line's tax lines**
  for the added units. Count each tax once by title + rate, or the tax on
  the difference doubles.
- **Never pick refund vs. charge from the weight math alone.** Weighwise
  blocks while totalOutstandingSet isn't zero (an authorization covering it
  exactly is fine): the merchant settles the balance on the order first.
- A weight refund (custom amount, no items) leaves totalOutstandingSet
  unchanged and flips the status to Partially refunded, which hides a
  balance still due.

The file has one rule about itself: when you learn a durable trap, add it here. A trap saved only in a dated note gets rediscovered the hard way.

My second Shopify app already follows those money rules. It didn't have to learn them from a customer.

Where this approach falls short

I don't want to oversell this.

I shipped money logic that didn't fail closed, then failed closed too hard. "Ship fast and learn" is fine for a settings page. For anything that refunds or charges, the first version should already refuse the states it doesn't understand, and only those. I learned both halves from one merchant, and lost the merchant. You don't have to.

The numbers are small. 22 installs and one upgrade are readings, not statistics. They tell me where to look, not how big the effect is.

Marketplaces own your distribution. They bring you buyers fast, and they also set the rules: review queues, ranking, required security settings, what your app is allowed to touch. Build there with your eyes open.

Some products shouldn't skip the slow loop. Regulated work, or B2B with a long sales cycle, needs real conversations before you build. Forum-mining a weekend won't cover it.

What I'd do this week

If you're a solo builder shipping with AI:

  • Before you build, spend a day reading what people already complain about: product pages, forums, reviews of the existing apps.
  • Log the first-run funnel from day one: install, setup, first real use.
  • For anything that moves money, block only the states you can't verify, say what to do next, and log every block so you see when one fires too often.
  • Rebuild your worst real incident on a test store and make it an end-to-end test.
  • Write every platform trap into a file your agents read before they start.

So what is the job now?

Here's my honest answer.

The code got cheap. Being wrong in production didn't.

A solo engineer used to be paid to turn a spec into working software. Now the software shows up fast, and the work is everything around it: finding out who needs it, watching what they do, catching what the docs never said, and making sure each lesson sticks.

You're not paid to type anymore. You're paid to close the loops.

If you shipped something fast with AI and it now carries real users or real money, this is the work I do when I take an AI prototype to production.


Building alone with AI agents? I'd love to hear which loop is costing you the most right now. Send me a note.

FAQ

How does AI change the job of a solo engineer?

Writing the first version stops being the expensive part. The work moves to four loops around the code: checking that people need it before you build, watching what they do after launch, finding what production knows that the docs don't, and saving every lesson so the next product starts ahead.

Should you ship fast if your app moves real money?

Ship fast, but fail closed on what you can't verify. Anything that refunds, charges or captures should refuse to act when the order is in a state it doesn't fully understand, and tell a human why. Don't block everything that looks unusual: a guard that's too broad stops the app doing its job, and users leave. Log every block so you see when one fires too often.

Are app marketplaces faster to validate than a standalone SaaS?

In my experience, often. A marketplace already has people searching for the job your app does, so a listing can bring real installs within days. My first real Shopify install came from the App Store listing, not from cold outreach. A standalone SaaS usually has to wait for SEO or pay for visitors before it has usage to learn from.

Questions about something you read here, or a project you want to move forward? I work with teams on full-stack builds, AWS and serverless consolidation, migrations, and messy systems.

Contact me