Back

Case Study

How an AI Agent Tracks Our Shipped Features at Kiwi Money

We shipped a feature. Instead of checking dashboards for weeks, our AI agent tracked it automatically — and found insights we would have missed.

5 min read

Shipping Fast, Measuring Slow

Our biggest problem was never shipping fast. We're two founders — we ship every week.

The problem was measuring what we shipped. We had no idea if the things we built actually worked. Features would go out, we'd move on to the next thing, and nobody went back to check.

When we did pause to measure, it was painful. Exporting data from Mixpanel, pulling subscription numbers from RevenueCat, pasting it into Claude, re-prompting until the analysis made sense — it would take us a legit full day to get to a conclusion. Time we could've spent building and talking to users.

So most of the time, we just didn't measure. We shipped and hoped.

That was the real mistake. Not shipping slow — flying blind on what we shipped.

The Backstory

Kiwi Money is a personal finance app for Sri Lanka. It automatically tracks expenses by reading bank SMS messages — your bank sends an SMS, Kiwi Money picks it up and categorizes it.

In February, our AI agent Pool ran a baseline health report. The findings were brutal:

  • 100% of free users missed the wow moment — they landed on a blank dashboard with no setup guidance at all
  • Even among trial users, 41% dropped off before seeing their transactions appear automatically
  • Free users had zero reason to upgrade because they never experienced what the app actually does

So we built a fix: a Dashboard Onboarding Setup flow that guides free users through granting SMS permission, importing messages, and seeing their transactions appear automatically — the “wow moment.”

We shipped it March 7. Then we told Pool to track it.

What the Agent Does

We sent one message in Discord:

“We shipped dashboard onboarding setup for free Android users. It guides them through SMS permission → import → wow moment. Track it.”

Pool automatically:

  • Defined the funnel (every step from setup viewed → setup completed)
  • Set the baseline (226 weekly active free users, 33 trial starts/week)
  • Set success criteria and a 4-week tracking schedule
  • Posted it to Trello so the whole team can see

No dashboards configured. No Mixpanel queries written. No 2-day measurement marathons. Pool figured out what to track from what we told it.

What It Found

Every Monday, Pool pulls data from Mixpanel and RevenueCat, compares to the previous week, and posts findings to Discord. After 2 weeks:

The good news: 68% of users who see the setup flow reach the wow moment — seeing their transactions appear automatically. Consistent both weeks.

The surprise: Zero in-flow conversions. Nobody subscribed during the setup flow itself.

Pool didn't stop there. It asked itself: “If nobody converts in-flow but trials are happening, where are they actually converting?”

It investigated and found:

  • 6 users converted to paid — but not during the flow. They converted minutes to hours later through other paywalls.
  • All 6 had rejected the paywall during onboarding. The wow moment changed their minds later.
  • Two patterns: Fast converters (3-7 min) went exploring and hit a paywall immediately. Slow converters (14-46 hrs) needed multiple paywall exposures over hours.

Pool's recommendation: Add a paywall right after SMS import completes — when motivation is highest.

What Shows Up in Discord

Every Monday, without asking:

📊 Feature Tracker: Dashboard Onboarding Setup — Week 2

• Setup viewed: 55 users

• SMS permission granted: 41 (75%)

• SMS import completed: 40 (73%)

• Setup completed: 39 (71%)

• In-flow subscriptions: 0

Wow moment rate: 68% (consistent with Week 1)

⚠️ Zero in-flow conversions. Users convert later via Explore paywall (fast) or Dashboard paywall (slow).

💡 Add a paywall at peak motivation — right after sms_import_completed.

What We Would Have Missed

Without automated tracking:

  1. The zero in-flow conversion problem — we'd have assumed the flow worked because overall trial starts looked fine
  2. The “second-chance” insight — all 6 converters had initially rejected the onboarding paywall. The setup flow gave them a reason to reconsider.
  3. The two conversion personas — impulse buyers vs deliberators need different paywall strategies

These are the kinds of insights that take 2 days of manual analysis to find. Pool found them on autopilot.

The Takeaway

AI agents aren't just chatbots you ask questions to. They do ongoing work — tracking, analyzing, reporting on a schedule, and surfacing insights you wouldn't think to look for.

For a 2-person team shipping every week:

  • Nothing falls through the cracks — every ship gets measured
  • Data comes to us — no more 1-day export-and-prompt marathons
  • Deeper insights — the agent asks follow-up questions humans forget to ask

We ship features. Pool tells us if they worked.

Kiwi Money is Sri Lanka's first automated money manager. Pool is their AI product analyst, built on Claude and connected to Mixpanel and RevenueCat.