Why We Almost Used Microsoft Orleans for Trial Balance — And What We Learned When We Didn’t

--

— A real exploration inside a financial product, where the obvious solution wasn’t the right one.

Press enter or click to view image in full size

Every time a single account changed, we recalculated everything. The entire trial balance. Every total. Every rollup. And our database had no idea what hit it.

This is a story about a financial product, a promising distributed systems framework, and the three weeks I spent figuring out why the obvious solution wasn’t the right one.

I want to be honest upfront: Orleans didn’t work for our use case. But the exploration taught me more about distributed state, object modeling, and the hidden cost of “elegant” abstractions than any greenfield project ever has.

1 The Problem: A Trial Balance That Recalculates Everything, Always

A trial balance lists every account in a company’s general ledger — assets, liabilities, equity, revenue, expenses — with their debit and credit balances. Every account feeds into totals. Every total feeds into higher-level summaries. It’s a deeply hierarchical, deeply interconnected data structure.

In our product, clients varied wildly. Some had a few hundred accounts. Enterprise clients had tens of thousands — with deeply nested sub-accounts, multiple currencies, and custom groupings layered on top.

The trigger was simple: one account changes → recalculate the whole thing.

That sounds fine until you realise what “recalculate” actually means at scale:

• Re-aggregate totals across every parent group
• Re-evaluate every formula referencing the changed account
• Re-validate cross-account rules (debits must equal credits)
• Persist the updated state — for every affected row

Multiply that by hundreds of concurrent users each editing different accounts simultaneously. Our database was not having a good time.

// What happened on every single account update

User edits Account #4821  (one field change)
           │
           ▼
Trigger: recalculate trial balance
           │
           ▼
SELECT * FROM accounts WHERE trial_balance_id = ?   ← full table read
SELECT * FROM account_line_items WHERE account_id IN (...)
SELECT * FROM groupings WHERE ...
           │
           ▼
Recalculate in application layer
           │
           ▼
UPDATE totals SET ... WHERE ...          ← N writes back
UPDATE account_summaries SET ...
UPDATE trial_balance SET last_calculated = NOW()// Meanwhile: 3 other users also just edited accounts.
// All 4 recalculations running simultaneously.
// All 4 hammering the same rows.

DB CPU spiking every save. Lock contention when two users edited the same trial balance. Recalculations stepping on each other, sometimes producing inconsistent totals. We needed a better model.

2 Why Orleans Looked Like the Perfect Answer

Someone on the team said it in a late-night architecture debate: “What if each account was its own grain?”

Orleans’ virtual actor model felt tailor-made for this:

❌ Our Problem

State scattered across DB tables

Every update triggers full recalculation

No isolation between concurrent edits

DB is the bottleneck for everything

✅ Orleans Promise

Each entity owns its own state in memory

Grains notify only their dependents

Single-threaded grain = no concurrency conflicts

State in memory, DB is just the backup

The mental model was clean: one grain per account. The account grain holds its fields, its line items, its subtotals. When it changes, it notifies its parent group grain. The parent recalculates only what changed. No full table scans. No lock contention. Beautiful.

Why Orleans Is Genuinely ExcitingOrleans makes distributed, stateful objects feel like local in-memory objects. You call a grain by ID. Orleans finds it in the cluster, activates it if needed, and routes your call. The complexity of distribution disappears behind a clean interface. For the right problem, it’s genuinely magical.

3 The Moment We Opened the Actual Account Model

We were excited. Then we looked at the actual data model.

An account in our system wasn’t a simple object. It was a deeply nested structure — many fields, and under each field, lists. Lists of line items. Lists of adjustments. Lists of mapped codes. Lists of custom attributes per client configuration.

// Simplified — real structure was far more complex

Account
  ├── id, name, code, type, currency
  ├── balances[]          ← list: one per period
  │     ├── period, debit, credit, adjusted
  │     └── adjustments[]  ← nested list per balance
  ├── lineItems[]         ← can be hundreds per account
  │     ├── description, amount, reference
  │     └── mappings[]     ← nested list per line item
  ├── groupMemberships[]  ← rollup groups this belongs to
  ├── customAttributes{}  ← varies per client schema
  └── auditTrail[]        ← append-only history log// One account grain = all of this in memory, per account.
// Enterprise client: 50,000 accounts × all of the above.
// = Gigabytes of grain state for a single tenant.

We started asking uncomfortable questions.

Does a lineItem get its own grain? It has its own fields and its own nested lists. But Orleans itself warns against making grains too fine-grained — activation overhead and message-passing costs add up fast.

And if we put everything inside the account grain, we’re loading hundreds of nested objects into memory for every activation — even for a single field change.

The Anti-Pattern We Were About to CreateOrleans explicitly discourages deeply nested grain hierarchies and overly chatty inter-grain calls. If every lineItem, every adjustment, every mapping becomes its own grain — you get thousands of tiny objects firing messages at each other per save. The framework that was supposed to simplify things becomes the hardest part of your system.

4 The Grain Boundary Problem — Where the Model Breaks

This is the core of why Orleans didn’t fit.

Orleans works beautifully when your entities have natural, stable grain boundaries. A user. An order. A game session. One grain = one clearly scoped thing with bounded, predictable state.

A trial balance account in financial software is the opposite of that. Its boundaries shift per client. Its nested lists are unbounded. Its relationships are complex and bidirectional.

ConsiderationOrleans ExpectationOur RealityState size per entitySmall and boundedUnbounded — varies per clientGrain boundariesClear and naturalAmbiguous — account vs lineItem vs mappingActivation costLow for simple stateHigh — loading nested lists on every activationGrain-to-grain callsFine in moderationWould be thousands per recalculationMemory predictabilityPredictable per grainWildly different per tenant

The more we mapped our domain onto Orleans, the more it felt like we were bending our problem to fit the framework — rather than the framework fitting the problem.

That’s always the warning sign.

A framework should make your problem easier to express, not force you to redesign your domain around its model. When you find yourself asking “how do I make this work with X” instead of “does X fit this” — that’s the moment to step back.

5 What We Did Instead: In-Memory State With Smart Invalidation

We stepped back and asked the real question: what is the actual problem we’re solving?

Not “how do we use Orleans.” But: why is the DB getting hammered, and what’s the minimum change that fixes it?

The answer was simpler than we expected. We didn’t need every account to be its own distributed actor. We needed the trial balance — as a whole unit — to live in memory per active session, with smart dirty-tracking so we only write back what actually changed.

// The approach that actually worked

// 1. Load once on open — cache entire trial balance in memory
TrialBalanceCache[clientId + tbId]
  = await LoadFullTrialBalance(clientId, tbId);// 2. On account update — mutate in-memory only
cache.GetAccount(accountId).Update(changedFields);
cache.MarkDirty(accountId);         ← track only what changed
cache.RecalculateTotals();           ← pure in-memory, ~0ms// 3. Write-back — only dirty accounts, debounced
DirtyWriteBackJob
  → fires every 2s (or on explicit save)
  → writes ONLY changed accounts to DB
  → clears dirty flags after success// Result:
// DB hits:        N writes per keystroke  →  1 batch per 2s
// Recalculation:  DB round-trip (~200ms)  →  in-memory (~0ms)
// Concurrency:    DB lock contention      →  handled in memory

Recalculation became instant — pure in-memory math, no DB round-trip. Writes became batched. Lock contention disappeared because concurrent edits now collide in memory, where we can handle them far more gracefully than at the database layer.

No actor framework. No grain boundaries to debate. A well-scoped cache, dirty tracking, and a sensible write-back strategy.

6 When Orleans Actually Is the Right Tool

I want to be clear: Orleans is a genuinely impressive framework. This wasn’t a verdict on Orleans — it was a verdict on fit.

Here’s my honest mental model for when Orleans shines:

• Natural, stable entity identity — a user, a game session, an IoT device, a chat room
• Bounded, predictable state per entity — you can reason about memory per grain upfront
• Entities need autonomous behaviour — self-triggered timers, reminders, lifecycle logic
• Massive fan-out — millions of independent entities each doing small, isolated work
• You’re in the .NET ecosystem — the integration story is genuinely excellent

For financial domain objects with unbounded nested structures, client-variable schemas, and tightly coupled recalculation logic — it adds a layer of complexity without proportional benefit.

7 The Real Lesson From Three Weeks of Exploration

The most valuable output wasn’t a working Orleans implementation. It was clarity.

We came out knowing exactly why our original approach was failing, exactly what properties the right solution needed, and exactly why a simpler in-memory model solved the actual problem without introducing a new layer of distributed systems complexity.

That clarity only came because we genuinely explored the alternative. Not dismissed it. Not blindly adopted it. Actually built the POC, hit the walls, understood why.

That is what senior engineering looks like. Not always picking the right tool on the first try — but being rigorous enough to know why something doesn’t fit, and confident enough to say it out loud.

We didn’t ship Orleans. We shipped something boring, fast, and solid.
The DB stopped screaming. Recalculations became instant.
And I walked away understanding Orleans better than most people who actually use it.

Sometimes the best engineering decision is the one you decided not to make.

If this resonated, follow for more ❤️

Real engineering stories from real products — not tutorials, not theory.

#Orleans#SystemDesign#Fintech#DistributedSystems#DotNet