An oral history of Bank Python

November 2021

The strange world of Python, as used by big investment banks

Today will I take you through the keyhole
to look at a group of software systems not well known to the public, which I
call “Bank Python”. Bank Python implementations are effectively proprietary
forks of the entire Python ecosystem which are in use at many (but not all)
of the biggest investment banks. Bank Python differs considerably from the
common, or garden-variety Python that most people know and love (or hate).

Thousands of people work on – or rather, inside – these systems but there is
not a lot about them on the public web. When I’ve tried to explain Bank Python
in conversations people have often dismissed what I’ve said as the ravings of a
swivel-eyed loon. It all just sounds too bonkers.

I will discuss a fictional, amalgamated, imaginary Bank Python system called
“Minerva”. The names of subsystems will be changed and though I’ll try to be
accurate I will have to stylise some details and – of course: I don’t know
every single detail. I might even make the odd mistake. Hopefully I get the
broad strokes.

Barbara, the great key value store

The first thing to know about Minerva is that it is built on a global database
of Python objects.

import barbara

# open a connection to the default database "ring"
db = barbara.open()

# pull out some bond
my_gilt = db["/Instruments/UKGILT201510yZXhhbXBsZQ=="]

# calculate the current value of the bond (according to
# the bank's modellers)
current_value: float = my_gilt.value()

Barbara is a simple key value store with a hierarchical key space. It’s
brutally simple: made just from
pickle and
zip.

Barbara has multiple “rings”, or namespaces, but the default ring is more or
less a single, global, object database for the entire bank. From the default
ring you can pull out trade data, instrument data (as above), market data and
so on. A huge fraction, the majority, of data used day-to-day comes out of
Barbara.

Applications also commonly store their internal state in Barbara – writing
dataclasses straight in and out with only very simple locking and transactions
(if any). There is no filesystem available to Minerva scripts and the little
bits of data that scripts pick up has to be put into Barbara.

Internally, Barbara nodes replicate writes within their rings, a bit like how
Dynamo
and BigTable work.
When you call barbara.open() it connects to the nearest working instance of
the default ring. Within that single instance reads and writes are strongly
consistent. Reads and writes from other instances turn up quickly, but not
straight away. If consistency matters you simply ensure that you are always
connecting to a specific instance – a practice which is discouraged if not
necessary. Barbara is surprisingly robust, probably because it is so simple.
Outright failures are exceptionally rare and degraded states only a little more
common.

Some example paths from the default ring:

Barbara also has some “overlay” features:

# connect to multiple rings: keys are 'overlaid' in order of
# the provided ring names
db = barbara.open("middleoffice;ficc;default")

# get /Etc/Something from the 'middleoffice' ring if it exists there,
# otherwise try 'ficc' and finally the default ring
some_obj = db["/Etc/Something"]

You can list rings in a stack and then each read will try the first ring, and
then, if the key is absent there, it will try the second ring, then the third
and so on. Writes can either always go to the first ring or to the uppermost
ring where that key already exists (determined by configuration that I have not
shown).

There are some good reasons not to use Barbara. If your dataset is large it
may be a good idea to look elsewhere – perhaps a traditional SQL database or
kdb+. The soft limit on (compressed)
Barbara object sizes is about 16MB. Zipped pickles are pretty small already so
this is actually quite a large size. Barbara does feature secondary indices
on object attributes but if secondary indices are a very important part of
your program, it is also a good idea to look elsewhere.

Dagger, a directed, acyclic graph of financial instruments

One important thing that investment banks do is estimate the value of financial
instruments – “asset pricing”. For example a bond is valued as all the money
that you’ll get for owning it, discounted a bit for the danger of the issuer of
the bond going bust. Bonds are probably (conceptually!) the simplest
instrument going and of much greater interest is the valuation of other,
“derivative”, financial instruments, such as credit default swaps, interest
rate swaps, and synthetic versions of real instruments. These are all based on
an “underlying” instrument but pay out differently somehow.

The specifics of how derivatives are valued does not matter, except to say that
there are both a lot of specifics and a lot of derivatives. The dependencies
between instruments forms a directed, acyclic graph. An example hierarchy for
some derivative financial instruments might look like this:

Dagger is a subsystem in Minerva which serves to help keep these data
dependencies straight. You write a class like so:

class CreditDefaultSwap(Instrument):
    """A credit default swap pays some money when a bond goes into
    default"""

    def __init__(self, bond: Bond):
        super().__init__(underliers=[bond])
        self.bond = bond

    def value(self) -> float:
        # return the (cached) valuation, according to some
        # asset pricing model
        return ...

Dagger tracks the edges in the graph of underlying instruments and
automatically reprices derivatives in Barbara when the value of the underlying
instruments changes. If some bad news about a company is published and a
credit agency downgrades their credit rating then someone in bonds will update
the relevant Bond object via Dagger and Dagger will automatically revalue
everything that is affected. That might mean hundreds of other derivative
instruments. Credit downgrades can be rather exciting.

Individual instruments are composed into positions. The Position class looks a
bit like this:

class Position:
    """A position is an instrument and how many of it"""
    def __init__(self, inst: Instrument, quantity: float):
        self.inst = inst
        self.quantity = quantity

    def value(self) -> float:
        # return the (cached) valuation, which basically is
        # self.inst.value() * self.quantity
        return ...

Again, note that a position is something you can also value. It is also
something whose value changes when the value of things it contains changes. It
it also automatically revalued by Dagger.

And a set of positions is called a “book” which is an immensely overloaded word
in finance but in this context is just a set of positions:

class Book:
    """A book is a set of positions"""
    def __init__(self, contents: Set[Valuable]):
        # the type Valuable is a "protocol" in python terms,
        # or an "interface" in java terms - anything
        # with value()
        self.contents = contents

    def value(self) -> float:
        # again, return the (cached) valuation, which is more
        # or less: sum(p.value() for p in self.contents)
        return ...

Books can contain other books. There is a hierarchy of nested books all the
way up the bank from the smallest bond desk to a single book for the entire
bank. To value the bank you would execute:

# this is the top level book for the whole bank which
# recursively contains everything else in the whole bank
bank = db["/Books/BigBankPlc"]

# this prints the valuation of the whole bank
print(bank.value())

That’s the dream anyway. In reality the CFO probably uses a different system
to generate the accounts. Valuations of subsidiary books are still well used
though.

If you understand excel you will be starting to recognise similarities. In
Excel, spreadsheets cells are also updated based on their dependencies, also as
a directed acyclic graph. Dagger allows people to put their Excel-style
modelling calculations into Python, write tests for them, control their
versioning without having to mess around with files like CDS-OF-CDS EURO DESK
20180103 Final (final) (2).xlsx. Dagger is a key technology to get financial
models out of Excel, into a programming language and under tests and version
control.

Dagger doesn’t just handle valuations. It also handles the various “risk
metrics” that banks use to try to keep a handle on how exposed they are to
various bad things that might happen. For example, Dagger makes it relatively
easy to find all positions on, say, Compu-Global-Hyper-Mega-Net Plc, which is
rumoured to be going bust. That’s counting all options, futures, credit
instruments and all of it “netted out” to find the complete position on that
company for the whole bank. Never again be surprised by your exposure to dodgy
subprime lenders!

Walpole, a bank-wide job runner

I’ve said so far that a lot of data is stored in Barbara. Time to drop a bit
of a bombshell: the source code is in Barbara too, not on disk. Remain
composed. It’s kept in a special Barbara ring called sourcecode.

Not keeping the source code on the filesystem breaks a lot of assumptions. How
does such a program run? The answer is Walpole, the bankwide job runner.
Walpole is a general purpose runner of jobs, like a mega Jenkins combined with
a mega systemd.

As with many things in Minerva, Walpole is not deployed per-team: there is but
one, single, bankwide instance. Walpole is suitable for both long
lived-services as well as periodic jobs and is even used for builds. Periodic
jobs come up a lot in banks: there are many, many, many end of day or weekly
jobs to run to update data, check things, send email digests, etc.

Walpole does all the usual stuff you need to run your software. It can restart
your software if it crashes and sends out alerts if it keeps crashing. It
stores logs. It understands dependencies between jobs (much like systemd does)
so if the job that generates the data your job needs fails, you job doesn’t
even try starting up but instead fires more alerts.

One real advantage is that Walpole considerably lowers the bar for getting your
stuff deployed. Anyone can put a job into Walpole – you need only a small
ini-style config file explaining what time to run your script, where your main
function is and your entire application is deployed with no further
negotiation.

This is a big deal because negotiating anything in large bank is an exercise in
frustration: lead times on hardware can be measured in months. Getting people
to agree with you takes of course much longer than that.

One of the great drawbacks of “Cloud Native Computing” as it now exists is that
it’s really, really complicated. It is often more complicated than the old,
non-cloud, sort of computing. In order to deploy your app outside of Minerva
you now need to know something about k8s, or Cloud Formation, or Terraform.
This is a skillset so distinct from that of a normal programmer (let alone a
financial modeller) that there is no overlap. Conversely, anyone can work out
an ini-file.

MnTable, the ubiquitous table library

I always feel that it’s a shame that programming languages rarely, if ever,
come with a built-in table datastructure. Programmers have an unfortunate
tendency to gravitate towards hash tables – particularly in Python and
Javascript where they are used to such extent that it is hard to find anything
which is not made out of hash tables.

Hash tables have some serious drawbacks. First, most implementations are
in-memory only and sit sparsely there, which makes it a pain in the bum to work
even with medium sized data sets; a problem Python programs very commonly run
into in practice. More importantly they require you to know your access
patterns up front and they really had better be by a single primary key.

Tables are the
reverse: they are memory-dense and easy to
spool to and from disk. They can use b-tree indices to allow efficient access
by any route; so you never end up having to invert your dictionary in the
middle of your program just so that you can access by something other than the
key. They can support bulk operations and can make use of lazy evaluation.

In open source land the popular library for this is
pandas but pandas has some serious drawbacks:

1. It did not exist when Minerva was originally implemented
2. It is less efficient than you might hope, particularly with memory
3. It’s not brilliant with datasets larger than memory
4. (Arguably) it has a baroque API

Instead of pandas there is a proprietary table library in Minerva: MnTable.

# make a new table with three columns of the types provided
t1 = mntable.Table([('counterparty', str),
                    ('instrument', str),
                    ('quantity', float)])

# put some stuff in the table (in place, tables are
# immutable by default)
t1.extend(
    [
        ['Cleon Partners', 'xlon:voda', 1200.0],
        ['Cleon Partners', 'xlon:spd', 1200.0],
        ['Blackpebble', 'xlon:voda', 1200.0],
    ],
    in_place=True)

# return a new table (without changing the original)
# that only includes vodafone.  this is lazy and
# won't get evaluated until you look at it
t1.restrict(instrument='xlon:voda')

MnTable gets used everywhere in Bank Python. Some implementations are lumps
of C++ (not atypical of financial software) and some are thin veneers over
sqlite3. There are many, many programs which start with an MnTable, apply some
list of operations to it and then forward the resulting table somewhere else.

This is convenient as data is everywhere in banks and most of it is “medium”
sized: in the gigabytes range. A lot is talked about high-frequency traders
but the majority of financiers are not looking at tick level or frankly even
intra-day level data. “Medium-sized” is big enough that you cannot create an
object for every row but not so big that you are going to need some distributed
compute cluster thingy.

A measure of the pain

It would be wrong to imply that working with any financial software is pure and
untrammelled joy. Minerva is no different.

New starters take an exceptionally long time to get up to speed – and that’s if
they don’t resign in fit of pique as soon as they see the special, mandatory,
in-house IDE (as I nearly did). Even months in, new starters are still
learning quite fundamental new things: there is a lot that is different.

Over time the divergence between Bank Python and Open Source Python grows.
Technology churns on both sides, much faster outside than in of course, but
they do not get closer. The rest of the world is not going to adopt any of
Minerva’s ideas, not least because they’ve never heard of them. Minerva is
also not adopting many of the ideas from the outside. There is an uncharitable
view (sometimes expressed internally too) that Minerva as a whole is a grand
exercise in NIH syndrome.

By nature, Minerva is holistic and all encompassing. That’s great if you’re
inside but if you’re outside, interacting with Minerva is a pain. Occasionally
a non-Minerva developer would ask me how he might read some specific piece of
data out of Barbara. I would tell him that the best way would be to use the
Minerva source code to do that. Ok, he would reply, maybe he could get away
with adding a Python script to a cronjob to do that – could I help him get the
code? That’s easy, I would reply: just read it out of Barbara.

I can just about understand why Minerva has its own IDE – no other IDEs work if
you keep your source files in a giant global database. What I can’t understand
is why it contains its own web framework. Investment banks have a one-way
approach to open source software: (some of) it can come in, but none of it can
go out. The github profiles of the bulge bracket investment banks are anaemic
compared to those of comparably sized companies in different industries. This
highly proprietary attitude has remained even as the Volcker
Rule has forced nearly all of the
proprietary trading out of investment banks. It is a curse.

It could be that the biggest disadvantage is professional. Every year you
spend in the Minerva monoculture the skills you need interact with normal
software atrophy. By the time I left I had pretty much forgotten how to
wrestle pip and virtualenv into shape (essential skills for normal Python).
When everything is in the same repo and all code is just an import away,
software packaging just does not not come up.

What makes it different

I haven’t covered everything that’s in a typical Bank Python implementation.
For example, I’ve skipped over things like:

• the proprietary timeseries data-structure
• the “vouch” system for getting your changes into prod
• time travel in Dagger
• the semi-bespoke (non-git) version control system
• the Prolog-based permission system
• replay-oriented financial message buses
• existential ennui arising from prolonged exposure to Windows 7 and MS Outlook
  2010

You’ll just have to use your imagination.

That said, I hope that I’ve given a view of the most important central parts:
Barbara, Dagger, Walpole and MnTable. Of those four subsystems, three pertain
to data. (The other can be seen as a database of jobs.)

One of the slightly odd things about Minerva is that a lot of it is
“data-first”, rather than “code-first”. This is odd because the majority of
software engineering is the reverse. For example, in object oriented design
the aim is to organise the program around “classes”, which are coherent
groupings of behaviour (ie: code), the data is often simply along for the
ride. Writing programs with MnTable is different: you group the data into
tables and then the code lives separately. These two lenses for organising
computations are at the heart of the object relational impedance mismatch which
has caused such grief. The force is out of balance: many
more programmers can design decent object-oriented classes than can bring a set
of tables into third normal form. This is a large part of the reason that that
annoying impedance mismatch keeps coming up.

The other unusual thing about Minerva is that it opts, in many cases, to have
one big something rather than many small somethings. One big codebase. One
big database. One big job runner. Clubbing it all together removes a lot of
accidental complexity: you already have a language runtime (and the version in
prod is the same as on your computer), a basic database and a place for your
code to run before you even start. That means it’s possible to sit down, write
a script and get it running in prod within the hour, which is a big deal.

Minerva is obviously heavily influenced by the technological path dependency of
the financial sector, which is another way of saying: there is a lot of MS
Excel. Any new software solution is going to be compared with MS Excel and if
the result is unfavourable people will often just use continue to use Excel
instead. Many, many technologists have taken one look at an existing workflow
of spreadsheets, reacted with performative disgust, and proposed the trifecta
of microservices, Kubernetes and something called a “service mesh”.

This kind of Big Enterprise technology however takes away that basic agency of
those Excel users, who no longer understand the business process they run and
now have to negotiate with ludicrous technology
dweebs for each software change.
The previous pliability of the spreadsheets has been completely lost. Using
simple Python functions, in a source controlled system, is a better middle
ground than the modern-day equivalent of J2EE. Financiers are able to learn
Python, and while they may never be amazing at it they can contribute to a much
higher level and even make their own changes and get them deployed.

Crib ideas from existing systems

One thing I regret about software as a field is how little time is spent
learning from existing systems and judging what they did well, or badly. There
are only a small number of books
discussing, in detail, real systems that exist.

Even when the public details of systems are available they can still be
strangely understudied. Email has been around a long time: it predates the
internet by a decade. And in that time it has not changed enormously fast and
is still mostly the same as it was in the
80s. Despite that, a lot of
programmers are still a hazy about what happens when you click “send”. Some of
them, I’m sure, will keep trying to “disrupt” email regardless.

This is a shame as foreign systems, like foreign countries, can be mind
expanding when experienced firsthand. Their customs can differ so enormously
from yours that it can lead you to rethink your own practices. But when you
just hear it second hand, it can sound like nonsense.

I once described Minerva’s “vouch” system, briefly, to another programmer who
had never seen it. I explained that when you had a code change, you just had
to convince any one of the code owners for the file in question to sign it off.
If the change was very urgent, they might sign off your change sight unseen,
based on your reputation alone. As soon as they clicked that “vouch” button –
bang – your new change was in prod: after all, there is no such thing as a
deployment step when your code is stored in a database. Disbelieving me, he
asked who in the world would trust such a bank. The answer is a lot of people.
They are a very big bank. You have certainly heard of them.

Contact/etc

Other notes

If you’re curious to try an MnTable-style table library, my friend Sal released
a pure-python, API compatible, version called
eztable.

I’ve mentioned that programmers are far too dismissive of MS Excel. You can
achieve a awful lot with Excel: more, even, than some programmers can achieve
without it. There exist trading systems in “tier one” investment banks where
the way that trades are executed is by clicking on special cells in certain
special xlsx files.

Even I would accept that that is too far but if you don’t already know Excel it
is one of the highest value things you can learn. For programmers the best way
to find out what you are missing is Joel Spolsky’s overview
talk, aimed directly at
programmers. If you decide to take the red pill after that, I’m told that
Coursera’s Excel Skills for Business
Specialisation is excellent.

One of things that tends to boggle programmer brains is while most software
dealing with money uses multiple-precision
numbers to make
sure the pennies are accurate, financial modelling uses
floats
instead. This is because clients generally do not ring up about pennies.

I’ve mentioned Barbara overlays. They also work for source code. You can tell
Walpole to mount your own ring in front of sourcecode when it’s importing
code for a job and then you can push source files to that instead of getting
them vouched into sourcecode. All manner of crazy, bananas, tutti frutti
hacks lie down this dark path. Do it, but only a little.