Question is a bit broad, but out of all the concepts under the financial engineering umbrella you're bound to explore the concept of ledgering eventually.
I've written a bit about it on my own co's product blog in an attempt to demystify some core concepts [1], [2], [3].
Still on ledgering and expanding into less mathematical and more applied concepts, I can also recommend a book called "The Accounting Game: basic accounting fresh from the lemonade stand" [4].
## *Trading and Exchanges: Market Microstructure for Practitioners by Larry Harris*
- comprehensive overview written in an accessible way
## *The Microstructure of Financial Markets by Frank de Jong and Barbara Rindi*
- 1st 1/4 of the book is generally useful. Then the math starts. This math is not needed to get a basic overview.
These are two books I wish someone gave to me when I started my first capital markets software engineering job. I recommend them to all the people I place in financial system engineering roles.
This is my passion. Message me if you want to talk more about this — see HN profile for contact info.
I would recommend the Jane Street youtube channel, the ocamal focused ones in my opinion you can skip. But otherwise their engineering videos will give you a good sense of it. When it comes to low latency systems this is a very private ecosystem. There are some open source projects for learning some of the mechanics of trading, market data and the order to settlement lifecycle. I am not aware of any being built for low latency.
Market data ingest, analysis and resulting order execution is chewing through way more data way faster than any banking transaction system. I have worked on both of them.
I work on low(ish) latency trading systems in FX. FIX is the standard communication protocol and familiarity with it is essential for me. Here you can look up the standard message types and tag values:
https://fiximate.fixtrading.org/
They also have docs for the standard message flows you can expect during trading. I use it regularly.
Also work in the FX space and could never find a good FIX specification resource. Always had to rely on whatever the broker shared with us, but it always felt incomplete. This is a super useful resource. Thanks for sharing!
Like someone else said your question is kind of broad. But I d recommend reading TigerBeetle's documentation to understand how financial transaction processing may differ from what they call general purpose databases like PostgreSQL.
There a free online accredited master's degree in Financial Engineering (https://www.wqu.edu/mscfe), probably not for OP but maybe some find it useful for them
A lot of settlement in financial markets is still pretty slow. That’s a big reason why there was so much fintech interest in blockchain.
You may be thinking of high frequency trading. In that case, traders interact directly with an exchange - e.g. via direct market access[1] - so it’s a pre-established two-party interaction. There’s no particular technical difficulty with making that fast. Usually, slow transaction times are a consequence of the structure of the market, not a technical issue particularly.
In the broad picture of engineering, I would consider this to be technically difficult. Many pieces of the puzzle need to interact correctly to remove latency, from physical location, network gear, decision about where software runs, removing unnecessary layers of everything, to algos and data structures, and doing razor-tight tradeoff analysis favoring low-latency at every step along the way. It’s also expensive. So it’s hard to agree that this is not hard. On the flip side, if you find this easy and want a job, msg me.
The OP mentioned that "most parties there take minutes to process messages". My point is really that if transactions are taking minutes to complete, that's likely to be a social/structural problem, not a technical one.
With today's technology and all the work in this space, it's not particularly technically difficult to get such times down into the seconds or milliseconds, depending on the constraints.
Yes, HFT has gone to extremes to get trade execution times down as low as microseconds. And sure, that involves technical challenges. But they're mostly sort of obvious ones - you mentioned some of the major categories - that yield to throwing engineering talent at. But I was referring to the Pareto 80% of gains here.
_1tan I have emailed your posted contact data should you see value.
I have personally founded, architected, and coded in entirety several acquired payment processing companies in the last 30 years of which were near 100% uptime as well as PCI Level 1 compliant among several other audit stipulations. Your mention of energy trading *immediately* caught my eye as I am now building an energy storage company that is founded in design on top of my decades evolved secure highly available API transaction engine that I am refactoring to support VPP to manage the energy storage array I am patenting.
My ying appears to align with your yang from my understanding of your words so if this is something of interest please feel free to engage from my outreach.
> Currently most parties there take minutes to process messages and I think there could be a lot to learn from financial systems engineering
Depends on what you mean by "financial systems". There are plenty of financial systems that takes days or weeks to fully process transactions.
You're probably referring to stock trading systems (which by their nature have to be extremely low latency). The term to search for is "HFT" (high-frequency trading).
Here is a recent paper on the topic (focused on C++, though most things you find will tend to be): https://arxiv.org/abs/2309.04259
I am mainly working on B2B message exchange, e.g. via AS4/EDIFACT/ebix/CIM files and most parties seem to still work with a lot of batch processing. With the ongoing digitization of e.g. metering hardware and faster settlement times in the energy markets (EU-centric) I think this might not cut it anymore in the future.
Currently most parties there take minutes to process messages
I suspect that the energy market is a bit different from markets where being ruthlessly fast might be a good long term behavior. So there might be good reasons for that way of doing business due to the specific nature of the energy market.
For example a limited number of counter parties might make stable B2B relationships more valuable. Or regulatory compliance might require extra steps. Or the messaging system might be noisy.
Or not. I wonder how executives at the trading companies would describe the delay.
One other thing that comes to mind - the players in the financial markets make so much money off that message-processing speed that it opens up a whole lot of options that may not be available to someone without those kinds of deep pockets. We're all still fighting Einstein, but the ability to just throw resources at the problem can surface some solutions that are out of reach when you manufacture something other than money.
I agree and scale is one of the reasons there's so much money to be made. The NYSE trades over $18 billion a day. A small edge can be exercised many many times.
I wouldn't look further than asking what happens currently in the processing of these messages. It might turn out for example that the minutes delay is deliberate: high frequency trading can become unstable more easily than low frequency trading. Perhaps there's a human in the loop?
Check out The Payments Engineer Playbook[0]. I don't work on financial systems but I'm subscribed and like the occasional distributed systems insights.
One of the posts turned up on HN front page a year ago[1]. Thats how I discovered it
I had come across his books when searching for ML books in C++. He has applied ML/Statistical/Deep Learning/etc. techniques to Market Trading Systems and other domains. Generally all his books contain implementations in C/C++.
The reviews of his books on Amazon gives you confidence that he really knows his stuff from a practical/industry pov.
Question is a bit broad, but out of all the concepts under the financial engineering umbrella you're bound to explore the concept of ledgering eventually.
I've written a bit about it on my own co's product blog in an attempt to demystify some core concepts [1], [2], [3].
Still on ledgering and expanding into less mathematical and more applied concepts, I can also recommend a book called "The Accounting Game: basic accounting fresh from the lemonade stand" [4].
[1]: https://www.formance.com/blog/engineering/how-not-to-build-a... [2]: https://www.formance.com/blog/engineering/debits-and-credits... [3]: https://www.formance.com/blog/engineering/ledgering-all-the-... [4]: https://books.google.com/books/about/The_Accounting_Game.htm...
# Books
## *Trading and Exchanges: Market Microstructure for Practitioners by Larry Harris*
- comprehensive overview written in an accessible way
## *The Microstructure of Financial Markets by Frank de Jong and Barbara Rindi*
- 1st 1/4 of the book is generally useful. Then the math starts. This math is not needed to get a basic overview.
These are two books I wish someone gave to me when I started my first capital markets software engineering job. I recommend them to all the people I place in financial system engineering roles.
This is my passion. Message me if you want to talk more about this — see HN profile for contact info.
I would recommend the Jane Street youtube channel, the ocamal focused ones in my opinion you can skip. But otherwise their engineering videos will give you a good sense of it. When it comes to low latency systems this is a very private ecosystem. There are some open source projects for learning some of the mechanics of trading, market data and the order to settlement lifecycle. I am not aware of any being built for low latency.
Market data ingest, analysis and resulting order execution is chewing through way more data way faster than any banking transaction system. I have worked on both of them.
If you want more in the weeds but still high level I gave a talk on the main concepts and systems you need to know to code low latency for markets. https://docs.google.com/presentation/d/1HIPJb0XX3JDHEYSrZC8v...
I work on low(ish) latency trading systems in FX. FIX is the standard communication protocol and familiarity with it is essential for me. Here you can look up the standard message types and tag values: https://fiximate.fixtrading.org/
They also have docs for the standard message flows you can expect during trading. I use it regularly.
Also work in the FX space and could never find a good FIX specification resource. Always had to rely on whatever the broker shared with us, but it always felt incomplete. This is a super useful resource. Thanks for sharing!
Do you really need anything more than the dictionary? https://www.onixs.biz/fix-dictionary.html
Implementations varies, no one forces the third party to use message types 100% as intended.
Like someone else said your question is kind of broad. But I d recommend reading TigerBeetle's documentation to understand how financial transaction processing may differ from what they call general purpose databases like PostgreSQL.
https://docs.tigerbeetle.com/single-page/#concepts
There might be some gems for you in this old thread: https://news.ycombinator.com/item?id=22573204
Book recommendations for learning financial systems.
I recently read Central Banking 101¹ and learned a lot!
¹ https://www.goodreads.com/book/show/56863052
There a free online accredited master's degree in Financial Engineering (https://www.wqu.edu/mscfe), probably not for OP but maybe some find it useful for them
A lot of settlement in financial markets is still pretty slow. That’s a big reason why there was so much fintech interest in blockchain.
You may be thinking of high frequency trading. In that case, traders interact directly with an exchange - e.g. via direct market access[1] - so it’s a pre-established two-party interaction. There’s no particular technical difficulty with making that fast. Usually, slow transaction times are a consequence of the structure of the market, not a technical issue particularly.
[1] https://corporatefinanceinstitute.com/resources/career-map/s...
In the broad picture of engineering, I would consider this to be technically difficult. Many pieces of the puzzle need to interact correctly to remove latency, from physical location, network gear, decision about where software runs, removing unnecessary layers of everything, to algos and data structures, and doing razor-tight tradeoff analysis favoring low-latency at every step along the way. It’s also expensive. So it’s hard to agree that this is not hard. On the flip side, if you find this easy and want a job, msg me.
The OP mentioned that "most parties there take minutes to process messages". My point is really that if transactions are taking minutes to complete, that's likely to be a social/structural problem, not a technical one.
With today's technology and all the work in this space, it's not particularly technically difficult to get such times down into the seconds or milliseconds, depending on the constraints.
Yes, HFT has gone to extremes to get trade execution times down as low as microseconds. And sure, that involves technical challenges. But they're mostly sort of obvious ones - you mentioned some of the major categories - that yield to throwing engineering talent at. But I was referring to the Pareto 80% of gains here.
_1tan I have emailed your posted contact data should you see value.
I have personally founded, architected, and coded in entirety several acquired payment processing companies in the last 30 years of which were near 100% uptime as well as PCI Level 1 compliant among several other audit stipulations. Your mention of energy trading *immediately* caught my eye as I am now building an energy storage company that is founded in design on top of my decades evolved secure highly available API transaction engine that I am refactoring to support VPP to manage the energy storage array I am patenting.
My ying appears to align with your yang from my understanding of your words so if this is something of interest please feel free to engage from my outreach.
Stay Healthy!
> Currently most parties there take minutes to process messages and I think there could be a lot to learn from financial systems engineering
Depends on what you mean by "financial systems". There are plenty of financial systems that takes days or weeks to fully process transactions.
You're probably referring to stock trading systems (which by their nature have to be extremely low latency). The term to search for is "HFT" (high-frequency trading).
Here is a recent paper on the topic (focused on C++, though most things you find will tend to be): https://arxiv.org/abs/2309.04259
I've heard from a friend in the energy trading space that they're a bit behind, but your question is still a bit too broad.
What in the financial space do you want to hear about? Networks, exchanges, settlement?
I am mainly working on B2B message exchange, e.g. via AS4/EDIFACT/ebix/CIM files and most parties seem to still work with a lot of batch processing. With the ongoing digitization of e.g. metering hardware and faster settlement times in the energy markets (EU-centric) I think this might not cut it anymore in the future.
Find out if they are moving to FIX, FIXFAST, or some sort of ITCH (binary) type feed.
Currently most parties there take minutes to process messages
I suspect that the energy market is a bit different from markets where being ruthlessly fast might be a good long term behavior. So there might be good reasons for that way of doing business due to the specific nature of the energy market.
For example a limited number of counter parties might make stable B2B relationships more valuable. Or regulatory compliance might require extra steps. Or the messaging system might be noisy.
Or not. I wonder how executives at the trading companies would describe the delay.
One other thing that comes to mind - the players in the financial markets make so much money off that message-processing speed that it opens up a whole lot of options that may not be available to someone without those kinds of deep pockets. We're all still fighting Einstein, but the ability to just throw resources at the problem can surface some solutions that are out of reach when you manufacture something other than money.
I agree and scale is one of the reasons there's so much money to be made. The NYSE trades over $18 billion a day. A small edge can be exercised many many times.
If you just want to get up to speed on the mathematics, Luenberger's Investment Science is a classic.
https://www.amazon.com/Investment-Science-David-G-Luenberger...
"The Disruptor is the result of our efforts to build the world’s highest performance financial exchange at LMAX."
https://lmax-exchange.github.io/disruptor/disruptor.html
I recently learned of Kdb+ and the associated ecosystem of financial tools.
May be worth getting a head start on these, I understand these are the tools that the HFTs and the quants use.
https://kx.com/products/kdb/
I wouldn't look further than asking what happens currently in the processing of these messages. It might turn out for example that the minutes delay is deliberate: high frequency trading can become unstable more easily than low frequency trading. Perhaps there's a human in the loop?
Check out The Payments Engineer Playbook[0]. I don't work on financial systems but I'm subscribed and like the occasional distributed systems insights.
One of the posts turned up on HN front page a year ago[1]. Thats how I discovered it
0. https://news.alvaroduran.com/
1. https://news.ycombinator.com/item?id=42269227
You might want to checkout the books by Timothy Masters - http://www.timothymasters.info/
I had come across his books when searching for ML books in C++. He has applied ML/Statistical/Deep Learning/etc. techniques to Market Trading Systems and other domains. Generally all his books contain implementations in C/C++.
The reviews of his books on Amazon gives you confidence that he really knows his stuff from a practical/industry pov.