Each week we find a new topic for our readers to learn about in our AI Education column.
New York and New Jersey have a bridge and tunnel problem, which means some of the most notoriously impatient people on the planet have to wait. What could possibly go wrong?
Manhattan, in case you weren’t aware, is an island, with a limited number of ways to get on and off of it, namely, bridges and tunnels. In fact, if you’re on a train going into Manhattan from New Jersey, you’re going through a tunnel to get there. The problem is that there are only a handful of train tunnels between New Jersey and Manhattan, and they were all built decades ago.
During that time, the population has flourished, and millions of more people live in New Jersey and work in New York. If you’re trying to get into Manhattan by train during a rush hour, you might have to wait your turn to get into the tunnel—and God help you if one of the tubes under the Hudson is out of service.
The shortage of tracks across the river is a bandwidth problem. The increasing waits, in minutes and hours per week, is downstream from the bandwidth issue. That wait, when a train has to slow down or stop, making people late for work, delaying deliveries, and causing other disruptions, is a form of latency—which is our topic for today’s AI Education column.
Artificial intelligence is usually discussed in terms of intelligence: How accurate is the model? How sophisticated is its reasoning? How much data can it analyze? How reliably can it detect fraud, forecast markets or answer a financial advisor’s questions?
Speed is just as important. How long does it take? That question brings financial institutions into the world of latency. In its simplest form, latency is delay—the elapsed time between an event and a response. It doesn’t even have to be caused by a bandwidth, or tunnel, issue. The time it takes one train to move from one station to the next, or a passenger to get from one destination to the next, is a form of latency. But that simple definition conceals an increasingly complicated issue for financial artificial intelligence. An AI system can encounter delays while retrieving data, moving information across a network, processing an inference, querying another application, generating language and returning its answer. An agentic system may repeat several of those operations before completing a task.
What Is Latency?
Latency, for our purposes, is the delay between cause and effect or between the initiation and completion of an operation. In computing, it generally refers to the amount of time required for a system to respond to an input or request. IBM defines network latency as the time required for data to travel from one point to another across a network, while Cloudflare similarly describes it as the time necessary for a packet to move between locations.
Latency is usually expressed in units of time—seconds, milliseconds, microseconds or even nanoseconds—depending upon the system. The important distinction is that latency measures time, not capacity.
That separates latency from two concepts with which it is frequently confused: bandwidth and throughput. Bandwidth describes the theoretical amount of information that can travel through a connection over a given period, while throughput measures the amount that actually gets through. Latency asks a different question: How long does the trip take? A system can therefore have enormous bandwidth and still suffer significant latency. Amazon Web Services notes that network performance depends upon both dimensions; high bandwidth by itself does not guarantee fast responses.
Latency in Technology and Computing
Computing systems contain numerous potential sources of latency. Network latency occurs while information travels between machines. Physical distance matters because signals cannot move instantaneously, while routers, switches, congestion, protocols and retransmissions introduce additional delays. MDN Web Docs notes that even ordinary web requests can incorporate DNS lookup, TCP handshakes, TLS negotiation, server processing and response transmission.
Processing or compute latency occurs while processors execute instructions. Storage latency measures the time needed to read from or write to storage. Database latency arises while applications query and retrieve information. Application latency incorporates the time software spends executing its own logic.
There is also data latency: the interval between data being generated and becoming available for use. That distinction will become increasingly important in AI. IBM defines data latency as the time between data generation or request and the point at which the information becomes available and ready for use.
Latency in Finance
Finance introduces additional definitions because financial activity is itself a sequence of signals, decisions, transactions, settlements and records. The most familiar is market latency: the delay involved in market data reaching a participant or an order reaching a trading venue. Closely related execution latency measures the interval between deciding to trade and executing that trade. Transaction latency can describe the time required to process a payment or other financial transaction.
In electronic markets, latency can become a competitive variable. TNS Waypoint defines financial-market latency in terms of the time required for market data, orders or system responses to move among the systems, venues and applications involved in a trading workflow. Infrastructure choices such as proximity to exchanges, hosting, network routes, market-data distribution and colocation can therefore influence performance.
But financial latency extends well beyond trading. There is payment latency, the time between initiating and completing stages of a payment; settlement latency, the delay before transactions become final; risk latency, the gap between a change in exposure and its recognition by risk systems; and information latency, in which decision-makers operate from information that trails actual events.
AI Brings Even More Latency
At the center of AI’s latency is inference latency: how long a trained model requires to process an input and produce an output. Moveworks notes that inference can include preprocessing, mathematical computation, transfers among processing units and postprocessing. Model complexity, hardware, memory bandwidth and software architecture can all influence the result.
There is also input or preprocessing latency, while raw data is cleaned, formatted, tokenized or transformed; retrieval latency, while a system searches databases, vector stores or other information sources; network latency, while information travels between users, applications and models; and postprocessing latency, while outputs undergo formatting, business-rule checks or other operations.
Generative AI introduces additional measures. Time to first token, or TTFT, measures how long users wait before an LLM begins producing its answer. Output tokens per second measures generation speed once the response begins. An application can therefore feel responsive because it starts answering quickly even though the complete response requires considerably longer.
What About Financial AI Latency?
Consider a bank using machine learning to determine whether to approve a card transaction. The clock begins before inference. Transaction information must arrive. Relevant customer or account information may need to be retrieved. Features must be prepared. The fraud model must execute. Rules or secondary models may be applied. A decision must then travel back through payment infrastructure.
The institution cares about end-to-end decision latency, not merely whether the machine-learning model itself scored the transaction in five milliseconds.
That distinction is emphasized by Feedzai, which defines transaction latency in fraud prevention as the time a transaction spends being processed by a particular system and warns financial institutions to understand exactly what vendor latency measurements include.
Now consider an AI financial-advice assistant. Its latency profile is completely different. It might authenticate a user, retrieve portfolio positions, query market data, search planning documents, call a language model, run compliance checks and construct a response. Millisecond performance is probably unnecessary, but a long delay can undermine the conversational experience.
A trading model presents another extreme. If an algorithm recognizes an opportunity only after faster market participants have acted, its prediction can be perfectly accurate about a world that no longer exists.
Fraud detection, payments, AML surveillance, algorithmic trading, customer-service agents, advisor copilots, portfolio analytics and back-office reconciliation therefore have different latency budgets. There is no universally correct target.
