Skip to main content
Back to blog

Sovereignty and Local Models: The Hidden Cost of Inference at Leboncoin's Scale

Published on August 31, 2026

  • Leboncoin's catalogue (89 million listings) can now be searched in natural language inside ChatGPT — but only as a search gateway: purchasing, messaging, and payment remain entirely on Leboncoin.

  • Contrary to a naive reading, Leboncoin does not pay for inference per token: OpenAI absorbs that cost, funded by ChatGPT subscriptions — and since dropping Instant Checkout in March 2026 (a conversion rate three times lower than a direct purchase on the merchant's own site), no transaction commission applies here either.

  • The real risk today isn't the bill, it's sovereignty: handing the conversational interface and intentional data to a third party for free, with no safety net against GDPR/the AI Act on personal information slipped into free text.

  • It's when a platform chooses to bring its own conversational agent in-house — rather than remain a passive source for a third-party aggregator — that the inference cost calculation at scale becomes real, up to $260,000/month at proprietary API rates: hence the appeal of local routing and cross-provider fallback.

The anatomy of an integration at massive scale

To grasp the scale of the architectural challenge, it's essential to look at the underlying infrastructure of a hyperscale platform. Leboncoin ranks as the second most-visited e-commerce site in France, drawing more than 30 million unique visitors every month [2]. Its technical infrastructure, having migrated to a cloud-native, microservices-oriented architecture, runs on more than 900 distinct services and 300 relational databases to handle up to 30,000 concurrent requests per second [8]. The data volume under management reaches 1.3 petabytes, with a publishing rate of six new listings per second, representing an estimated exchange value of €27 billion in 2024 [2].

The shift from classic search to a conversational interface radically changes this infrastructure's load profile. Historically, search on classifieds platforms relied on indexed databases, queried through strict keywords, category trees, and boolean filters such as geographic location or price ranges [5]. While computationally efficient, this method requires users to already know the site's taxonomy and to iterate manually — a tedious process [4]. Integration into a conversational agent removes this friction: the user delegates the sorting and contextualization phase to an assistant able to understand implicit intent, interpret context, and draw on session history to refine suggestions within seconds [4]. This shift from filters to conversation isn't limited to general classifieds, either: it more broadly reshapes how qualified visitors convert, a pattern already documented for other e-commerce verticals (Search filters vs. conversational agent). However, this ease of use comes from the probabilistic generation of word fragments (tokens) by vast neural networks — a process whose per-request computational cost is in a different league from a simple database query.

The context protocol and the loss of the interface

To dynamically connect a catalogue of 89 million listings to an external language model, the industry now relies on interoperability standards such as the Model Context Protocol (MCP), introduced by Anthropic in late 2024 and widely adopted by 2026 [11]. MCP acts as a universal, bidirectional abstraction layer, unifying how AI models access external data sources, use tools, and plug into structured workflows — eliminating the need to build a custom integration for every new system [11].

The Model Context Protocol's architecture rests on a strict separation of responsibilities across three entities. The Host (MCP Host) is the execution environment and user interface, such as an integrated development environment or a consumer-facing conversational application. The Client (MCP Client), located within the host, handles routing requests between the language model and external servers. Finally, the Server (MCP Server) acts as a gateway exposing specific resources, executable tools, and prompt templates in a standardized way via JSON-RPC messages [12]. In Leboncoin's integration with ChatGPT, the commerce platform primarily acts as the MCP server, opening its catalogue and search functions to the external agent [3].

This architectural setup creates a fundamental strategic vulnerability. By delegating the host role to an external AI aggregator, a company gives up the direct user relationship and the primary cognitive interface. The platform is relegated to a mere data supplier (merchant as supplier), while the aggregator captures the entire customer journey, from discovering a need through to the final recommendation [14]. To preserve its competitive edge, a company must bring the MCP host in-house, which means deploying its own conversational AI to orchestrate requests internally — and therefore requires mastering language-model inference at scale.

Who actually pays for inference, today

Contrary to a naive reading of the business model, Leboncoin settles no per-token bill for this integration. The experience is limited to a search gateway: clicking a result sends the user back to the Leboncoin site or app, where messaging and payment happen exclusively — nothing is transacted inside ChatGPT itself [65]. OpenAI, moreover, dropped its native "Instant Checkout" payment flow in March 2026, after it produced a conversion rate three times lower than a direct purchase on the merchant's own site [66]; the model that has since prevailed for integrated retailers (Walmart, Etsy) amounts to "own your agent, rent the distribution" — each keeps its own interface and customer relationship, with ChatGPT serving only as a discovery channel [66]. Neither OpenAI's official Apps SDK documentation nor the information made public about the Leboncoin deal mentions any commission or API fee charged to the merchant [67]: the inference calculation in the next section is therefore borne by OpenAI, funded by ChatGPT subscriptions — not by Leboncoin, as things stand today.

This apparent free ride doesn't cancel the risk, it shifts it from the financial terrain to the strategic one: a company relegated to a mere catalogue supplier hands over the conversational interface and intentional data for free — the subject of the next section. And it's precisely when a platform chooses to take back control by hosting its own conversational agent, rather than remaining that passive source, that the inference cost calculation becomes real: that alternative scenario is what the section below models.

Modeling the inference cost chasm

So it isn't Leboncoin's actual bill being modeled here, but the one any comparable platform would face if it chose to bring its own conversational agent in-house rather than remain a mere data supplier for a third-party aggregator. The viability of conversational search then runs head-on into the unit cost of inference. Estimating the order of magnitude of billing tied to exclusive use of proprietary APIs reveals figures that become technologically and economically paralyzing once applied to tens of millions of users. The industry bills access to language models based on the number of tokens processed as input (the user's query plus the context pulled from the database) and the number of tokens generated as output (the model's formulated response).

The 2026 inference market is segmented between ultra-capable frontier models, mid-tier models for everyday tasks, and ultra-efficient architectures built on open models or mixture-of-experts (MoE) designs offering rock-bottom pricing.

Provider and Model (2026 Generation)

Input Cost (per million tokens)

Output Cost (per million tokens)

Architecture and Role

OpenAI GPT-5.5

$5.00

$30.00

Omnimodal frontier model for complex reasoning [17]

Anthropic Claude Sonnet 4.6

$3.00

$15.00

Reference model for agentic workflows [17]

Google Gemini 3.1 Pro

$2.00

$12.00

Premium model with a very large context window [17]

OpenAI GPT-5.4 Mini

$0.75

$4.50

Fast mid-tier model for general tasks [17]

Mistral Large 3

$0.50

$1.50

Multilingual European model with aggressive pricing [17]

Meta Llama 4 Scout

$0.17

$0.66

Open model optimized to absorb massive volumes [18]

DeepSeek V4 Flash

$0.14

$0.28

Ultra-efficient model redefining the global pricing floor [17]

To quantify the financial impact in a verifiable way, we need to model a typical conversational search session using the Retrieval-Augmented Generation (RAG) approach. When a user asks for a sofa recommendation, the system first queries the traditional database to pull relevant listings, then injects that raw data into the prompt submitted to the language model. The model has to process the conversation history, the system's behavioral instructions, and the text content of the retrieved listings in order to produce a natural, synthesized response.

An average query structure in e-commerce production breaks down as follows. System instructions and enforced formatting consume roughly 400 tokens. The user's query and a summary of recent history add another 100 tokens. The database context — assuming ten detailed listings are injected, each weighing 150 tokens — adds 1,500 tokens. Total input tokens thus reach 2,000 per request. The assistant's generated response, detailing the options and justifying the selection, typically runs to 250 output tokens.

Applying these metrics to Leboncoin's traffic, which totals 30 million monthly unique visitors, and assuming a conservative 10% adoption rate for the conversational interface, the platform would see 3 million active users on this feature [4]. At an average of five interactions per user over the month to refine a search, total volume comes to 15 million LLM-type requests per month. This model results in a massive monthly consumption of 30 billion input tokens and 3.75 billion output tokens.

Using public API rates, the financial projection becomes explicit. On the API of a 2026 frontier model such as OpenAI's GPT-5.5, monthly billing would reach $150,000 for context ingestion and $112,500 for generation — a staggering total of $262,500 per month (more than $3.1 million annualized) [17]. Switching to a highly capable competing model like Claude Sonnet 4.6 would cut that burden to $146,250 per month, which remains prohibitive for absorbing the full traffic [17]. Using state-of-the-art optimized models such as DeepSeek V4 Flash or Llama 4 Scout would bring the cost down to roughly $5,000–$7,500 per month — the viability threshold for a free consumer service [17].

API providers frequently tout prompt caching mechanisms to soften these costs. At Anthropic and Mistral, repeated prefixes earn a pricing discount of up to 90%, while DeepSeek V4 Flash offers a cache-hit rate as low as $0.0028 per million tokens [17]. In a retrieval-augmented-generation architecture applied to commerce, however, the real gain is marginal. While system instructions (400 tokens) can be cached, the bulk of the query's context — the 1,500 tokens representing the specific listings returned by the database — is by nature dynamic and unique to each search, leaving the overwhelming majority of input load ineligible for cross-user caching discounts [19]. Relying exclusively on cloud APIs to process the entirety of organic traffic therefore creates untenable financial pressure, justifying bringing part of the inference capacity in-house.

The sovereignty of intentional data and the compliance imperative

Beyond the economic constraint, routing every user request to external AI providers raises critical challenges around privacy, regulatory compliance, and the preservation of the company's intangible capital. The very nature of conversational search changes the type of data the platform collects.

In a traditional purchase journey, algorithms try to infer consumer intent by analyzing clicks, browsing time, and cart abandonment (first-party behavioral data). The conversational interface, by contrast, encourages users to explicitly state their needs as free text, generating what the industry calls intentional data, or "Zero-Party Data" [21]. Users voluntarily provide extremely rich context, mentioning family structures, strict budgets, timelines, or even health issues or personal vulnerabilities to steer the system's recommendations [23].

Extracting and classifying these intentions is the most powerful conversion rate optimization (CRO) lever available to e-commerce businesses in 2026 [25]. Analyzing these signals allows offers to be personalized deterministically rather than probabilistically, driving substantial increases in conversion rates [24]. Delegating the analysis of these raw queries to a third-party API — particularly to players whose business model relies on exploiting data to continuously train their own models — amounts to giving up a fundamental competitive advantage [6].

More constraining still, Europe's legal environment imposes strict barriers on outsourced processing. With the EU's AI Act taking effect in 2026 and GDPR's ongoing obligations, transmitting potentially identifying data (Personally Identifiable Information — PII) to extraterritorial cloud infrastructure exposes organizations to heavy penalties [27]. Free-text queries inevitably contain email addresses, phone numbers, or precise geolocation information that traditional filters struggle to catch [6].

To legally operate a solution that occasionally relies on external APIs, the architecture must include a layer for detecting and masking sensitive information. Several design patterns have emerged to address this at the network edge, each with its own trade-off between added latency and loss of semantic context.

PII Masking Pattern

Median Added Latency

LLM Context Alteration

Recommended Industry Use Case

Hard Strip

~2 ms

Destructive

Handling generic queries and FAQs [6]

Typed Placeholder

~8 ms

None

Standard masking of isolated entities (emails, numbers) [5]

Token Vaulting

~15 ms

Partial

Regulated industries requiring cryptographic isolation of PII [5]

Smart Contextual Classification

~25 ms

Selective

Detection requiring semantic understanding of context [5]

Deploying a security gateway capable of applying typed-placeholder replacement or token vaulting makes it possible to detect a sensitive entity, replace it with a substitute token before sending it to the large language model, then restore the original information in the response delivered to the end user. This parsing step, however, adds computational overhead and does not protect the company against the external model inferring implicit sensitive data on its own. The only absolute guarantee of sovereignty and confidentiality lies in processing raw data locally (on-premise).

The architectural pattern: intelligent routing and localized models

To simultaneously solve the hyperscale economic equation and the regulatory compliance imperatives, software engineering is moving toward intelligent LLM routing architectures. This paradigm breaks with the monolithic approach of sending all traffic to a single, expensive model. It relies on deploying a primary router, often hosted locally, tasked with evaluating each request's difficulty in real time. The system assigns simple or sensitive tasks to cheap local models, while preserving the ability to fall back to proprietary frontier models for requests that demand complex reasoning [30].

Foundational research on routing frameworks, such as RouteLLM or FrugalGPT, demonstrates the dramatic effectiveness of this approach. Algorithmically optimizing model choice can cut the overall inference bill by more than a factor of two (2x+), and up to 98% in certain specific contexts, while keeping overall response quality equivalent to 95% of the models' native capabilities [30]. Selection strategies fall into several architectural approaches, including confidence cascading, where models are queried sequentially until a response deemed reliable is obtained — a very economical method, but one penalized by the accumulation of successive inference latencies [33].

The state-of-the-art approach favors pre-inference classification. In this model, an ultra-lightweight classifier analyzes the incoming request and predicts the target model before any text generation takes place, guaranteeing a single call per interaction [33]. This principle underpins the ComplianceGate architecture, specifically designed for multi-tier routing in industries subject to heavy regulatory constraints.

The ComplianceGate architecture deploys a bidirectional transformer encoder, calibrated at roughly 134 million parameters, which acts as the system's security checkpoint [35]. This pre-trained neural network evaluates each incoming request by generating a probability distribution across a matrix crossing two key dimensions: the level of cognitive complexity required (Simple, Medium, Complex) and the data sensitivity level (presence or absence of PII) [35]. What makes this architecture groundbreaking is its application of compliance by design. Any request classified as containing sensitive data is structurally confined and routed exclusively to inference models running on servers physically located within the authorized jurisdiction (for example, EU territory), blocking any leakage to extraterritorial cloud infrastructure.

This type of classifier's empirical performance demonstrates the viability of strict local routing. The encoder adds a nearly imperceptible latency penalty, measured at just 7 milliseconds on average. It reaches a classification accuracy of 99.2%, with a recall rate on identifying personal data close to exhaustive. On top of that, the system incorporates a safety mechanism grounded in algorithmic certainty: if the classifier's confidence score drops below a configurable threshold (typically set at 80%), the router defaults to the most restrictive label, forcing the uncertain request onto the most powerful local cluster to prevent any accidental violation [35].

By routing the vast majority of everyday searches ("I'm looking for a used bike in Paris") through this triage gate, the router diverts traffic toward small, highly responsive local models (Small Language Models — SLM). The analysis shows this massive diversion not only saves between 33% and 52% of API costs, but also cuts the system's median latency by 39% compared to systematically sending the entire flow to a heavy model [37].

Extreme optimization of dedicated servers and local inference

For this localized architectural model to be economically justified, the internal hardware infrastructure and the software handling inference must operate at maximum efficiency. Unlike cloud APIs' usage-based billing, hosting open models (such as Llama-3, Qwen, or Mistral's variants) on bare-metal servers involves fixed, unavoidable costs. The profitability of sovereignty therefore depends entirely on engineering's ability to saturate GPUs to maximize token throughput per second [38].

Sizing European GPU clusters

Technological independence requires provisioning high-performance computing hardware. Looking at European cloud computing offerings, from providers such as Scaleway or OVHcloud, makes it possible to build a precise sizing grid to handle primary routing and specialized models. The amount of video memory (VRAM) dictates the size of model that can be loaded, while raw compute power determines generation speed.

Hardware Accelerator (GPU)

Unit VRAM

Recommended Architectural Role

Estimated Monthly Pricing (Europe, 2026)

NVIDIA L4 PCIe

24 GB

Lightweight routing models (< 10B parameters) for fast primary triage [40]

~€677 (long-term commitment) [41]

NVIDIA L40S

48 GB

Inference for mid-tier models and high-context RAG queries

~€1,100–€1,500 [41]

NVIDIA H100 SXM5 / PCIe

80 GB

Processing nodes for complex models (70B+) or large consolidated volume [37]

~€1,817–€2,482 [40]

Acquiring or renting servers equipped with L40S cards proves particularly well-suited to edge deployments. These chips, purpose-built for cutting-edge inference, support reduced-precision data formats (such as FP8), artificially doubling in-memory processing capacity with no significant degradation of the model's predictive quality [43]. The crossover point between paying for a proprietary API and renting an H100 GPU cluster is generally reached once a platform continuously generates more than one billion tokens per day — a volume easily reached by organic traffic on Leboncoin's scale [39].

The vLLM revolution and the PagedAttention algorithm

The real hosting challenge, however, lies not in acquiring hardware but in managing memory in software. Early implementations of transformer-type models suffered from a crippling flaw in how they managed the key-value cache (KV Cache). This cache stores the state of attention computations for every token already processed, sparing the processor from redoing the work from scratch for each newly generated word [46]. Historically, inference engines statically pre-allocated a contiguous block of memory matching the maximum allowed length for every incoming request [46]. If a user asked a short question requiring a brief answer, almost all of the reserved memory space sat empty. This naive allocation caused catastrophic fragmentation, wasting between 60% and 80% of available VRAM, which drastically limited the number of requests processed in parallel [30].

Deploying the open-source vLLM framework radically solved this bottleneck through the PagedAttention algorithm [46]. Inspired by operating systems' virtual-memory paging mechanisms, PagedAttention splits the KV cache into small logical blocks (typically 16 tokens per block) that are dynamically mapped onto non-contiguous physical spaces in GPU memory [46]. Blocks are only allocated when the model strictly needs them during generation. As soon as a request completes, its blocks are instantly freed and returned to the available pool [50].

This architecture eliminates internal and external fragmentation outright, cutting the memory waste rate to under 4% [46]. The reclaimed memory allows the size of concurrent request batches to grow exponentially, multiplying overall throughput by a factor of two to four compared with earlier generations of inference servers — instantly paying back the cost of the GPU cluster [30].

Continuous batching and prefix caching

Paired with PagedAttention, vLLM popularized continuous batching (also called in-flight batching). Rather than waiting for an entire batch of requests to finish — forcing short tasks to wait while the longest one completes its generation — the engine admits and releases requests individually at every iteration step of the model. The GPU is thereby kept in a state of constant saturation, ensuring optimal use of expensive hardware [30].

In addition, application-level query optimization benefits significantly from Automatic Prefix Caching (APC). In production environments, many requests share identical initial segments, most notably the long behavioral instructions developers insert (System Prompts). APC computes a cryptographic fingerprint (hash) for each context block. When a new request enters the system, the engine checks whether that initial sequence already exists in memory [49]. On a match, the system reuses the physical cache directly without running any tensor computation. This feature, enabled by default in recent vLLM releases, eliminates the computational cost of long, recurring instructions, collapsing Time-to-First-Token latency and preserving bandwidth for pure generation [20]. Other engines, such as SGLang with its RadixAttention technology, deliver similar performance by exploiting prefix trees, proving especially formidable for multi-branch workflows or autonomous agents [46].

Multi-model orchestration and cross-provider fallback

A sovereign, high-performing architecture cannot do without a centralized orchestration layer. Deploying a hybrid primary routing model — interconnecting local servers running vLLM with multiple remote API providers (such as OpenAI, Anthropic, Gemini, Mistral, or xAI) — requires a flow-management gateway, commonly called an "AI Gateway" [58]. This gateway acts as the application's single entry point, standardizing communication protocols, managing authentication, and guaranteeing the overall service's high availability.

Dedicated solutions, such as Orq.ai, Requesty, EvoLink, or the open-source LiteLLM project, structure this indispensable intermediate layer [28]. Their critical function lies in implementing cross-provider fallback policies — a vital resilience mechanism in an ecosystem where the stability of commercial APIs remains subject to significant fluctuation.

The fallback logic works as a dynamic safety net. When a complex request slips past the local router and is sent to a preferred provider (Anthropic, for example), the gateway closely monitors the call's execution [32]. If the remote provider suffers an infrastructure outage (returning HTTP 500 errors) or if the company suddenly exceeds its usage quotas (HTTP 429 rate limits), the circuit breaker mechanism built into the gateway activates. The request is instantly interrupted, preserved, and transparently rerouted to the best-performing alternative provider (OpenAI or Mistral, for example) defined in the fallback chain [32]. This system guarantees that the end user experiences no service interruption, fully masking failures in the underlying infrastructure [62].

To optimize these switchovers without sacrificing the benefits of pricing caches at external providers, advanced gateways use sticky session routing. Under normal conditions, a multi-turn conversation could be fragmented across several providers depending on load balancing, destroying any chance of benefiting from prompt-caching discounts [62]. Sticky session routing attaches a unique identifier to a user exchange and forces the entire conversation to route to the same specific provider. Only in the event of a genuine critical incident at that provider does the whole session switch over and rehydrate its context with a new API, ensuring optimal budget management [27]. This orchestration layer illustrates a pattern already implemented by conversion-focused conversational agents such as Gamaro, which natively orchestrates multiple providers (OpenAI, Gemini, Mistral, xAI, Anthropic) with cross-provider fallback — confirming that model diversity isn't passive redundancy, but the very foundation of system reliability.

Conclusion

The ambitious integration of a commercial catalogue the size of Leboncoin's at the heart of a conversational AI foreshadows the next evolution of user interfaces. As things stand, it costs Leboncoin no inference bill at all: OpenAI bears that calculation, funded by ChatGPT subscriptions. The real price is paid elsewhere — handing the conversational interface and intentional data over to a third-party aggregator for free, without capturing any of that value, all while facing regulatory obstacles around exfiltrating that data to third-party servers. For any platform that chose instead to take back control by hosting its own conversational agent rather than remaining that passive source, the analysis shows that exclusively and continuously routing traffic from tens of millions of users to proprietary language models would, this time, become a structurally untenable path — the financial chasm driven by token pricing, made worse by the inability to cache the large volumes of dynamic data injected in real time.

The only architecture capable of sustaining this shift in scale while guaranteeing brand sovereignty rests on designing a hybrid inference pipeline organized around pre-inference intelligent routing. Integrating dedicated compliance classifiers (ComplianceGate) makes it possible to instantly route sensitive, everyday requests to localized models, hosted on European bare-metal servers and supercharged by the dramatic efficiency of engines like vLLM and PagedAttention. Commercial frontier models, often seen as the alpha and omega of AI deployment, thereby find their proper place: specialty engines, reserved for complex resolutions and orchestrated transparently by security gateways capable of absorbing failures. By adopting this sovereign, multi-tier design pattern, organizations ensure they create no constraining third-party dependency, turning generative AI from an uncontrollable operating cost into a lasting strategic advantage.

Citation sources

Cet article est aussi disponible en français