Specialized LLMs for Finance: Development and Training Approach

More and more financial institutions are using domain-specific LLMs trained on financial data, news, filings, market activity, to handle tasks like sentiment analysis, forecasting, summarizing reports, or answering client questions.  Deploying an LLM in a financial setting requires careful alignment with the client’s needs and constraints. Financial institutions operate in a high-stakes, regulated environment, so an AI model must be accurate, secure, and compliant. Below, we outline best practices and then compare strategies for customizing models (fine-tuning vs. training from scratch).

The article is prepared in collaboration with experts from a custom software development company Belitsoft. A good financial LLM needs to fit the business’s data, workflows, and risk boundaries. Belitsoft’s team helps financial firms build tailored language models that are production-ready from the start.

Define Use Cases and Requirements

Start by figuring out what the client actually needs from the LLM — customer support, portfolio advice, fraud detection, research summaries. Each use case demands different strengths. A financial advisor bot needs tight instruction-following and fresh market context. A fraud detection model needs to catch patterns in numbers and avoid hallucinations. Early on, define what success looks like — accuracy, speed, response quality — and lock in any compliance constraints, like avoiding unauthorized financial advice.

Domain Adaptation and Training

To address client-specific needs, plan how to impart the organization’s proprietary knowledge and policies to the model. There are generally two techniques: (a) Fine-tuning the model on the client’s data/task, or (b) Continual pre-training on a large corpus of financial text relevant to the client (or a combination of both).

Training a New Model from Scratch

Train a language model from random initialization using a large corpus of texts, including financial domain data (and possibly some general data for coverage). This means performing the full pre-training process specifically for the financial domain, effectively creating a new foundation model. BloombergGPT is an example where a company collected a massive dataset (data from Bloomberg’s terminals, filings, news, etc.) and trained a 50B-parameter model from scratch​. The result is a model inherently imbued with financial knowledge from the ground up.

Cost & Resources

Very high cost – training a state-of-the-art LLM from scratch is one of the most resource-intensive endeavors in AI. Estimates suggest models cost on the order of millions of dollars to train​. BloombergGPT’s training run took roughly $1+ million in compute​. Additionally, the effort required (data engineering, training infrastructure, weeks or months of training time) is non-trivial. Only organizations with substantial ML budgets or unique data access typically attempt this. Moreover, after the base model is trained, one often still needs to do instruction-tuning and RLHF to make it user-friendly, which is another phase of training. Thus, building from scratch is a lengthy multi-phase project. In terms of data, one needs hundreds of billions of tokens of text. BloombergGPT used ~700 billion tokens (a mix of general and finance data)​. If purely focusing on finance domain text, one might use on the order of 50–100B tokens (as FinLLaMA did with 52B tokens) for a mid-sized model, but the more the better to reach high knowledge coverage. Such volumes require extensive data collection and cleaning (e.g. parsing years of SEC filings, news archives, research reports, etc. with careful filtering)​.

Performance & Capabilities

A model pre-trained from scratch on financial data can potentially achieve very specialized expertise. Because it sees an enormous amount of domain-specific text, it may pick up on subtleties that a general model might miss.

It can also be tailored in architecture. If certain tools or numeric capabilities are needed, the model design could incorporate that from the start. In theory, a scratch-trained model could outperform a fine-tuned general model if the domain data is rich and the model is large.

However, in practice, we see many fine-tuned models matching or beating models like BloombergGPT on benchmarks​. This is because general models are already very strong, and fine-tuning/adaptation can get you close to the performance of a from-scratch domain model in most cases. One area where a from-scratch model shines is complete control over its content: since the training data is curated, the model might have fewer irrelevant facts. But note, it will lack knowledge outside its training distribution. For example, a model trained only on financial texts might know finance in detail but be clueless if asked a general question (whereas a fine-tuned model might recall general info from its base). Whether that matters depends on the use case.

Use Cases

Training a new financial LLM is justifiable in a few scenarios:

• A client has unique proprietary data at massive scale that gives a competitive edge if encoded in an LLM. Bloomberg is a case where they had decades of proprietary financial data unavailable elsewhere – training their own model made sense to leverage that data fully​
• Strict compliance or IP requirements might push an organization to avoid using any external base model. For instance, a government treasury or central bank might decide they want an LLM built entirely from data they trust to minimize risks of hidden biases or backdoors. By training from scratch, they know exactly what went into the model (and can ensure no data leaves their premises).
• Need for architectural customization: if the application demands a model with a specialized architecture (say, an LLM that natively handles structured financial data or integrates with a database), building a bespoke model could be beneficial.
• Research and prestige: a large financial firm with resources might invest in its own LLM to be at the forefront of AI innovation (much like big tech companies did). However, this is often more of a strategic choice than a purely practical one.

Pros

Full control over training – can integrate the exact data you want, and the end model is entirely owned by the organization (no dependency on external weights or licenses). The model can be optimized for finance from the ground up, potentially achieving higher accuracy on certain niche tasks. Also, if done well, it contributes to the open research community (if released) and builds internal expertise.

Cons

Extremely high cost and time investment. Requires a highly skilled ML engineering team. The outcome is not guaranteed to surpass a fine-tuned existing model, especially if the fine-tuned model can also incorporate a lot of domain data. From-scratch models still need subsequent fine-tuning for alignment, which is additional effort. Moreover, once you train it, you bear the maintenance burden – if the data or needs change, you’d have to retrain or at least do continual training. In short, this is only feasible for top-tier players in the industry or collaborations (as was the case for BloombergGPT). Most others will find it prohibitively expensive and unnecessary.

Recommendations for Choosing an Approach

Use Fine-Tuning for the majority of projects, especially when you have limited data but need the model to excel on specific tasks. Leverage open-source checkpoints (ensuring the license permits your use) and apply domain-specific training. This could include supervised fine-tuning on examples of the task, as well as unsupervised continued training on any in-domain text the client can provide (sometimes called domain adaptive pre-training). Fine-tuning is also the way to go when time-to-deployment is short – a model can be customized and deployed in weeks rather than months.

Consider Training from Scratch only if the situation matches the special cases described above. The client should be prepared to invest heavily and wait potentially months for a result. Ensure that the benefit (significantly better performance, or meeting a non-functional requirement that an existing model cannot) clearly justifies this route. If the main concern is that an external base model might contain unwanted biases or data, an alternative is to select an open model known for high quality and perform a thorough evaluation and filter of its outputs rather than reinventing the wheel.

Hybrid approach

Often the best practice is not a binary choice. You can take an existing model and pre-train it further on a large financial corpus (which is like training from scratch but starting from a halfway point) – this injects domain knowledge – and then do a fine-tuning on instructions or downstream tasks. This approach has been validated by the community (the AdaptLLM project did this and improved a 7B model’s prompting ability without full retraining​). It provides a strong domain expert model at a fraction of the cost of training anew. Thus, for clients with moderately large financial datasets, a continual training + fine-tune pipeline can be recommended.

Leverage Retrieval Augmentation When Possible

Instead of fine-tuning on all your financial data, use RAG. The model stays general: no updates to the weights. It just pulls in the right documents when a question comes in.

That’s enough for a lot of client use cases  like a chatbot answering from a knowledge base. No retraining needed when the docs change.

It also helps with compliance: the model doesn’t store sensitive data, but fetches what it needs, when it needs it.

To make RAG work you need a real search system under the hood.

RAG is a good fit when it’s about recall,  like surfacing policy docs or pulling up past filings. It works when the answer already exists and just needs to be found.

Incorporate Alignment and Guardrails

A financial LLM needs more than just knowledge — it has to follow user intent and stay within ethical and compliance boundaries. Techniques like RLHF (Reinforcement Learning from Human Feedback) help fine-tune behavior. For example, RLHF can teach a robo-advisor to tailor suggestions to a user’s risk profile. The FinGPT team calls it the “secret ingredient” for personalization. Even without full RLHF, you should still train on compliance prompts — refusing disallowed requests (like insider info) and following the required tone (adding disclaimers to recommendations, etc.). Add a system prompt or policy layer to enforce these rules in production.

Test and Iterate

Before deployment, evaluate the model rigorously on client-specific scenarios. Use hold-out examples of the firm’s data (research reports, customer emails, transaction logs, etc.) to see how the LLM performs. Benchmark against human experts if possible. Key things to check are factual accuracy (does it get numbers and names right?), tendency to hallucinate (making up nonexistent financial facts is unacceptable), and compliance (does it inadvertently output sensitive data or advice without caveats?).

There are emerging finance-specific evaluation benchmarks (e.g. the Open Financial LLM Leaderboard​) that can be useful for standardized testing. Based on evaluation, refine the model – possibly by additional fine-tuning, adjusting prompts, or filtering its knowledge base. Plan for a feedback loop post-deployment: monitor model outputs in production and allow users (or compliance officers) to flag mistakes, which can then inform further training.

Data Privacy and Compliance

Financial client data is often highly sensitive (PII, trade secrets, etc.). A deployed LLM must be handled like any critical infrastructure. Ensure all fine-tuning uses properly sanitized data (no leakage of client identities unless necessary, and then only in secure environments). Host the model in a secure environment – many banks will insist on on-prem or private cloud deployment with encryption and access control​. The organization’s compliance team should review the LLM’s behavior to ensure it conforms to regulations (for example, an LLM that provides investment advice might trigger regulatory requirements to disclose certain information or be licensed).

One best practice is to have the model refrain from certain types of output entirely. For example, it should not give outright buy/sell recommendations on securities unless specifically allowed, and it should include disclaimers that its outputs are AI-generated and not official financial advice. Thorough legal review of the deployment is advised​.