LLMs in Financial Analysis: From Promise to Practice

In this session of Dean’s Distinguished Speaker Series, Professor Charles C.Y. Wang, Tandon Family Professor of Business Administration at Harvard Business School, examined the practical realities of harnessing Large Language Models (LLMs) for financial analysis. He shared that while AI is a powerful tool for reading and processing text, analysts must be strategic in its application, avoiding over-reliance on models for extensive arithmetic or ambiguous reasoning paths.

Through actionable guidelines, Professor Charles showed how decomposing complex queries into sequential prompts and actively managing the provided context can mitigate the risks of “confident but eloquent” AI blind spots. Moderated by Associate Professor Kelvin Law from the Division of Accounting at Nanyang Business School, the session highlighted the importance of triangulating across different models and using clear, unambiguous terminology to ensure AI serves as a reliable asset rather than a liability in financial decision-making.

The following is an edited transcript of the Q&A segment:

Prof Kelvin: Thank you very much. I learned a lot from the talk. I have a number of burning questions from when I was listening to it. I’m not sure how many we can cover because of time, but let’s talk about the first one. You talked about financial analysis and tried to showcase how it works. Can you quickly walk us through it? If you want to do a financial analysis, what is the start, what is the finishing line, and how does it work briefly?

Prof Charles: I’ll describe a very simple workflow that I have now. It is a very basic agentic workflow with a five-agent setup. It ingests the 10-K, extracts the financials, and puts it into Excel. One agent is a strategist. It’s a specialist for doing qualitative analysis, understanding industry competition, and where the company sits in that competition. There is an assembler that puts everything in Excel, an analyst that does a DuPont decomposition and forecasting, and a modeler that puts together the DCF.

Prof Kelvin: Do you have a smart agent sitting at the top and then the other agents at the bottom? I think that’s what you’re saying.

Prof Charles: I’m the orchestrator. This is a very basic setup, but even at this level, it changes things. Something that in my evaluation course would take students or take me personally a few hours to put together for a new company, now takes five to seven minutes from end to end for the first run. It's very helpful because now you spend more of your time thinking about the really important stuff. Is the forecasting really right? Did we get the economics right? What are all the different scenarios that need to be thought through?

Prof Kelvin: For this workflow, which stage do you think is the most important one? Which one will easily go wrong among these five? Sometimes people say extracting the data is where it goes wrong.

Prof Charles: I’ve tried this a bunch of times. It’s been tuned in a way where there are small errors but not big errors. If you want to play with this, I have a GitHub repository where you can download these instructions. They were written in Gemini because, at the time, Gemini was the best model. You can take this to all the other LLMs. It’s native to Gemini and uses Google Sheets and Google apps. You can turn this into a Claude code skill as well, though I haven’t tested how good the skill is. You could play with this. I think it's decently good. The most important part is what happens afterwards.

Prof Kelvin: You covered a lot on fundamental analysis. One thing a lot of our audience probably has questions about is technical analysis. You mentioned some advanced developments using vision to do technical analysis.

Prof Charles: This is not an area that I have researched myself, but what I can tell you is that our colleagues in finance have done really cool work. They are trying to examine whether similar models, using neural networks that can see pictures, can be trained to help us do technical analysis. The simple answer is yes, it seems able to do it quite well. They can match patterns, such as head and shoulders or other terms I don’t quite understand, to subsequent returns. It can figure out how to interpret charts live and then predict returns.

Prof Kelvin: That is quite good. Another question is about education. You mentioned DeepSeek and Gemini. There are so many AI models everywhere. How do you see students or faculty using these tools later on to develop their own skills? How can we prepare the next generation of leaders?

Prof Charles: My advice for students is to go and play with this every day, but play with it with a skeptical mind. Hopefully, I’ve shown you enough ways in which the answers LLMs give can distort your decisions or your thinking. Use it to build and play with it, but be mindful of outsourcing actual thinking. I know lots of folks are using LLMs to do homework. That's fine, but why are you coming to a university? Is it just for a grade, or are you here to think? Use it to help you do the boring stuff.

Earlier, some of us were talking about the transition from a world with no calculators to a world with calculators. I grew up in Taiwan in the 80s, where we had to learn how to use an abacus. I was always so proud of the fact that I could compute in my head. I felt sorry for people who used calculators. But now I feel totally differently. Calculation is a skill, but it’s a mechanical skill. When you study mathematics long enough, you realise the super interesting stuff is not calculation. It’s the really deep patterns, understanding broader structures that were not so obvious, and figuring out how to prove those things. That is where the human mind can be quite beautiful.

I’m not worried if you’re using large language models to help you do the boring stuff. I get worried if it’s substituting the mind training that helps us do the more beautiful and important things. For the instructors and professors, I would say don’t shy away from this. It’s here. The tsunami is already here. You can try and resist it, but you’re not going to succeed. Our best course forward is to figure out how to embrace it and adapt our pedagogy in ways that accentuate what we do best, which is to teach critical thinking.

Prof Kelvin: We have a few minutes left, so let’s open the floor. If there’s any question from the audience, go ahead.

Audience Member 1: Thank you so much. Very exciting talk and results. I have a question regarding the technical math side. You mentioned there is statistically significant data proving that ChatGPT and DeepSeek predict differently on US and Chinese stocks. I wonder if you cluster them based on well-informed Chinese companies – from a US news perspective versus under-informed ones, will some trend change? Will there be some sort of Simpson's paradox where you reverse the result?

Prof Charles: If I understood your question correctly, you’re asking if there is variation in that finding relative to the amount of information the US knows. Yes. We tried a version where we looked at cross-listed firms, like the Alibabas of the world, where you might naturally think there’s more coverage and news in the US. We found exactly that. For cross-listed firms, that baseline optimism shrinks by about 30 to 40%. We also tried to explore the variation in what kind of news is missing. When it’s missing a lot of positive news, ChatGPT gets really pessimistic. It's not a baseline bias of loving Chinese companies. It’s entirely a function of what information seems to be missing.

Audience Member 2: Thanks, Professor. I’m doing project finance for renewable assets, which are quite non-standard. From one country to another, the projects are very different in terms of engineering designs. Financial modelling has always been quite painful for us, taking time and figuring it all out. I’ve been talking to friends in financial advisory firms. Some are developing AI plug-ins for Excel. Some are in the pilot stage and making progress. I tried a few, and it’s not quite there yet, but it seems they are making progress. My question for you is, do you think in due course this will work well enough for all these non-standard situations?

Prof Charles: I’m optimistic, but I’ll say that with reservation because I don’t know the particular context you have in mind. I’ll describe the following. This set of agent instructions was designed to produce something called the residual income valuation model, which is quite non-standard. For regular finance, we have our usual DCF that investment bankers are familiar with. But since accounting professors need jobs, we need to teach something different from our finance colleagues. I used templates and examples to train and develop this set of instructions, and that seems to apply quite well across companies. If there’s a particular archetype, you can train it to do that archetype. But if it has to be adaptive and change the archetype depending on a dynamic situation, I’m less certain.

Audience Member 3: I have a question about what you mentioned regarding the corpus being important for these models and the kind of information they have access to. I know you looked at it from the side of analysing the data. But in terms of generating the data, are companies also using AI to generate these reports and information? At what point does it just become AI feeding information to AI, creating a recursive model? Does that introduce a way to game the system? Similar to how Tesla embedded an image, are there ways to fool these AI analysts into giving more positive outlooks on companies based on how they are trained to look at the data?

Prof Charles: We have early evidence suggesting that AI is now generating financial reports, or at least the textual portions of them. We have evidence suggesting that's also true for analyst reports. You’re asking an important question about what happens when the machine is trained on machine output. Early on, when people were studying large language models, they noticed that when learning from machine-generated text, the model's intelligence ultimately collapses. I don’t think that result is true anymore given recent technological developments, but I'm going to use that intuition to end by talking about us.

If we are some approximations of these neural networks, and we are recursively learning from machine-generated text because all these books are now written by machines, what does that mean for the collective intelligence of our species over time? When do we collapse intellectually?

The optimistic side of me says no, that won’t happen. There is always a young, enterprising set of folks, smart Singaporeans, who will differentiate themselves on the basis of human creativity. When they do, people will notice, and they will become the next leaders in the world. That’s why we won’t end up in that equilibrium. But the part that worries me is, what if the people evaluating talent no longer have the capacity to differentiate what’s truly good and truly beautiful? Can we escape from that depressing reality? I don't know. This is why I encourage everyone to use these tools. Let’s get more efficient. But we all need to collectively figure out how to maintain and sharpen our critical thinking skills. That is going to be the single most important thing we do in the age of AI.

Prof Kelvin: That was a great discussion on using synthetic data. Finally, here’s a quick summary of what we covered today. We talked about use cases, and we discussed the reasoning and mental models you saw on the slides. A main takeaway was differentiating core and non-core earnings using various LLMs. We also learned something new about Tesla using Bitcoin and embedded images, and fortunately, some vision models are able to capture that. We had audience questions about the agentic workflow. Lastly, we learned that in some contexts, a non-reasoning model might actually outperform a reasoning model. Those are the key takeaways. Thank you all for coming and thank you to everyone joining us online today.