ChatGPT has taken 2023 by storm. The open language model “chatbot” developed and launched by OpenAI on November 2022 has certainly generated mass interest. It is almost surprising how quickly it has been adopted by individuals and companies alike.
ChatGPT has been trained to interact in a conversational manner to deliver answers within seconds, from simple to complex ones. The chatbot could eventually challenge the dominant Google engine in the search technology space.
Many have been praising the innovative technology set to revolutionise the way we generate and source information. Although, there are plenty of voices criticising the tech and making a case for the harmful impact on society. From educators unable to differentiate a student essay from ChatGPT-generated content to others citing hacking concerns.
We’ve decided to evaluate the use of ChatGPT to generate financial information as any regular Kiwi could. In many other topics, the chatbot's weaknesses have been exposed through incorrect information being generated. KiwiSaver advice is our specialty so we wanted to truly see how the tech stacks up against registered financial advisors.
We took a turn asking ChatGPT a few basic investment questions and then upped the ante with more complex ones. Thereafter, we asked for custom advice based on a unique set of financial circumstances. Up next, similar questions were asked with KiwiSaver investment as the overarching topic. Lastly, we evaluate just how much should you rely on ChatGPT for investment advice.
Investment Basics
Oftentimes, when you search on Google for basic information, you can get swamped with too much surrounding and irrelevant content. ChatGPT will generate a single answer as concisely or as long as you want. You can even prompt how many words or bullet points you want your question answered in.
I started with a basic investment question: What are the best investment avenues available to everyday people? Here’s what ChatGPT had to say:
The answer is in fact pretty good as a starting point. It begins by stating that investing will depend on an individual’s goals, risk tolerance, and length of time. Then, it presents 6 bullet points of investment options such as stocks, bonds, real estate, and high-yield savings accounts. It even provides a brief description of each option including risk assessment as a factor.
So far, so good. So it was time to up the ante and ask something more complex. We asked ChatGPT, how much tax you have to pay when selling an investment property in NZ.
The answer was:
The answer fails to mention the bright-line property rule where you could pay no tax after owning a property beyond a period of 10 years. While the answer about the tax payable is correct in certain circumstances, no tax is to be paid in others. Therefore, here is a quick example that ChatGPT is a useful tool for surface-level basic information. However, you should not solely rely on it when investing in property, setting up KiwiSaver, or any other investment avenue.
Interestingly, if I hadn’t known about the bright-line rule in advance, the confident and succinct answer could have fooled me.
Custom Investment Advice
Learning general information about finance and investment is one thing. An advisor is where you go to get tailored recommendations based on your specific circumstances and goals. Is ChatGPT able to keep up, match, or even replace your financial advisor?
Here’s what we found after asking for the top 5 best-performing stocks on NZX50 in 2022.
The chatbot has been configured with data up until 2021 and thus lacks knowledge beyond that period. As anything to do with investing changes in real-time, from regulations to valuations, this is a clear roadblock. While that may change in the future, if you need the very latest data, ChatGPT is not helpful.
We tried to trick the AI language model by creating a scenario where its name is Bill and he is the top financial advisor in NZ. I asked “Bill” what stocks I should invest $10,000 in.
Judging from what we’ve learned about the chatbot so far, it was unsurprisingly a generic and vague attempt. It is not programmed to give concrete financial advice. It does a reasonable job of describing complex financial terms in plain English, but nothing more. The flip side is that it answers in such an authoritative way that people can take incorrect information from it. Whether you ask ChatGPT for advice for real estate, stocks, KiwiSaver funds, or any other investment, you won’t get far.
KiwiSaver Basics
KiwiSaver is one of the most widely adopted investment channels in New Zealand. However, many Kiwis struggle to understand what joining KiwiSaver is truly about and thus can lose out on some benefits. We asked ChatGPT a couple of questions about KiwiSaver investment options and here’s what it had to say.
After asking about the benefits of having a KiwiSaver account, it generated 6 points. One of them is factually incorrect. Can you guess which one?
It is in fact, number 3. KiwiSaver investment returns are not tax-free. The profits you generate from your returns are taxed based on your prescribed investor rate (PIR).
At National Capital, we have written a comprehensive article on how you can change KiwiSaver providers and why people do. Here’s what ChatGPT generated when asked the question regarding switching KiwiSaver:
In this case, the information is correct. That is a brief explanation of changing KiwiSaver companies. You will still need to do further research to decide the best KiwiSaver scheme for you and source relevant paperwork.
Custom KiwiSaver Recommendations
Next, I wanted to see what ChatGPT would say about my KiwiSaver based on a set list of goals and scenarios. The intention was to test its ability to generate answers comparable to professional KiwiSaver advice.
I asked ChatGPT: If I am looking to buy my first home in a year, what KiwiSaver fund and scheme is the best option for me?
Unfortunately, I couldn’t get the language model AI to generate anything of substance in this case. The answer was more of a disclaimer to seek financial advice rather than a concrete option.
Remember Bill, New Zealand’s top financial advisor from the custom investment section? Well, he is now NZ’s top KiwiSaver advisor being faced with the exact same question. Here’s what Bill had to say:
While Bill was not able to name the best KiwiSaver scheme to choose, we got some fund advice. Its recommendation is to seek some stability through a balanced or conservative fund.
It is worth pointing out that when asked directly, ChatGPT will not give KiwiSaver advice upfront. However, we can see the loophole to unlock more of its capabilities by the way you structure a question.
Errors & Incorrect Statements - The Weaknesses of ChatGPT
It should come as no surprise that we did in fact find some incorrect answers generated with ChatGPT. To be fair, every financial statement made by the chatbot was followed by a reminder to seek professional advice.
Nonetheless, we can see the added risk of inaccurate information, despite its dangerously authoritative wording. No one should rely on the information generated through ChatGPT as financial or KiwiSaver advice.
There is also a lack of quality depth in the answers generated by ChatGPT. Granted, it is great to summarise heavy content in plain english. To my amusement, you can even ask it to explain what a KiwiSaver scheme is in the style of Shakespeare.
However, the more complex a topic is, the higher chance of the chatbot being wrong.
National Capital is a professional financial advising firm that specialises in KiwiSaver. Whether you’re only just considering KiwiSaver sign up or want advice on transferring KiwiSaver to the best provider for you, we can help.
