This is a follow-up from the previous article on my 3 months spent with ChatGPT.
There is no complex topics I don’t address first with ChatGPT. The target is to build a first layer of analysis that I will be able to challenge based on my experience and the questions that would arise from it. This means I will challenge whatever ChatGPT is providing me with, using several tools, methods and process. And the beauty of this way of working is that ChatGPT will, in the end, challenge itself using a toolkit that I have prepared for this type of play.
Building, reviewing, looking for gaps and running tests
What I do with ChatGPT may be totally in the wrong. This is based only on my experience and I don’t intend to build a playbook for anybody to use. This is just sharing in a blunt and honest way what I am doing with it in order to sort out things, identify hypotheses, challenge them, look for gaps in thinking, and challenge everything I have done through test scenarios. And even re-test everything with stress tests, leveraging and utilizing events that happened in the real world to make sure my results are the closest to bullet proof.
Before elaborating on how I do all of this, we need to discuss a few options that are only available with premium subscription, be in with Anthropic or OpenAI. I didn’t test Open source LLMs as I don’t have the required level of expertise to start from scratch in that direction.
The first thing is to build a base level of things you want the LLM to use and go back to in order to answer to your questions. You can see this step as feeding the LLM with how you brain learned and how you figure out things, which is a great piece by itself as this will require you to dive into what you actually use and do in order to solve problems, how you process information, how you build your convictions, and how you share your thinking with the rest of the world.
I built my very own version of personalization request to ChatGPT with a lot of methods, frameworks and processes that the strategy consulting industry is using (2×2 matrix, 3 horizons, SCQA, RAPID, Blue Ocean, …). I poured into this already messy pool of knowledge a bit of project management methods (including agility at scale) and stopped for a bit. A few projects down the road, I came back to it in order to make some adjustments. I asked ChatGPT to review all the work we done so far and identify gaps and potential challenges in using all of these tools and methods. I updated and adjusted based on the feedback and insights it shared with me, once again using the methods and tools I had fed it with. Then re-ran it with the newly updated toolkit to make a side by side analysis and ask ChatGPT to provide me with a gap analysis of the 2 results I had in hand. Then I revisited my toolkit based on the gap analysis, leveraging the business model canvas as a framework to highlight how I was doing in all compartments of my reasoning.
Challenging the challenge
As I review what I have written, this may really well seem totally blurry. Just copy/ paste this in your favorite LLM and ask it to rephrase it, make it into a step-by-step approach, and help you actually build your own customization. And while at it, ask it to challenge it and provide you with options for improvements, options for test scenarios to have a feeling of whether is could work for you. And many other questions. It will never falter and will always provide you with answers. But at some point, you may feel it is writting things that are totally irrelevant and, when you would want to trace back your train of thoughts, it will tell you without hesitation it has misled you because it needed to simplify or that your work was too long of a document so it needed to truncate it.
If you are like me, you may want to insult it as it didn’t tell you anything before doing it. As I went that way, I can tell you it is your fault for not asking how it will deal with complexity. In fact, ChatGPT is programmed as a general public tool, focused on simplifying and helping people access knowledge and information in a simple way. Even if it means over simplifying. And watch out, because it can compound in that direction, leading you to progressively build on something that is totally differnet from what you expect and end up with a piece of work that can directly go to the bin, as you won’t be able to back track where it started to send you in the wall.
Once you are clear with that, and that is part of the reason I go back to my personalization content from time to time to update and include additional rules for ChatGPT to not burn my work, you can really focus on iterations. Which is the bread and butter of LLMs to me: you make sure you’re starting point is right, you build a first layer of additional information that give you several different leads to follow, investigate them and make decision to keep pushing or close them because they don’t provide enough value. You build, build, build and at some point you will have refined your understanding at the level you want and can move on.
Don’t get lost in the search and watch out for caveats
The real challenge is to pay attention of the level of refinement you want to create. LLMs can be really helpful to connect different types of expertises, connect the dots, and even generate new ideas. But it may go way too fast and you will suffocate because of the volume of new information it can feed you with. And if you’re not ready for arbitration and quick decision making, you will drown before you know it. The easy way will be to include in your persoanlization prompt to not overwhelm you with information unless you request for more. I have tested this but this is not really helpful as it doesn’t “get” what overwhelm would mean at the quantitative level. It will learn by doing, which may result in you being bothered.
This is one of the many caveats you will encounter when learning how to lead an LLM. Other ones may be that, even though OpenAI generated an impressive feature with the canvas (close to a collaborative doc with ChatGPT), it doesn’t know how to navigate the docs and how to structure them. You will end up with a lot of docs, potentially doobloons, and will have a hard time managing all the documentation that can be generated out of any LLMs. It can really quickly become a tsunami if you don’t pay attention to it. I’m doing it when I want to dive into a topic and receive a large amount of information, in order to get a feeling of the topic, but you will need to run it as a side conversation when the main one will be about building a documentation that is more straightforward and direct.
Additionally, ChatGPT can’t manage very long document, you will need to build understanding in the conversation (the chat) and then build a consolidated version only in the canvas. I requested the LLM to only update the canvas when I explicitely asked for it, and even with that it went back to not following that rule from time to time. I had to spend about 2h with it to get to the bottom of it and understand why it happened and fix this.
Key takeaways from using ChatGPT
Using ChatGPT, I can do the following while spending the same amount of time:
- 2x more research to get a good feeling of a topic
- 5x more benchmarking (you need to feed ChatGPT with the .pdf format of anything you want it to read for you, then leverage your toolkit to extract whatever needed; don’t ask it to simply recap, it will end up poorly)
- 5x faster in running analysis methods (watch out, you need to create a dedicated routine/ request so that ChatGPT is providing you with the full process that you can read to prevent the blackbox effect and its negative impact)
- no additional performance for .ppt design and completion, it’s not made for this
Bottom line, LLMs don’t give you more capabilities, they give you more arbitration and request more focus. It can totally be used by Junior profiles to accelerate production at consultancy but it still requires a lot of judgement and experience to make decisions. With a team of 5, you would now be able to run 4-6 projects more accurately and with less stress. When in another industry, you would need more than an LLM to perform a drastically more relevant work as you would require more dashboards for decision making and more forecasting tools to run scenarios and see how today’s decision can end up in a few weeks or months.
Leave a Reply