While I’m sure AI will continue to improve, it still suffers from the same issue as the search engine, which is that you need to know enough to ask the right questions.

I have run into this countless times when using AI. I asked for ideas around a topic on how to solve a problem, and it seems to miss a really good solution. I bring it up, and it the says something like, “oh yeah, that is much better.” On the flip side, if I lead it with some ideas, it has trouble breaking free of that and it tells me I already have the best idea.

If the topics coming together are seemingly unrelated, it takes a good prompt to get the AI to link those ideas on the path toward a solution.

Just today I was asking Copilot about different ideas on how to structure a new project. I laid out some pseudo code with my initial idea, and it gave it back to me with a more complex syntax. I asked why, and if there were any advantages to the way it did it, and then it told me no, my way was better, cleaner, and the preferred way for the language. Though after pushing it some more it did suggested another alternate suggestion, which is tried to dismiss as worse, until I explained why it would actually be better. As far as I’ve seen, at least with Copilot (which is all I’m currently allowed to use at work), it’s no match for a person with some experience and knowledge when it comes to more abstract thinking.

I haven't thought hard about it yet, but perhaps I should be a bit worried.

My other "superpower" is digging into documentation and figuring out how to actually use stuff I've never seen before. Another thing that might be under threat from AIs soon. I've certainly used AIs for my own hobby projects in this regard, sometimes with good result, so it's surely a matter of time.

Though at least at my current job, my most valuable skill is being able to understand the customer's needs, and being able to come up with solutions that solve their problems well without breaking the bank. Part of that is finding out how to best utilize existing code, which means I like to work on varied parts of the code base. Part of it is probing the customers to understand how they operate, which limitations they have and so on.

I think part of that is thanks to the same drive that lead me to all these obscure topics, which drives me to want to understand the existing code and the customers domain, which in turn puts me in a much better position to help and guide our customers to a good solution.

Not sure if AI's will do that too soon, time will tell.

This is actually where I get the biggest benefits from AI. It's not really good at making those connections itself. But it is good at fuzzy searching. By that I mean I can describe things but not know the name or proper terms for that. So I describe to an LLM, it can figure that out, and then I go off and search for depth and nuance (don't use the LLM for that...). This is something that traditional search is really bad at. But LLMs are also really bad at traditional search.

I'm not sure why we aren't trying to make this more complementary. I really don't want my LLM to be direct search just as I don't want my direct search to be an LLM. Frankly, the context of what I'm looking for matters. Don't make it an end-to-end thing. Just give me a fucking toggle switch for direct search or fuzzy search (hell, throw in an "I don't know" and make it the default)

I'm not worried about the AI replacing me here because the "superpower" (and I assume the gp's) here isn't having broad awareness. It is the ability to abstract and connect seemingly unconnected things. Once the AI is there, it's going to have already been able to replace a lot more stuff. The "superpower" is creativity combined with broad knowledge-base. LLMs seem more complementary here than replacing.