The few times I've used LLMs as question answering engines for anything moderately technical, they've given subtly-but-in-important-ways incorrect information such that taking them at face value would've likely lost me hours or days of pursuing something unworkable, even when I ask for references. Whether or not the "references" actually contain the information I'm asking for or merely something tangentially related has been rather hit or miss too.

The one thing they've consistently nailed has been tip-of-my-tongue style "reverse search" where I can describe a concept in sufficient detail that they can tell me the search term to look it up with.

> Now with LLMs, I simply prompt the same with a little bit more of context "pros and cons of using mysql vs mongodb when storing photos. Link references".

In near future, companies will probably be able to pay lots of money to have their products come up better in the comparison. LLMs are smart enough to make the result seem "organic" -- all verifiable information will be true and supported by references, it will only be about proper framing and emphasis, etc.

I use it the same way. The feeling is that I'm back in ~2010 when Googling stuff felt like a superpower. I could find anything back then.

Of course, it didn't last long, and trying to Google now is an exercise in pain and frustration. Lots of people have complained about the various things Google and marketers have done to get there, idk, I just don't like how it works now.

Top LLMs feel amazingly good at rapidly surfacing info online, and as I go through the references they're usually pretty good. I guess the same forces as before will apply, and there will be some window of opportunity before it all goes away again.

I do this. But the killer usecase for me is writing all boilerplate and implementing some half-working stuff keeps my attention on the issue which makes me able to complete more complex things.

A recent example is when I implemented a (Kubernetes) CSI driver that makes /nix available in a container so you can run an empty image and skip a lot of infra to manage.

I talked to it a bit and eventually it wrote a Nix derivation that runs the CSI codegen for Python and packages it so I could import it. Then I asked it to implement the gRPC interface it had generated and managed to get a "Hello World' when mounting this volume (just an empty dir). I also asked it to generate the YAML for the StorageClass, CSIDriver, Deployment and DaemonSet.

So LLM left me with a CSI driver that does nothing in Python (rather than Go which is what everything Kubernetes is implemented in) that I could then rewrite to run a Nix build and copy storepaths into a folder that's mounted into the container.

Sure implementing a gRPC interface might not be the hardest thing in hindsight, but I've never done it before and it's now a fully functional(ish) implementation of what i described.

It even managed to switch gRPC implementations because the Python one was funky with protoc versions in Nix(Python bundles the grpc codegen it's so stupid) so i asked it to do the same thing for grpclib instead which worked.

> Am I the only one using LLMs as if they were a search engine?

Nope, you're not alone as I also do this. I'm also not using any AI IDE's (yet).

It's funny, I actually recently failed a live technical interview where I was using my LLM of choice to answer Google-like queries instead of an IDE like cursor. The interviewer told me that he had actually never seen anyone use AI for coding like that before. Up to that point, I assumed that most coders were mainly using AI as a search engine and not necessarily using AI IDE's yet. Are we really that rare?

> I would read the official documentation of each db, forums, blog posts, stackoverflow entries, etc. It was time consuming on the searching side. The time it took to read all the sources was fine for me (it's learning time, so that's always welcomed).

This learning time that you welcomed is what you will now miss out on. The LLM gives you an answer, you don't know how good it is, you use it, and soon enough, if running into a similar issue, you will need to ask the LLM again since you were missing out on all that learning the first time which would have enabled you to internalize all the concepts.

It's like the regex engine example from the article. An LLM can create such a thing for you. You can read through it, it might even work, but the learning from this is orders of magnitudes less than what you get if you build this yourself.

In the past you looking around documentation, SO answers etc would have hopefully helped you learn more about the tools and develop skills required to independently analyze the pros and cons. If you ask an LLM (or a search engine or a colleague) and take their words as the ground truth then you won’t develop the skills. Worst, sooner or later no one will have enough knowledge nor analytic skill to form an opinion on any sufficiently deep subject and they will all be dependent on corporate chatbots to spoon feed them information that may or may not be biased in a way against your interest. Now imagine if ChatGPT tells you to use azure and Gemini tells you to use GCP…

It is nice when it works, but sometimes I run into trouble where I don't know a right word to put in the prompt to get the answer I'm looking for. I've recently been playing around with Raku and had a really cryptic type signature error and Claude was of absolutely no help because it didn't know about the interaction of Signature literals and 'Slurpy' sigils in method parameters. Only when I learned about it and included the word Slurpy in my prompt would it actually regurgitate the information I was looking for but at that point I already knew it.

> Am I the only one using LLMs as if they were a search engine?

Google agree with you and now at the top there is a AI generated answer, clearly labeled as AI generated, and it cites the sources. I was trying to escape AI, but I have to recognize the implementation by Google is quite good.

This, and I use them for code review and rapidly generating prototypes that I heavily edit. Almost none of the LLM code survives usually. You could ask "Why dont you just write it yourself then?" but sometimes getting started with the skeleton of a working project is the most difficult part of it.

This plus easier search of poorly/not-at-all documented APIs is like 80% of my usage too. Besides that, a lot of “here’s my design for xyz system, am I a stupid idiot for using this architecture?”.

Yup this is where 90% of the productivity benefits come from for me. Instead of needing to spend an hour scouring documentation I can ask an LLM and have an answer in 5 minutes

Best metaphor I have found to how I use them is as "a hunting dog".

They can get into small crevasses and the foliage and whatnot and they don't mind getting wet. They can fluster rabbits out. And are somewhat smart. But you still have to make the kill and you have to lead the dog, not the other way around.

"Copilot" is a great marketing name, but a bit deceiving.

Even when using them to code, I use them as a search engine. Rather than telling them to implement feature X, I clone a repo which has a similar feature and say:

"explore the repo I've cloned at /src/foo and explain how it achieves barFeature. Now look a the project at /src/baz and tell me why it would be difficult to use foo's approach in baz"

I rarely have it do anything novel, just translate ideas from existing projects into mine. Novel work is for me to enjoy coding directly.

i'm currently using it in large part as a question asking tool. feed it a bunch of documents, then ask it to ask me questions to figure out whats been missed -- stuff that i probably know, or could find out, or look for more documents that have the answer, but arent clear from the documents i had handy.

this afternoon i spent some time feeding context, with the idea of making a QnA context for maybe 50 different website routes, so i can have it ask me similar questions per route, so i can have a hell of a lot of pretty accurate documentation for an intern to pick up a couple to work on, or get some LLM to take a try at filling out the details.

im feeding it knowledge, with the output being documents, rather than me getting knowledge from it.

I learned about window functions in SQL using an LLM. I hadn't written SQL in over a decade and never ran across them. It explained how they work and the trade-offs. It was great!

Cursor is awesome as search engine for your codebase. Extremely helpful if you are not familiar with it.

My main use for LLMs is to ask it to give me some boilerplate configuration for some library/tool I am not very familiar with and then look up the docs for the options it spits out. Like:

"give me the terraform configuration for an AWS ECS cluster with 2 services that can talk to each other, with one of them available publicly and the other one private"

Occasionally to give some small self-contained algorithms, for example:

Give me a way to format the difference between dates as human-readable text. For example: 1 day, 16 hours.

> Am I the only one using LLMs as if they were a search engine?

I quite like the search-first LLMs like Gemini and Copilot for this reason. They give you links which you can use to verify the output and seem to be less prone to directing you to SEO spam than Google Search/Bing.

Although I do partly think that search engines today are delivering poor performance compared to what they delivered historically so LLMs are benefiting from that.

It's kind of scary how good it is. I haven't completely switched over, but I think a lot of that is just not wanting to admit that this is the new paradigm and the implications. The ability to not only find what you're looking for faster, but have it tailored to your specific context? It's hard to go back from that.

My most common use is to treat it like one of those recipe clippers that grabs a recipe out of an overly long blog post. Last weekend I punched '28 years after credits?' into Kagi and got back only what I wanted. Finally I can have a recipe clipper for any topic. I'm happy with it.

My LLM use case preference is exactly that - a google replacement. I've modified my prompts to ask for links to direct source material, thereby allowing me to go deeper. I find this in no way different then a search engine.

The downside however, is that at least 50% of the links cited no longer exists. This points to a general issue with LLMs, temporal knowledge. I tend to at least go to archive.org on a per-link basis when so inclined.. But clearly that's not ideal.

Does anyone have a better solution, short of just asking the LLM to remove the extra step, and link to the archive.org cache for the ingestion date (which frankly, I also don't have faith in being accurate).

That's basically how I use it though I cannot wait for Gemini to be a click away in my browser with the current screen/page(s)/tab(s) embedded so I can ask it stuff about the current long article/documentation page. We're becoming very, very lazy.

> Am I the only one using LLMs as if they were a search engine?

That's basically how I use them.

I tell to myself to use them as "teachers", not "interns", i.e. ask them questions to guide my process or look for the sources of knowledge needed to understand or do something, instead of asking them to get things done (except tedious, simple tasks).

Nope, I do this too (most of the time.) I don’t like working on code I don’t understand. I have started to ask it to use a client API I’ve written to figure out how clients would work with the stuff I write though. It’s great.

I also treat chatgpt as analogy to google. works relatively fine.

I think one of the major issues I (and others) have with LLM's as search is they are heavily biased towards outdated data. The other day I was trying to have it get me the most updated versions of X, X, X from an old project. I tried several providers and all of them without exception gave me an "upgrade" to an older version than was in the project for at least 1 item(by years). I'm sure they are choosing LTS releases over the new ones because that is most popular overall in a dataset. However no matter how "hard" I prompt they keep giving me those old ones over "latest"

Same. By appending a ? to a query, Kagi returns a summary with references. Quick and easy.

This work great for me too, when learning something new the LLM brings in all sorts of perspectives and contexts I don’t know about yet.

It pulls in info from the docs, forums, code samples, shows me how things are connected, and may even suggest why something is good or bad.

Plenty of opportunity to look up docs I didn’t even know about yet.

It's a suggestion engine. An autocomplete (I am not saying GPT is just autocomplete of course it is more advanced) but end if the day you need to verify everything it does right now. May not be true in the future though.

Search finds you sources then you can decide if you trust them. AI generates code and you have no idea if what it generated was perfect, slop or almost perfect but bad in some major way.

For really basic overviews of different technologies I've found YouTube videos to be useful.

For code examples I've always found GitHub code search to be super useful.

I haven't really found a use case for LLMs where it's any faster than the research tools already available on the internet.

A good example is writing AWS CDK code, which is a pain in the ass. But there are so many examples to pull from on GitHub code search that are being used in real projects that I've found this method to be faster than prompting an LLM that may or may not be correct.

I just used an LLM to sleuth out a concurrency issue earlier - I could see what was happening in the debugger, I just couldn’t really see why - asked the LLM to talk me through lifecycle stuff, and boom, first thing it brought up was apparently the answer. Thank you, glorified fuzzy search! Much quicker than pouring through the docs.

I got lucky sure, and sometimes I don’t get so lucky - but it works often enough to have become a part of my regular approach at this point. Why is the thing broke? Maybe the robot knows!

Nope! I use llms like I used to use Google and StackOverflow. Really great to bounce ideas off of, brainstorm, yes save time too ... But the at the end of the day, what will separate programmers in the future is those that bother to put an ounce of effort into things, read, and learn. Versus those who want the easy button and answer given to them. They're going to be the next generation of people who leak your passwords, social security numbers, PII, and create massive global security incidents.

To be clear. AI doesn't kill people. People kill people. In this case, lazy people. Welcome to Idiocracy my friends.

Well now the search engines have LLMs too, which are often useful, even if only to avoid cookie popups.

I use perplexity when I want sources for my info, chatgpt if I want a quick overview.