Thoughts on AI

This is not a fully baked collection of thoughts that I spent a ton of time crafting. It is just a simple list of why I, as a software product professional, dislike the current state of AI (Artifical Intelligence) in the tech industry:

1) It burns too many resources for the value it offers - We are burning a massive amount of electrical power to run all the AI training and apps. We are doing so at a time when we need to be improving the energy management across our world. If AI could improve its energy efficiency, I'd be more open to it.

2) Technology companies are inappropriately using it as product branding - Tech products matter because they solve problems. The actual tech used to solve the problem is not the selling point of good solutions. No consumer truly cares if the SaaS app they love is coded in React or Angular. Yet tech companies are promoting new features of their products by waving the flag of "AI-Driven!". This is silly. Sell products because of what they do, not because of what tech you use to make them do it. If you have to call out "AI" to make it sell, you didn't really create a compelling feature.

3) We simply don't need it - I've always been a bit of an oddball in the tech world because I question whether it is improving our world. I think time has proven my skeptical instincts correct as we see the negative impact of social media and misinformation campaigns across our society. I believe we need to be asking whether technology improves our lives or not when we develop new products. And I don't see many things coming out of the current AI excitement that meet that criteria of being an overall improvement. Sure, they are clever, and maybe even helpful. But are they better? Are the results better or just faster? Are the people working in jobs that are moving to AI better for it? Are the people who receive the work product created by AI better for it? Or are they just clever shiny new toys? Do we really need such things, especially if we look back to my first point of the energy cost to build them?
I am not saying AI should not exist - it know that it can improve the world when applied correctly. There are use cases where those questions can be answered with an emphatic "Yes, this is better in all ways." My favorite positive example was a hearing aid I head about that can allow you to focus on one person in a crowd and filter out all noise other than their voice. Yeah, sign me up for that one. And find more products like that which solve a real problem.

But most companies putting out AI-driven products do not fall into that category. I'd encourage everyone in this industry to put much more thought into their usage of AI than what I'm seeing today.

Six months ago, I was not sold on AI. Most of my rationale still holds true, but at the same time AI is advancing to the point where being a luddite about it is not realistic as a tech professional.
So to balance the professional need to understand how to use LLMs in software development, but without sending either my data or dollars to one of the AI platforms, I have spent the last week or so working with running different models through Ollama, and integrating them into VS code using continue.dev.

My first lesson was... upgrade my memory. I was running with 32GB RAM, which was enough for smaller models, but it crashed trying to run a 40B model. So I upgraded, and it now works.
FWIW, I'm running a Nvidia 3070 card, which is decent but a couple years old - it can handle running 40B models, but the responses are slow. So I am learning which models deliver what levels of quality, at what speeds, and am adjusting my work accordingly.

My findings so far have led me to only use the chat UX:

• Autocomplete is annoying. You need to use a small model for it to be fast enough to not be disruptive to your flow. And the small models are only helpful for trivial code snippets that you either could just type out yourself with little hesitation, or could throw at the chat UX to get a quick answer. At the same time, the UX was just... bad. If you did not want the answer it gave, you should be able to just type over it, ignore it, escape it away, or something. No, it injected itself as test and whitespace, so it took work to ignore a bad autocomplete. Presumably, this will get better in the future.
• Editing file directly - Again, no. It is not reliable enough, and I spend too much time fixing my files, even if 90% of the code is decent.

What is making it useful for me is to have many models configured, so I can tailor the responses I get to what I need. continue.dev has an easy drop-down in the chat UX, so I can easily target a specific model for each thing I ask, and copy/paste resulting code when appropriate:

• Small, easy questions, such as some syntax I forgot are easily answered by a 1.5B model. Almost instant replies.
• I switch to a 14B model for most questions that are working at a file level - I'll write up pseudo-code or specs, highlight it, and ask it to deliver code for me. It works a little slow - maybe 2-3 minutes for a response, but that is just fine when it would have taken me 5-10 to get my head wrapped around the right solution. Most of the time, I take about 50% of the code it produces, and overall it is helpful and speeds things up.
• I will run a larger model when I want more accuracy on a more complex feature. The response will be slow enough that I'll kick it off, then do something else for 10-15 minutes. While this sound glacially slow compared to the Web UIs provided by AI platforms, it works fine for me. I spend a few minutes thinking through a feature, I go take a break (or write up a blog post like this), and when I return I have a first draft of code to look at and get rolling on the feature. It isn't speedy, but it uses my breaks in my day to be productive. As an example, GPU is currently churning on writing up a web page that will import a wordpress XML file into this web site. Nothing in the code is terribly difficult, but I just don't want to write it. So I spent 5 minutes detailing exactly what I wanted to code to do, how to map XML fields into the JSON object I need, etc. I want to copy/paste its results, so I don't mind waiting the extra 10 minutes to get code that is going to be more like 90% useful vs. the smaller models.
• Sometimes I find that it is not code that I need, but general direction on a completely novel feature. Even having written code for decades now, not every features springs fully baked from my noggin. I certainly could spend an hour thinking through the design, but AI can speed that up. If I post the goals of the feature into a 14B model, that 2-3 minutes it takes to give me an approach to the problems saves me 20-30 minutes of brainstorming. It rarely is correct, but it is a first/worst idea that I can react to, which is more effective than a blank slate. I point to the new DeepSeek models for this, as I'm seeking the thought process about the approach, and DeepSeek gives me that process. I almost don't care about the final answer in these cases - it is the thinking that matters to me.

I would like to encourage people to find the middle ground between "AI can do everything - anyone not using it to help with all their tasks is doomed", and "AI is all hype, just keep doing what you always have."

The truth is that most things life have a healthy middle ground that work better than living at any extremes of opinion. AI is no different. It cannot do everything. It can be a good partner for brainstorming ideas or some basic research. It has accuracy problems and sometimes is so far off-track that what it says is useless, so you cannot rely on it. Yet when you are still hashing through ideas in your mind, it provides a decent sounding board and might help you refine your thoughts.

I don't know if there is a "correct" level of us for AI, but I do know that if you either always use or never use it, you are not doing it right.