When ChatGPT came out there was a real moment for most people who use computers daily about what compute was really capable of. A lot of manual typing and hard graft creative process disappeared overnight for slop. I have learned more since then about LLMs but for a lot of people I think the magic isn't quite realised, especially for domain specific and problem-specific interaction like locomotion.
I wanted to start to take advantage of where the frontier was really reaching ahead, inference and programming. It's crazy to me that my computer is generally capable of a fair amount of decision making and coordination in large part on its own, how capable is the hardware that I have today? I have seen people run LLMs on an esp32. Surely my computer can run similar model performance seen in Fable, GLM, Qwen locally if I'm willing to wait. In my simple view of the world, it's a lot of matrix multiplication that can be done on the CPU too, it's probably just a lot slower.
I thought I'd have a go at some local AI models first, before I even consider spending some money on something. I tried some of the earlier image generation models like stable diffusion but the tech in AI moves fast.
Personal Compute in general in my view has missed a couple upgrade cycles to datacentres and before technology got expensive, my current personal computer has 32GB of RAM and a 6GB VRAM 1660 NVIDIA graphics card, for AI this is not a lot to work with. I went out and bought the latest NVIDIA 5060 Ti 16GB card.
I had a play with a few models, Gemma was probably the most capable. When I started working with AI, MOE models started gaining popularity, I strongly believe there is a better model that can be run locally on my hardware but the selection currently of essentially Gemma E4B, Qwen3.6-35B-A3B is limiting. I've heard Qwen3.8 is on the way (I've used it now - glad to see it lived up to my expectations).
I think there may be some alpha to mixture of expert models and I could see it bringing down the cost dramatically for those of us who don't have a great amount of VRAM at our disposal. If a fast storage medium costs as less than it does to buy a specific AI Graphics card surely that is a workable solution.
I am also interested in how AI models could be baked into silicon, if we can put an AI model baked into a chip on a phone or something, that intelligence unlocks so much potential near realtime 'intelligence' at the cost of a static model. We've already seen Google make dedicated hardware decisions around AI and I think we'll start to see this be incorporated into CPUs or handed off to tensor processor cores in the near future.
I am left wondering what the cost of compute will be in the future, we are already seeing the cost of computing go up as server based hardware is already dedicated to compute datacentres and increased consolidation of hardware through so called capex based thinking. I have committed to a new graphics card, and I think it may be time to start thinking about RAM, how far can this drought of consumer chips last?
AI and its improvements in every day life are not fully understood, I do think for software generation and speed of development, their improvements speak for themselves, being able to leave my computer to it, and have my ideas thought about for a couple minutes and come back to a mostly sufficient implementation makes programming for me much more enjoyable.
After playing with a few models, I can sometimes get the job done using Qwen.36 A3B 32B, it's around 10 tokens a second handing most of it off to the CPU, but I've not tried running the model on an NVME SSD yet, I still reckon I can eek out a bit more performance using a stronger hardware setup and find something that is usable and is both smarter and quicker, whether it exists yet I am not sure.