The Future of AI
I was reading a post over on macrumors.com where they interview Ed Zitron who's a pretty well established AI skeptic. You can read the article here.
I have a lot of thoughts about where the technology was going and I wanted to dig in on some of what he's saying.
Consumers and enterprises alike have been trained to pay a monthly fee for a service, and while these services might have limits or strictures, basically nobody buying software expects to have a metered service, let alone one that's both metered and with hard to measure costs.
This is not entirely true. AWS has been selling services to enterprise customers entirely based on a usage model for years. That has worked well. Certainly there are stories of people blowing budgets accidentally, but by and large the Internet runs on AWS quite well.
For consumers, yes, this is true. That said, in the consumer space there's likely a lot more wiggle room for model choices. Free chat is probably going away except on a trial basis. Lower price points will get you the faster/lighter models.
That said, how can you model things to work with a monthly fixed price?
1990s Dial-Up
A good mental model for AI is the way that Internet service was sold in the 90s. You can't afford to give everybody their own modem to dial into. Instead you over-subscribe. Rather than have each mode serve one person, it serves 10, and you're assuming not everybody will dial in at the same time. Sometimes you hit a busy signal when the math doesn't work or the dialup provider pushed the limits.
AI lends itself better to this because the AI is only being used for that slice of time to answer your one question, or write that one piece of code. That time that you spend coming up with your next prompt is all free reign for somebody else to use that resource.
The True Cost of Tokens
The thing to keep in mind is that for an AI company, their cost is the capital they spend to build out the data center, and the runtime costs for that data center. For them, they want to make sure they are maximizing the revenue from that data center. That means minimizing the idle time of those processors, and getting as much revenue as they can per unit of work.
Right now if I sign up for a $100/mo plan from Anthropic, I can easily burn that much in a single day if you look at token prices. The question becomes: what's the true cost of a token.