Could AI Could Just Rewrite Every App for You on the Fly?
AI screen-monitoring tools could rewrite apps in real-time, creating custom interfaces without API access. This interesting thought experiment shows how software might transform on-the-fly to match individual needs.
The future is hard to predict, and that is probably a good thing, although it does make stock picking a bit difficult. Also, strangely enough, a lot of things we didn't predict seem so obvious in hindsight. How is that possible? Well, there's no answer to that, because we're not going to be able to predict the future in any real meaningful way, and yet we're still left with questions. Many, many questions.
One question I have is exactly how far will we be able to go with generative artificial intelligence and its ability to program. How much code, how fast, how well, how weird, and what are the unforeseen applications?
Screen/Stream
The right side of the "can AI code?" spectrum, the "yes" side, is pushing forward and getting more and more fantastical. (HINT: See "vibe coding.")
Recently I've seen some diatribes around the idea of AI completely rewriting an app on the fly. You would see one thing, the original app, and you would see another thing, your app, as created, customized, and personalized by the AI. The AI-ified app could do whatever, or look like whatever, or hide whatever.
We know the following:
- AI can write code (yes, definitely–though the question is still how well...)
- We can set up your computer so that the AI can see the screen and do things based on what it sees there. (Yes, concerning from a privacy standpoint.)
- AI does not need a programmatic interface, it doesn't need an API, to see what is in an image or understand what is happening on the screen. (Multimodal.)
Given the truth of points 1 and 2 (they are possible now) it is conceivable that the AI could rewrite the entire application you are using, on the fly, simply based on what it sees on the screen, or from a separate stream of your monitor. Thus it could present the information to you in a completely different way. The screen/stream becomes the data used in the new AI-ified version of the app.
Amazing! Terrifying! Unbelievable! These could all be true at the same time.
Speed would also be an issue. Could an AI tool essentially rewrite a screen on the fly? On the one hand, apps are getting slower and slower. Many web applications take a second or two to redraw...could that be in the realm of what real-time AI rewriting could do?
Conclusion
It's a fascinating thought experiment: what if an AI could not only look at your computer screen, the data stream that you use to do your work, to live many parts of your life (including your phone), but also recreate that stream based on what it "thinks" is best for you. Warping the semi-reality of the computer screen.
This is pretty fantastic–in part because rewriting the screen could mean that absolutely anything is possible. After all, it is only a few thousand pixels. Yes, speed is a concern, we expect screens to change quickly, be sharp and in focus, but then again, if it were possible, or even if it was just a subset or a specific app, it doesn't seem completely implausible. We know that people will try to have generative AI tools create video games on the fly, so why not apps?