Local AI on Omarchy OS – Easy Web UI, Local Models & GPU Acceleration

In this video, I show a simple way to run local AI on Omarchy OS using a clean web-based interface, instead of relying on command-line tools or cloud services.

https://buymeacoffee.com/gnugent

This isn’t about training models or hardcore AI development.
It’s about making local AI easy to use on the desktop, in a way that feels familiar if you’re used to tools like ChatGPT or Claude — but running entirely on your own machine.

⸻

🔹 What This Video Covers

I walk through a script I’ve written that:
• Installs a local AI backend
• Adds a web UI on top of it
• Detects your discrete GPU automatically
• Ignores integrated GPUs when appropriate
• Works across NVIDIA, AMD, and Intel (with caveats)
• Uses simple keyboard shortcuts to start and stop everything

The aim is to make local AI something you can actually use day-to-day, not something hidden behind terminals and config files.

⸻

🔹 Why Local AI?

Sometimes you don’t want to:
• Send files to the cloud
• Rely on an internet connection
• Share private data

Local models let you:
• Work offline
• Point the AI at local files
• Experiment freely
• Keep everything on your own hardware

This setup makes that practical.

⸻

🔹 Models & Hardware

Local AI performance depends heavily on your GPU:
• 16GB GPUs are the sweet spot
• 8GB GPUs are usable but limited
• Tested on:
• Intel Arc 12GB
• AMD RX 9060 XT
• NVIDIA RTX 5060 Ti

All testing in this video is done on desktop systems.
Laptop behaviour may vary due to hybrid graphics.

⸻

🔹 Using the Interface

Once installed:
• A keyboard shortcut launches the web UI
• Another shortcut cleanly shuts everything down
• Models can be downloaded and swapped easily
• You can run multiple models depending on VRAM
• Tokens per second and GPU load can be observed live

I demonstrate this by:
• Loading different models
• Explaining scripts with AI
• Showing GPU usage during inference
• Using a multimodal model to analyse images

⸻

🔹 Intel, AMD & NVIDIA Notes
• AMD works well using ROCm
• NVIDIA works as expected with CUDA
• Intel works via an additional compatibility layer
• Some models work better than others
• Gemma models behave well
• Some larger models are not compatible

These limitations are explained clearly in the video.

⸻

🔹 Why This Matters

This setup gives you:
• Cloud AI when you want it
• Local AI when you need it

Both can live side by side on Omarchy, giving you flexibility without changing your workflow.

⸻

💬 Final Thoughts

This is about lowering the barrier to local AI.

If you’ve been curious about running AI on your own machine but didn’t want to fight with command-line tools, this is a simple, practical way to get started.

⸻

🔔 Subscribe to Linux Out of the Box for more Omarchy tools, AI workflows, Linux gaming, and real-world desktop setups.