I’ve always believed that to truly master any technology, you have to understand what’s happening “behind the scenes.” You need to know what moves information from point A to point B and how the “machine” actually processes data.
Lately, Artificial Intelligence has sparked such a massive wave of excitement in such a short time that very few people have had the chance to really dig into the core concepts. If you try to go even deeper, you quickly run into high-level statistics and complex mathematics, a barrier that narrows the field to even fewer enthusiasts.
However, there is a shortcut to mastering any new skill: just dive in. In the world of IT, a standard Personal Computer is often all the gear you need to get started.
But you might be thinking, “That can’t be right, surely none of us can actually run Artificial Intelligence on a home setup?“
Well, think again.
GGUF
From the website https://huggingface.co/docs/hub/gguf
GGUF format, a binary format that is optimized for quick loading and saving of models, making it highly efficient for inference purposes. GGUF is designed for use with GGML and other executors. GGUF was developed by @ggerganov who is also the developer of llama.cpp, a popular C/C++ LLM inference framework. Models initially developed in frameworks like PyTorch can be converted to GGUF format for use with those engines.
In other words, a single GGUF file can house an entire Artificial Intelligence*
(*) a language model, but let’s go on with simplifications!
LLAMA.CPP
The creator of GGUF also developed LLAMA.CPP, an efficient tool designed to run models directly on your personal computer. This software is accessible even for those with a standard CPU, requiring no high-end GPU or TPU. It’s the perfect setup for testing!
Step 1: download the model GGUF, for example “LFM2.5-1.2B-Instruct-BF16” (less than 3GB)
Step 2: download LLAMA.CPP (less than 30MB)
Step 3: run the server and enjoy 😅
llama-server.exe --model LFM2.5-1.2B-Instruct-BF16.gguf
Models and Quantization
In this famous 1994 photo, Bill Gates remarked: “This CD-ROM can hold more information than all the paper below me!” back when that meant just 700 MB. It’s a great reminder of why we shouldn’t be shocked by how much information even a small LLM can pack.
The smaller the model, the faster it performs and the easier it is to run on any hardware.
How do you downsize an LLM?
For starters, you can build a small, specialized model from scratch designed for a single task. Alternatively, you can take a massive model (the kind behind ChatGPT 😅 that consumes as much energy as an entire nation) and shrink it down through Quantization.
From the website https://huggingface.co/docs/transformers/main_classes/quantization
Quantization techniques reduce memory and computational costs by representing weights and activations with lower-precision data types like 8-bit integers (int8). This enables loading larger models you normally wouldn’t be able to fit into memory, and speeding up inference. Transformers supports the AWQ and GPTQ quantization algorithms and it supports 8-bit and 4-bit quantization with bitsandbytes.
Think of it like a low-resolution, grainy image: it’s not perfect, but you still get the picture!
As of January 2026, the Hugging Face 🤗 platform has surpassed a staggering 2.4 million models. This vast library features specialized AI tools trained to generate high-quality text, images, videos, and much more, marking a significant milestone in the evolution of open-source artificial intelligence.
Training a model for specific tasks
Let’s take a practical example. If I want to use Artificial Intelligence to write a program for Business Central 😅, I don’t need a model trained in C#, nor do I need a massive, expensive model trained to answer every possible question.
Many believe that AI models will eventually become “commodities,” likely losing the high value currently attributed to them. Hugging Face partially proves this theory, but common sense leads us to the same conclusion.
If we imagine AI as a library, or even just a bookshelf at home, it is made up of many “small” books, each with its own specific content. No one has ever thought of creating a giant book with billions of pages that is heavy and difficult to open! Despite the fact that hardware continues to surprise us every day, we will soon reach a limit that demands more sustainable AI models.
Let’s go back to the Business Central example. As partners, we all own one or more vertical solutions that sit outside Microsoft’s standard offering. If we want AI to effectively assist our developers, we must train it on the specific technical content of our own products.
Accomplishing this through “context” alone is impossible, as it is strictly limited by token counts. Instead, it requires a specific model. This is where Fine-Tuning a GGUF model comes into play: it allows us to take a generic model trained by others and teach it entirely new, specialized knowledge.
https://huggingface.co/docs/transformers/training
Ah, LLAMA.CPP works seamlessly with Visual Studio Code and Continue is a powerful open-source assistant to pair with it!
Private AI
Last but not least is the critical issue of data ownership. When sharing information with public AI tools, concerns regarding confidentiality, intellectual property, and personal privacy naturally arise.
This is where LLAMA.CPP shines: it runs locally on your own PC and functions entirely offline. For those looking to provide private AI services to employees or clients, the barrier to entry is surprisingly low. For less than $4,000, you can purchase a “small” supercomputer from NVIDIA to power your operations securely in-house.
Safe Agents
Building an AI Agent from scratch provides a deep understanding of how the LLM, the user, and various tools interact. Today, most users rely on agents based on the Model Context Protocol (MCP). However, behind the scenes, MCP is essentially a framework for the “tools” called by the LLM.
By setting up a local server, you can intercept and track an Agent’s actions at a low level. Alternatively, it allows you to develop your own innovative Agent, iterating and testing as much as you need without spending a single cent on tokens.
Conclusion
The work Hugging Face is doing is nothing short of amazing; I believe they represent the true meaning of “Open” AI. We are seeing a new wave of processors for standard personal computers equipped with TPUs capable of running LLM models at ever-increasing speeds.
AI has long moved beyond being a cloud-only technology. Day by day, it is becoming more accessible and easier to integrate locally. Seeing is believing!