What do gguf, q4ks, A3B mean?

If you have ever tried downloading a local LLM, you have probably seen model names that look like this: 

Qwen3.8-27B-A3B-It-2507-gguf-q2ks-mixed-AutoRound

At first, it looks like meaningless technical shorthand. 

It isn’t! 

Every part of that name tells you something about the model: how large it is, how it is built, how much of it is used at a time, how its weights are stored, and what format the file uses. 

Once you understand those pieces, choosing a local model becomes much easier. 

1. 7B, 14B, 35B… How Large Is the Model?

The first number you usually see is the model’s parameter count. 

The B means billion. 

So: 

• 7B = 7 billion parameters 
• 14B = 14 billion parameters 
• 35B = 35 billion parameters 
• 70B = 70 billion parameters 

Parameters are the learned values that make up the model. 

For local AI, parameter count matters because a larger model generally requires more memory to run. 

But there is an important complication. 

A model with 35B parameters does not necessarily use all 35 billion every time it generates a token. 

That brings us to MoE models. 

2. MoE: Does the Model Use Everything at Once?

There are two broad types of models you’ll encounter: 

Dense models and Mixture-of-Experts (MoE) models. 

A dense model uses essentially its entire parameter set for each token. So, a 35B dense model uses roughly all 35B parameters during inference. 

An MoE model works differently. 

It contains a much larger pool of parameters, divided into different experts. A routing mechanism decides which experts should be used for a particular token. 

This means an MoE model can have a large total parameter count without using all those parameters at once. 

And that is where the next part of the name comes in. 

3. A3B: How Many Parameters Are Active?

You might see a model called: 

35B-A3B 

The first number still means: 

35B = 35 billion total parameters 

The A3B tells you approximately how many parameters are active for each token. 

So: 

35B-A3B 

means roughly: 

35B total parameters → 3B active parameters per token 

The A refers to the activated parameter count. 

This is why an MoE model can have a large total parameter count without requiring the same amount of computation as a dense model of the same size. 

For example: 

35B dense 

→ 35B parameters active 

35B-A3B MoE 

→ 35B parameters available 

→ ~3B active for each token 

The model still has 35B parameters. A3B does not mean the model is a 3B model. 

4. Base vs Instruct: How Was the Model Tuned?

You may see two versions of the same model labelled something like: 

Qwen3.5-35B-A3B-Base 

and 

Qwen3.5-35B-A3B-Instruct 

The difference is how the model was trained after its initial pretraining. 

A base model is the raw pretrained version. It has learned patterns from its training data, but it hasn’t been specifically tuned to behave like a helpful assistant that follows user instructions. 

An instruct model has gone through additional training, commonly called instruction tuning or instruction fine-tuning, to make it better at following commands, answering questions and carrying out tasks in a conversational format. 

So, broadly: 

• Base model → learns to predict and generate text 

• Instruct model → further tuned to follow instructions and interact with users 

This means the two versions can have the same architecture, parameter count and quantization, while behaving quite differently. 

For example: 

35B-A3B-Base-Q4 

and 

35B-A3B-Instruct-Q4 

can both be 4-bit versions of the same underlying model, but the Instruct version is generally the one you’d want for a chatbot or general interactive use. 

The important thing to remember is that Base vs Instruct has nothing to do with model size or quantization. 

It describes how the model was trained to behave. 

5. FP16, BF16: How Precisely Are Those Parameters Stored?

Now we have established how many parameters the model contains. 

The next question is: 

How much information is stored for each parameter? 

This is where you’ll see terms such as: 

FP16 and BF16 

Both use 16 bits per value, but they represent those values differently. 

 For example, a model with 35 billion parameters stored at 16 bits requires roughly: 

35B × 16 bits ≈ 70 GB 

just for its weights. 

That is far too much for many consumer machines. So people compress the weights. 

6. Q4, Q5, Q6, Q8: Quantization

This is where Q4, Q5, Q6 and Q8 come in. 

These are different levels of quantization. 

Instead of storing model weights using 16 bits, quantization stores them using fewer bits. 

You will commonly see: 

Q8 → roughly 8-bit 
Q6 → roughly 6-bit 
Q5 → roughly 5-bit 
Q4 → roughly 4-bit 
Q3 → roughly 3-bit 

The lower the number, the smaller the model generally becomes. 

That can make an enormous difference. 

A 35B model at 16-bit precision is roughly: 

70 GB 

At roughly 4 bits per weight, the same model is closer to: 

18 GB 

The exact size varies because real quantization schemes have additional metadata and don’t always use exactly the nominal number of bits for every value. 

But the principle is simple: 

Lower-bit quantization reduces memory requirements, usually at the cost of some model quality. 

You may now encounter something like: 

Q4_K_M

You already know what Q4 means: it is a 4-bit-class quantization. 

But what are K and M? 

They identify the specific quantization scheme. 

Modern quantization methods don’t necessarily store every weight in exactly the same way. They can use different groupings, scales and precisions to achieve a better balance between model size and quality. 

That is why you’ll encounter names such as: 

• Q4_K_M 
• q2ks (Same thing just with underscores removed)
• Q6_K_s 
• Q8_0 

You don’t need to memorize the implementation details of every variant. 

For most users, the useful information is: 

Q4_K_M = a commonly used 4-bit-class quantization designed to balance size and quality. 

So when comparing two versions of the same model, Q4_K_M and Q6_K, you’re primarily comparing different quantization levels and schemes. 

8. GGUF: What Is the File?

Finally, you may see: 

GGUF 

This is different from everything we’ve discussed so far. 

GGUF is a model file format. 

It tells the software how the model is packaged and stored. 

That means a filename like: 

Qwen3-30B-A3B-Instruct-2507-q2ks-mixed-AutoRound-gguf

Can be read as: 

Qwen3 → which model
30B → how many parameters exist
A3B → how many are active per token
Instruct → how it was tuned
2507 → version/date identifier
gguf → container/file format
q2ks → quantization format
mixed → not every layer gets the same bit width
AutoRound → quantization algorithm

That’s the entire “alphabet soup.” 

Putting It All Together

Now take the scary-looking filename again: 

Qwen3.5-35B-A3B-Q4_K_M-GGUF 

Read it from left to right. It is basically a spec sheet compressed into one line. 

The Cheat Sheet

Frequently Asked Questions