Why we created HivenetQuant

An open model doesn't go straight from Hugging Face into production.

Before it can run efficiently on a particular GPU fleet, someone has to make a series of decisions. How much precision does the model need? Which layers can be compressed safely? What happens to coding, reasoning, vision, or tool use when they are? Does a smaller checkpoint actually run faster on the hardware that will serve it?

We make those decisions for the models we run at Hivenet.

Now we're starting to publish the work behind them.

HivenetQuant is our new Hugging Face account for Hivenet's model-optimization work. It is where we will publish optimized model checkpoints alongside the benchmarks, serving configurations, and reproducibility information we use to evaluate them.

Our first releases focus on NVFP4 quantization for NVIDIA Blackwell GPUs. The goal is practical: make strong open models easier and more efficient to run while measuring what changes in the process.

That last part matters most.

A model that uses less memory isn't useful if the compression quietly damages the capabilities you needed in the first place.

Quantization is part of deploying a model

Large models are usually trained and released at relatively high numerical precision. That helps preserve accuracy, but it also means more memory use and more data moving through the GPU.

Quantization reduces that precision.

A 16-bit weight, for example, can be represented using four bits instead. Applied across billions of parameters, the difference is substantial. Smaller weights can make room for larger models, longer contexts, more concurrent requests, or fewer GPUs.

The difficult part is deciding what can safely become four-bit.

Different parts of a model respond differently to reduced precision. A uniform recipe can work well for one architecture and poorly for another. It may preserve general knowledge while hurting code generation, or reduce model size without producing the expected speed improvement.

So we don't treat quantization as a file-conversion step.

For the models we optimize ourselves, we measure which parts of the network are sensitive, choose precision accordingly, rebuild the model, and then compare the result directly with the original weights.

HivenetQuant is where we make those decisions visible.

‍

‍

Starting with NVFP4 and Blackwell

Our current work centers on NVFP4, NVIDIA's four-bit floating-point format for Blackwell GPUs.

NVFP4 stores values using a four-bit floating-point representation while using finer-grained scaling to preserve numerical range. Groups of 16 values share an FP8 scale. The format is supported directly by Blackwell Tensor Cores, including the RTX 5090 GPUs we use for this work.

That combination makes it interesting for inference. It reduces memory requirements while allowing the hardware to perform the lower-precision calculations directly.

But choosing NVFP4 still leaves an important question unanswered.

Where should we use it?

Our first published Qwen3.6-27B checkpoint is a good example.

What we changed in Qwen3.6-27B

For HivenetQuant/Qwen3.6-27B-NVFP4, we don't simply quantize every available layer to the same precision.

The model's larger MLP layers use NVFP4 for both weights and activations. More sensitive attention and recurrent DeltaNet layers remain at FP8. The vision encoder and MTP speculative-decoding head stay in BF16, along with several other components that we deliberately leave untouched.

The split comes from measurement.

Our pipeline estimates how much error each layer introduces when reduced to NVFP4. It then considers how much of that error GPTQ can recover during reconstruction and keeps higher precision where it produces the largest accuracy benefit.

The result is a mixed-precision model rather than a blanket four-bit conversion.

That distinction would mean little without testing it afterward.

So we compare the resulting checkpoint with the original FP16 model using the same prompts, evaluation harnesses, decoding settings, and serving environment.

We care less about producing an impressive standalone benchmark score than answering a more useful question:

What did quantization change?

The losses are small, but they aren't uniform

Across the tasks we tested, Qwen3.6-27B stayed close to the original FP16 model.

A few examples:

‍

The average tells part of the story. The individual tasks tell us more.

Knowledge, competition mathematics, vision reasoning, and long-context retrieval remained close to FP16 in these tests. Coding was more sensitive.

We think that difference is worth publishing.

A small average regression can hide a larger change in the workload somebody actually cares about. If you're deploying a coding model, a four-point coding loss matters more than an average assembled from mathematics, knowledge, vision, and unrelated tasks.

It also gives us something concrete to improve.

Quantization isn't useful to us as a competition for the smallest possible checkpoint. The relevant trade-off is the amount of efficiency we gain for the capability we give up.

Smaller models can be faster, but the workload matters

We also benchmark the optimized checkpoints on the hardware they are designed to run on.

For Qwen3.6-27B on RTX 5090 GPUs, the NVFP4 version reached 116 tokens per second in our single-stream 1K-context decode test, compared with 83 tokens per second for FP16 under the same configuration.

With ten concurrent requests, aggregate output throughput increased from 484 to 688 tokens per second.

Both are improvements of roughly 1.4×.

Prompt processing behaved differently. With a 1K-token input, the speed difference was small. At a 10K-token context, prefill performance increased from 6,266 to 8,701 tokens per second.

This is why we avoid describing quantization with one universal speed number.

Context length, batching, memory bandwidth, model architecture, and the kernels doing the work all affect the result.

The same lesson became even clearer when we started testing different model architectures.

One recipe doesn't fit every model

Our initial dense-model work suggested that reducing both weights and activations to four-bit could give us a useful combination of memory savings and speed.

That held for dense models such as Qwen3.6-27B and the Gemma 31B model we tested.

Mixture-of-experts models changed the answer.

An MoE model can contain tens of billions of parameters while activating only a fraction of them for each token. That changes where the computational bottlenecks are.

In our tests, reducing activation precision on these models provided much less benefit. The additional quantization and kernel overhead could outweigh the work saved.

For the MoE architectures we've tested, a hybrid approach using four-bit weights with higher-precision activations produced the better trade-off.

That changed our pipeline.

We still run sensitivity-based precision selection where the measurements show that it helps. For packed-expert MoE models, we use a more direct hybrid recipe instead.

There is no reason to run an optimization because it sounds sophisticated. It should earn its place in the results.

This is one of the broader reasons we created HivenetQuant. Model optimization becomes much more useful when it is treated as experimental engineering rather than a fixed recipe.

Calibration matters too

The model architecture isn't the only variable.

Post-training quantization needs calibration data to determine how the original numerical values should be represented at lower precision.

The calibration set therefore influences the checkpoint.

We found that calibrating on a single type of material did not transfer well enough across the range of workloads we wanted to preserve. Our current mix includes web text, encyclopedic material, code, mathematics, and conversational data.

We also check that this calibration material doesn't overlap with the benchmark questions we later use to evaluate the model.

That separation is easy to overlook.

If evaluation material leaks into calibration, the resulting benchmark can make a quantization recipe look better without giving us a better picture of how it will behave on unseen work.

Good model optimization depends as much on measurement discipline as it does on numerical formats.

Publishing the evidence with the checkpoint

This is where the Hugging Face account becomes important.

We don't want HivenetQuant to become a collection of model names followed by claims that they are "optimized" or "near-lossless."

For the checkpoints we publish, we want developers to be able to see how those conclusions were reached.

That means publishing the precision choices, comparisons against the original model, serving configuration, and the information needed to reproduce the evaluation.

Our Qwen3.6-27B model card, for example, includes the FP16 comparison across knowledge, reasoning, coding, vision, tool use, long-context retrieval, and speculative decoding. It also documents the vLLM configuration used for serving and points to the detailed reproducibility setup.

The benchmark includes the regressions as well as the wins.

We think that is the useful way to publish quantized models.

If coding gets worse, the card should say so.

If a format saves memory but doesn't improve a particular phase of inference, that belongs in the results too.

And if a future model needs a completely different recipe from the one that worked last week, the process should be able to discover that.

What comes next for HivenetQuant

HivenetQuant is new, and the account will grow alongside our inference work.

We will use it to publish optimized open models that we have built or independently verified, together with the measurements behind them.

Some will use NVFP4. Different hardware and model architectures will lead us to other approaches over time. The format itself isn't the point.

We are interested in the layer between an open model release and the system that eventually serves it: precision choices, hardware behavior, performance, reproducibility, and the small engineering decisions that determine whether an efficient model still behaves like the model you intended to deploy.

Most of that work usually stays inside infrastructure teams.

We think some of it is worth publishing.

You can follow the models and benchmark results on HivenetQuant on Hugging Face.