The 3-Layer Hack That Doubles LLM Reasoning (Without Retraining)

Discovering Hidden "Thinking Circuits" in Transformer Models

A new open-source discovery reveals that transformer models contain discrete "reasoning circuits"—and duplicating just 3 specific layers boosts logical reasoning from 0.22 to 0.76 on standard benchmarks. No retraining. No weight changes. Just routing hidden states through the same circuit twice.

This is the kind of "why didn't anyone think of this before" discovery that changes how we understand language models. The research, built on David Ng's RYS method, shows that transformer models organize into contiguous blocks of 3-4 layers that act as indivisible cognitive units. Find the right block, duplicate it, and reasoning capability jumps dramatically.

The Discovery: Contiguous Layer Blocks as Cognitive Units

Researchers replicated the RYS (Rank Your Steps) method and found something unexpected: transformer models have sharp boundaries between functional regions. Shift a layer by one position and the effect disappears. But hit the exact circuit, and capabilities transform.

The discovery locations vary by model:

These aren't arbitrary ranges. They're specific contiguous blocks where the model has concentrated its reasoning capabilities. The boundaries are precise—move one layer in either direction and you lose the effect.

The Results: Massive Gains Without Training

The performance improvements are striking:

The BBH Logical Deduction result is particularly notable—a 245% improvement on a standard reasoning benchmark without any parameter updates. The model isn't learning anything new. It's just using its existing reasoning capability twice.

Nothing degraded. All improvements came from routing hidden states through the same circuit twice, effectively giving the model more "thinking time" within its existing architecture.

Different Patterns Unlock Different Capabilities

The research revealed that how you apply the duplication matters as much as which layers you duplicate:

Double-pass routing boosts mathematical reasoning. Running the same circuit twice in sequence improves performance on math-heavy tasks like GSM8K.

Triple-pass routing boosts emotional reasoning. Three passes through the circuit improve performance on sentiment and emotional intelligence tasks.

Interleaved doubling creates pure math specialists. Alternating between the reasoning circuit and other layers produces models optimized specifically for mathematical reasoning.

Same model. Same VRAM requirements. Different routing patterns produce different capability profiles. This suggests that transformer models have more latent capacity than we typically access—a kind of "dark matter" capability that can be unlocked through architectural tweaks rather than training.

What This Means for Open-Source LLMs

The implications for developers building with open-source models are significant. This discovery suggests we may be dramatically underutilizing existing models. A GGUF model running on consumer hardware could potentially match GPT-4's reasoning capabilities on specific tasks—just by routing through the right layers differently.

The open-source tooling (github.com/alainnothere/llm-circuit-finder) makes this discoverable for any GGUF model. Developers can:

This levels the playing field. You don't need a $100M training run to improve model performance. You just need to understand the architecture you already have.

The Architecture Implications

Traditional thinking about transformer scaling has focused on depth—adding more layers, more parameters, more compute during training. This discovery suggests that how we use existing layers matters as much as how many layers we have.

The "contiguous block" nature of reasoning circuits hints at how transformers actually process information. These aren't uniform stacks of identical operations. They're specialized pipelines where different regions handle different cognitive functions.

Future model architectures might be designed with explicit "reasoning loops"—built-in mechanisms for running specific layer blocks multiple times when facing complex problems. Instead of hoping the model reasons correctly in a single forward pass, we could give it natural "thinking time" within the architecture itself.

FAQ

How do I find the reasoning circuits in my model?

The open-source llm-circuit-finder tool on GitHub provides methods for identifying reasoning circuits in GGUF models. The process involves analyzing layer activations across reasoning benchmarks to identify contiguous blocks where reasoning capabilities concentrate.

Does this work for all transformer models?

The research has been validated on Devstral-24B and Qwen2.5-32B. The technique likely applies to other decoder-only transformers, though the specific layer ranges for reasoning circuits will vary by model architecture and training. The methodology for finding circuits should generalize.

Do I need more VRAM to use this technique?

No—this technique doesn't increase model size or VRAM requirements. You're routing through existing layers differently, not adding new parameters. The model's memory footprint remains identical.

Can this be combined with quantization?

Yes, the technique works with quantized models (GGUF, AWQ, etc.) since it doesn't modify weights. You're changing inference routing, not model parameters.

Will this be integrated into inference engines?

It's likely that popular inference frameworks (llama.cpp, vLLM, etc.) will eventually support custom routing patterns or "reasoning modes" based on these findings. Until then, the technique requires custom implementation using the available open-source tools.

Does this replace the need for larger models?

Not entirely, but it narrows the gap. This technique extracts more capability from existing models, but there are still fundamental limits to what a smaller model can represent. However, for many practical applications, an optimized 24B model may now be competitive with much larger alternatives.

What is the RYS method this research builds on?

RYS (Rank Your Steps) is a technique developed by David Ng for analyzing transformer layer contributions. The original method is documented at dnhkng.github.io/posts/rys/. The circuit discovery research extends this methodology to identify contiguous functional blocks rather than individual layer importance.

Sources: LLM Circuit Finder, RYS Method, Hacker News Discussion