Will LLMs Replace GPU Experts? What K-Search Reveals About Transferring CUDA Knowledge to Apple Silicon

This is also an infrastructure question. As AI shifts from a single large model toward distributed and runtime-based systems, performance depends on the combined behavior of accelerators, memory, interconnects, and execution layers—not on raw GPU throughput alone. Chinoba: AI Infrastructure & Hardware

Can an LLM write highly optimized GPU kernels without the help of GPU specialists?

And can the optimization knowledge accumulated over years in CUDA for NVIDIA GPUs be transferred to hardware with a fundamentally different design philosophy and programming environment, such as Apple Silicon?

If the condition is that “the LLM writes the code once,” the answer remains close to no. Kernel optimization depends not only on understanding the algorithm, but also on memory hierarchy, parallelism, synchronization, instructions, compilers, input sizes, and the behavior of the physical hardware.

Yet the work extending K-Search to Apple Silicon, in collaboration with UC Berkeley Sky Lab, reframes the question. The important point is not whether an LLM can produce a finished solution in a single attempt. It is whether optimization knowledge can be transferred to a different architecture by treating the LLM as an explorer that forms hypotheses, and real-hardware measurement as the basis for selection and learning.

According to BAIR’s report, an attention kernel evolved through this process reached roughly 97% of the performance of Apple’s native MLX attention kernel. A Mamba SSM kernel achieved up to roughly 20× faster prefill performance than the community MLX-LM implementation. These kernels were not the result of manual, case-by-case tuning by GPU performance engineers; they were produced by combining an LLM with measurement-driven search.

BAIR: From CUDA to MLX: How K-Search Brings Decades of Kernel Expertise to Apple Silicon

This is also an infrastructure question. As AI shifts from a single large model toward distributed and runtime-based systems, performance depends on the combined behavior of accelerators, memory, interconnects, and execution layers—not on raw GPU throughput alone. Chinoba: AI Infrastructure & Hardware

What Is a GPU Kernel?

A GPU is a processor designed to execute a vast number of small operations simultaneously. LLM inference repeatedly performs many uniform computations, including matrix multiplication, attention, normalization, and activation functions. A GPU kernel is a small program that executes one such operation in parallel on the GPU.

Consider attention. It calculates relevance scores from input vectors, normalizes them, and aggregates value vectors using those weights. If this operation is implemented as a sequence of separate steps, every stage may read from and write to memory. A high-performance kernel can execute the same mathematics dramatically faster by fusing operations, keeping necessary data close to the processor, overlapping memory loads with computation, and reducing synchronization.

The difficulty is that there is no universally fast form of code.

• When arithmetic dominates, the key question is how fully the compute units can be utilized.
• When data movement dominates, memory bandwidth and regularity of access patterns become decisive.
• For small inputs, the overhead of parallelization and synchronization can make execution slower.
• For large inputs, partitioning, pipelining, and tile size can determine the outcome.

CUDA is NVIDIA’s programming platform for GPUs. CUDA kernels embody a large body of experience in allocating threads and using shared memory, registers, Tensor Cores, and asynchronous copies. Apple Silicon, by contrast, runs Apple GPUs with unified memory and the MLX machine-learning framework and kernel environment. It is therefore natural that directly porting CUDA code does not automatically produce fast results.

Not “Code Porting,” but “Porting Optimization Intent”

The core of K-Search is not simply to rewrite source code incrementally. It separates optimization intent from the program that implements it.

For example, the search tree can retain intents such as:

• If memory bandwidth is the bottleneck, fuse operations and eliminate unnecessary intermediate outputs.
• If data will be reused, tile it and retain it in nearby memory or registers.
• If the processor is waiting for memory, overlap the next data load with current computation.
• If the input is small, reduce partitioning and synchronization to limit launch overhead.
• If the input is large, decompose work into more parallel units to keep the compute units occupied.

At this stage, the LLM forms hypotheses about the kernel’s structure, its bottleneck, and the next structural change worth trying. It then generates code appropriate to the target architecture and compiles it on the target hardware. Only candidates that pass correctness tests are measured, and their latency and throughput are fed back into the search tree.

The K-Search paper describes this mechanism as a world model that is updated together with the LLM. Even if an intermediate modification is temporarily slower or an implementation attempt fails, the underlying optimization direction is not discarded immediately. This makes it possible to pursue multi-step transformations—for example, changing the memory layout before applying vectorization. In the CUDA-to-MLX extension, CUDA kernels are not reused as finished code; the design knowledge they contain is used as prior knowledge for the search.

K-Search paper

Why 97% and 20× Can Both Be True

The two figures use different baselines, so they need to be interpreted separately.

The roughly 97% result for attention is measured against Apple’s highly optimized native MLX attention kernel. It means that automated search can come very close to an implementation refined by experts. It does not mean that 97% applies to every input shape, every Apple device, or every model. It is a result under specified benchmark conditions.

The roughly 20× result for Mamba SSM is measured against a community MLX-LM implementation. It represents an improvement over an implementation with more optimization headroom; it does not mean that it is 20 times faster than Apple’s optimized reference. The report identifies implementation of the state-space model recurrence as a parallel prefix scan as a major factor.

The significance of presenting these results together is clear. Automated search may either approach an already highly optimized reference implementation or substantially improve an implementation that has not yet been fully optimized. It is not, however, a universal code generator that always surpasses expert implementations.

What Do Similar Cases and Benchmarks Tell Us?

Looking only at K-Search could create the impression that LLMs have nearly solved GPU optimization. The broader evaluation landscape is more cautious.

K-Search reports an average 2.10× improvement over OpenEvolve on complex FlashInfer kernels such as GQA, MLA, and MoE, and a 14.3× improvement for MoE. But the paper also shows that the generated kernels do not consistently surpass expert-optimized FlashInfer kernels. This is an important practical boundary.

K-Search evaluation results

KernelBench measures whether an LLM can replace PyTorch implementations with CUDA or DSL kernels that are both correct and fast, across 250 tasks. Its importance lies in evaluating actual correctness and measured performance rather than code appearance or compilation success alone.

KernelBench

ParallelKernelBench addresses 87 more production-oriented multi-GPU problems. Here, it is necessary to consider not only computation but also data movement over NVLink, hardware topology, deadlock avoidance, and the fusion of communication with computation. A June 2026 report found that, even after three attempts, the best frontier model produced solutions that were both correct and faster than a naïve baseline for only 31% of the problems. This is a crucial counterpoint to the expectation that an LLM alone can write advanced GPU code.

ParallelKernelBench

KernelFoundry combines MAP-Elites search to preserve candidate diversity, evolution of the prompts themselves, and template-parameter optimization across both SYCL and CUDA. Here too, the central capability is not the LLM’s first answer, but a closed loop of candidate generation, validation, measurement, and selection.

KernelFoundry

Will Experts Become Unnecessary?

The answer is: not anytime soon. But the center of gravity of expert work will change.

Traditionally, experts designed kernel structures, repeatedly experimented with tile size, memory layout, synchronization, and pipelines, and analyzed the reasons behind performance. As search-based systems mature, specialists may move from writing every candidate by hand to designing the problem setting and evaluation process.

The following responsibilities will become more important, not less:

• Define target workloads and the real input distributions they must serve.
• Set evaluation criteria that include correctness, numerical error, reproducibility, power consumption, and memory use.
• Define the permitted search space and constrain optimizations that cannot safely be used.
• Detect shortcuts that make a benchmark appear fast without carrying out the intended computation.
• Revalidate selected candidates under multiple production conditions and ensure that they remain maintainable.

This resembles the decision problem of the AI-agent era. Rather than delegating the final decision entirely to an LLM, it separates hypothesis generation, execution, evaluation, and adoption—and leaves a trace for each. In GPU kernel search, compilation logs, correctness tests, measurements, input conditions, and adoption rationales correspond to a Decision Trace.

Conclusion: LLMs Are Not Only Writing Code

K-Search does not show that LLMs have suddenly and completely replaced GPU specialists.

Rather, it shows the emergence of a mechanism for reinterpreting CUDA-grown optimization knowledge through validation on target hardware and adapting it to environments as different as Apple Silicon. The source of value lies less in the LLM’s code-generation capability itself than in the loop that connects hypotheses, implementation, correctness verification, performance measurement, and the next search step.

One-shot LLM generation still exhibits high failure rates in complex GPU programming. Yet evolutionary search grounded in real-hardware evaluation is beginning to turn specialist knowledge into a reusable search asset.

GPU optimization may be shifting from a problem in which people write the right answer once to a problem in which people and AI converge on the right answer through measurement.

Chinoba

Chinoba

Intelligence as Relationship

Research Platform

founded by

Masao Watanabe

AI Systems Architecture
Decision Trace
Human–AI Coordination
Algorithmic Governance