FAANGineering - BlogFlock

Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned - Netflix TechBlog - Medium

2026-07-13T22:44:11.000Z

By Parth Jain, Rakesh Sukumar, Yingwu Zhao, Renzo Sanchez-Silva & Nathan Fisher
A deep dive into the engineering challenges of building a real-time service dependency map at Netflix scale — from streaming architectures and distributed aggregation pipelines to time-travel queries and the methodology that made it work.

Introduction

In our first post, we introduced the problem: engineers at Netflix needed a unified, real-time view of service dependencies to troubleshoot faster, understand blast radius, and navigate our distributed architecture. We described our multi-source approach — combining eBPF network flows, IPC metrics, and distributed tracing into physically separate graph layers that can be queried independently or merged into a comprehensive view.

That post explained what we built and why. This post is about how — the engineering reality of building this system at Netflix scale.

Here’s the truth: the first version worked perfectly… in our local environment. Production was a different story. Kafka consumers fell behind. Instances ran out of memory. Some nodes received 100x the traffic of others. Garbage collection pauses consumed more CPU than actual business logic.

What you’ll learn in this post isn’t a success story — it’s a learning journey. We’ll walk through the architecture decisions that enabled scale, the production challenges that tested those decisions, the optimization methodology that guided us through, and the lessons that apply to any distributed system. Along the way, we’ll share the innovations that made it possible to process millions of flow records per second, reconstruct topology at any point in time, and provide sub-second query responses — all while maintaining near real-time freshness.

Architecture Deep-Dive: Building for Streaming and Scale

Streaming-First: Why Real-Time Matters

Traditional service topology systems use batch processing — aggregating data hourly or daily, then storing complete snapshots. This approach works at a modest scale but has a fundamental problem: by the time you see the data, it’s already old. During a production incident at 3am, an hour-old dependency map is archaeology, not observability.

Our key architectural decision was to build streaming-first. Instead of batch jobs that process historical data, we continuously ingest flow records from multi-region Kafka streams and IPC metrics as Server-Sent Events, process them through reactive pipelines with backpressure handling, and provide near real-time topology updates — typically within tens of minutes, compared to the hours-old or day-old data that batch processing approaches provide.

This wasn’t just about freshness — it was essential for our use cases. Live events can’t wait for the next hourly batch. Incident response needs current data. Change validation requires seeing immediate impact. The architecture had to support continuous updates while handling massive scale without falling behind.

How Backpressure Enables Real-Time Processing
The streaming approach created new challenges, but also required solving a fundamental problem: how do you process millions of flow records per second in real-time without losing data when downstream systems slow down?

Traditional approaches fall short at our scale:

Unbounded queues: Simple but dangerous. Keep buffering until you run out of memory, then the instance crashes.
Drop-based flow control: Discard data when buffers fill. Fast, but now your topology is incomplete — you’ve lost connection information.
Batch processing: Process everything, but hours later. By then, the incident is over (or worse, still happening with stale data).

We needed something different: the ability to slow down gracefully under load without losing data. This is where reactive streams with backpressure became essential.

Here’s how it works: when Stage 3 can’t write to the graph database fast enough, it signals Stage 2 to slow down. Stage 2 signals Stage 1. Stage 1 signals the Kafka consumer to pause. The data waits in Kafka until downstream capacity returns.

When a downstream stage can’t keep up, it signals upstream to slow down — backpressure flows in the opposite direction of the data

Backpressure propagates naturally through the entire system. When any stage becomes overwhelmed — from traffic spikes, GC pauses, or external slowdowns — the pipeline automatically slows to a sustainable rate. No data is lost in most cases, no instances crash, the system degrades gracefully.

This is what enables “real-time” at our scale. During normal operation, we process with minimal latency. During load spikes or temporary slowdowns, we slow down rather than fall over. The data still gets processed — just a few seconds or minutes later instead of immediately. For topology updates, this trade-off is acceptable: slightly delayed real-time updates are vastly better than hour-old batch data or incomplete topology from dropped records.

The cost of this approach is complexity. Reactive streams are harder to reason about compared to traditional synchronous blocking models (we’ll discuss this more in the challenges section). But at Netflix scale, backpressure isn’t optional — it’s the mechanism that keeps the system running reliably under production load.

Multi-Layer Architecture: Physical Separation for Independent Optimization

As we covered in our first post, our multi-source approach uses three physically separate topology layers with different storage optimized for each:

Network Layer: eBPF flow logs in graph database partition — comprehensive coverage but lacks application context
IPC Layer: Application metrics in a different graph database isolated from the one for Network Layer — rich endpoint details but only instrumented services
Tracing Layer: Distributed traces in columnar storage (Parquet) — actual request paths but sampled.(We cover the tracing layer and its integration in our next post).

Flow logs and IPC metrics travel through two independently-optimized pipelines into separate graph stores, unified behind a single API

Physical storage isolation enables independent optimization — each layer has different throughput, query patterns, and evolution timelines. At query time, we execute parallel queries across relevant storage systems and merge results, providing unified views with sub-second latency while maintaining flexibility to evolve each layer independently.

The Three-Stage Distributed Aggregation Pipeline

The heart of the network layer ingestion is a three-stage distributed pipeline. This architecture solves a fundamental challenge with network flow logs: they only show individual network hops, not the true application-level connections we need to build a useful topology.

The Core Problem: Network Intermediaries

In cloud environments, traffic between applications rarely flows directly — it traverses intermediate network components like load balancers, NAT gateways, API gateways, and proxies. Network flow logs show individual hops: App A → Load Balancer and Load Balancer → App B appear as separate flows. But what engineers need is the logical dependency: App A → App B. Without resolving these intermediaries, our topology would be cluttered with infrastructure components rather than showing the service-to-service relationships that matter for troubleshooting.

The three-stage pipeline solves this:

The flow log pipeline in detail — three stages connected by SSE, with enrichment applied just before the final graph write

Stage 1: Initial Aggregation (FlowLog Ingestion Service)

Multi-Region Kafka (4 regions)
  → Filter invalid flow logs
  → 5-minute time-window batching
  → Create initial aggregators per window
  → Distribute via consistent hashing
  → Stream to Stage 2 via SSE

Stage 1 consumes flow logs from multi-region Kafka, filters invalid records, batches them into 5-minute time windows, and creates initial aggregator objects. At this stage, we’re still working with raw network hops — identifying which flows involve intermediaries but not yet resolving them. Aggregators stream to Stage 2 for resolution.

Stage 2: Network Intermediary Resolution Layer (Intermediate GraphEntity Ingestion Service)

Stage 1 Aggregators (via SSE streams)
  → Group flows by intermediary (load balancer, NAT gateway, proxy, etc.)
  → Identify pairs: (Source → Intermediary) + (Intermediary → Destination)
  → Resolve to direct edges: Source → Destination
  → Track which intermediaries were traversed
  → Aggregate metrics across both hops
  → Re-distribute via consistent hashing
  → Stream to Stage 3 via SSE

This is the key step. Stage 2 performs graph resolution:

Collect flows by intermediary: Group aggregators where an intermediary is either source or destination — creating maps of flows going TO intermediaries (Source → Intermediary) and FROM intermediaries (Intermediary → Destination)
Resolve direct edges: For each intermediary, join its incoming and outgoing flows to create direct application edges (App A → App B), combining metrics from both hops
Result: Clean application-level topology showing App A → App B instead of App A → Load Balancer → App B

This resolution happens at aggregation time, not query time, with resolved edges flowing to Stage 3.

Why can’t we do this in a single stage? The fundamental issue is data locality. To resolve App A → Load Balancer → App B into App A → App B, we need both flows on the same instance to perform the join. But in Stage 1, flows are scattered across instances based on Kafka’s partitioning. Stage 2’s critical function is to redistribute aggregators by intermediary identifier — all flows involving “Load Balancer X” route to the same instance for resolution. This is the classic map-reduce pattern: Stage 1 maps, Stage 2 shuffles and reduces by intermediary, Stage 3 performs final aggregation.

A concrete example of why a single stage isn’t enough — Stage 1 scatters flows by partition, Stage 2 reshuffles by intermediary to resolve direct edges, and Stage 3 persists the final result.

Stage 3: Final Aggregation and Enrichment (GraphEntity Ingestion Service)

Stage 2 Aggregators (via SSE streams)Flow
  → Final aggregation across time windows
  → Enrich with external data (query key-value stores)
  → Convert to graph entities
  → Persist to graph database (throttled writes)

Stage 3 performs final aggregation of resolved edges, enriches graph nodes with external data sources (application health, ownership, metadata), converts aggregators to concrete graph entities (nodes and edges with all properties populated), and persists them to the distributed graph database with controlled throttling to respect storage system limits.

Why Three Stages, Not Two?

We initially used two stages: aggregate in Stage 1, resolve and persist in Stage 2. This worked in testing but failed at production scale — Stage 2 became overwhelmed by data concentration.

The problem: intermediary resolution requires collecting ALL flows involving an intermediary on the same instance.As a result, the instances handling flow logs for popular applications and their intermediaries became ‘hot nodes’ due to significant data concentrationCompounding this, data enrichment (querying external stores for health and metadata) meant the busiest instances were also doing the most I/O.

The solution: split responsibilities into three stages. Stage 2 focuses purely on resolution and redistributes. Stage 3 handles enrichment and persistence. This graduated redistribution — distribute, resolve, distribute again, persist — spreads load across multiple instances and isolates compute-heavy resolution from I/O-heavy enrichment. Even when intermediaries see 100x typical traffic, no single instance becomes a bottleneck.

Why Server-Sent Events Instead of gRPC or Message Queues?

We initially used gRPC but it became a performance bottleneck — serialization overhead, connection pool management, and memory pressure for streaming responses consumed more CPU than business logic. Message queues added infrastructure complexity without benefit for our use case.

SSE proved ideal: lightweight HTTP-based protocol with minimal serialization, natural backpressure integration with reactive streams, and simpler connection model. The lesson: industry best practices like “use gRPC for service communication” don’t apply universally. For streaming large volumes of pre-aggregated data, lighter-weight alternatives may be more appropriate. Measure, don’t assume.

Why IPC Doesn’t Need Three Stages

The IPC pipeline mirrors the same pattern as the flow log pipeline, but needs only a single stage.

The IPC layer uses single-stage aggregation because: (1) IPC metrics are already at application level — no intermediaries to resolve, and (2) data is partitioned correctly from the start — each node receives all IPC metrics for its assigned applications via consistent hashing, eliminating the need for redistribution. This highlights a key principle: data partitioning strategy determines processing architecture. When data arrives with the right partitioning, you can aggregate directly; when it doesn’t (like network flows requiring intermediary resolution), you need shuffle/redistribution stages.

Dynamic Load Distribution: How Hashing Works with Auto-Scaling

How do we decide which instance receives which aggregator when our Auto Scaling Groups dynamically add or remove instances? Traditional approaches assume static clusters — requiring explicit rebalancing, coordination services, or manual data movement when cluster size changes.

Our Approach: Dynamic Consistent Hashing

We use consistent hashing with dynamic instance discovery from our service registry. Each instance queries the registry to get the current list of healthy ASG instances, maintains them in sorted order (ensuring all instances have the same view), and uses this list for the hash function findOwnerInstance(aggregator.primaryKey). When ASG scales up or down, the hash function naturally redistributes aggregators based on the updated instance list — no explicit coordination needed.

The key insight: leverage existing infrastructure. Our service registry already tracks ASG membership for health checking. Using it as our source of truth gives us dynamic cluster membership for free. Consistent hashing provides stable partitioning (most aggregators stay on the same instance during membership changes), while the sorted list ensures consistency.

The Result

Load follows infrastructure automatically. During traffic spikes or live events, new instances immediately receive their share. During deployments, aggregators seamlessly shift to healthy instances. This pattern proved crucial for production stability — no manual intervention, no coordination protocol, just automatic rebalancing.

The V1 Journey: Major Challenges at Production Scale

Getting the initial version (V1) to production taught us that scale changes everything. What works in development breaks in production. Every assumption gets tested. And fixing one bottleneck reveals the next.

Challenge 1: Kafka Consumer Lag

The Problem: Our multi-region Kafka consumers started falling behind — consumer lag grew from seconds to minutes, then hours. Flow logs were arriving faster than we could process them. If this continued, we’d never catch up, and our “real-time” topology would become increasingly stale.

Investigation: We instrumented Kafka consumer metrics heavily. Key findings:

Kafka had fewer partitions than optimal for our consumer group size
Each fetch operation retrieved relatively few records
Network socket buffers weren’t right-sized for our throughput
Cross-region read latency added overhead

Solutions Applied:

Increased Kafka partitions: More partitions enabled more parallel consumers in our consumer group, distributing load across more instances.
Tuned fetch parameters: Increased records per fetch operation, reducing the number of network round-trips. This trades off per-message latency (we fetch larger batches) for throughput (more records processed per second).
Increased socket receive buffer size: Ensured network buffers never limited fetch operations. At our scale, default buffer sizes were too small.

Results: Throughput improved significantly, and lag reduced to acceptable levels — typically under a minute even during peak traffic.

Lesson: At scale, you can’t optimize in isolation. Fixing Kafka lag revealed the next bottleneck: our instances themselves couldn’t keep up with the higher ingest rate. The pipeline moved faster, which exposed downstream capacity problems.

Challenge 2: Hot Nodes and Data Amplification

The Problem: This was the most severe production issue we faced. Some instances in our Auto Scaling Group were receiving 100x more traffic than others. Memory usage spiked. Garbage collection pauses became frequent and long. More CPU time was spent in GC than in business logic. Eventually, hot instances would go DOWN, triggering cascading failures as their load redistributed to other instances.

Root Cause Investigation:
Flow logs for popular services dominate traffic volume. A service like our authentication layer or recommendation API is called by hundreds of other services, generating orders of magnitude more flow records than typical services.

Our initial architecture used consistent hashing to determine which instance owned aggregation for each destination service. All flow logs for a given destination are routed to the same instance — the “owner” for that destination. This design seemed reasonable: group related data for efficient aggregation.

But popular destinations created hot nodes. One instance might own authentication services, another might own a rarely-used backend service. The load distribution was wildly uneven — some instances handled 100x the flow records of others.

Worse, data amplification occurred during redistribution. Consider a service called by 100 upstream services across 10 instances. All 10 instances receive flow logs for that destination (because they all have local clients calling it). When they route aggregators to the owner instance, that instance receives 10 separate aggregators it must merge. The data volume multiplied during shuffling.

When many instances route data for the same key to one owner, the volume multiplies right where it lands — the root cause of hot nodes.

We profiled extensively using async-profiler and heap dump analysis. The results were clear: hot instances spent most of their CPU on garbage collection, trying to manage the rapid allocation and deallocation of aggregator objects as flow logs poured in faster than they could be processed. Memory pressure led to GC thrashing, which consumed CPU, which slowed processing, which increased memory pressure — a vicious cycle.

Solution: The Three-Stage Pipeline’s Dual Benefits
The three-stage pipeline we described earlier — designed primarily for proxy resolution — turned out to be exactly what we needed to solve the hot nodes problem as well. Here’s why:

Stage 1 performs initial aggregation locally before any distribution. Instead of sending every flow log to a remote instance immediately. Each instance performs online aggregation of raw flow logs into time-windowed aggregators (over 5-minute periods) directly in memory; this allows the raw flow to be discarded and garbage collected quickly,, significantly reducing memory pressure, and ensures only the aggregation results are transferred across the network to downstream stages.

Stage 2 focuses on proxy resolution but also provides intermediate redistribution. Aggregators from Stage 1 distribute via consistent hashing to Stage 2 instances. Now we’re moving compressed aggregators, not individual flow logs. After resolution, Stage 2 redistributes resolved edges again to Stage 3, providing a second hashing operation that further spreads load.

Stage 3 receives resolved aggregators that have been compressed twice and distributed twice. Even for extremely popular services, load has been spread across enough distribution points that no single instance becomes overwhelmed.

The key insight: architectural decisions driven by one requirement (proxy resolution) often solve other problems (load distribution) as beneficial side effects. The three-stage pipeline with graduated redistribution achieves both goals — it resolves proxies to show clean application-level topology AND prevents hot nodes by spreading load across multiple distribution points.

Switching from gRPC to SSE
As described earlier, this challenge also revealed that gRPC wasn’t the right protocol for inter-stage communication at our scale. We replaced gRPC with Server-Sent Events, dramatically reducing resource consumption on both sender and receiver sides.

Results:

CPU usage became evenly distributed across instances — no more hot nodes with 10x the load of others
Network bandwidth usage dropped significantly due to better aggregation and lighter-weight protocol
Memory pressure decreased as we reduced the object allocation rate
The system scaled gracefully with Auto Scaling Group changes

Lesson: Technology choices must match your specific use case. gRPC is excellent for request-response RPC patterns. For streaming large volumes of aggregated data in a pipeline, lighter-weight alternatives can be more appropriate. Let measurements guide the decision, not industry hype or existing team expertise.

Challenge 3: Memory and Garbage Collection

The Problem: Even after fixing hot nodes, we still saw high heap usage, frequent garbage collection pauses, and instances occasionally going DOWN. GC logs showed pauses consuming significant CPU time — in some cases, more than our business logic.

Root Cause: Multiple factors contributed: objects accumulating in heap while waiting for 5-minute aggregation windows to complete, unnecessary conversions between different object types as data flowed through stages, and immutability overhead — following Scala best practices, we used immutable data structures for aggregators, but every update created new objects, overwhelming the garbage collector at millions of records per second.

Investigation: Heap dumps and GC logs revealed flow log objects retained beyond their useful lifetime, unnecessary intermediate conversion objects, and constant creation/disposal of immutable aggregator versions. Minor GCs occurred every few seconds, major GCs took hundreds of milliseconds — the JVM spent more time on garbage collection than business logic.

Solutions Applied:

Faster processing: Process flow logs immediately, aggregate quickly, release references. Optimized Pekko stream stages to minimize object lifetime.
Eliminate unnecessary conversions: Route aggregators directly between stages instead of converting to intermediate types.
Mutable structures on hotpath: This was controversial — Scala best practices emphasize immutability. But at our scale, immutability created too many objects. We pragmatically chose mutable aggregators on the hotpath (immutability elsewhere), prioritizing performance over convention. Switching to mutable aggregators reduced heap allocation by over 50% and cut GC pause time significantly, though it required more careful code review.
Tuned time windows: Balanced data freshness against memory pressure.

Results:

Heap usage decreased substantially
GC pauses reduced to acceptable levels (tens of milliseconds instead of hundreds)
CPU freed up for business logic instead of garbage collection
Instance stability improved — no more instances going DOWN due to memory issues

Lesson: “Best practices” are starting points, not absolute rules. At unique scale, you may need to diverge from conventions. But do it deliberately, with measurement justifying the decision, and with awareness of the trade-offs. Don’t abandon immutability everywhere — just where performance data proves it’s necessary.

Challenge 4: Reactive Streams Complexity

The Problem: Our Pekko Streams pipelines would stall unexpectedly. Backpressure propagation didn’t work as expected. We struggled to debug why certain streams would stop processing without obvious errors. The reactive programming mental model — with its emphasis on async boundaries, backpressure, and demand-driven processing — proved harder to master than anticipated.

What We Learned:
Reactive streams with backpressure are powerful tools for building systems that handle load spikes gracefully. When downstream consumers slow down (due to temporary load, GC pauses, or external system slowdowns), backpressure allows upstream producers to slow down rather than overflow buffers or drop data.

But this power comes with complexity:

Non-intuitive behavior: Traditional imperative code flows top-to-bottom. Reactive streams are demand-driven — downstream consumers pull from upstream producers. This inversion of control isn’t intuitive.
Async boundaries: The .async operator in Pekko Streams creates a boundary where processing moves to a different thread. This can improve parallelism but also introduces complexity around buffer sizing, demand signaling, and error propagation. We initially misunderstood when to use .async and ended up with over-parallelized streams that created more overhead than benefit.
Debugging difficulty: When a stream stalls, there’s no stack trace pointing to the problem. You must understand the internal mechanics — demand signals, buffer states, materializer state — to diagnose issues.

Our Approach:

Deep learning investment: We invested significant time in understanding reactive streams concepts deeply. Reading documentation, experimenting with small examples, and building team expertise.
Simplified patterns: Where possible, we simplified our stream graphs. Complex branching and merging patterns are powerful but hard to debug. We preferred linear flows with clear stage boundaries.
Better monitoring: We added metrics at stream boundaries — tracking buffer sizes, element throughput, backpressure events. Visibility into stream internals helped diagnose issues.
Team education: We documented our learnings, shared patterns that worked, and built institutional knowledge about reactive streams.

Lesson: Powerful abstractions require investment. Don’t assume you understand a framework without validation. Build your mental model deliberately, test it with experiments, and be humble about your understanding. Reactive streams are worth mastering for systems that need to handle load gracefully, but expect a learning curve.

V2 Evolution: Continuous Refinement

V1 got us to production. The major architectural challenges — Kafka lag, hot nodes, memory pressure — were solved. But production at full scale revealed new optimization opportunities. V2 represents the continuous refinement that turns a working system into a production-ready system.

Challenge 5: Persistent Heap Pressure

The Problem: Despite V1 optimizations, we still observed higher-than-desired heap usage. GC metrics improved but weren’t optimal. Memory profiling showed room for improvement.

Root Cause: Deeper analysis revealed we were still doing unnecessary object conversions between stages. We’d convert aggregators to full graph entities (with all properties populated) before routing to the next stage, even though the next stage just needed the compressed aggregator state.

Solution: Architectural change to route aggregators directly through all stages, only converting to final graph entities at Stage 3 immediately before persistence. This eliminated two intermediate conversion steps and the associated object allocation.

Result: Heap usage dropped further, GC pauses became even less frequent, and memory headroom improved.

Challenge 6: Serialization Complexity

The Problem: Custom serialization logic for SSE messages caused occasional erratic errors that were hard to reproduce and debug. Different parts of the codebase used inconsistent serialization approaches.

Solution: Standardized on JSON encoding throughout the pipeline. While slightly less efficient than binary serialization, JSON’s human readability made debugging far easier, and the overhead was negligible compared to other operations. Consistency eliminated an entire class of bugs.

Result: Serialization-related errors disappeared. Debugging became easier because we could read SSE message contents directly.

Challenge 7: Stream Processing Inefficiencies

The Problem: Even after understanding reactive streams better, our Pekko configurations weren’t optimal. We had over-parallelized some stages and under-parallelized others. The .async boundaries weren’t placed optimally.

Solution: Through continued profiling and experimentation, we tuned parallelism parameters, adjusted buffer sizes, and refined async boundary placement. We added monitoring at stream boundaries to identify bottlenecks.

Result: Throughput improvements and more consistent processing latency.

Challenge 8: Uneven Graph Database Throughput

The Problem: Write distribution to our graph database wasn’t even. Some partitions received heavy write traffic while others sat idle. This caused throttling to kick in unevenly and limited overall write throughput.

Solution: Implemented batching of aggregators before writing to the graph database and improved distribution logic across partitions. Rather than writing each aggregator immediately, we batch them and write multiple entities in coordinated operations.

Result: More consistent write throughput and better utilization of database capacity.

Challenge 9: Data Enrichment at Aggregation Time

Beyond the core topology graph, we needed to enrich nodes with additional context. At Stage 3, before persisting graph entities, we integrate enrichment data from external sources — application health status, ownership information, and other metadata. Performing this enrichment at aggregation time rather than at query time avoids the performance overhead of post-query joins and ensures every topology node has full context when queried.

Pattern Recognition

Each V2 challenge followed the same pattern: production revealed an assumption, profiling identified the root cause, targeted fixes improved specific metrics. Measure, hypothesize, validate, iterate. This is how you build at scale — not by getting everything right upfront, but by continuous learning and improvement.

Time Travel: Continuous Topology Reconstruction

One of the most powerful capabilities we built enables querying historical topology: “What did the call graph look like when this incident happened?” This time-travel feature required solving an interesting architectural challenge — how to efficiently store and reconstruct topology across time.

The Problem

Engineers need to answer temporal questions: What did the topology look like during an incident? How have dependencies evolved? Traditional approaches — full snapshots or event sourcing — either have exponential storage costs or require slow log replay.

Our Approach: Time-Windowed Aggregators with Mutation Tracking

We combine two mechanisms:

1. Time-Windowed Aggregator Snapshots: Every aggregator stores startTs and endTs timestamps for its 5-minute window. These immutable aggregators persist in the graph database keyed by (entity_id, timestamp), providing checkpoint states every 5 minutes.

2. Property-Level Mutation Tracking: The graph database maintains mutation history at the property level — storing only changed properties with timestamps. This is much more efficient than full entity copies and provides sub-window precision beyond the 5-minute aggregation boundaries.

3. Query-Time Reconstruction: When querying historical topology, we query the mutation history API for the time range, retrieve all mutations, and reconstruct topology state by applying mutations in order.

This approach provides efficient storage (compressed aggregator states + sparse property mutations), fast retrieval (indexed mutation history, no log replay), and flexible analysis (arbitrary time ranges without pre-computing all possibilities).

Query-Time Re-Aggregation: We can further aggregate historical data at query time using the same aggregator classes from ingestion. This enables arbitrary groupby dimensions (availability tier, business domain, deployment cluster) that weren’t pre-computed, allowing exploratory analysis without exploding storage costs.

Lessons for Distributed Systems

While these challenges were specific to service topology, the lessons apply broadly to distributed systems at scale.

Scale Changes Everything

What works at 100 requests per second fails at 100,000 requests per second. The change isn’t linear — it’s qualitative. Approaches that are fine at modest scale hit fundamental walls at extreme scale.

Examples from our journey: immutable data structures create GC pressure at millions of allocations per second; single-stage aggregation fails catastrophically with power-law traffic distribution; standard gRPC becomes heavyweight for streaming aggregation at volume.

The lesson: be willing to break conventional wisdom when scale justifies it. But do it based on measurement, not speculation.

Optimize One Bottleneck at a Time

Distributed systems have cascading bottlenecks. Fix Kafka lag, and you discover hot node issues. Fix hot nodes, and you discover GC problems. Fix GC, and you discover serialization inefficiencies.

This isn’t failure — it’s the nature of complex systems. Each optimization raises throughput, which stresses the next weakest point. The approach: prioritize based on impact, fix the current bottleneck thoroughly with measurement confirming resolution, then move to the next one. Optimization at scale is continuous, not one-time.

Distribution Is Key to Scale

Single aggregation points are inevitable bottlenecks. Consistent hashing distributes load but doesn’t prevent concentration when data itself is unevenly distributed (power-law distributions like ours).

Our three-stage pipeline with graduated redistribution solved this. Load spreads across multiple distribution points at each stage. Even with highly skewed data, no single instance becomes overwhelmed. The general principle: use multi-stage processing with redistribution at each stage when dealing with skewed data at scale.

Current State and Impact

Service Topology operates in production today, processing flow logs, ipc metrics and traces from multiple regions and serving queries with sub-second latency. Teams across Netflix use it daily for incident investigation, blast radius analysis, dependency understanding, and production change management. The system has become essential infrastructure for maintaining reliability at scale.

Conclusion

Service Topology at Netflix represents a journey through building distributed systems at scale. We started with engineers struggling to understand dependencies across scattered tools. We built a multi-layer architecture using streaming aggregation, network intermediary resolution, and time-travel capabilities. And we learned that optimization at scale is continuous — measure, iterate, validate, repeat.

The challenges we faced — Kafka lag, hot nodes, memory pressure — required breaking conventional wisdom when data justified it. Each fix revealed the next bottleneck. But that iterative process, guided by constant measurement, is what makes systems work at extreme scale.

In our next post, we’ll explore the tracing layer integration, unified querying across heterogeneous storage, and how all three layers combine to provide comprehensive topology visibility.

Acknowledgements

Service Topology was built by Parth Jain, Rakesh Sukumar, Yingwu Zhao, Renzo Sanchez-Silva, and Nathan Fisher.

Special thanks to the many engineers across Netflix who made this possible — the Observability team who built the broader system, the graph database platform team who provided the storage foundation, and the Platform Modernization Engineering, and Live teams who provided invaluable feedback and use cases throughout development.

Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler - Engineering at Meta

2026-07-13T16:00:50.000Z

TL; DR

At Meta’s scale, a few milliseconds of latency degradation can have a significant negative impact on ads performance.
When a Linux kernel upgrade risked regressing latency across Meta’s ad serving fleet, we turned to sched_ext — the upstream, BPF-based extensible scheduling framework — to build a scheduling policy customized to the Ads delivery workload.
The result: a 28% reduction in ads retrieval stage tail(99th percentile) latency, 3.28 megawatts(MW) power saving, and a 1.1% increase in the number of ads ranked, proving that workload-specific scheduling optimization can directly drive business value.

Why Ads Latency Matters

Meta’s ads serving fleet handles more than 5 million requests per second on average at the serving platform entry point, which is over 400 billion per day across all monetized surfaces¹. Every millisecond shaved off the p99 latency makes the ads more relevant for people on our platforms, and better matches mean stronger ROI for advertisers.

This provides a real opportunity to reduce latency through workload-specific scheduling. That is why our Ads and Linux Kernel teams have been working together to build a scheduling policy customized to the ads delivery workload using sched_ext, the upstream, BPF-based extensible scheduling framework. Until now, we have been using the general-purpose schedulers typically integrated in the Linux kernel (CFS and EEVDF) that balance threads across CPUs with no understanding of the workload. However, here we know the purpose and importance of each thread. With sched_ext, we can encode this knowledge directly into the scheduler. Work that improves the p99 request latency is scheduled first, and everything else takes a back seat.

sched_ext at Meta

sched_ext is an open-source, BPF-based scheduler framework that officially entered kernel v6.12. We developed it by partnering with the authors of Google’s ghOSt to design a scheduler suitable for upstream Linux integration. It has already been deployed in several services at Meta, delivering meaningful reductions in scheduling latency.

While upgrading our fleet to the latest stable version of Linux (kernel v6.9) we observed that the new Earliest Eligible Virtual Deadline First (EEVDF) scheduler introduced in Linux kernel v6.6 was causing a latency regression which reduced the number of ads ranked in response. As a result, a subset of ads hosts were forced to remain on the older v6.4 kernel, creating technical debt and operational fragmentation.

Given its already strong performance, sched_ext was a great candidate to address these scheduling regressions.

Custom Scheduling with sched_ext

Sched_ext lets scheduler developers implement their preferred scheduling policy as a BPF program. When a host starts running the ads workload, an ads-optimized policy is applied. From that point on, the kernel calls into the BPF scheduler through a set of event-driven callbacks to handle common scheduling events, such as:

Thread wake-up: choose a CPU when a thread becomes runnable.
Enqueue: place a thread in a run queue.
Dispatch: select the next thread when a CPU becomes idle.
Idle transitions: respond to CPUs entering/leaving idle states.

At a high level, the policy soft-partitions CPUs into two pools, one for threads on the latency-critical request path and one for less latency-sensitive work. Which thread goes into which pool is part of the domain-specific knowledge encoded inside the policy. The size of each pool is adjusted dynamically using load-based heuristics. This approach tends to keep related work on the same CPUs over time, improving last-level cache (L3) locality and reducing costly DRAM access.

The policy is packaged as a user-space binary that loads the BPF program. That design makes experimentation and performance optimization much faster. To roll out a change, we can simply restart the scheduler process to unload the old policy and load the new one, without rebuilding or reinstalling the kernel.

Results and Impact

The initial launch took place to switch from kernel 6.4 with the CFS scheduler to kernel 6.9 with sched_ext on the largest ads serving server type. Based on the backtest experiment, the launch delivered:

+1.1% on weighted-ads-ranked (metric for number of ads retrieved and ranked).
3.28 megawatts of power savings across the fleet.
28% reduction in service p99 latency on the ads retrieval path².

Compounding improvements. Two follow-on scheduler-policy updates, delivered as purely user-space changes, extended the win:

Additional 60% reduction in service p99 latency.
18% reduction in timeout errors on the critical path.

This is a non-trivial win delivered with no dependency on kernel releases. Each follow-on iteration above shipped in days rather than months because the scheduler policy lives in user space as a BPF program. That cadence is what turned sched_ext from a “kernel upgrade unblocker” to a continuous-optimization platform for ads serving.

From Short-Term Fix to Strategic Asset

What started out as a targeted response to a very specific operational issue has turned out to be much more strategic, and widely applicable, than we originally anticipated. sched_ext delivers some key benefits to Meta:

A parallel and decoupled scheduler optimization path. Upstream Linux scheduling naturally evolves over time, sometimes in larger steps (such as the CFS-to-EEVDF transition), which can be disruptive to downstream consumers. sched_ext gives Meta the flexibility to continuously improve these custom schedulers alongside that evolution. We run and refine our own BPF-based scheduling logic, tailored to the unique demands of our production workloads, so our critical services stay optimized regardless of what happens upstream.

Independent deployment and reduced overheads. Scheduler improvements ship as BPF program updates, shipped in days rather than months. The resulting reduction in the cost of experimentation is transformative. Ideas that previously required a kernel patch and months of validation — local-cache-aware placement, ROI-based executor routing, NUMA-aware steering — become tractable iterations rather than major projects.

A shared industry asset. sched_ext was upstreamed into Linux v6.12, so the same mechanism Meta used here is now available to the entire Linux ecosystem. Any operator with a workload that doesn’t fit the general-purpose model — hyperscaler, cloud provider, embedded systems team — can ship workload-specific scheduling policies without forking the kernel.

Future Plans

sched_ext is already allowing us to see opportunities for further improvements in ads performance, by giving the application more fine-grained control over the behavior of the scheduler. For example, the ads services have important context about the relative importance of service requests, and are potentially able to signal to the scheduler when a thread starts working on an important request. When the scheduler receives this hint, it can take appropriate steps like increasing this thread’s scheduling slice or ensuring it’s always at the top of the queue.

Acknowledgments

Special thanks to Samuel Nair, Usama Arif, GP Musumeci, Praveen Alevoor, Ye Wang, and the broader Ads capacity efficiency and kernel team for their contributions and collaboration.

Ads Infra Leadership Team: Uladzimir Pashkevich, Varna Puvvada, Prabhakar Goyal, Neeraj Agrawal, Tak Yan, Liz Shepherd, Drew Lackman

The post Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler appeared first on Engineering at Meta.

Better tools made Copilot code review worse. Here’s how we actually improved it. - The GitHub Blog

2026-07-10T15:57:47.000Z

Give an agent better tools and it should do better work. That’s the instinct, anyway.

When you open a pull request, Copilot code review reads the diff and explores the surrounding code to find the problems that matter before they ship. To do that, it used its own code exploration tools. So when we swapped in the better-maintained, shared tools that power the Copilot CLI, grep, glob, and view, we expected a clean upgrade.

Instead, in our benchmarks, we found that the cost of reviews was higher and fewer issues were being caught.

But the tools weren’t the problem. The instructions were. Once we rewrote them for the way a reviewer actually reads a pull request, the regression flipped into a win: roughly 20% lower average review cost, while maintaining the same review quality.

This is the story of how adjusting the workflows around the tools led us to a fix.

Same tools, wrong instincts

If you’ve built on top of an agent framework, you’ve probably inherited its tools too. They work, so you keep them, until the day your use case drifts far enough from what they were designed for that they quietly start working against you. That’s the situation we were in. Before trying to use the shared CLI tools, Copilot code review used its own code exploration tools. That tool layer was inspired by earlier agentic systems, including ideas from SWE-agent-style repository navigation and GitHub Copilot Autofix: list directories, search files, search directories, and read code. Those tools worked, but they were specific to Copilot code review, and they were designed for how models behaved at the time. Earlier agentic coding models made fewer tool calls and were worse at automatically pulling in necessary context. This meant it was more important to include all relevant information in the few tool calls that the model made.

Meanwhile, the Copilot CLI harness has a shared set of Unix-inspired code exploration tools: grep, glob, and view. That harness is also used by a growing number of Copilot agent products, including GitHub Copilot cloud agent, so harness improvements can benefit more than one product. We wanted to clean up and share infrastructure where possible, so we experimented with using the tools from the Copilot CLI harness in Copilot code review. The goal was to reduce duplicated tool implementations, create one shared place to improve code exploration tools, and make it easier to carry those improvements across Copilot products.

On paper, the migration looked simple:

Old Copilot code review	GitHub Copilot CLI	Purpose
list_dir	glob	Discover candidate files and directories before opening code.
search_file and search_dir	grep	Search code for matching text, symbols, or call sites.
read_code	view	Read the relevant file contents once a path or range is known.

The existing review tools were not thin wrappers. When searching for a directory or reading a code range, they could return the matched or requested lines plus extra surrounding code context. That added token cost, but it also matched how earlier models often benefited from having nearby context included automatically.

Initially, we hoped this would be a simple migration: swap one set of tools for another. But when we tested the shared tools in offline benchmarks, the review agent became less efficient and less effective. Average cost increased, and the number of useful comments dropped.

The trace revealed a browsing loop

Our internal Copilot code review benchmarks were useful because they show more than a final score. They show the path the agent took, including which tools it called, how much output came back, where errors happened, and whether it was narrowing toward evidence or widening the search.

When we first tried the shared Copilot CLI tools in offline benchmarks, the agent often behaved as if it was browsing a repository instead of investigating a pull request. It would search broadly, guess likely paths, read broadly, find more things to search, and carry that extra context forward.

Figure 1: Before — a simplified illustration of the general-purpose behavior we observed: widening the search, guessing paths, and accumulating context.

That pattern is understandable. Broad exploration can be useful when the task is “understand this repo.” But it’s not how a reviewer would usually review a pull request.

When I review a pull request, I start from the diff and ask targeted questions:

Where is this function called?
Is this config key used anywhere else?
Is there a test or helper with the same pattern?
What is the smallest nearby code range that explains this behavior?

I do not want to open a large part of the repository before I know what I am looking for. I want the minimal context needed to answer the question, without overloading the review with unrelated code.

That matters because every tool result becomes part of the agent’s working context. Extra file contents can be carried forward into later reasoning, increasing cost and sometimes making the review less focused. A tool result is not a disposable printout; for an agent, it’s extra tokens that stay in the context window.

The traces made that difference visible. The shared tools were not the problem. The instructions were giving the agent the wrong instincts to do an efficient and effective review.

The tools themselves worked, but their instructions were tuned for their use within the Copilot CLI and implied the wrong workflow: the agent used grep, glob, and view like a broad coding assistant instead of a reviewer. A coding assistant may map a whole area before making a change to ensure it doesn’t break some other corner of the code. On the other hand, a reviewer usually starts from the diff, asks whether the change introduced a problem, and then looks for the narrowest nearby evidence required to confirm or dismiss it.

General coding-assistant tool instructions, like the ones used by Copilot CLI or Copilot cloud agent, make sense for an interactive assistant. A developer may ask it to understand a repository, plan a change, edit files, and continue over multiple turns.

Copilot code review has a narrower job: start from a pull request diff, gather enough surrounding evidence to decide whether a change introduces a real issue, and avoid loading context that is not needed for that review question.

It was therefore clear that we couldn’t simply replace the previous Copilot code review tools with the tools from the Copilot CLI without additional prompting work. The problem became: how do we design tool instructions that use these shared tools effectively in a code review setting?

Rewriting the tool instructions for a reviewer’s workflow

The next iterations made the guidance specific to code review. The workflow we wanted Copilot code review to follow was:

Start from the diff and form specific review questions.
Use glob when the path is uncertain and grep to find candidate files, symbols, and call sites.
Batch cheap discovery before reading files.
Use view only when the agent knows which file or line range it needs.
Batch focused reads instead of alternating between one search and one read.

In oversimplified form, this was the behavior we encoded:

Generic posture: Use the available tools to inspect repository context that may be relevant.

Review-shaped guidance: Start from the diff. Narrow first with grep and glob; read exact evidence with view. If grep fails to find relevant context, retry with a simpler escaped search. If a path is wrong, pivot to glob instead of guessing nearby paths.

For example, imagine the diff changes an authorization helper that decides whether an operation is allowed. A relevant review question is not “show me the full contents of every file that calls this helper.” It could instead be the narrower: “are any request-handling callers relying on the old behavior?”

The intended path is short:

start from the helper changed in the diff 
grep for callers of that helper 
glob for likely route, handler, or controller files 
view the most relevant caller ranges 
decide whether any caller changes the risk

The guidance also changed how the agent recovered from failed searches. If an input made grep fail, the better next step was one simpler, corrected search. If a path was wrong, the better next step was glob, not guessing neighboring paths and reading whatever happened to exist. That nudged the agent away from letting a small tool failure turn into a larger exploration loop.

Figure 2: After — a simplified illustration of the review-shaped behavior the prompt guided toward: stay anchored to the diff, narrow with grep and glob, then read focused ranges with view.

The change was small in wording and large in effect. It changed the rhythm of the agent from “browse, read, search again” to “ask, narrow, read, decide.”

Benchmarks let us debug behavior, not just scores

The shared harness gave us the tools. The internal Copilot code review benchmarks gave us the feedback loop.

We could run the same review examples, compare tool traces, update the instructions, and run again. That let us ask concrete questions:

Did the agent narrow first, or read broadly first?
Did it batch independent searches?
Did it call view only when it had a reason?
Did a tool-instruction change reduce tool errors, or just move them somewhere else?
Did the trace stay focused on evidence from the diff?
Did the review still preserve the quality metrics we cared about?

The most useful signal was not “the instructions are better.” It was more concrete. The agent was making a similar number of tool calls, but spending more of them on relevant evidence instead of repeatedly expanding the search.

That connected product-level outcomes to understandable engineering behavior. Instead of guessing why a score moved, we could inspect the workflow that produced it.

The result: roughly 20% lower average review cost

In production, the tuned behavior showed roughly 20% lower average review cost compared with the control. Importantly, it did not show a quality signal that could block shipping.

The reduction did not come from the tools by themselves, it came from the workflow around them. Shared code exploration tools, Copilot code review custom tool instructions, and internal benchmarks made the agent’s behavior visible enough to tune.

That framing matters when building with agents. It can be tempting to treat tools as implementation details by swapping one tool for another, then comparing the final answer. But for an agent, the tool surface is part of the product experience. It changes what the agent notices, how it searches, how much context it carries forward, and when it decides it has enough evidence.

Tool descriptions and system instructions are closer to API documentation. Unclear API docs can leave a developer confused and lead to inefficient or wrong decisions. Unclear tool prompting can do the same for an LLM; a small wording change can affect cost, quality, and the shape of the investigation because it changes how the agent spends its attention.

Same tools, different job

We also tried to apply the same kind of focused tool instructions in the CLI, where it did not produce the same kind of win. That is a useful counterexample, and an important guardrail for the lesson.

Copilot code review is anchored to a diff and a review question. Copilot CLI handles broader, interactive coding tasks where exploration can be part of the job. There may be no single diff anchor, the user may change direction over multiple turns, and the right context may not be obvious at the start. The same grep, glob, and view tools can support both products, but the workflow around those tools has to match the product.

The takeaway is that shared tools scale when the instructions and benchmarks match the job.

Try it out yourself using GitHub Copilot code review.

The post Better tools made Copilot code review worse. Here’s how we actually improved it. appeared first on The GitHub Blog.

How GitHub gave every repository a durable owner - The GitHub Blog

2026-07-09T16:29:37.000Z

GitHub has over 14,000 repositories across our primary internal GitHub organization. As of early 2025, there were over 11,000 non-archived repositories, the vast majority of which with no clear owner. For repositories attached to production services, we have historically had robust durable ownership, but for repositories with no associated service, there was no reliable way to tell who the owner is.

That gap became a recurring problem during our secret scanning remediation effort: while we could technically rotate a secret, doing so without knowing the repository owner was risky and often disruptive, and we had no clear way to route remediation work. Over the course of a month and a half, we validated ownership for every active repository, archived about 8,000 repositories that were no longer in use, and changed repository creation so that ownership was required from the start.

Our original ownership model

For years, GitHub has been tracking ownership for deployed services through our internal Service Catalog. Each service entry recorded metadata like which repository it lived in, which gave us a mapping from service to repository; the owning team; executive sponsor; and support information.

Here’s an example of the Repo Ownership app’s service ownership entry:

- team: github/repo-ownership-dev 
  repo: https://github.com/github/repo-ownership 
  name: repo-ownership 
  kind: moda 
  long_name: Repo Ownership 
  description: Service enforcing repo ownership across the org 
  maintainer: mrecachinas 
  exec_sponsor: stephanmiehe 
  ...

Having this rich metadata enables service-centric workflows, such as incident response, on-call routing, vulnerability management, and compliance scoping.

Unfortunately, that relationship was many-to-one (i.e., a service could only be attached to a single repository, but a single repository could have multiple services). That meant if you started from a service, you could find the repository and its owners. But if you started from a repository and needed to find an owner, you had to reverse the lookup, and that only worked for repositories that mapped to a service in the first place.

That left a significant ownership gap that included team repositories, documentation repositories, internal tools, one-off project repositories, personal experiment repositories, and anything else that didn’t back a deployed service. Every time we needed to contact the owner of one of these “unowned” repositories, it required manual work: check the commit history, read the README, ask around in Slack, or make a guess based on the repository name.

For a one-off effort, that kind of ambiguity is annoying but manageable. For recurring security workflows that fan out across the entire organization, it presents a real risk. During our secret scanning cleanup, we spent too much time trying to find the right owners before we could make informed decisions about alerts.

Designing the new ownership model

Fundamentally, we needed repository ownership to be a first-class property. We considered storing ownership in a dedicated file within each repository or maintaining it in a centralized repository, but ultimately chose GitHub custom properties. This approach provided a native, structured, and organization-wide queryable way to manage ownership. It also enabled us to enforce enterprise and organization policies and rulesets selectively according to ownership type.

We created two custom properties: ownership-type and ownership-name.

ownership-type accepted three values: “Service Catalog,” “Hubber Handle” (a “Hubber” is what we call a GitHub employee), and “Team.” These covered the realistic range of repository ownership at GitHub. A repository either belongs to a service (with an on-call team and a defined lifecycle), a team (like a shared documentation repository or internal tool), or an individual (like a personal project or experiment).
ownership-name was a text field with light validation. Our GitHub App validated every value: Hubber handles were checked against actual membership in our GitHub organization, teams were verified to exist in the organization and have at least two members, and Service Catalog entries were confirmed against our Service Catalog itself. We were intentionally permissive on formatting. If someone typed @my-team instead of my-team, we accepted it. We wanted to make it frictionless to add ownership and lean on robust validation to catch invalid entries like nonexistent teams, former employees, and services that had been decommissioned.

Day-one coverage

Before we asked anyone to do anything, we built a periodic sync from Service Catalog to repository custom properties. Every repository that backed a known service had its ownership-type set to “Service Catalog” and its ownership-name populated automatically. That took care of about 1,500 service-backed repositories, leaving team repos, docs repos, one-off projects, and personal repos remaining.

The rollout

To roll this out, we built a GitHub App backed by a Kubernetes CronJob. The enforcement logic needed access to Service Catalog, the GitHub API, and a few internal systems, so a simple GitHub Actions workflow wasn’t sufficient.

The diagram below shows the repository ownership enforcement flow, from initial ownership scan through warning issue creation, automatic closure, or archival after 30 days.

We scheduled the first run of the CronJob for a Saturday morning thinking nobody would be paying attention… Big mistake! Issues started appearing in repositories across the organization, and people began jumping on Slack, asking about this new issue in their repository saying it would be archived. At a globally distributed company, someone is always online.

After the 30-day grace period, we archived any repository that still didn’t have ownership set. We chose archiving because it’s reversible and non-destructive: the repository becomes read-only and GitHub Actions stops running, but nothing is deleted. If someone needs it again, we provided an easy way for them to unarchive it, set ownership, and continue. That enabled us to safely apply archival broadly instead of debating every edge case.

Once the initial grace period passed and the bulk of archiving was done, we tightened the enforcement loop from 30 days to one hour. A new repository that somehow bypassed the creation-time ownership requirement would get flagged almost immediately.

The sharp edges

This mostly rolled out seamlessly, with two minor internal incidents exposing some interesting edge cases.

The first incident was caused by archiving a repository where ownership had not been applied. Datadog had been configured to open issues in that repository as part of a monitoring workflow. When the repository was archived and Datadog couldn’t create the issue, our internal monitoring service noticed and automatically paged the owning team, and they escalated to us.

That incident exposed a gap in how we notified. The ownership issues were landing in repositories, but nobody was getting notified directly. We fixed this by @-mentioning repository administrators and assigning all users with write access as a fallback on ownership issues. That way the issues couldn’t be buried or overlooked, and the people who could actually set ownership saw them immediately.

The second incident was a data reliability problem. While we were robust against a Service Catalog outage, we didn’t consider that it might return stale data or corrupted data. If bad data caused the app to think a batch of repositories had lost their Service Catalog entries when they hadn’t, we’d be mass-archiving repositories with perfectly valid owners.

To mitigate the risk of archiving legitimate repositories, we added a low water mark. During each run, prior to performing any actions, the app would tally how many archives it was about to perform and issues it was about to open. If the number exceeded a conservative threshold, it would bail out entirely and trigger a Datadog monitor rather than risk a bad run. If Service Catalog was unreachable, the job would skip Service Catalog validation and only check what it could verify independently.

Results, by the numbers

We finished with approximately 3,000 active repositories and 11,000 archived (up from about 3,000 archived at the start). The entire effort took under 45 days from the first (Saturday morning) run to steady state. Every active repository now has a validated owner, or it gets archived.

Many of those newly archived repositories hadn’t seen a commit in years: abandoned experiments, completed hackathon projects, and even one-person prototypes from 2008! Archiving them ultimately reduced our surface area and made the active repository inventory reflect reality.

Making ownership stick

Getting to 100% coverage is only useful if it stays at 100%, so we enforced ownership properties across every repository creation workflow, including the repository creation page (shown below), internal tooling and automation, making them mandatory for all new repositories.

We also tightened the enforcement loop: repositories that lose their ownership are now flagged within one hour rather than the original 30-day grace period.

Each ownership type has its own durability characteristics, and we designed around them. Service Catalog entries follow service lifecycle: when a service is deprecated, its repositories typically get archived too, and that’s the intended behavior. Teams are validated for having at least one member, and team existence and membership tend to be reasonably stable. Individual Hubber handles only become invalid when someone leaves the company, which usually means their personal repositories should be archived regardless. For any repository critical enough to outlast a single person, ownership should be a team or a service, not an individual.

What this means for you

You can implement a similar ownership model today using GitHub custom properties. Here’s the approach we’d recommend:

Define your ownership taxonomy. Decide which types of owners make sense for your organization. Services, teams, and individuals worked for us, but your categories might look different.
Create custom properties at the organization level. Set up an ownership-type property as a single-select with your allowed values, and an ownership-name property as text. Custom properties are queryable through the API and visible across the organization.
If you have a service catalog or asset inventory, sync it. Populating ownership for repositories you already track is the fastest way to get meaningful coverage before you start asking people to fill in gaps.
Enforce ownership at repository creation time. Make the properties required so the inventory stays clean going forward.
Build a grace-period workflow for existing repositories. Open issues with a reasonable deadline (we used 30 days), then archive repositories that go unclaimed. Because archiving is reversible and non-destructive, it’s a safe default.
Don’t run your first enforcement pass on a Saturday!
Build guardrails before you trust automation at scale. The low water mark and the @-mention fallbacks weren’t in the original design. They came from real incidents. If you’re building a system that archives repositories or opens issues at scale, assume your data sources will occasionally be wrong, and your notifications will sometimes get lost. Design for that from the start.

For more on custom properties, see the custom properties documentation.

The post How GitHub gave every repository a durable owner appeared first on The GitHub Blog.

Automating cross-repo documentation with GitHub Agentic Workflows - The GitHub Blog

2026-07-08T21:11:56.000Z

“Where are the docs?” It’s a question nobody on a product team enjoys answering. The honest reply is usually some variant of “behind.” A writer is staring at a closed pull request, trying to reverse-engineer what changed. The pull request’s author has already moved on. By the time the doc actually publishes, the feature has shipped, sometimes more than once.

That used to be us on the Aspire team (we’re a small team of 10 building dev tools for distributed apps). A few months back, we were trying to figure out how to safely bring AI into automations we already trusted. That’s when we discovered GitHub Agentic Workflows. I started bolting prototypes into microsoft/aspire.

Here’s what that bought us, in numbers pulled straight out of GitHub: for Aspire 13.3 and 13.4, 82 feature-docs pull requests merged at a median of 44.8 hours after the product pull request, every one of them reviewed by the engineer who shipped the feature. No new headcount. No process retraining. Just a different way of asking “who writes this?”

🔒 The constraint: cross-repo automation is the hard part

Our product lives in microsoft/aspire and our docs site lives in microsoft/aspire.dev—different repo, deploy target, and review chain. Most teams figure out same-repo automation pretty quickly; cross-repo automation is where things get sharp. Broad repo-scoped tokens belong in a museum, and any responsible security posture (ours included) restricts them accordingly. That’s a good thing. It’s also a real bottleneck if the place where you write the docs isn’t the place where you write the code.

The default workflow for years was:

Engineer ships a feature in microsoft/aspire.
Docs writer notices weeks later.
Docs writer opens the pull request, reads the diff, and pings the engineer to clarify what changed.
Engineer is on the next feature, vaguely remembers, replies with half the picture.
Docs draft ships, sometimes against a release that’s already out.

This is the reverse-engineering tax. We needed automation that crossed repos without handing an agent a write-everywhere token. GitHub Agentic Workflows turned out to be the answer.

🤖 Why GitHub Agentic Workflows

GitHub Agentic Workflows is a project from the GitHub Next team that I keep describing to people as “GitHub Actions, but with a model as the work-item processor and guard rails that satisfy security review.” That’s reductive, but it’s close.

The shape of it:

You author a workflow as a single markdown file (.github/workflows/my-thing.md). YAML-style frontmatter on top, an English-language prompt underneath.
You run GitHub Agentic Workflows compile, and it generates a sibling .lock.yml (a normal GitHub Actions workflow) that you commit alongside.
At runtime, the workflow runs an agent against your prompt with a constrained toolset.
Critically, the agent doesn’t write to GitHub directly. It emits intent (a JSON blob describing the pull requests, issues, and comments it wants to create), and a separate, narrowly scoped job (the safe-outputs handler) materializes that intent against a per-workflow GitHub app.

That last bullet is the unlock. The agent gets read access and a prompt. Writes go through a tiny verifiable pipeline with explicit allow-lists. Security review nods. We ship.

💚 A small aside: kindred stacks

I love when the tools you’re using to build are built with the same tools you’re using to build with. The GitHub Agentic Workflows docs are built with Astro and Starlight. So is aspire.dev—Astro with Starlight, dressed up with the wider Starlight plugin ecosystem (astro-mermaid, starlight-llms-txt, starlight-sidebar-topics, starlight-image-zoom, the gorgeous @catppuccin/starlight theme, and more. Shout-out to Chris Swithinbank and the Starlight maintainers, the entire ecosystem feels designed by people who genuinely care).

There’s a real kinship there. The tool we use to automate docs and the docs site we automate into share the same foundation. Convenient, because the Mermaid sequence diagram in the next section renders the exact same way in both worlds.

The end-to-end pipeline

Here’s the flow we landed on. The protagonist is a workflow called pr-docs-check.md living in microsoft/aspire.

A run starts on pull_request: closed against main or release/*, gated by merged == true. From there, the workflow first runs a deterministic target branch resolver in plain bash before the agent ever wakes up:

Pull request milestone title (e.g. 13.4 → release/13.4 on aspire.dev).
Linked-issue milestone title (parse Fixes/Closes/Resolves #N from the body, fetch each issue, take the first non-empty milestone).
Pull request base ref, if it matches release/X.Y[.Z].
Fall back to main.

This is the linchpin. Milestones in the product repo map cleanly to release branches in the docs repo. When the agent finally runs, it knows exactly where the docs should land without any creative writing about target branches or guessing.

The agent reads the diff, scans linked issues, and decides: does this need docs? If yes, it drafts the actual content in a checked-out microsoft/aspire.dev workspace, following our existing doc-writer skill (voice, MDX conventions, Starlight components). It then emits a create_pull_request safe-output and hands off.

The safe-outputs handler takes over:

Title prefix: [docs]
Label: docs-from-code
draft: true (we never auto-merge)
Base branch: agent-supplied, restricted to main or release/*
Target repo: microsoft/aspire.dev
Reviewer: the SME identified from the source pull request’s reviews—i.e., whoever the product team trusted to approve the feature, now gets asked to approve the doc for that feature.

A companion job posts a marker comment back on the source pull request with the docs pull request link and minimizes any older pr-docs-check comments on re-run. The engineer who just hit Merge gets a notification within a few minutes: “Here’s the docs draft. Look it over?”

🔐 The safe-outputs contract

The whole security story comes down to a small, boring stretch of frontmatter:

tools: 
  github: 
    toolsets: [repos, issues, pull_requests] 
    min-integrity: approved          # only run pinned, integrity-checked actions 
    allowed-repos: 
      - microsoft/* 
    github-app: 
      app-id: ${{ secrets.ASPIRE_BOT_APP_ID }} 
      private-key: ${{ secrets.ASPIRE_BOT_PRIVATE_KEY }} 
      owner: "microsoft" 
      repositories: ["aspire.dev", "aspire"] 

safe-outputs: 
  create-pull-request: 
    title-prefix: "[docs] " 
    labels: [docs-from-code] 
    draft: true                      # human-in-the-loop, always 
    base-branch: main 
    allowed-base-branches: [main, release/*] 
    target-repo: "microsoft/aspire.dev" 
    protected-files: blocked         # AGENTS.md, manifests, security config: hands off 
    fallback-as-issue: true

That’s the deal in plain text. The agent gets a GitHub App token whose installation is scoped to exactly two repositories—the product repo and the docs repo—and nothing else in the org is reachable. It can only land pull requests against main or release/*. AGENTS.md and dependency manifests are off-limits by policy. If the pull request creation fails (network blip, conflict, anything), the framework falls back to filing an issue, so nothing is silently dropped.

This is the part security review actually liked. The agent’s reasoning is fuzzy. The action surface is not.

📊 By the numbers

Here are the stats from a rolling 30-day window (May 3 – June 2, 2026) spanning the back end of the Aspire 13.3 release and the run-up to 13.4:

Metric	Value
Product pull requests merged in microsoft/aspire	396 (338 main / 50 release/13.3 / 8 release/13.2)
pr-docs-check workflow runs	396
Draft docs pull requests created on microsoft/aspire.dev	82
– Merged	82 (100%)
– Closed without merge	0
– Still open	0
Docs pull requests target branches	52 → release/13.3, 27 → release/13.4, 3 → main
Median time-to-merge (docs)	44.8 hours
Merged within 24 h / 7 days	38% / 96%

Note: Numbers captured at the time of writing; the workflows keep running, so the totals only go up.

A few of those numbers deserve a second look:

396 runs → 82 pull requests is not a defect. The workflow runs on every merged pull request; most of them are internal refactors, test fixes, or dependency bumps with no user-facing surface. The agent saying “no docs needed” 300+ times is a feature.
100% merge rate says the agent’s docs picks are right. The tighter prompt we shipped after the v1 false-positive phase is paying off.

✅ What worked, ❌ what didn’t

What worked

✅ Milestone → release-branch mapping. This was the single highest-leverage choice we made. Engineers already set milestones on pull requests and issues; we got accurate target-branch routing for free.
✅ Draft-only, SME-as-reviewer. The agent never merges. The engineer who shipped the feature is the one who confirms the docs are right. We’ve stopped reverse-engineering features at the doc layer. The engineer just tells the docs draft what to say, in the place where they already are.
✅ Scoped GitHub app per workflow. Each workflow gets its own app token with explicit repo and permission scopes. Security review approved. We approved too; the first time we needed to rotate keys.
✅ protected-files: blocked. The agent cannot touch AGENTS.md, package manifests, or repo security config. Period.

What didn’t (at first)

❌ The agent’s “is this docs-worthy?” gate was too generous in the first version. It drafted pull requests for changes that were genuinely internal, such as a CI tweak or a logging refactor. The result: 9 closures of 69 pull requests (≈13%), so we tightened the prompt’s user-facing-change definition and added explicit negative examples (CI, internal helpers, tests-only). Now, the rate is trending down.
❌ Cross-repo pull request creation needed a mirrored checkout pattern that wasn’t obvious from the docs. The agent works in one repo; safe-outputs needs to find the target repo to push a branch. We solved it by checking out microsoft/aspire.dev twice—once as the current workspace, once under _repos/aspire.dev—so the safe-outputs handler can rediscover it deterministically.
❌ Big diffs blow prompt budgets. We pre-extract pull request metadata (linked issues, milestone, base ref) in pre-agent-steps bash, so the agent gets a small, structured summary instead of a giant payload. This is GitHub Agentic Workflow’s designed-in pattern, and it works.

Wrapping up

The changes we made shifted our thinking. A feature wasn’t considered done until the docs were. Docs no longer trail along behind it like a tin can on a string. The engineer’s review is the gate; the bot does the typing.

Critically, this doesn’t replace docs writers; it un-burdens them. Our writers used to spend most of their time reverse-engineering features. Now they spend their time on the things only a human can do well: narrative pages, sample programs, conceptual walkthroughs, the parts of the docs that don’t fall out of a diff. The bot handles the mechanical “this new option was added; here’s the reference page update” work that was never enjoyable for anyone.

Huge thanks to the GitHub Next team for GitHub Agentic Workflows (and for making the safe-outputs primitive a first-class part of the design), and to Chris Swithinbank and the Starlight maintainers for the docs platform we automate into. A genuine thank-you, too, to the security folks whose guardrails forced us to design this the right way the first time. The boring secret of good automation is that strong security constraints make the system more trustworthy and more correct.

If you build a product in one repo and ship docs in another—and especially if you have to do it inside any nontrivial security boundary—GitHub Agentic Workflows is worth a serious look. Start with one workflow, such as pr-docs-check, and watch what happens to your median time-to-docs.

🔗 The other workflows

pr-docs-check is the one I wrote this post about, but it’s not running alone. If you’re curious about the rest, the source is public:

milestone-changelog.md: runs every two hours, picks up newly merged pull requests in the active milestone, and maintains a 13.x-Change-log wiki page (new features, improvements, notable bug fixes) with a companion editorial-feedback issue. 346 runs.
release-update-support-mdx.md: on a stable Aspire release, drafts a [support] pull request on aspire.dev that updates the support policy page (promotes the new version, demotes the previous one, refreshes the “Last updated” badge).
update-integration-data.md: lives in the docs repo; runs pnpm update:all daily, refreshes NuGet metadata + GitHub stats + sample data, and opens a chore: Update integration data PR with supersede-and-close logic for stale runs. 27 runs, eight merged pull requests.
repo-pulse.md: a rolling three-day repo dashboard pinned to a single issue and updated in place: recent merges, pull requests awaiting review, new issues, discussion activity. One issue, always fresh.

Happy automating, friends! 🤖🚀

The post Automating cross-repo documentation with GitHub Agentic Workflows appeared first on The GitHub Blog.

GitHub availability report: June 2026 - The GitHub Blog

2026-07-08T19:35:51.000Z

Last month, we added a new section to our monthly reports with updates on GitHub’s availability work and the infrastructure investments behind it. The response and the questions we heard made one thing clear: customers want more of this, not less—including when the news is mixed.

The short version of June: We made real structural progress, we paused deliberately when a ramp went sideways, and we missed a target that we’ve now re-baselined.

Monolith traffic in Azure peaked at 45% in Central US this month. That number is lower than we’d hoped, because we paused the ramp for roughly a month after a stability incident on May 21 made it clear the environment wasn’t ready for more traffic. We restarted the ramp on June 17 with a new per-turnup stability gate that requires the environment to be verifiably healthy before each step up. Going more slowly with more confidence is the right trade—a controlled pause is preferable to relearning the same lessons at higher load.

Git in Azure grew from 30% to a peak of 43% (HTTP and SSH combined) over the month, and we missed our June target of 50%. We expect Git to plateau near 45% for now because of two deliberate decisions to avoid added user latency: we are waiting on additional vPoP traffic to route to Central US rather than backhauling IAD HUB Git traffic, and we are routing only HTTP for now because SSH has no read/write split at the edge. We continue to prioritize this, but we don’t have a new target to share. We’ll keep working as quickly and safely as possible and will report the specific numbers in our next update.

Underneath those headline numbers, we made progress that we are particularly excited about. Our new extracted pull requests service, pullsd, is now handling 100% of anonymous pull request reads in production; this traffic is no longer served by the monolith. Reposd, our new extracted repository service, became the first extracted service to serve production REST traffic from Azure, ramping to 50% of read traffic before we proactively turned it down for a Redis capacity constraint. There was no incident and no rollback under duress. It will re-ramp once that capacity work completes. Our new users service is now offloading roughly 500,000 queries per second at peak from our primary database, with the physical migration of authentication and authorization tables landing in early July. Our API rate limiting is now approximately 97% handled at the Gateway, so rate-limiting decisions no longer contend with request-serving workers inside the monolith. Client-side database load shedding is running against 5% of real production traffic, which means we now have live evidence that we can shed low-priority queries under stress before they cascade into user-facing failures. And two-person confirmation is now required end-to-end for interactive production access and ChatOps changes, with a unified audit trail behind it.

The incident write-ups that follow are the other half of the picture: what the system did well and what it didn’t, what we’ve already changed as a result, and what we’re still changing. The same principle continues to guide us: availability, then capacity, then features.

In June, we experienced six incidents that resulted in degraded performance across GitHub services.

June 04 17:30 UTC (lasting 1 hour and 25 minutes)

On June 4, 2026, from 17:30 to 18:55 UTC, Copilot code review experienced elevated failures for review requests on github.com. Affected users saw “Copilot ran into an error” on pull requests when requesting a code review.

During the incident window, an average of 81.6% of Copilot code review requests failed, with a peak failure rate of 93.9%, and a total of approximately 36,800 code review requests failing. GitHub Enterprise Cloud with data residency was not impacted.

The issue was caused by a newly released dependency used by the Copilot code review processing workflow. The release introduced an incompatibility with the runtime environment. Because the workflow automatically consumed the latest release, the incompatible version was picked up without sufficient compatibility validation and caused review processing to fail. Affected review jobs did not fail fast; many continued running until they timed out.

We mitigated the incident by removing the problematic dependency version and redeploying the affected processing service. New code reviews began recovering at 18:44 UTC, and the failure rate returned to baseline by 18:55 UTC. Remaining timed-out work drained by 19:59 UTC.

To reduce the risk of recurrence, we are pinning the dependency version instead of automatically consuming the latest release, adding compatibility checks for future releases, improving fast-failure behavior when the review processor cannot start, adding shorter timeout controls for review workflows, and improving monitoring for review completion failures.

June 08 06:30 UTC (lasting 2 hour and 06 minutes)

On June 8, 2026, between approximately 06:30 and 08:36 UTC, signed-out users experienced sustained elevated HTTP 504 errors when accessing pull requests, issues, releases, patch diffs, and other related github.com pages. During the incident, approximately 17% of unauthenticated requests to the affected github.com endpoints returned gateway timeout errors, peaking at roughly 34% of requests at around 06:50 UTC. Some GitHub Actions workflows were also affected when they depended on release downloads or related github.com endpoints. The impact lasted approximately two hours and was isolated to unauthenticated traffic; signed-in users were not affected.

The issue was caused by a significant increase in abusive, automated anonymous traffic to specific github.com endpoints. Because unauthenticated requests are served by a dedicated pool of web application servers, it degraded our ability to respond to unauthenticated requests, causing requests to queue beyond timeout thresholds and return gateway timeout errors.

We mitigated the incident by identifying the anomalous traffic pattern and applying targeted blocks at the load balancer and application layers. Once the blocks took full effect, error rates returned to normal and affected services were fully restored by 08:36 UTC.

To reduce the likelihood and impact of similar incidents in the future, we are improving automated detection and blocking for these traffic patterns, improving our emergency traffic-blocking deployment path, and evaluating routing changes for endpoints used by both signed-out users and automated workflows.

June 10 15:05 UTC (lasting 1 hour and 20 minutes)

On June 10, 2026, between 15:05 and 16:25 UTC, GitHub API services experienced degraded availability due to sporadic authentication failures affecting approximately 9% of requests. Both REST and GraphQL API requests were affected. Customers experienced intermittent “logged out” behavior as erroneous 401 (unauthorized) responses caused first- and third-party app integrations to trigger repeated authentication flows. Because only requests routed through the affected infrastructure failed, the same client could succeed on one request and fail on the next, producing the intermittent behavior. Affected requests also experienced approximately 800ms of additional latency, as the gateway retried authentication before returning an error.

A memcached proxy service, rollout to our internal API infrastructure caused our authentication service to pick up an incorrect host configuration, leading to intermittent authentication lookup failures. We mitigated the incident by deploying a configuration change to memcached service to use the correct host.

To prevent similar issues in the future, we plan to migrate our authentication system to the new caching infrastructure to improve resilience and strengthen overall reliability posture. We are also improving how the gateway distinguishes transient authentication-system errors from genuinely invalid credentials, so that temporary lookup failures no longer appear to users as being logged out.

June 16 17:20 UTC (lasting 55 minutes)

On June 16, 2026, between 17:20 and 18:15 UTC, the Opus 4.8 model experienced degraded availability in GitHub Copilot. During this window, some requests to Opus 4.8 failed or errored. Other Copilot models were not affected and remained available as alternatives. This was caused by an issue with an upstream model provider.

While the issue was ongoing, we enabled degraded-mode messaging to inform affected users. The upstream provider resolved the issue, and we monitored Opus 4.8 until success rates returned to normal. The incident is fully resolved.

We are reducing our reliance on any single inference provider for a given model and balancing capacity across providers, so traffic can fail over to healthy capacity during an upstream outage. Separately, we hardened our public status-page tooling to ensure incident updates publish reliably, improving how we keep customers informed during mitigation.

June 17 03:50 UTC (lasting 54 minutes)

On June 17, 2026, between approximately 03:50 and 04:44 UTC, GitHub Copilot was degraded and most of its frontier chat models were temporarily unavailable across all regions. During this window, affected models either disappeared from the model picker in the web, editor, and CLI experiences, or returned a “model not available” error when selected. Customers could continue using GitHub Copilot by selecting one of the models that remained available. The incident occurred during off-peak hours, which limited the number of customers affected.

This was due to a configuration change that our production system deemed invalid. We mitigated the incident by reverting the configuration change, after which the affected models returned automatically as the service reloaded the previous configuration.

We are working to roll out configuration changes gradually with stronger validations, alerts on sudden drops in the number of available models, and automatically roll back configuration changes that trigger these alerts.

June 25 17:33 UTC (lasting 23 minutes)

On June 25, 2026, between 17:33 and 17:55 UTC, our background job service experienced degradation which increased delays to pull requests, repository pushes, actions workflows, and webhooks, with delays peaking at 7 minutes. The issue was caused by underlying hypervisor issues and an incoming traffic spike, causing service timeouts which led to a connection storm and continual rebalances.

The issue was mitigated by replacing the impacted node at 17:49, after which all services saw recovery by 18:07.

To reduce the likelihood and impact of similar incidents in the future, we have made background job processing more resilient to sudden traffic spikes, reduced the possibility of connection churn in degraded cases, removed the co-location of critical nodes so that one unhealthy node cannot affect others, and added earlier alerting on the conditions that led to this degradation.

Follow our status page for real-time updates on status changes and post-incident recaps. To learn more about what we’re working on, check out the engineering section on the GitHub Blog.

The post GitHub availability report: June 2026 appeared first on The GitHub Blog.

How GitHub Copilot enables zero DNS configuration for GitHub Pages - The GitHub Blog

2026-07-08T16:00:00.000Z

Custom domains make a project feel real. But for many developers, DNS, the last mile, is also the most frustrating: A records, CNAME entries, TTLs, and that long wait where you’re never quite sure if the internet is broken or you are.

In this post, I’ll walk through how I took a project from an empty repository to a live website on a custom domain, secured with HTTPS, in about 14 minutes without manually editing a single DNS record. The trick is to let GitHub Copilot CLI drive the work, with a community Namecheap skill handling the DNS automation through the registrar’s API.

Here’s what you’ll learn how to do:

Publish a site with GitHub Pages
Register an inexpensive domain
Enable your registrar’s API and connect it to Copilot CLI
Point the domain at GitHub Pages and verify it end to end

What you’ll need

A GitHub account (the free tier works)
GitHub Copilot CLI, installed and authenticated with GitHub Copilot
A Namecheap account, for buying the domain and using its API

No prior DNS expertise required. That’s the whole point. Let’s get started. ⤵

Step 1: Publish a site with GitHub Pages

Every deployment needs something to deploy, so start with a home for the site: a new public repository.

With the repository in place, you don’t have to hand-write an index.html, commit it, and then click through the pages settings yourself. Instead, describe the outcome you want to Copilot CLI and let it create the landing page and enable GitHub Pages for you.

The site is now live on a github.io URL. That’s a solid start. Now let’s give it a proper address.

Step 2: Register an inexpensive domain

You don’t need a premium .com to ship a side project. For this walkthrough I chose one of the cheapest top-level domains available, .click, and searched for an available name.

ghpagesblog.click was available, so I moved to checkout.

The total came to USD $2.00, or about CAD $2.46. That’s a low-risk price for trying a custom domain on a side project.

Step 3: Connect the domain to GitHub Pages

This is the step developers tend to dread. Here, an AI assistant does the repetitive work while you stay in control of the decisions.

Enable Namecheap API access

Before Copilot CLI can update your DNS, you need to turn on Namecheap’s API. In your Namecheap account, go to Profile → Tools, scroll to Business & Dev Tools, and select Manage under Namecheap API Access.

You can also navigate directly to the API access settings page (note that this URL may change over time).

On that page, complete three steps:

Toggle the API to ON.
Add the public IP of the machine that will call the API to the IP allowlist (Namecheap labels this field Whitelisted IPs).
Copy the API Key and store it somewhere safe. You’ll need it shortly.

For more detail on what the API offers, see Namecheap’s API introduction.

Install the Namecheap skill

Next, give Copilot CLI the ability to talk to Namecheap by installing the Namecheap skill. It’s a single command:

gh skill install github/awesome-copilot namecheap --scope user

The first time you ask Copilot to do something like “list my Namecheap domains, it confirms the skill is configured and prompts you for your username.

Then it asks for the API key you copied earlier.

With credentials in place, Copilot returns the list of domains in your account. It’s a quick way to confirm everything is wired up correctly before making any changes.

Point the domain at GitHub Pages

Now connect the domain to the site. Ask Copilot to configure the custom domain using the skill.

A good automation asks before it acts. The skill pauses to confirm the change before touching any records.

Once you approve, it replaces the existing parking records with the GitHub Pages A records and a CNAME for the WWW subdomain, which is the exact configuration GitHub Pages expects. This matches GitHub’s documented steps for configuring a custom domain for your GitHub Pages site.

It also handles the repository side, committing a CNAME file that tells GitHub Pages which custom domain the site should answer to.

Not using Namecheap? The same approach works with any registrar that offers an API. You don’t need a purpose-built skill: point Copilot CLI at your registrar’s API documentation and ask it to read, understand, and use that API to set the GitHub Pages records for your domain. The registrar changes; the workflow doesn’t.

Step 4: Verify the deployment

Rather than assuming success, Copilot CLI checks its own work. First, it confirms the domain resolves.

Then it confirms that the site returns a healthy HTTP 200 response.

If you’d like to review every prompt and response, the full Copilot CLI session is available as a gist.

Now for the timeline. The domain was purchased at 11:21:27 a.m. ET.

The site was live on the custom domain, served over HTTPS, at around 11:35 a.m. ET. That’s roughly 14 minutes from owning nothing to a fully deployed site, including API setup, skill installation, DNS configuration, propagation, and verification.

Wrapping up

DNS isn’t hard, exactly, but it’s fiddly, easy to get wrong, and slow to give feedback. By pairing GitHub Pages with GitHub Copilot CLI and the Namecheap skill, the repetitive parts of a custom-domain deployment fade into a short conversation: you make the decisions and approve the changes, and the tooling handles the plumbing.

If you’ve been putting off a custom domain because the DNS step feels like a chore, this workflow removes the friction. To go further, explore the GitHub Pages documentation and the guide to configuring a custom domain for your GitHub Pages site, then try it on your next project.

The post How GitHub Copilot enables zero DNS configuration for GitHub Pages appeared first on The GitHub Blog.

Q1 2026 Innovation Graph update: Open source collaboration is accelerating worldwide - The GitHub Blog

2026-07-07T16:00:00.000Z

Q1 2026 was a banner quarter for open source collaboration, as shown by the GitHub Innovation Graph’s economy collaborators metric, part of the latest data release. Outbound collaboration, defined as the sum of git pushes and pull requests sent from developers in one economy to public repositories in another economy, grew by 16% quarter-over-quarter from Q4 2025 to Q1 2026.

That’s the second highest quarter-over-quarter growth rate we’ve seen since 2020. The highest was in Q2 2020, with a 21% growth rate, when many of us suddenly and collectively decided to use our computers more.

In third place was Q1 2023, with a 9% growth rate. This was the first quarter after a research lab blogged about a new website they made, and they enticed users to sign up and file bug reports by offering a chance to win up to $500 in API credits. Evidently, sweepstakes are unreasonably effective motivators.

Metrics by economy

While it’s clear that collaboration is growing globally, plotting these and other metrics separately by economy highlights the different trajectories of the world’s developer communities:

Now, it’s your turn to analyze the underlying datasets yourself for interesting pursuits, such as improving how economic growth is measured or estimating the impact of a research lab’s new website. While we have no sweepstakes to offer (for now), we think there are fascinating data stories strewn throughout these CSVs just waiting to be told.

One example might be the impressive recent growth in Syria starting in Q4 2025, which coincides with changes we made to enable broader access to GitHub functionality following the relaxation of sanctions and export controls on the country:

We’re grateful to the developers who advocated for these changes and the communities that continue to help spread the word, such as GitSyria. Thanks to their efforts, we’re happy to share that we’ve been able to provide the GitHub Student Developer Pack to over 8,000 verified Syrian students in just the last six months.

Helping maintainers manage collaboration

While greater collaboration leads to many benefits, we recognize that the rapid increase in contribution volume has strained several communities. Ashley Wolf, our Director of Open Source Programs, wrote about the Eternal September of Open Source in February 2026, describing how maintainers are managing new contribution dynamics and what we’re doing to try to help. Features we’ve shipped include:

Pull request limits: You can set a maximum number of open pull requests that users without write access may have open in your repository at one time, giving you a more proactive way to manage contribution volume.
Repo-level pull request and issue controls: Gives maintainers the option to limit pull request and issue creation to collaborators or disable pull requests and issues entirely.
Pinned comments on issues: You can now pin a comment to the top of an issue from the comment menu.
Banners to reduce comment noise: Experience fewer unnecessary notifications with a banner that encourages people to react or subscribe instead of leaving noise like “+1” or “same here.”
Pull request performance improvements: Pull request diffs have been optimized for greater responsiveness and large pull requests in the new files changed experience respond up to 67% faster.
Faster issue navigation: Easier bug triage thanks to significantly improved speeds when browsing and navigating issues as a maintainer.
Temporary interaction limits: You can temporarily enforce a period of limited activity for certain users on a public repository.

Do you have feedback on the directions we’re exploring? Share it in the community discussion.

Share what is working for your projects, where the gaps are, and what would meaningfully improve your experience maintaining open source.

The post Q1 2026 Innovation Graph update: Open source collaboration is accelerating worldwide appeared first on The GitHub Blog.

How GitHub used secret scanning to reach inbox zero - The GitHub Blog

2026-07-02T16:00:00.000Z

Several years ago, GitHub Security launched an initiative to assess and improve our overall secrets hygiene. As part of that effort, we piloted the Secret Scanning capability that was under development at the time. That’s when we found more than 20,000 secrets spread across our 15,000+ repositories.

The number was significantly higher than we anticipated, but it quickly became clear that success would depend on identifying which alerts represented real risk, assigning ownership, and remediating them safely. Nine months later, we reached zero open alerts.

New secret scanning customers often ask us: “How do you manage this internally? How did you actually clean up your existing secrets?”

Like many long-running software companies, GitHub’s approach to secrets management evolved over time. GitHub was founded in 2008, before today’s centralized vaults, automated secret scanning, and dedicated secrets-management platforms were common across the industry. As engineering practices matured and GitHub grew, we continued investing in stronger controls, better tooling, and systematic risk reduction for legacy patterns. This work reflects our ongoing commitment to improving security, reducing exposure, and ensuring our internal practices meet the same high standards we expect across the industry.

This blog post shares what worked for us during this effort, and highlights strategies you can apply to better protect your own secrets.

Cutting out the noise

The first thing we discovered was that the alert count was a bit misleading—i.e., 20,000 alerts did not mean 20,000 equally risky problems.

When we dug into the data, we discovered that just five repositories accounted for roughly 18,000 of those alerts, and every one of those secrets was inactive: test fixtures, deactivated credentials, and fake-but-valid-looking secrets used for testing. (We build secret scanning, so naturally we have repositories full of legitimate-looking secrets in tests.)

That left over 2,000 alerts that needed attention: potential live credentials and thousands of decisions about risk, rotation, and remediation.

Secrets don’t just live in code

Secret remediation touched more than source code. We found secrets in support tickets (customers occasionally include tokens), bug bounty reports (researchers disclose what they found with complete reproductions, including API requests with tokens used), incident notes, and wiki pages.

We partnered with customer support, security incident response, and our bug bounty program to develop shared playbooks. Across all these workflows, we had to ensure we weren’t creating new problems, like opening issues or pushing commits containing the very secrets we were trying to remediate.

Our phased approach

We were not going to close 20,000 alerts by asking a few security engineers to grind through them one by one. We treated it like any other operational backlog: stop new debt, then work down what already exists with a workflow that’s repeatable, measurable, and not dependent on one person’s institutional knowledge.

Phase 1: Enable everywhere, stop the accumulation

Before cleaning up existing secrets, we had to stop new ones from piling up.

We enabled secret scanning and push protection across all of our enterprises and organizations. Thanks to GitHub Advanced Security’s organization-level settings, this wasn’t a repository-by-repository slog across 15,000 repositories. We enforced the setting so individual repositories and teams could not quietly opt out.

Push protection blocked new secrets at the source. That kept the backlog from growing faster than we could burn it down.

Phase 2: Understand and triage

We broke down the 20,000+ alerts by repository, secret type, and age so we could separate noise from work.

When we dug in, we discovered that just five repositories accounted for roughly 18,000 of those alerts, and every one of those secrets was inactive: test fixtures, deactivated credentials, and fake-but-valid-looking secrets used for testing. (We build secret scanning, so naturally we have repositories full of legitimate-looking secrets in tests.)

For high-volume, low-risk alerts, we developed criteria for bulk closure. If a secret was in a dedicated test repository, had never been active, and matched a known test pattern, we could confidently mark it resolved. In a matter of days, we closed out roughly 18,000 alerts.

The hard questions

We had to make strategic decisions about how to remediate secrets. When a secret lives in an issue, do you edit the body (and potentially remove revision history), or preserve the audit trail? When a secret is committed to a repository, do you rewrite git history? Anyone who’s tried rewriting git history at scale knows what happens next: force-pushes break open pull requests, invalidate commit SHAs, and generally interrupt developers.

A common question was: “Can we just delete the repository if it’s no longer in use?” Our answer was generally no. A deleted repository takes its audit trail with it. If a secret in that repository was ever leaked or the repository was ever compromised, you lose the forensic record you’d need during incident response. Rotate the secret, archive the repository if appropriate, but keep the history.

Whenever possible, we rotate or revoke the exposed secret first. The harder question is whether the residual risk warrants rewriting git history, or whether a revoked secret in history can safely be left in place. These are the types of questions and decisions present with each alert that product security teams wrestle with.

Phase 3: Validate what’s actually live

A credential sitting in a repository might have been rotated years ago, or it might still unlock production systems. You can’t prioritize without knowing the difference.

At the time, secret scanning didn’t have native validity checking, so we built our own approach. The goal was narrow: determine whether a credential still worked and, when appropriate, collect enough metadata to route the alert or notify the right owner.

For example, for a GitHub token, a representative check could make a single authenticated request to a low-impact endpoint like GET /user:

response="$( 
  curl -sS -w '\n%{http_code}' \ 
    -H "Authorization: Bearer $TOKEN" \ 
    -H "Accept: application/vnd.github+json" \ 
    -H "X-GitHub-Api-Version: 2022-11-28" \ 
    https://api.github.com/user 
)" 

status="${response##*$'\n'}" 
body="${response%$'\n'*}" 

case "$status" in 
  200) 
    login="$(jq -r '.login // empty' <<< "$body")" 
    echo "token appears active for GitHub user: $login" 
    ;; 
  401) 
    echo "token appears invalid or revoked" 
    ;; 
  403|429) 
    echo "unable to determine validity; rate-limited or blocked" 
    ;; 
  *) 
    echo "unable to determine validity: HTTP $status" 
    ;; 
esac

Remember, our goal was to answer the smallest useful set of questions: does this credential still work, and who needs to know about it? We treated ambiguous responses as inconclusive, and we avoided follow-on requests to repositories, organizations, or other private resources.

This required close partnership with our privacy and legal teams. Even a “read-only” validity check can have implications when you’re touching a credential you may not own.

As we worked through this manually, our product team built the solution natively, which made the remaining work much faster. Validity checking is now built into GitHub secret scanning.

Phase 4: Figure out who owns what

That cross-functional work also exposed an ownership problem: even after we knew a credential was active, we still had to figure out who could rotate it.

We partnered with customer support, security incident response, and our bug bounty program to develop shared playbooks for secrets reported outside of code. That included redacting secret values before routing work to teams, determining whether a credential belonged to GitHub or a customer, and notifying affected customers or researchers so they could rotate tokens under their control. Across all these workflows, we had to ensure we weren’t creating new problems, like opening issues or pushing commits containing the very secrets we were trying to remediate.

For GitHub-issued credentials like personal access tokens, we worked with our product team to surface secret metadata directly in the alert: who created the token, when, and what scopes it had. That meant we didn’t need to use the token itself to figure out who it belonged to.

For everything else, ownership was harder, and this exposed a deeper problem: not all repositories had clear owners.

Our internal engineering standards (the Engineering Fundamentals program) enforce durable ownership on services, and we maintain a mapping between services and repositories, but not all repositories map cleanly to a service. The pain we experienced led to a broader repository ownership initiative (using GitHub’s Custom Properties), plus a parallel effort to ensure all secrets in our credential manager have durable owners. You can’t rotate a secret if you can’t find the owner.

Phase 5: Manual triage for the long tail

Even with validation and metadata, a long tail of alerts required human judgment. For each one: what does this grant access to, has it been rotated, who owns the connected system, and what’s the remediation path?

For every alert we dismissed, we ensured an accurate disposition (e.g., revoked, used in test, false positive) was recorded, along with a comment containing relevant context, such as a link to a remediation issue or an approved security exception.

This phase required close collaboration across teams to identify system owners, validate remediation status, and assess residual risk where automated signals alone were insufficient.

Phase 6: Systematize and drive accountability

As patterns emerged, we made the work scalable:

We routed alerts into our internal vulnerability management platform for centralized tracking and reporting.
Different credentials need different remediation steps. We documented playbooks by secret type so teams could self-serve.
We automated notifications, routing alerts to the right teams based on repository ownership.

The final piece was accountability. We tied secret remediation to GitHub’s Engineering Fundamentals program, making it a security fundamental that teams were measured against. We set clear expectations and gave teams visibility into status. When secret hygiene is part of how engineering health is measured, it becomes a shared responsibility across the organization.

Nine months after we started, we hit inbox zero.

Lessons learned

Don’t panic at the number. Our initial count was 20,000+ alerts, but 90% were not valid. The raw count is almost never the real scope of work.
Enable and enforce everywhere, no exceptions. Partial rollouts create blind spots. We enabled and enforced secret scanning and push protection at the enterprise level, without allowing anyone to opt out.
Validate before you escalate. Not every detected secret is live. Validation helps you create a prioritized to-do list.
Metadata saves hours. For GitHub credentials, secret metadata cut down the necessary detective work. If you’re working with third-party providers, push them to surface similar metadata, or build your own enrichment layer.
You can’t remediate without ownership. Invest in durable ownership infrastructure early.
Automate the workflow after detection. Detection gets you started, but the operational challenge was routing alerts, tracking owners, and closing the loop. Invest in the workflow layer.
Make it everyone’s problem. Security teams can’t remediate thousands of alerts alone. We tied secret hygiene to our Engineering Fundamentals program. When leadership watches the dashboards, teams find time to fix things.
Document your decision framework. You’ll encounter secrets without clean remediation paths. Document how you decide: When is rotation sufficient? When do you rewrite history? When do you accept residual risk?

What this means for you

You don’t need to reinvent most of what we built. Many of our manual workarounds, including validity checking, ownership identification, and bulk triage, are now native features in secret scanning.

If you’re starting today:

Enable and enforce secret scanning and push protection everywhere.
Triage the backlog by repository and secret type; bulk-close what you can prove is noise.
Validate what’s live before you escalate.
Route alerts to owners, and track remediation like any other engineering work.

Ready to get started? Learn how to enable secret scanning and push protection with GitHub Advanced Security.

Coming soon: How we tackled repository ownership at scale, and why durable ownership of repositories and secrets is the foundation everything else depends on.

The post How GitHub used secret scanning to reach inbox zero appeared first on The GitHub Blog.

Meta’s AI Storage Blueprint at Scale - Engineering at Meta

2026-07-01T16:00:36.000Z

Over the past several years, model capabilities and training dataset sizes have experienced exponential growth. During the past year or so, the time between new-frontier-model releases has gone down from months to weeks. Reliable and fast access to storage is important to both the speed and computational cost of this AI innovation. If AI is the brain, storage is the memory: Capability and speed are highly dependent on the size of memory and speed of retrieval.

Yet while AI compute performance has roughly tripled every two years, storage and interconnect performance growth have been more modest. As a result, storage bottlenecks continue to be one of the primary contributors to GPU stalls for AI workloads, directly impacting expenditures and time to market. Aside from GPU utilization, storage architecture also directly impacts the speed of iteration in AI research; with GPUs increasingly becoming geo-distributed and dataset sizes increasingly becoming massive, researchers spend a significant amount of time ingesting and moving data across regions, thus impacting research velocity. In this blog post, we discuss how Meta’s BLOB-storage architecture evolved to address two primary challenges: maximizing GPU utilization and maximizing research velocity.

Storage Architecture Overview

Meta operates hundreds of exabyte-scale storage clusters that serve all of Meta’s external and internal products, including Facebook, Instagram, Reality Labs, Meta AI, Ads, Data Warehouse, and internal Databases. Our storage service exposes object storage, file systems, and block-device APIs, and these API abstractions are built on top of a horizontally scalable foundational block layer called Tectonic. The Tectonic layer is a regional, multi-tenant storage fabric that provides high durability and availability leveraging erasure-coding techniques, supports tiering across media types (e.g., HDD and flash), and manages smart placement of hot, cold, and warm data for efficient utilization of I/O across tenants. The BLOB-storage layers that operate on top of Tectonic expose a global, infinitely scalable storage fabric, and expose policies that let users make tradeoffs between durability and availability.

In a previous @Scale talk titled, “Training Llama: A Storage Perspective,” we discussed how Meta trained Llama directly over the Tectonic block layer by exposing an NFS-like FileSystem interface on top of it. While this architecture continues to be used widely within Meta, our modern training stack has been migrating slowly on top of the BLOB-storage interface, as is the case across the industry. This transition is motivated by the need for unified storage access to massive data lakes in the BLOB-storage layer as well as the need for high performance.

Maximizing GPU Utilization

Modern AI workloads are “data hungry” and have very different workload characteristics than traditional web applications: bursty and sustained high throughput, predictable and bounded pMax latencies, and variable I/O patterns. The focus for BLOB storage, in recent years, has largely shifted to maximizing GPU utilization.

Why Latency Matters

To see why bounded and low-pMax latencies are important, let’s consider model training. During that training, hundreds of thousands of GPUs iterate over vast amounts of data in storage multiple times (i.e., over multiple epochs), and the GPUs train datasets in batches. Periodically, after every certain number of steps or batches, the GPUs synchronize their state among themselves. If one GPU is slow, this step will slow down all GPUs as well as the entire training.

Figure 1 shows a data-loading pipeline across two GPUs. The dataloader in every GPU host prefetches the next dataset batch, while the GPU is processing the current batch for maximum compute or I/O overlap. In the case of GPU1, the storage-fetch latency is well within bounds, so the GPU is never stalled waiting on I/O. In the case of GPU2, there are two instances where storage fetch exhibits high latency, stalling GPU. As a result of these stalls, the overall step-completion time is delayed.

Figure 1: Dataloading across two GPUs.

Legacy BLOB-Storage Architecture Wasn’t AI-Ready

Over the years, BLOB storage evolved organically, adding layers on top of layers in a true service-oriented fashion. Many of these layers were stateful and maintained their own metadata stores. While these metadata-access latencies typically weren’t the bottleneck for the traditional use cases served by global HDDs, they were showstoppers for AI workloads with millisecond access to data in flash. Figure 2 shows the request flow for a typical getObject(“/bucket/path”) API. After the request arrives at the API server, the server does many metadata lookups across the namelayer, volumeslayer, and containerlayer before resolving the path to a set of (blockId, offset, size) tuples. Some of these lookups can cross regions, and it’s not uncommon for latencies to add up to hundreds of milliseconds; one slow response from any of the lookups was sufficient. After the lookups, the API server proxies the data from the Tectonic layer to the client.

Figure 2: Old request flow for getObject API.

While this architecture served conventional workloads well, the foundational assumptions that dictated design tradeoffs have since shifted. Some of these are:

Performance and latency: As discussed, while latency needs for conventional workloads were modest, AI workloads demand predictable and bounded latencies all the way up to pMax.
Reliability and durability: The legacy architecture was designed to be highly durable and available, even in the face of region outages; data and metadata were globally replicated by default. While AI workloads demand very high availability, the global-by-default design choice no longer holds.
Cost efficiency: Legacy stack was built on top of HDDs and highly optimized for cost per byte. The IOPS demands for AI workloads necessitate flash, and in addition, the computational cost of storage becomes negligible relative to the computational cost of GPUs.
Power efficiency: With GPUs, datacenters are increasingly power constrained rather than space constrained. Every kilowatt of power spent on storage is power not spent on GPUs. This is a new constraint with AI workloads.

In short, the tradeoff space has shifted enough for us to rethink the entire architecture.

Rebuilding the Foundation

As we set out to build the new foundation, we made the following major design choices:

Unified metadata schema: We rewrote the metadata subsystem and collapsed the metadata spread across different layers into one unified and flat schema backed by ZippyDB. This paves the way for O(1) lookup to resolve paths to storage addresses, which is a step-function improvement.
No dataplane proxy: We eliminated the dataplane proxy and built a fat client SDK that is capable of streaming bytes directly from storage servers to the clients. This helps with power-efficiency goals and also helps achieve higher throughput/lower latency.
Regional deployment: The BLOB-storage stack is now lean with flexibility to be deployed as a regional or global service. We now deploy a regional BLOB-storage stack colocated with GPUs in every AI region.

Figure 3: New request flow for getObject API.

Figure 3 shows the new request flow for getObject(“/bucket/path”). When the SDK on the client receives this API call, it now issues a getReadPlan(“/bucket/path”) request to the API server. The API server does O(1) lookup per chunk to the new metadata store to map the path to (blockId, offset, size) tuples. It then returns the ReadPlanResult to the SDK. The SDK has Tectonic BlockClient embedded within it, and so is now able to stream data from these blocks directly from Tectonic. With these changes, we have rebuilt the foundations and met the goal of adding zero overhead on top of Tectonic. By eliminating the data proxy, we also stay within budget for the power footprint.

Dealing With Spikes and Hot Spots

During data and checkpoint loading, AI workloads are known to access data concurrently across hundreds of GPUs. Subsets of data such as model weights are often “hot,” and events such as GPU restarts trigger sharp traffic spikes. With the foundations now fixed, our next problem was dealing with those spikes and hot spots. Luckily, the BLOB-storage layer has had experience dealing with hot spots over the years, so we adapted existing solutions to AI workloads here. Specifically, we employed two approaches:

Distributed data cache: We leveraged the spare memory on the GPU hosts as a distributed data cache for frequently and concurrently accessed data. To achieve this, we reused components from Meta’s Owl subsystem: We integrated the peers in the Owl subsystem directly into the BLOB-storage client SDK so that all data access goes through this data cache.
Readplan metadata cache: Readplan refers to the mapping from path to storage address. We now cache the read-plan for frequently accessed BLOBs in a distributed-memory store similar to memcache.

In practice we observe an average cache hit rate of 80% on the distributed data cache, and the read-plan cache provides 1-2 ms access to metadata. In essence, these simple mechanisms do three things:

Absorb the spikes and reduce the I/O requirements from storage.
Solve the problem of metadata hot shards.
Improve p50 and p99 latencies by serving from memory.

Protocol Optimizations

What we’ve discussed so far got us 80% of the way. We achieved the remaining 20% by identifying and fixing bottlenecks across the stack. Below are some noteworthy problems, though not an exhaustive list by any means:

Laggards: One slow storage node contributing to tail latencies. This is a well-understood problem, and we resorted to hedged reads on the client side to mitigate this.
Egress spikes: During checkpoint events, it is common for the client to create sharp egress spikes. This in turn can cause congestion, timeouts, and retries, eventually stalling GPUs. We resolved this by building dynamic concurrency control on the client SDK to automatically tune parallelism based on application-level congestion signals.

With all of the above, the new BLOB-storage stack is now capable of serving AI workloads without causing GPU stalls, adding negligible overhead on top of the Tectonic layer. Our next focus shifted to research.

Maximizing Research Velocity

GPUs are scarce and increasingly becoming geo-distributed; at the same time, training workloads need data colocated with GPUs for performance reasons. This creates an interesting challenge for researchers: They are now on the hook for ingesting and moving datasets across regions.

At Meta, a typical training-job submission involves the following:

A researcher curates data from various sources, enriches them and persists them in BLOB storage.
The researcher picks a region where they want to run the job.
The researcher submits a data-ingestion job, which creates a snapshot of the training datasets onto the target region in a file format optimized for data loading from within the GPU host.
The researcher then waits for ingestion to finish; depending on the dataset size, that can take hours.
The researcher submits their training job and monitors their run.
The researcher analyzes outputs, tweaks datasets, and iterates again, starting with Step 3.

Steps 2 through 4 can take hours and directly impact the speed of iteration for researchers. Ideally, we like our researchers’ time to be spent on tuning models, not waiting for storage. Currently, researchers copy snapshots before starting their jobs to colocate data with GPUs, which results in the most optimal performance. While this optimization for performance makes sense for large-scale training jobs that span weeks or months, the vast majority of jobs are much smaller; the researchers owning these jobs are more than willing to trade off occasional performance degradation for iteration speed.

And so, we needed a system where researchers are able to ingest data once and access data anywhere without thinking about regional boundaries. We needed a workflow that allows researchers to iterate in minutes and not hours. As we went back to the drawing board, the write-once, read-many characteristic of these datasets rang a bell. What if we think of storage as a disk in a planet-scale computer and borrow ideas from the operating-system world? When a Linux process running on a CPU core attempts to read a file from disk, the operating system transparently hydrates data on demand across the various layers of the cache—page cache in memory and L2 and L1 CPU caches. This intuition led to the architectural evolution in Figure 4:

Figure 4: Dataloading architecture evolution.

The core idea is to leverage the various on-host and off-host storage resources as a tiered cache with global BLOB-storage fabric backed by HDDs as the ultimate source of truth. Specifically, we leverage the memory and flash on the GPU host as L1 and L2 caches. And we leverage the regional BLOB-storage fabric backed by flash as the L3 cache dataloader continues to access storage through the familiar BLOB-storage SDK. To effectively hide latencies and to simplify the data life cycle, we rely on the following:

Dataloader prefetch: Dataloaders prefetch the next batch of datasets into memory while processing the current batch. This prefetch will surface as a read operation at the BLOB-storage SDK level.
Deep prefetch: We expose an explicit prefetch() API as part of the BLOB-storage SDK. The dataloader will trigger explicit prefetch of the data needed during the next few minutes by invoking the prefetch() API in the background. This API triggers hydration of data from remote storage onto the local region L3 cache and also prewarms the metadata cache.
Automatic data life cycle: Data in the L3 regional disaggregated flash tier is typically held for a configured period of time to allow reuse across epochs in a training cycle. We support custom eviction policies, including TTL and LRU policies. The eviction policies are also capacity/quota aware.

We saw rapid adoption of this new data-loading paradigm as soon as production rollout started, and we continue to support both of the data-loading paradigms in production today. To illustrate the impact in numbers, Figure 5 shows roughly the ingestion times before and after the rollout across all workloads:

Figure 5: Ingestion times before and after the rollout.

In a world where new frontier models get released in weeks, this shift in the data-loading paradigm is a much-needed change to move even faster.

Key Takeaways

Modern AI workloads are data hungry, and storage plays an important role in both the computational cost and speed of innovation. Storage bottlenecks directly impact GPU utilization and computational cost, and in a world with geo-distributed GPUs, time spent on cross-region data ingestion directly impacts the speed of iteration in research. The BLOB-storage architecture at Meta was built to serve Meta’s family of apps, and we needed a step-function improvement in performance to serve AI workloads. This led to rethinking the entire architecture. By rebuilding the metadata subsystem and by adopting a tiered caching architecture with prefetching/on-demand hydration, we are able to meet the needs of today’s workloads effectively.

Future Work

We are continuously evolving storage at Meta to keep up with hardware evolution and workload demands. Some future work in this area will include:

Scaling storage to network limits.
Supporting checkpointing without stalling GPUs at even higher scale.
New challenges for inference workloads, which we are starting to tackle.

The post Meta’s AI Storage Blueprint at Scale appeared first on Engineering at Meta.

6 security settings every GitHub maintainer should enable this week - The GitHub Blog

2026-07-01T15:59:29.000Z

At GitHub Security Lab, we spend a lot of our week talking to maintainers. Some find the settings page dense and the docs sprawl. Most maintainers we talk to weren’t hired to be security engineers. While this is true, ignoring a project’s security settings completely will lead into leaving a lot in the table in terms of automation and scalability, leading into a poor security posture, and before you realize it to vulnerabilities that pile up, exposing your users.

Here’s the short version. Six settings, free to use, updated in less than half an hour. We’ve bundled them into a guided flow called Protect Your Project so you can do them in one pass, and we walk through each tool you’ll use below.

1. Add a SECURITY.md file

This is the lightest-lift setting on the list and the one that makes everything else easier.

A SECURITY.md file tells the people who find bugs in your project where to send them. Without one, your options for a well-meaning reporter are a public issue (now a public exploit) or your personal email (if they can find it).

You don’t need to write much. We suggest adding a communication mean such as an email so that those reporting vulnerabilities can reach you directly without posting about them publicly. Then, you can state what bugs are in scope, alongside anything else a reporter should have in mind when contacting you. For reference, we point maintainers to the the systemd project’s security policy that we consider a complete example. It sets clear expectations about reproducers and doesn’t assume you have a 24/7 response team when you don’t. Borrow the structure, change the contact details, commit it.

Ten minutes, tops.

2. Turn on private vulnerability reporting

SECURITY.md tells reporters where to go. Private vulnerability reporting (PVR) gives them a private place to make their report.

Once enabled, a researcher can file a confidential advisory on your repo. You triage it out of the public eye and disclose on your timeline. The setup is one checkbox in Settings → Security.

If you only do one thing tonight, do these first two together. They are free, and are the fastest signal to your community that you take this seriously.

3. Turn on secret scanning, with push protection

This is the one with the most embarrassing failure mode.

GitGuardian’s State of Secrets Sprawl 2026 found 28.65 million new secrets leaked on public GitHub in 2025, a 34% jump over the prior year and the largest single-year increase on record. AI-assisted commits are leaking secrets at roughly twice the baseline rate. The average cost of a data breach now sits at $4.44 million globally ($10.22 million in the US) per IBM’s 2025 Cost of a Data Breach Report.

Secret scanning catches keys and tokens that slip into your repo by blocking them locally before they’re pushed to your repository. It doesn’t matter if your repo is public or private, because once secrets leave your local development, then they are available to anyone with access to your repo.

4. Turn on Dependabot and dependency review

Your project isn’t just your code. It’s the dozens (often hundreds) of packages your code pulls in.

Looking at WordPress, for example: this search for reviewed, critical-severity advisories mentioning WordPress returns a long list of plugins with known vulnerabilities. If you’re running a WordPress site, Dependabot helps ensure none of these plugins are sitting in your dependencies.

Dependabot alerts you when a package you depend on has a known vulnerability. Dependency review shows you, inside a pull request, exactly what’s being added or upgraded and whether any of it has an open advisory. Together they turn an opaque package.json diff into a two-minute review.

5. Turn on code scanning

Code scanning runs static analysis on your repo and flags the patterns that lead to real bugs. SQL injection. Command injection. Dangerous deserialization. The usual cast.

Code scanning with CodeQL can detect unsafe GitHub Actions workflows. CodeQL is the engine, and we built code scanning. We made it free for open source in 2019, and it now ships as a one-click default setup in your Security and Quality tab.

This is the setting most maintainers skip because it sounds like it needs configuration. It doesn’t. Default setup picks the right query pack for your language and runs on every pull request.

6. Turn on branch protection on your default branch

This is the simplest, least-flashy setting, but it will yield the biggest impact starting as soon as you turn it on. This is about requiring a pull request before merging to main with minimum one approval.

This catches the worst-case scenario: a compromised credential, a confused contributor, or a tired version of you pushing straight to production. It’s also what makes the other five settings actually bite, because now Dependabot alerts and code scanning findings block a merge instead of sitting in a tab you never open.

In conclusion

These six settings will not make your project unhackable. Nothing will.

What they will do is close the easy doors, the ones being walked through right now by people scripting through public repos at scale.

Turn these on, and your project will be meaningfully harder to attack than it was this morning. So will every project that depends on it.

The post 6 security settings every GitHub maintainer should enable this week appeared first on The GitHub Blog.

How GitHub maintains compliance for open source dependencies - The GitHub Blog

2026-06-30T17:28:16.000Z

Every day, GitHub engineers introduce new dependencies into the GitHub platform, internal applications, and open source projects. GitHub is not just the home of open source; it is powered by open source! And an important part of using open source responsibly is respecting the licenses that govern the projects you depend on.

At GitHub, we are committed to upholding our obligations to the open source community and to the dependencies we use. Here’s how our Open Source Program Office (OSPO) uses the new GitHub License Compliance feature to manage thousands of dependencies.

Managing the open source license compliance process

Nearly all software carries some kind of license agreement. The license gives you permission to use a project, provided you comply with its obligations. Those obligations may be as simple as giving credit to the original author in your documentation, or they may require you to distribute all your source code when shipping your program. In some cases, licenses may also restrict certain activities or categories of use.

Your organization likely has its own policies about acceptable licenses based on your business model, software ecosystem, and distribution strategy. For example, suppose your organization sells a commercial, closed source binary application. You may want to prevent dependencies that would require you to open source your proprietary code.

Or, you may have a project that you plan to release as an open source package. In this case, you may want to avoid including dependencies governed by commercial or incompatible open source licenses.

If you can’t comply with the obligations required in either scenario, you should avoid the dependency to prevent legal or operational risks. It may require engineering effort to remove these licenses after the fact. For enterprise software, the business risk of noncompliance is huge because it can lead to costly litigation and reputational damage.

Traditionally, license reviews have been performed manually or with third-party software. But now, GitHub has introduced a license compliance feature for GitHub Advanced Security customers, enabling you to review new dependencies directly on pull requests. This review helps ensure that the licenses for those dependencies’ comply with your policy, while also giving you the flexibility to expand your policy to allow new licenses or individual projects.

Two months ago, GitHub’s OSPO migrated from internal-only tools that we’d built to manage compliance onto the new feature. As early adopters, we gave the development team quick feedback and helped ensure the feature would clear the bar for large, fast-moving enterprises with complex compliance requirements.

Setting up for policy success

Because GitHub had built internal license compliance tools prior to the introduction of the product, we had an existing list of acceptable licenses to use as our initial policy. You’ll likely find that many dependencies use common permissive licenses such as MIT, Apache 2.0, and BSD-3-Clause, which are a good starting list to seed your policy. We initially rolled the feature out using the “Evaluate” mode on an organization-wide ruleset, which generated annotations in pull requests without blocking merges, so we were able to get developers accustomed to the new workflow without impeding their productivity. Running the old and new tools in parallel also let us see if their behavior diverged. After about a month of this mode of operation, we got to a state where the alerts were mainly on packages with unusual, missing, or explicitly disallowed licenses.

How GitHub license compliance works

Under the hood, license compliance checks are enabled via rulesets. We target repositories via a custom property, where the value of the property determines whether license checks are enabled in “Active” or “Evaluate” mode. In repositories that are targeted by a ruleset, pull requests that modify a project’s dependencies trigger a scan that looks up the licenses used by each of the new dependencies. If the new dependencies’ licenses are already permitted, or there are package-specific exceptions, the checks pass. If there are failures, either in the direct or transitive dependencies, the tool comments on the pull request with alerts for each problematic package.

The developer then reviews the alerts. If they decide the dependency is unacceptable, they can update their code or close the pull request to remove it. If they believe the license or package should be allowed, they can raise an exception request which will notify a specific team in the organization who can decide whether and how to amend the policy.

A day in the life of the license policy team

GitHub’s license policy team consists of OSPO members and engineers with expertise in license reviews and supply chain analysis. Since we are a worldwide company, our policy review team has members across time zones to review alerts in a timely manner. We are in the process of formalizing an SLA for reviewing license requests, but in practice it’s rarely more than a couple of hours before we can triage an incoming request.

Team members receive email notifications of new review requests and can also access a dashboard to see the backlog of pending requests.

When approving a request, we have two decision points: first, whether to permit the license or the package. Then, decide what scope – enterprise or repository – to use. If it’s a safe license that simply hasn’t shown up before, we’ll add it at the enterprise level and thus allow dependencies with that license anywhere at GitHub. Some packages carry a commercial license which can’t be permitted everywhere but should be allowed in the repository owned by a team which has paid for the software, so those policy amendments get added at the repository level. Package exceptions are useful for internal software which usually doesn’t have license data associated with it. Helpfully, the tool supports wildcard matches for package exceptions. For example, we’ve permitted everything in the @github-ui/* React namespace, so we don’t need to approve those packages one by one.

Making it easy for developers

To support this process, we’ve established procedures about contacting the GitHub OSPO, and how to use an emergency “break glass” override. These situations should be rare, but a clear emergency override process is essential for critically time-sensitive pull requests. As we mentioned above, the license policy enforcement happens via ruleset, and the ruleset condition keys off a custom property. So toggling the value of the property can temporarily turn off enforcement if there’s a critical fix that’s blocked by a license alert. So far, we’ve only needed to use this once, but it was very helpful to have the option.

We’ve also provided internal documentation and training to help developers understand the importance of license compliance. Ultimately, it’s everyone’s job to help ensure compliance and manage risk and it’s our job to make that as easy as possible.

Wrapping up

License compliance is a critical part of managing our software supply chain. By helping developers make informed dependency choices aligned with GitHub’s license policy we prevent costly rewrites and potential legal problems. We’ve been enthusiastically using and providing feedback on the new GitHub License Compliance feature for several months. Now that it’s in public preview, we are excited to see more companies adopt it and hope our experience provides some guidance if you’re just getting started.

GitHub Enterprise Cloud customers can use the License Compliance feature across repositories which have an active GHAS Code Security license. For more information, see About open source license compliance.

The post How GitHub maintains compliance for open source dependencies appeared first on The GitHub Blog.

10 Years of Meta’s Commitment to Python - Engineering at Meta

2026-06-30T16:00:46.000Z

This year marks Meta’s 10th consecutive year as a sponsor of the Python Software Foundation (PSF), the charitable organization dedicated to advancing, supporting, and protecting the open-source Python programming language and the community that sustains it. Python is one of the world’s most influential programming languages, and we use it across our engineering stack, from the backend of our apps and products like Instagram and Threads to cutting-edge AI research.

We recognize the vital role the PSF plays in sustaining the language, nurturing its global community, and driving innovation. After a decade, it felt like the right moment to reflect on why we, as an organization of engineers, are committed to funding the PSF. By supporting the PSF, we aim to help ensure that Python remains robust, innovative and accessible for generations of engineers to come. We hope our involvement will inspire other individuals and organizations to join us in strengthening the foundation that supports so much of today’s technology.

The Importance of Python at Meta

Python is the most used programming language at Meta. It powers infrastructure across our most important products and initiatives and supports a wide range of teams across the company. Some of the core maintainers of Python are Meta engineers who have authored new features and Python Enhancement Proposals (PEPs) for the Python community. PyTorch, one of the world’s most widely-used machine learning frameworks, was originally developed at Meta in partnership with the community before being spun off into its own independent foundation. Meta also builds open-source Python developer tools to help developers write better quality, more performant Python. This includes projects like Pyrefly, an incredibly fast type checker and language server.

Supporting the continued growth and sustainability of Python is a natural fit for Meta’s technical vision. It will continue to play an important role in helping us achieve our goals as we invest further in AI, build new data-driven products, and further scale our infrastructure.

Why Meta Sponsors the Python Software Foundation

At Meta we understand that using open source software like Python comes with a shared responsibility to help ensure the language and its ecosystem remain healthy, secure, and innovative for everyone. Every product shipped, every model trained, and every insight generated with Python is made possible by the collective work of the open source community, backed up by the organizational support and infrastructure maintained by the PSF. For Meta, supporting the PSF is a strategic investment in the future of Python, and hence the long-term stability of our own technology stack.

Our sponsorship of the PSF has helped fund impactful initiatives such as the Developer-in-Residence program, which employs full-time developers who are focused on improving the Python programming language and its ecosystem. This program has been transformative, allowing critical work to happen that would otherwise fall to overstretched volunteers or go unaddressed entirely.

PSF funding also goes towards strengthening the core infrastructure of the Python ecosystem, most notably the Python Package Index (PyPI), where our sponsorship has helped fund essential security enhancements. These improvements are vital for protecting the global Python community and ensuring that developers everywhere – including our own engineers – can safely share and consume packages.

Beyond purely technical investment, Meta’s support also helps fund educational programs and community events like PyCon US, where we’ve provided free and discounted passes to PyCon, supported workshops and summits, and contributed to fundraising efforts for groups like PyLadies. These investments help grow the Python community and foster the new talent that is essential for Python’s long-term sustainability.

In short, sponsorship of the PSF is a valuable investment in the tools and community that make our work possible.

How Can You Support the Python Software Foundation?

There are several ways you as an individual, or your organization as a whole, can contribute to the ongoing success and sustainability of the PSF:

Make a one time donation: You can give any amount as a one off donation.
Become a PSF member: By becoming a member you can vote in discussions on the direction of the Python language. There are different donation tiers available, including donating your time.
Become a sponsor: For organizations looking to make a sustained impact, the PSF offers annual sponsorship tiers, each with increasing levels of recognition and benefits.

As an organization the most meaningful way for you to support the PSF is through annual sponsorship. Besides benefitting from the continued success of the Python language itself, there are a range of additional benefits depending on your sponsorship amount. Sponsors of the PSF receive public recognition, with their names and logos featured on the PSF website, in annual reports, and at major events. Sponsorship also provides valuable opportunities for community engagement, allowing organizations more opportunities to connect with the global Python community, participate in events, and demonstrate their commitment to open source. Higher-tier sponsors benefit from increased brand visibility through prominent logo placement and may be invited to speak or participate in special initiatives.

Thank You!

Finally, we want to say thank you to the Python community: the maintainers, contributors, educators, and advocates who make Python what it is today. Your passion and dedication are the foundation of Python’s success, and we’re proud to be able to support you, both as collaborators and sponsors.

Visit our website to learn more about Meta Open Source. You can also subscribe to our YouTube channel, or follow us on Facebook, Threads, Bluesky, LinkedIn, and X.

The post 10 Years of Meta’s Commitment to Python appeared first on Engineering at Meta.

Highlights from Git 2.55 - The GitHub Blog

2026-06-29T17:25:25.000Z

The open source Git project just released Git 2.55 with features and bug fixes from over 100 contributors, 33 of them new. We last caught up with you on the latest in Git back when 2.54 was released.

To celebrate this most recent release, here is GitHub’s look at some of the most interesting features and changes introduced since last time.

Repacking with incremental multi-pack indexes

Returning readers of this series may recall our coverage of incremental multi-pack indexes and incremental multi-pack reachability bitmaps. In case you could use a refresher, here’s the short version.

Git stores the contents of your repository as individual objects: commits, trees, and blobs. Those objects usually live in packfiles, which are compressed collections of objects. A packfile has a corresponding pack index that lets Git locate any object inside the pack quickly. But large repositories do not usually have just one packfile: over time, fetches, pushes, maintenance tasks, and repacks can leave many packs behind.

A multi-pack index (or MIDX) gives Git a single index over many packs. Instead of opening and searching each pack’s individual index, Git can ask the MIDX which pack contains a given object and at which offset. This is especially useful for large repositories, and it is one of the building blocks behind GitHub’s repository maintenance strategy.

As we covered when Git 2.47 introduced the incremental MIDX format, a repository can store its MIDX as a chain of layers instead of as a single MIDX covering every pack. A single-file MIDX is simple and efficient to read, but it has an important maintenance cost; since that file includes every pack it covers, even a small update can require a large write in an already-large repository.

Incremental MIDXs address that by storing a chain of MIDX layers. Each layer covers some collection of packs, and the chain file records the order of those layers. Appending a new layer to the tip of the chain does not invalidate the older layers, so Git can index newly created packs without rewriting a single MIDX that covers the entire repository.

Git 2.55 teaches git repack how to write those incremental MIDX chains directly:

$ git repack --write-midx=incremental

Without any other options, that mode is append-only: Git writes a new layer for the packs created by the repack and leaves the existing layers alone. That is already useful when you want to minimize how much metadata gets rewritten during a maintenance run.

But an append-only chain cannot grow forever. If each maintenance run adds a new layer, then eventually the chain itself becomes the thing you need to maintain. Git 2.55 also supports combining --write-midx=incremental with geometric repacking:

$ git repack --write-midx=incremental --geometric=2 -d

When those modes are used together, each repack creates a new tip layer, then decides whether adjacent layers should be compacted together. The default rule is controlled by repack.midxSplitFactor: if the accumulated object count in newer layers grows large enough relative to the next older layer, Git merges those layers into a single replacement layer. Otherwise, the older layers are left untouched.

At a high level, the algorithm works like this. Below, $N$ refers to the repack.midxNewLayerThreshold value, and $f$ refers to the repack.midxSplitFactor value:

Pick the un-MIDX’d packs as geometric repacking candidates. If the tip MIDX layer has at least $N$ packs, include those as candidates too.
Apply the usual geometric repacking rule to that candidate set, and write a new tip MIDX layer covering the resulting packs.
Compact adjacent MIDX layers while the accumulated object count of the newer layer(s) exceeds $1/f$ of the next deeper layer’s object count.

To see how the pieces fit together, let’s start with a repository that already has an incremental MIDX chain. The older layers are on the left, and the tip layer is on the right. Meanwhile, normal repository activity keeps producing new packs. Those packs are not covered by any MIDX layer yet, which means the next maintenance run has two jobs: decide what to repack, and decide how much of the MIDX chain to rewrite.

Ordinarily, those un-MIDX’d packs are the only geometric repacking candidates: Git can write a new pack and a new tip MIDX layer without disturbing any existing layer. The figure below shows the more interesting case, where the current tip layer has accumulated enough packs to meet the configured repack.midxNewLayerThreshold. Once that threshold is met, packs from the tip layer can join the newly written packs as geometric repacking candidates.

Geometric repacking then asks a local question about the newest candidate packs. Geometric repacking then asks a local question about the newest candidate packs: is the pack immediately to the left of some suffix of packs ( $\mathcal{P}$ ) large enough to preserve the geometric progression if Git rolls up $\mathcal{P}$ ? In the first attempt below, $\mathcal{P}$ contains the smallest pack from the current tip layer along with the new un-MIDX’d packs. But the pack to the left of the split is only 30,000 objects, which is smaller than twice the size of $\mathcal{P}$ , so this split is too far to the right.

So Git moves the split one pack earlier and asks the same question again. Now $\mathcal{P}$ includes one more pack from the tip layer. The pack immediately to the left has 100,000 objects, which is at least twice the size of the selected suffix. That is the point where the geometric invariant holds, so Git can roll up exactly those packs into a new pack.

After writing that new pack, Git writes a new tip MIDX layer over the surviving pack from the previous tip layer and the newly written roll-up pack. At this point, the packfiles themselves are in good shape, but the MIDX chain may still have accumulated too many small adjacent layers. Git applies the same “newer compared to older” instinct to the MIDX layers themselves: if the newer layer is large enough relative to its neighbor, compact their metadata into a replacement layer.

That compaction step is deliberately metadata-only. Git does not repack the objects from those layers again; it writes a new MIDX layer that covers the same packfiles. Then it considers the next older layer. Here, the compacted layer is still smaller than half of the deeper layer, so Git stops. The older layer remains untouched, which is the key property that keeps this maintenance incremental.

The result is a compromise between two extremes. A single-file MIDX minimizes lookup complexity, but can require large rewrites during maintenance. A purely append-only incremental MIDX minimizes each write but allows the chain to grow without bound. Geometric incremental repacking keeps the number of layers logarithmic in the total number of objects, while ensuring that the newest, smallest layers are rewritten more often than older, larger ones.

This also integrates with Git’s existing repack machinery. Newly written packs that are not yet covered by the MIDX chain are always candidates for the geometric repack; packs in deeper MIDX layers are left alone. Packs in the tip MIDX layer join the candidate set only after the tip layer has at least repack.midxNewLayerThreshold packs. If the tip layer is still smaller than that threshold, Git skips disturbing it entirely and simply appends a new layer for the newly written packs.

For repositories that receive a steady stream of new objects, this means routine maintenance can update the repository’s pack metadata incrementally, without forcing each maintenance run to rewrite a single MIDX covering the entire object store.

[source]

Fixing up earlier commits with `git history`

Anyone who has polished a commit series before sending it for review has probably had this experience: you notice that a change in your working tree really belongs in an earlier commit, not at the tip of the branch.

Today, one common way to handle that is to create a fixup commit and then autosquash it:

$ git commit --fixup=<commit> 
$ git rebase --autosquash <commit>^

That works, but it asks you to spell out the mechanism instead of the intent. Git 2.55 builds on the experimental git history command, which Git 2.54 introduced, by adding a new fixup subcommand. It applies the changes currently staged in the index to an earlier commit:

$ git history fixup <commit>

Here is a small example. The first commit introduced a pancake recipe, followed by a few more commits on top. Later, we realize that the recipe was missing maple syrup. After staging that one-line change, git history fixup <commit> folds it into the original recipe commit and replays the descendant commits on top.

Here the staged change becomes part of the target commit itself. The target commit keeps its message and authorship by default, unless you pass --reedit-message, and Git rewrites the commits that follow so the branch ends at an equivalent history with the fix in the right place.

Like the rest of git history, this command is still experimental. It is also intentionally conservative. Because fixup reads from the index, it needs a working tree and cannot operate in a bare repository; if applying the staged change would produce a conflict, the command aborts instead of leaving you in the middle of a stateful rewrite.

[source]

The tip of the iceberg…

Now that we’ve covered the largest changes in more detail, let’s take a look at a selection of some other new features and updates in this release.

Returning readers of this series may remember our coverage of config-based hooks from Git 2.54, which let you define hooks in your Git configuration rather than only as executable files in $GIT_DIR/hooks. Hooks are the scripts Git runs at well-known points in your workflow, like before creating a commit or after receiving a push. Moving them into configuration makes those hooks easier to share, compose, and selectively disable without copying scripts into each repository’s hooks directory.
Git 2.55 extends that work by allowing compatible configured hooks to run in parallel. For example, a project might have independent pre-commit hooks for linting and unit tests; if both declare hook.<name>.parallel = true, Git can run them at the same time. The number of concurrent jobs can be controlled globally with hook.jobs, per event with hook.<event>.jobs, or on the command line with git hook run -j. Hooks that need shared state, like commit-message hooks or other hooks that inspect the index or working tree, continue to run serially.
[source]
If you have ever run git status only to be greeted by a long pause at your terminal, you may have used Git’s built-in filesystem monitor to speed things back up. When core.fsmonitor is enabled, commands like git status can ask a long-running daemon which paths have changed instead of scanning the entire working tree.
Until now, that built-in daemon was available only on macOS and Windows. Git 2.55 adds support for Linux, where the implementation uses inotify. That works without elevated privileges, but requires one watch per directory, so very large repositories may need to raise the fs.inotify.max_user_watches limit. As on other platforms, the daemon is conservative around network-mounted repositories, which remain opt-in.
[source]
Reachability bitmaps are one of the tricks Git uses to answer questions like “which objects are reachable from this commit?” without walking the entire object graph from scratch. They make object traversals faster, but Git still has to build and update those bitmaps during maintenance tasks like git repack --write-midx-bitmaps.
Git 2.55 makes that generation path faster by avoiding unnecessary tree recursion, reusing already-computed selected bitmaps, caching object positions, and sorting bitmaps before XORing them together. In benchmarks from the patch series, those general improvements reduced bitmap generation time in one large repository from about 612 seconds to about 294 seconds.
The same series also improves pseudo-merge bitmaps, which group related references together so Git can combine precomputed bit arrays during a traversal instead of rediscovering the same objects repeatedly. In one benchmark, pseudo-merges made a full git rev-list --objects --use-bitmap-index traversal nearly 20 times faster, but previously nearly doubled bitmap generation time. After these changes, pseudo-merges keep most of their traversal speedup while adding much less work to the bitmap generation path.
[source, source]
If you use partial clones, filtered packs, or other workflows where Git intentionally omits some objects, pack size still matters. The git pack-objects --path-walk mode, introduced in Git 2.51, groups objects by path before performing a second compression pass, which can produce better deltas when path locality matters.
In Git 2.55, --path-walk can be combined with filters including blob:none, blob:limit=<n>, tree:0, object:type=<type>, sparse:<oid>, and compatible combine: filters. That makes packing using --path-walk available in more partial-clone and filtered-pack workflows. In one benchmark on Git’s own repository, a blob-less path-walk repack produced a pack roughly 16% smaller, at the cost of a slower fresh-delta computation.
[source]
Git learned a new experimental command, git format-rev, for pretty-formatting revisions from standard input. Unlike git log, which walks a range of history, git format-rev is designed for cases where you encounter commits one at a time or embedded in other text.
For example, suppose you’re using git last-modified to print the commit that last modified each path in some directory. What if you wanted to know who last modified each path, not just which commit did it? You could replace those commits with author names by piping its output through something like this:
```
$ git last-modified | perl -F'\t' -lane ' 
  chomp($F[0] = qx(git show -s --format=%an $F[0])); 
  print join "\t", @F 
' 
Junio C Hamano	builtin/commit.c 
[...]
```
That works, but it has to start a new Git process for each row just to format the commit. In Git 2.55, git format-rev can handle that part as a normal pipeline:
```
$ git last-modified | 
  git format-rev --stdin-mode=text --format=%an 
Junio C Hamano	builtin/commit.c 
[...]
```
The command’s text mode can also rewrite full commit object names found in freeform text, which makes it useful for commit-message hooks or other scripting workflows.
[source]
When you push your repository somewhere, you may have noticed output that starts with remote::
```
$ git push origin main 
Enumerating objects: 5, done. 
[...] 
remote: Resolving deltas: 100% (2/2), completed with 1 local object.
```
When a client fetches or pushes, Git can multiplex different streams over the same connection: one sideband carries packfile data (the actual objects being transferred), another carries progress messages that the client usually prints to stderr, and a third carries errors from the remote.
Those progress messages are useful, but they also come from the other side of the connection and may be printed directly to your terminal. Before Git 2.55, that meant a server could include arbitrary terminal control sequences in sideband output, including sequences that move the cursor or erase text. Git now masks most of those control characters by default while still allowing ANSI color sequences, so colored progress output continues to work.
[source]
Suppose you are halfway through editing a file when you realize that you started from the wrong branch. If the branch you want to switch to changed the same path, a plain git checkout <branch> will refuse to move and risk clobbering your work. git checkout -m <branch> is the “try to carry my local edits with me” version of that operation.
But what happens when the other side has modifications against that same path? Previously, git checkout -m gave you one chance to resolve the resulting conflicts immediately. Git 2.55 makes that safer by using an autostash internally, so the conflicted local changes are saved as a stash entry that you can either resolve right away or reapply later.
[source]
Some projects need to publish the same branch to more than one place, like a primary host and one or more mirrors. Remote groups have long been available to git fetch, where a group is configured with remotes.<name> as a whitespace-separated list of remotes. Git 2.55 lets git push use the same shorthand:
```
$ git config remotes.publish "github gitlab mirror" 
$ git push publish main
```
This is equivalent to pushing to each remote in the group in sequence. Since atomicity can only be guaranteed for a single transport connection, --atomic is not supported when pushing to a group.
[source]
git log --graph is great for visualizing branch structure, right up until the graph itself gets too wide to read. In repositories with many parallel branches, the graph lanes can consume most of the terminal before you get to the commit subject.
Git 2.55 adds --graph-lane-limit=<n> to git log --graph and related commands. Lanes beyond the limit are replaced with `~`, making graph output more manageable in repositories with very wide histories.
[source]
Suppose you want to list the 10 most recent commits on your branch. That is easy enough: git log -n 10 does exactly that. But what if you want the 10 oldest commits? If you are thinking, “it surely isn’t git log --reverse --10,” then congratulations: you’re a veteran Git user! Instead of reversing the history and then printing 10 commits, Git takes the 10 most recent commits and reverses their order.
You can get there by post-processing the whole range (for example with git log --reverse <range> | tail -10) but doing so still asks Git to print and format all of the commits that the shell is going to throw away. Git 2.55 adds a new --max-count-oldest=<n> option to git rev-list and the git log family of commands, which selects the oldest n commits in a range instead.
[source]
During a fetch, the client and server negotiate by having the client advertise commits it already has as have lines. That lets the server avoid sending objects the client can already reach. But in repositories with many references, the negotiation algorithm may skip a ref that is especially important for finding common history.
Git 2.55 adds new controls for which references participate in negotiation. The new include and restrict options, along with corresponding remote.* configuration, allow users to require certain refs to be sent as have lines or to limit negotiation to a specific set of refs.
[source]

…the rest of the iceberg

That’s just a sample of changes from the latest release. For more, check out the release notes for 2.55, or any previous version in the Git repository.

The post Highlights from Git 2.55 appeared first on The GitHub Blog.

Inside the Advisory Database and what happens when vulnerability volume breaks records - The GitHub Blog

2026-06-29T16:10:20.000Z

In May 2026, the GitHub Advisory Database published 1,560 reviewed advisories—more than five times our typical monthly output and the highest in its history.

And it still wasn’t enough to keep up.

Over the past few months, the vulnerability ecosystem has shifted in a fundamental way. Input across private vulnerability reports, repository advisories, and CVE requests has increased simultaneously, pushing the entire system to a new operating scale.

This blog builds on an ongoing GitHub community discussion tracking the evolving nature of vulnerability reporting, as well as PVR and Advisory Database roadmap developments. A recurring theme in that thread is the downstream impact of platform changes on advisory curation and data quality. This aligns with GitHub’s broader shift toward emphasizing quality and shared responsibility in vulnerability reporting, which in turn directly shapes how advisory data must be curated and maintained.

TL;DR

Review times for new advisories are longer because vulnerability volume and complexity have increased significantly. Advisory quality has not changed: reviewed advisories are still human-validated, and existing alerts continue to function normally. If you want to help, focus on three things: submit complete vulnerability data, coordinate closely with maintainers and researchers, and request CVEs only when there is a clear intention to publish.

Record output and unprecedented input

May was not a one-time spike. From March through May, we sustained more than 6,000 advisory decisions per month. This included updating existing advisories, publishing new advisories, and reviewing inbound advisories, and exceeded any prior three-month peak.

At the same time, inflow accelerated across every source:

Private vulnerability reports across the platform increased from ~550/week in January to more than 3,000/week for most of May.
Repository advisories scaled from ~650/week to more than 5,000/week.
GitHub CNA CVE requests reached almost 4,000 in May alone, nearly 10x year –over year.
The CVE program has already published 30,000+ CVEs in 2026.
More than 1.7 million total repositories have enabled private vulnerability reporting.

This is not a localized surge. It reflects structural change across the vulnerability disclosure ecosystem.

The impact

Since mid-April, due to this surge, we have not consistently met our internal goals for publication. Processing times extended first to about a week, then to multiple weeks for a meaningful share. Longer publication times can increase exposure windows. We take that seriously, and timeliness is a core part of the value this database provides.

What’s still working

Our data pipelines and publishing infrastructure have continued to operate through this period. Imports are running, data integrity is intact, and published advisories are accurate. Advisories that reach reviewed status today meet the same quality standard as before.

CVE assignment quality has remained strong. Our assignment rate has held between 91–94% through the entire surge, consistent with or better than historical norms and showing that there hasn’t been a clear degradation in the requests we receive.

The issue is throughput. The system that validates, enriches, and publishes advisory data is functioning; it is now operating beyond the volume and complexity it was designed to handle.

The work isn’t uniform

Not every security advisory requires the same level of effort. Some arrive well formatted: the advisory details clearly name the affected package and its relevant ecosystem, the version range is documented, and the fix is tagged. A curator can validate and publish these in under a few minutes.

But a growing share of incoming advisories require more investigation:

Package disambiguation. The advisory details say “foo”, but is that foo on npm, python-foo on PyPI, or the unrelated foo on Maven? When upstream data doesn’t specify an ecosystem, our curators figure it out.
Version range reconstruction. Many security advisories arrive with no affected version range, or with ranges that don’t match actual release history. Curators trace commits, changelogs, and tags to determine what’s actually affected.
Multi-ecosystem advisories. Some projects ship packages to multiple registries, like a library with both a .NET implementation (NuGet) and a JavaScript implementation (npm) of the same functionality, where a vulnerability in the shared logic affects both. This requires independent verification across multiple data sources.
Conflicting upstream data. When the CVE record, the maintainer’s advisory, and the commit history disagree about what’s affected, someone has to determine the truth.

Historically more straightforward advisories dominated, and the harder ones could be absorbed. When volume surges, the queue fills with both, and the complex ones take disproportionately longer, creating a compounding effect. The mix now matters much more. This isn’t just more work; it’s significantly more complex.

What “reviewed” actually means

A reviewed advisory is not simply a republished record; it’s the result of verification.

Curators:

Map vulnerabilities to the correct ecosystem package
Validate affected and fixed versions against release history
Confirm upstream accuracy
Check for duplication and consistency
Validate classification and scoring

This is what allows downstream tools to rely on the data without additional validation.

Publishing faster by skipping verification would increase false positives at scale, which can create more risk than delay.

A broader ecosystem shift

This trend extends beyond GitHub.

The volume of reported and published vulnerabilities continues to grow rapidly, and organizations across the ecosystem are adapting to that change.

The system is working as designed. More vulnerabilities are being reported, disclosed, and tracked than ever before. That creates pressure downstream, including during advisory curation.

What we’re doing now

Improving community contribution quality and throughput. Community contributions are an important part of how we improve the Advisory Database, and each is reviewed against the same validation standard as any other advisory. We’ve strengthened triage and prioritization, so high-quality submissions are identified earlier, reviewed more consistently, and moved through the queue faster. This helps us respond to current volume while reinforcing our primary goal of maintaining a high-quality, trusted dataset.
Scaling the systems behind curation. We’ve increased aspects of the capacity of our backend curation systems to handle higher sustained throughput and we’re continuing to modernize the data infrastructure that supports analytics and queue management.
Building AI-assisted research tools. We’ve developed and deployed tooling that gives our curators AI-powered assistance during the research phase of advisory review. Curators still make every decision, but routine research can be completed faster for higher quality advisories.
Expanding automation where it helps the most. We’ve improved automation for extracting more data from upstream CVE information and for handling how community contributions interact with already-reviewed advisories. That work reduces time per decision without lowering the quality bar.
Investing in documentation and training. We’ve significantly expanded our operational documentation. This enables us to bring new team members up to speed faster and improves consistency across the team.

What we’re building next

To support this new scale, we are investing in:

Reducing time-per-advisory for the most common cases. A significant portion of incoming advisories require research that follows predictable patterns, such as identifying the correct package, confirming the version range, and checking for a fix. We’re investing in tooling that accelerates these patterns, so curators can spend their time on genuinely ambiguous cases that require human judgment.
Making risk-based review prioritization smarter. We’re exploring additional risk signals for prioritization, such as package usage, evidence of active exploitation, and ecosystem impact to ensure the advisories that matter most reach users first.
Improving the feedback loop with upstream data sources. A significant share of curation time is spent correcting incomplete or inaccurate upstream data. We’re investing in tighter integration with the sources we ingest from, especially through increased repository GitHub Security Advisory and Private Vulnerability Reporting data validation, so that data quality issues get resolved closer to the origin rather than in our review queue.
Continuing to be transparent. We’ll share updates on our progress as we make it. If things improve, we’ll tell you. If we hit new challenges, we’ll share that too.

What this means for you

Dependabot users: Existing alerts are unaffected. New advisories may take longer to trigger, with critical issues prioritized.
API and feed consumers: Reviewed data remains accurate; unreviewed advisories are visible but not yet validated.
Maintainers: Repository advisories continue to flow into the global database; prioritization is based on several factors, including project impact and severity.

How you can help

Include complete data in vulnerability reports. Providing affected version ranges, root cause, and clear reproduction steps makes a direct difference in how quickly and accurately advisories can be reviewed. When this data is complete, curation can take minutes. When it isn’t, curators must reconstruct missing details from source code, release history, and conflicting upstream signals. At this scale, those gaps compound quickly. High-quality upstream data is one of the most effective ways to improve both speed and accuracy across the ecosystem.

Include the right advisory details. Our best practices guide covers ecosystem categorization, package names, and version range formatting, but a few additional details make a direct difference in how quickly and accurately advisories can be reviewed and published to the GitHub Advisory Database.

Use the package name as it appears in the registry. Advisory package names must match the registry, not the repository or project name. Downstream systems rely on registry identifiers to match advisories to affected dependencies. If the name is incorrect or missing, alerts cannot be reliably generated, and affected users may never be notified. Using the registry name ensures the advisory can be correctly linked, indexed, and distributed.
List all affected packages. Some vulnerabilities impact multiple packages within a project. Each affected package should be listed separately with its own ecosystem, package name, and version range. Advisories are consumed at the package level, so missing a package means missing the users who depend on it. Including all known affected packages improves coverage and ensures alerts reach the full set of impacted users.
Provide a complete CVSS vector string. The GitHub Advisory Database supports CVSS 3.1 and 4.0. A severity label such as “High” is a quick summary, but a complete CVSS vector string includes a richer set of attributes, such as attack complexity, required privileges, and user interaction, which describe the vulnerability in greater detail. This structured information allows severity to be validated, interpreted consistently, and used by downstream tools for prioritization and automation. Without it, scoring is less precise and harder to compare across advisories. If you include a score, use the official calculators and include the full vector.
Include relevant CWE classification. A CWE identifies the underlying weakness behind a vulnerability, such as cross-site scripting, SQL injection, or deserialization of untrusted data. Unlike a narrative description, a CWE gives downstream tools and security teams a standardized way to understand what kind of issue they are dealing with. That matters because CWE data can be used to categorize, filter, prioritize, and compare vulnerabilities across large datasets. It helps organizations group related issues, apply policy or reporting rules, and understand patterns in the vulnerabilities affecting their software. The more specific the CWE, the more useful the advisory becomes for downstream consumers.

For more guidance, see the best practices for writing clear, complete security advisories.

Be intentional when requesting CVEs. Requesting a CVE ID signals that a vulnerability will be disclosed and tracked publicly. When requests are made without plans to publish, it can divert time and attention from advisories that are actively moving toward release. Aligning CVE requests with clear publication intent helps ensure that effort is focused on where it has the most immediate impact and keeps the system responsive for everyone.

Coordinate closely with maintainers and other researchers. High-quality advisory data depends on shared context. Aligning affected packages, version ranges, and fixes helps reduce ambiguity and conflicting information across sources. At this scale, small gaps in coordination can become large inconsistencies downstream.

Improve advisory quality by contributing pull requests to the Advisory Database. Every correction to version ranges, package mappings, or fixes improves the accuracy that developers rely on.

Recognize the scale of this ecosystem shift and take part in it. The increase in vulnerability reporting reflects real progress. More issues are being found, fixed, and disclosed than ever before. Maintaining quality at this scale depends on researchers, maintainers, and data consumers and producers working together toward the same goal.

The bigger picture

Two years ago, the database published ~270 advisories per month.

In May 2026, it published over 1,500 while processing thousands of additional decisions across the system.

This reflects a broader shift:

More repositories are enabling responsible disclosure.
More researchers are reporting vulnerabilities.
More maintainers are publishing fixes and advisories.
The vulnerability ecosystem is scaling toward greater transparency.

That growth creates pressure on systems like ours. But it also represents meaningful progress.

Every advisory improves visibility. Every alert reduces risk.

We are scaling to meet that reality, and we will continue to share progress as we do.

The post Inside the Advisory Database and what happens when vulnerability volume breaks records appeared first on The GitHub Blog.

GenPage: Towards End-to-End Generative Homepage Construction at Netflix - Netflix TechBlog - Medium

2026-06-29T13:01:02.000Z

Authors: Lequn Wang, Jiangwei Pan, and Linas Baltrunas

Figure 1. Autoregressive homepage generation. GenPage builds a Netflix homepage one row or entity at a time, each one conditioned on what’s already on the page and the user’s context.

Introduction

The Netflix homepage is the first thing users see when they open the app and the primary way they discover content to enjoy. Almost every part of it is personalized, including which rows appear, which entities show up within those rows, and how everything is arranged on the page.

Constructing that homepage is a genuinely hard problem. It is not simply producing one ranked list. The homepage is a structured, two-dimensional layout, made up of recommendation rows and the entities within them. Here, an entity can be a movie, show, game, live event, or other recommendable item. Each choice can affect the value of the others. Traditionally, it is built through a complex, multi-stage pipeline, with separate components for candidate generation and ranking at both the row and entity levels.

We saw an opportunity to rethink this design. Large language models have shown that a single generative model can perform diverse tasks just by generating a response to a prompt. Inspired by this prompt-response paradigm, we trained a single generative model to build the homepage by directly answering one question:

Given everything we know about this user and this request, what homepage should we generate to maximize user satisfaction?

We call this approach GenPage. It treats the user history and request context as the prompt, and autoregressively generates the entire homepage as the response (Figure 1). Unlike most generative recommenders, such as TIGER, HSTU, and OneRec, which generate flat ranked lists, GenPage generates the rows, entities, and layout together.

This shift is motivated by several goals:

End-to-end modeling. A single transformer model that constructs the page from raw input signals can replace a complex multi-stage recommender stack. This reduces the number of ML models to maintain, avoids misaligned objectives across stages, and eliminates much of the traditional feature engineering.
Whole-page optimization via reinforcement learning (RL). Autoregressive page generation makes it possible to optimize for page-level rewards with RL. This can capture interactions across rows and entities, such as diversity or the balance between rows with different stopping power. For example, a Continue Watching row near the top of the page may strongly satisfy a user’s immediate intent, but also reduce how much of the page they browse. Modeling these interactions at the page level lets us align the system more directly with user satisfaction than entity-level objectives alone.
Better scaling behavior. A generative transformer model gives us a clearer path to improving quality through more data, compute, and model capacity, without repeatedly redesigning the system.
Flexibility and extensibility. The prompt-response paradigm is flexible by design. By simplifying feature engineering and enabling whole-page optimization, GenPage makes it easier to support new product experiences, such as additional content types like live events, games, and podcasts; layouts beyond the current two-dimensional structure; personalized UI components; and per-entity artwork personalization, all with fewer architectural changes.

Bringing GenPage into production at Netflix also required solving challenges specific to industry-scale recommender systems. Because the homepage is generated in real time, serving latency is a primary engineering constraint. We also need to handle entity cold start in a constantly evolving catalog, keep the model fresh as user interests and cultural trends shift, and enforce complex product and business rules on the generated output.

Despite these challenges, GenPage has already had substantial production impact. In an online A/B test against a mature, highly optimized multi-stage production recommender, GenPage delivered statistically significant gains on the core user engagement metric we use for launch decisions, while reducing end-to-end serving latency by 20%.

Offline, two findings stood out. First, enriching the prompt helped more than scaling model capacity in our current regime. Second, RL post-training increased homepage diversity even though diversity was not part of the objective.

We expect this approach to generalize to many personalization settings. In this post, we focus on Netflix homepage construction as a concrete case study, sharing our design, trade-offs, and lessons learned.

Data

Moving from a traditional recommender to a generative transformer requires us to rethink how the data is represented. Similar to how an LLM turns text into tokens, GenPage represents both the user context and the generated homepage as one sequence of discrete tokens (Figure 2). This sequence includes the full structured homepage layout, with multiple rows and the entities inside them, so the model can generate the page holistically rather than scoring each row or entity in isolation.

Figure 2. Tokenization of Netflix homepage construction data. The context tokens function as the prompt, drawing from diverse data sources including user history, profile attributes, and request context, with example tokens shown for each source. The page tokens represent the generated response, encoding the structured layout of rows and entities.

Each training example represents a homepage impression and consists of three components:

Context: user engagement history, profile attributes, and request context.
Page: the recommended rows and entities shown on the homepage, in layout order.
Feedback: user interactions with that page, such as play, thumbs-up, or abandonment for entities on the page.

Only the context and page are tokenized as model inputs and outputs. Feedback is used to derive supervision signals via our internal reward system (see the Reward system section).

Instead of using an off-the-shelf text tokenizer, we build a domain-specific tokenizer for the homepage construction data. This is a proven approach in recommender systems and other specialized domains including computer vision, biology, and chemistry, where the raw data is not naturally represented as text. Compared with generic text tokenization, this gives us two key advantages:

Computational efficiency. Custom tokenization significantly reduces sequence length, lowering inference cost and latency. For example, representing the event “User watched Orange Is the New Black for 50 minutes 30 days ago.” would require 16 tokens with the GPT-5 tokenizer, whereas our scheme compresses it to 4 tokens: [Entity_ID], [Action_Type], [Action_Time_Bucket], and [Action_Duration_Bucket].
Product control. A direct mapping between tokens and product concepts, such as rows and entities, makes it easier to control what the model can generate. This is crucial for enforcing business rules on the final homepage.

Context tokens

Context tokens encode user engagement history, user profile, and request context.

We represent user history as a sequence of user actions. For each action, we extract key metadata, including the action type, entity ID, timestamp, and duration. These actions include both explicit signals, such as play, add to My List, and thumbs-up, and implicit signals, such as trailer views or visits to a details page.

User profile tokens capture attributes such as language and profile type. Request context tokens encode signals like time of day, day of week, and device.

Some data sources are too long to include directly as raw token sequences. A user’s full impression history, for example, would be prohibitively expensive to represent in full. In these cases, we use a summarized version. This is a pragmatic trade-off: while GenPage aims to operate on raw inputs as much as possible, handcrafted summaries still introduce a form of prompt engineering into the pipeline. Learning to compress these long data sources end to end is an important direction for future work.

To help the model distinguish between data sources, we insert special tokens that mark the start of each segment. Continuous signals, such as timestamps and durations, are bucketized into discrete ranges to keep the vocabulary finite.

Page tokens

Each entity, such as a show, movie, or game, and each row, such as Korean TV Shows, is represented as a single token. The homepage is serialized in layout order: left to right, then top to bottom. We update the entity and row vocabulary daily to incorporate newly added entities and rows. Entities that are still out of vocabulary at serving time are handled through semantic embedding fusion and fallback tokens, both described later.

In principle, the same paradigm can extend to any output that can be expressed as a linear token sequence. This includes layouts beyond the current two-dimensional structure, such as one-dimensional feeds or mixed layouts, as well as personalized UI components and per-entity outputs such as personalized artwork. We leave these extensions to future work.

Paginated recommendation

To make recommendations responsive to in-session user preferences, the homepage is often generated incrementally, a few rows at a time. Before each pagination request, we append the page tokens from previously generated rows to the prompt, along with the user’s latest engagements on those rows from Netflix’s real-time event-logging infrastructure. This allows the model to generate the next set of recommendations using both the user’s long-term preferences and their most recent in-session behavior.

Reward system

To quantify the long-term value of a recommendation, we rely on an internal reward system described in prior work. The reward system is tuned through online A/B testing to align with long-term user satisfaction and serves as the primary supervision signal for both supervised and reinforcement learning.

The reward system processes user feedback and assigns a scalar reward for every impressed entity on the homepage. For instance, a TV show binge-watched in one night reflects stronger user satisfaction and receives a higher reward than a movie watched for only 10 minutes. An impressed entity that the user abandons receives a negative reward.

We define the page-level reward as the sum of rewards across all impressed entities on the homepage.

Model architecture

GenPage uses a standard decoder-only transformer architecture, the same general architecture behind many modern LLMs. This choice keeps the model simple and flexible, while also letting us benefit from the broad ecosystem of tooling around transformer training and serving.

One architectural detail is that we untie the input embedding and output projection weights. This is useful because pretraining and post-training place different demands on the logits. Next-token prediction pretraining optimizes a softmax over the vocabulary, while weighted binary classification (WBC) post-training optimizes per-token sigmoid scores, as described below. Untying the weights gives the model more flexibility to adapt to both objectives.

Training recipe

Our training pipeline mirrors the LLM recipe: we first teach the model the “language” of the Netflix homepage through pretraining, then align its outputs with user satisfaction through post-training. For post-training, we explore two alternative approaches: weighted binary classification (WBC) and reinforcement learning (RL).

WBC is simpler to optimize and aligns directly with the entity-level objectives of our production ranking models. RL is harder to evaluate and optimize, but it is the key path to GenPage’s full vision of page-level optimization, with the flexibility to incorporate test-time reasoning and multi-token entity representations.

Pretraining via next-token prediction

We pretrain the model with a standard next-token prediction objective: given the context tokens and a prefix of page tokens, the model learns to predict the next page token. This stage focuses on representation learning, teaching the model the relationship between user contexts and successful homepages. Note that our context-page training examples resemble the prompt-response pairs used in LLM supervised fine-tuning (SFT) more than the raw text used in LLM pretraining. We nonetheless call this stage pretraining because we train the model from scratch rather than fine-tuning from an existing checkpoint.

Unlike LLMs, which often face a scarcity of high-quality labeled data, recommender systems have an abundance of user feedback. For pretraining, we use homepage impressions that received positive feedback when served in production, bootstrapping the model to generate pages similar to those produced by the existing production system.

However, pretraining mainly teaches GenPage to imitate the production system. It does not directly optimize the magnitude of the reward, and as GenPage becomes part of production, repeatedly training on pages generated by earlier versions of the model can risk model degeneration. To address these limitations, we explore two post-training approaches.

Post-training via weighted binary classification

One effective way to align the generative model with user satisfaction is weighted binary classification (WBC). At a high level, WBC turns generation into token-level value prediction: given the user context and the tokens generated so far, the model learns to estimate the value of generating each possible next row or entity token.

This objective is easier to optimize than page-level RL. By decomposing the homepage into per-token targets, WBC provides token-level credit assignment by construction, rather than requiring RL to infer how each generated decision contributed to the final page-level reward.

This training setup is enabled by our custom tokenization. Each page token corresponds directly to a specific entity or row, making it straightforward to assign a reward. For every impressed entity on the page, our reward system provides a scalar reward based on user feedback. For each impressed row, we derive a row-level reward by aggregating the rewards of the entities in that row.

From each reward, we derive a binary label from its sign, such as positive engagement versus abandonment, and a weight from its magnitude, such as binge-watching receiving a higher weight than a short play. We then optimize a weighted binary cross-entropy loss on the logit for the corresponding token. Under this setup, the logit for a token can be interpreted as the model’s value estimate for generating that token at that position.

Although the model is trained as a value predictor, it can still generate pages autoregressively. At each step, the model scores the candidate next tokens, greedily selects the token with the highest value, and appends it to the prefix. This process repeats token by token until the full homepage is generated.

Post-training via reinforcement learning

Our second post-training approach is reinforcement learning (RL). WBC is effective for optimizing entity-level metrics, but it does not directly optimize the homepage as a whole. RL treats page generation as a sequential decision-making problem, allowing the model to optimize a page-level reward while preserving the flexibility of autoregressive generation.

This opens the door to several important capabilities:

Whole-page optimization. RL directly optimizes an aggregate page-level reward, allowing the model to account for interactions across rows and entities, such as diversity, stopping power, and page-level business constraints.
Test-time reasoning. Analogous to its application in LLMs, RL can optimize reasoning capabilities for generative recommendation. Reasoning outputs can also be viewed as a form of automated feature engineering.
Multi-token entity support. In our current tokenization, each entity and row is represented as a single token, so rewards map cleanly to individual tokens. In more complex settings, however, an entity may require multiple tokens, such as [Show_ID] plus [Episode_#] for an episode, or a sequence of semantic ID tokens. In that case, WBC’s per-token labeling becomes ambiguous because a single entity-level reward must be distributed across multiple tokens. RL avoids this issue by optimizing the sequence-level return, making it a more natural fit for variable-length, multi-token entities.

Inspired by the RLHF recipe used to align large language models, we adopt a two-step approach. First, we train a reward model that predicts the page-level reward for a generated page. This reward model is distinct from the reward system described earlier. The reward system converts observed user feedback into a scalar reward for a page that was actually shown, whereas the reward model predicts the page-level reward for a generated page without showing it to the user. This prediction is what lets RL optimize against arbitrary candidate pages during training.

Training against a reward model avoids the high variance of off-policy correction on logged or predicted propensities, but introduces the risk of reward hacking. Since the reward model is trained on data generated from the production policy, it is most reliable on pages similar to those the production policy generates. We therefore use a KL penalty to keep the policy close to the pretrained checkpoint, which itself was trained to mimic the production policy. This keeps the pages within the reward model’s region of coverage and limits opportunities for reward hacking.

For the RL algorithm, we adopt Dr. GRPO, a variant of GRPO that mitigates biases in the training objective. To train the model within this framework, we need the following components:

Prompts: production user requests, represented by context tokens.
Policy and reference models: both are initialized from the pretrained checkpoint; the reference model anchors the KL penalty discussed above.
Reward model: a dedicated transformer-based reward model, also initialized from the pretrained checkpoint, predicts the page-level outcome reward, using the sum of entity-level rewards from our internal reward system as the supervision target. We also incorporate rule-based format rewards to guide the RL policy. For example, the page should resemble a list of rows, and business-critical rows or entities should not appear too low on the page.

Addressing production challenges

Cold start

New entities lack the rich interaction data needed to learn robust token embeddings. We address this through two complementary strategies:

Context injection. We inject metadata about new or time-sensitive entities (e.g., Live Now events) directly into the context tokens, providing the model with semantic and time-sensitive information.
Semantic embedding fusion. Rather than relying solely on entity ID embeddings learned from user interaction data, we represent each entity as a fusion of its ID embedding and a content-based embedding derived from semantic information such as synopses, cast, transcripts, genres, and video content. This fused embedding serves as the input embedding for the entity’s token in the transformer. During training, with small probability, we randomly replace an entity ID token with the generic fallback token (described below), so the model learns to make recommendations from the content-based embedding alone. This ensures that a new entity has a meaningful representation in the same latent space as established entities as soon as its content metadata is available — even before it has any interaction data.

Multi-cadence incremental training

At Netflix scale, daily retraining of a large transformer from scratch is prohibitively expensive, but recommendation models must remain fresh to capture shifting trends and new catalog additions. We address this with a multi-cadence incremental training strategy (Figure 3).

Figure 3. Multi-cadence incremental training. Periodic large-scale pretraining and post-training passes run on a broad historical window. Between them, daily incremental updates combine the latest day’s data with a sampled subset of past data to keep the model fresh while avoiding catastrophic forgetting.

Our training pipeline operates on a cyclic schedule with two distinct rhythms. At a tunable cadence, we conduct a large-scale pretraining and post-training pass on data from a broad historical window. Between these passes, each day we perform an incremental update by continuing post-training from the previous day’s checkpoint, using a mix of the latest day’s data and a sampled subset of past data. This helps the model stay current with new trends and catalog changes while preventing overfitting and catastrophic forgetting.

To manage the daily influx of new tokens (e.g., new entities, rows), we employ fallback tokens. New tokens are initialized using fallback tokens of their type (e.g., [Row_Fallback_Token] for new rows, [Entity_Fallback_Token] for new entities). During training, we randomly replace a small percentage of known tokens with fallback tokens, teaching the model to handle unknown tokens gracefully.

Enforcing business rules

A Netflix homepage must satisfy structural constraints (e.g., organized as a list of rows) as well as product logic such as deduplication, row pinning, and category consistency (e.g., entities in a Comedy row must be comedies). While training signals can encourage rule adherence, they cannot guarantee strict compliance.

We enforce these rules at inference time through constrained decoding. At each autoregressive generation step, we compute a mask of eligible tokens based on the applicable business rules and apply it to the output logits, allowing only rule-compliant tokens to be generated. This is greatly simplified by our custom tokenization: because each entity and row is a single token, business rules map directly to token-level masks, avoiding the multi-token bookkeeping that constrained decoding requires over a text vocabulary. For example, to pin a specific row (e.g., popular games) at a fixed position (e.g., row position 2), we simply mask out all other tokens at that position.

Hybrid row decoding

Autoregressive generation ensures that each newly generated token is conditioned on the full preceding context, but generating every entity token one at a time can be expensive. We leverage the structure of the homepage to balance inference efficiency with the amount of contextual information available to each generated token.

Within each row, the first few entities are especially important: they receive the most user attention and strongly shape the row’s perceived quality and theme. To reduce inference latency, we use a hybrid row decoding strategy. The model autoregressively generates only the first few entities in each row. Conditioned on this generated prefix, we obtain logits for all eligible entities in a single forward pass and select the top-scoring remaining entities, subject to the same inference-time business-rule constraints described above.

This approach preserves autoregressive conditioning where it matters most while avoiding the latency and cost of decoding long rows token by token.

Offline experiments

We ran a series of ablations on Netflix internal data to understand how different components of GenPage affect model quality. Because the system was developed iteratively, individual ablations span different training configurations and data snapshots, so we report only relative comparisons within each study. Unless otherwise noted, experiments use ~200M-parameter models and report results on a held-out evaluation set.

Does pretraining help?

We compare WBC post-training with and without a preceding next-token-prediction pretraining stage. Figure 4 shows that pretraining yields substantial improvements across all metrics.

Figure 4. Relative improvement from pretraining (versus WBC post-training without a pretraining stage), across loss reduction, row AUC lift, and entity AUC lift. Loss is the weighted binary cross-entropy; Row and Entity AUC are sample-weighted ROC-AUC over row and entity targets.

The gains may look small in absolute terms, but they are large in our production regime: setting aside the sample weighting, an Entity AUC lift from 0.91 to 0.92 means that for a randomly drawn pair of impressed entities, the model’s misranking rate drops from 9% to 8% — a magnitude of improvement we rarely observe from a single change on a mature production system. Pretraining the model on the “language” of the Netflix homepage provides a strong initialization for post-training, mirroring the pretrain-then-post-train recipe behind modern LLMs.

How does performance scale with model size?

We sweep model size from ~120M to ~900M parameters (Figure 5) and report the next-token-prediction loss from pretraining and the WBC loss from post-training. Both losses decrease in a power-law-like fashion, mirroring the scaling trends seen in LLMs. This confirms that the generative approach scales favorably with model size, suggesting that recommendation quality can be further improved by scaling capacity.

Figure 5. Pretraining and WBC post-training losses as model size scales from 120M to 900M parameters. Both decrease in a power-law-like fashion, mirroring LLM scaling trends.

How does performance scale with information in the user context?

Over the course of development, we progressively enriched the prompt, both by adding new data sources to the context and by refining how each source is tokenized. With model size held fixed, the WBC post-training loss decreases substantially as the context is enriched (Figure 6).

Figure 6. WBC post-training loss as we progressively enrich the user context tokens. Loss is normalized to the first step (= 1.0).

The model-size sweep and the context-enrichment sweep span different axes and are not strictly comparable: the model-size study covers roughly an order of magnitude in parameters, while the context study spans the full trajectory of our prompt design. Even so, the gap between the two is striking. Scaling the model from 120M to 900M parameters reduces WBC loss by roughly 1.3%, whereas the cumulative effect of enriching the context is around 6.9%. In several cases, a single well-designed context addition delivers a larger improvement than the entire ~7.5× model-capacity scaling.

This suggests that, in our regime, enriching the prompt — both what we put in the context and how we tokenize it — yields a substantially larger improvement than scaling model capacity. Personalization quality appears to be bottlenecked first by the information and representation available to the model, and only then by capacity. We expect context enrichment to dominate until the context is saturated, at which point model capacity becomes the primary driver.

Does RL post-training optimize at the page level?

In offline evaluations (Figure 7), RL post-training consistently improves the page-level reward over the pretrained checkpoint, but this is largely confirmatory: the reward is computed using the same model the policy is optimizing against. More interestingly, although diversity is not part of the RL objective, homepage diversity — measured via pairwise embedding distance among entities on the page — also increases over the course of training. This suggests that the RL-trained policy is optimizing the page as a whole rather than myopically optimizing each token in isolation.

Figure 7. RL post-training dynamics. Reward and diversity are shown relative to the initial checkpoint (1.0). Reward rises as expected; diversity also rises, despite not being part of the RL objective.

Online evaluation

We conducted an online A/B test against the current production homepage recommender using GenPage. In this test, GenPage decoded over the existing production row and entity candidate sets, which help handle many business rules (such as eligibility).

Figure 8 shows the result: all variants delivered statistically significant improvements on the core user engagement metric we use for launch decisions (p < 0.001) against a mature, highly optimized multi-stage production baseline. The variants differed in their training-data configurations; that they all delivered comparable lifts suggests the gain is robust to these design choices rather than dependent on a particular configuration.

Figure 8. Daily core user engagement metric over a 14-day online A/B test. The figure shows the average treatment effect of several GenPage variants (differing in training-data configurations) against the production baseline. Shaded regions are 95% confidence intervals. All variants delivered statistically significant improvements over production.

Alongside the engagement wins, we observed unintended shifts in the distribution of impressed entity categories (e.g., new vs. established titles, TV shows vs. movies). These shifts are not necessarily negative, but they are not something we explicitly optimized for, and they warrant deeper investigation. We suspect these shifts reflect GenPage personalizing more precisely than the production stack — consistent with an increase in homepage impression efficiency, i.e., users engaging with what they saw using fewer impressions. This sharper personalization appears to surface production-inherited components (such as the reward system) that aren’t yet aligned with the new generative paradigm. We plan to characterize the drivers of these shifts and, where appropriate, tune these components so the resulting distributions better align with desired product behavior.

We also observed strong responsiveness to in-session signals: the latest in-session actions quickly influenced subsequent recommendations and faded back to long-term preferences after a day or two, confirming that the model effectively attends to action timestamps. This responsiveness emerges naturally from the generative formulation, without the extensive manual feature engineering used in our production stack.

Contrary to the common assumption that generative models are slower, GenPage reduced end-to-end serving latency by 20% relative to the baseline. By replacing multiple ranking stages and heavy feature computation with a single transformer operating on raw tokenized inputs, we eliminated substantial serving complexity and computational overhead. Custom tokenization and hybrid row decoding further reduced the number of decoding steps, and thus latency. The 20% reduction was achieved without exhausting the available optimizations; further reductions are possible, and this headroom can be reinvested in capacity or richer prompts.

Conclusion

We presented GenPage, an early step toward end-to-end generative Netflix homepage construction: representing user context as a tokenized prompt and generating the entire homepage autoregressively in real time. This collapses the traditional multi-stage recommender stack into a single transformer that can be optimized end-to-end.

In online A/B tests against a mature, highly optimized multi-stage production system, GenPage delivered statistically significant gains on the core user engagement metric we use for launch decisions, while reducing end-to-end serving latency by 20%. Achieving this required adapting the LLM training recipe — pretraining followed by WBC or RL post-training — together with a set of domain-specific techniques: custom tokenization for serving efficiency and product control, context injection and semantic embedding fusion for entity cold start, multi-cadence incremental training for model freshness, constrained decoding for business-rule enforcement, and hybrid row decoding for inference efficiency.

Two offline findings stand out. First, in our current regime, enriching the prompt yields a substantially larger improvement than scaling model capacity — a takeaway we expect to generalize to other industry-scale personalization settings, at least until the available context is fully exploited. Second, RL post-training increases homepage diversity even though diversity is not part of the objective — an indication that page-level optimization captures interactions across rows and entities.

Several pieces of the full vision are still in progress: long context still relies on handcrafted summarization, and broader LLM-style capabilities — language, multimodality, and reasoning — have not yet been incorporated. One promising direction here is a hybrid tokenization combining our domain-specific tokens with generic text tokens, retaining structured control while inheriting the strengths of general-purpose LLMs; conceptually, this introduces an additional recommendation modality into an LLM.

More broadly, we expect many advances from the LLM ecosystem to transfer naturally to this setting, and the boundary between an LLM and a recommender system may increasingly blur. Our results suggest this is a viable path toward simpler recommender systems that align more directly with user satisfaction.

Acknowledgments

Contributors to this work (in alphabetical order): Abhishek Agrawal, Baolin Li, Casey Stella, Daneo Zhang, Dan Zheng, Donnie DeBoer, Fengdi Che, Fernando Amat Gil, Grace Huang, Inbar Naor, Ishita Verma, Jason Uh, Jimmy Patel, Justin Basilico, Lanxi Huang, Lingyi Liu, Liping Peng, Louis Wang, Michelle Kislak, Nathan Kallus, Nicolas Hortiguera, Paran Jain, Qusai Al-Rabadi, Rein Houthooft, Ryan Lee, Santino Ramos, Scarlet Chen, Shaojing Li, Sheallika Singh, Si Cheng, Wei Wang, and ZQ Zhang.

GenPage: Towards End-to-End Generative Homepage Construction at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

GitHub and UNDP team up to advance development priorities in Ghana with open source - The GitHub Blog

2026-06-26T16:53:57.000Z

Open source software is commonplace. Many people use it without even knowing, whether for everyday web browsing or building tools to improve efficiency. At its core, open-source software is built on code that is publicly available for anyone to use, adapt, and improve. It’s more complex, however, when a government sets out to adopt open source at scale to drive development impact.

This discussion is happening in Ghana, where the Ministry of Communications, Digital Technology, and Innovation (MoCDTI) is undertaking an ambitious and deliberate digital reset to create more jobs, grow enterprises, and empower youth. This effort includes advancing more than a dozen legislative reforms at once, covering areas such as cybersecurity, data protection, electronic communications, data exchange and emerging technologies. Some are new legislation, while others replace existing laws. Together, they are redefining the country’s ICT legal framework.

What does long-term sustainability look like in practice? Will the systems built on these foundations be open and auditable? Or will they end up trapped behind proprietary walls, with a limited number of vendors able to assess and support them?

Governments pivot to open source adoption

For governments and other large organizations, making a strategic shift towards adopting open source is a significant undertaking that presents both challenges and opportunities.

A ministry might deploy an open-source tool for a particular project, however systems-wide questions on managing licensing compliance, building internal maintenance capacity and coordinating across other ministries within a government may be unanswered. As a result, this adoption of open source, whilst transformative in value, could remain in isolation–without ever becoming the institutional default.

This is the gap the OSPO (Open Source Programme Office) model is designed to fill. OSPOs are a common setup in the private sector, providing structured governance for open source: policies, compliance, community engagement, and skills development. The model is now increasingly being adopted by universities, civil society, and within the public sector that has underscored the need for readiness to ensure effective adoption.

The Open Source Programme Office Readiness Assessment, known as OSPORA, is a UNDP-led initiative that does exactly this for countries. Supported by the government of France, OSPORA is a structured diagnostic approach that helps governments assess their readiness for open source adoption and governance. Critical for identifying practical steps, it could be thought of as the equivalent of running an audit before an architecture migration except the architecture is institutional, not technical.

OSPORA asks: What policies exist? What’s the technical capacity? Who are the internal champions? Where are the coordination failures? Does procurement deter open source adoption? And crucially: what’s politically realistic given the current government’s priorities?

Ghana demonstrating what’s possible

Early last month in Ghana, the GitHub Policy team teamed up with UNDP to carry out of these assessments. Over the course of a week, the team ran interviews and workshops with diverse stakeholders, including:

Senior officials at MoCDTI, including the National IT Agency and the Kofi Annan-India Center of Excellence, who are leading the digital transformation and legislative reform process
Heads of IT departments across government ministries
Community tech groups building open source within Ghana’s developer ecosystem
The local Linux user group, which bridges global open source governance, community and local implementation

The OSPORA methodology draws on Public Digital’s framework on open source in government, which is a structured set of questions covering not just technical readiness but institutional structures and policies, procurement practices, legal frameworks, and political will.

What emerged was nuanced. Ghana has political commitment to digitalization, clear champions backed by over a decade of open source delivery experience, and a growing tech community eager to contribute. Importantly, the case for open source is being made from within—by officials who see it as an essential means to build a more digitally sovereign future, and a closely linked ambition to grow Ghana’s national digital economy and local technology capabilities.

At the same time, there are gaps: the lack of a clear, centralized policy on open source; coordination challenges between the National Information Technology Authority (NITA) and individual ministries operating in siloes; and under-resourced IT teams, especially in rural areas. In instances where progress have stalled, it was rarely attributed to technical reasons—the more significant barriers are institutional inertia and resistance to change, particularly where existing procurement patterns and vendor relationships are well entrenched.

These findings are an opportunity to make meaningful improvements and expand the delivery of public services through open source to better serve the people who rely on them.

A call to action for development impact

Ghana is home to one of West Africa’s fastest-growing tech communities, as well as some of the region’s first accredited Digital Public Goods. It also has the second highest number of GitHub developer accounts in West Africa. With Ghana’s One Million Coders initiative underway to skill up a massive developer workforce by 2028 and others digital development initiatives emerging, the foundations are being built, and the talent pipeline is growing.

Open source governance shapes what gets built. Codes that get contributed can support the building of a national infrastructure. For example, UNDP maintains open source projects on GitHub that governments deploy such as the National Carbon Registry to help countries implement and manage carbon markets. The policy decisions being made in Ghana and other countries today about data exchange standards, content liability, emerging tech regulation will determine the future of an open source in expanding choices for countries on their digital transformation journeys.

Ghana’s story illustrates what open source sustainability looks like at the national level. While conversations around open source frequently focus on maintainer burnout and funding models, OSPORA represents a different piece of the puzzle on how institutions can sustain open source adoption over time and across administrations at scale. It’s also why GitHub alongside UNDP is excited to participate in the United Nations Open Source Week taking place this week in New York City.

The GitHub team is deeply grateful to the UNDP Digital, AI and Innovation Hub and the Country Office team in Ghana whose vision, persistence, and on-the-ground leadership have made this possible. Partnerships like this don’t happen in the abstract, they happen because people show up, do the hard work of building trust across institutions, and stay committed to the long game. This collaboration underscores our shared belief that open source is a powerful catalyst for sustainable development and the growth of digital public goods.

Open source offers a structurally different path.

Explore UNDP’s open source projects, learn more about UNDP Ghana, or check out GitHub’s Social Impact programs.

The author is grateful for contributions from Cynthia Lo from GitHub, and Laura Hildebrandt, Benjamin Bertelsen, and Dwayne Carruthers from UNDP.

The post GitHub and UNDP team up to advance development priorities in Ghana with open source appeared first on The GitHub Blog.

Transitioning as a Hubber - The GitHub Blog

2026-06-26T12:00:00.000Z

When I joined GitHub, my legal name was Ursula—but my handle was gleeblezoid. Now, as Arthur, I’m still gleeblezoid.

Since our remote-first culture primarily uses handles, transitioning at GitHub was easier than it would have been earlier in my career. I previously worked in IT at companies that only used names for identification, which can be challenging for professionals transitioning.

I started my career doing IT support and operational work, but being interested in computers meant teaching myself how to code. A colleague from a previous role referred me to GitHub, and I started out five years ago on the IT Engineering team. After repeatedly bothering various security teams with issues and pull requests, I got adopted into the Enterprise Security team after just six months. I’ve been there ever since.

While here, I’ve been proud to work with my team on migrating our main SaaS platform to infrastructure as code, and to be a guest speaker a handful of times at Oxford University on the subject of version control.

But another great thing about working here: GitHub offers gender affirming care related benefits to all employees in terms of covering healthcare; I can expense voice training, HRT prescriptions, and therapy among other things.

There are also less obvious things that made GitHub a safe place for me to transition. Being a remote-first company means I don’t need to agonize over what to wear to the office or who will see me on the way there. Most of my work is captured in writing within Slack or GitHub itself, so when I started voice training and eventually having my voice break on HRT I wasn’t spending the whole day talking to people out loud.

We have the kind of culture where my main avatar can be a cartoon frog in a suit and nobody bats an eye, which removes the entire problem of people guessing my gender by my appearance.

I know a lot of people in the tech industry, and more trans people than the average person probably does. I know people who are closeted at work, people who go through bureaucratic nightmares on changing their name, and people for whom coming out at work is something they end up doing on a recurring basis with every new set of people they interact with.

I’ve not experienced that. Outside of the understandable bureaucratic friction of changing my name in places like payroll it’s been smooth. My team call me what I want to be called and treat me like a regular human being—as has everyone else at work I’ve interacted with. I updated my name and pronouns on our internal systems, and that was that.

Being trans isn’t easy or universally accepted. I’m not sure when I’ll next get to see my overseas teammates in person, for example. It’s also not an experience solely defined by hardship or social barriers. There is a great deal of joy in showing up as yourself and in sharing that joy with others. I nearly cried on a Zoom call when I heard someone use my name and refer to me as “him” for the first time at work, and I absolutely did cry when one of my teammates sent me a shaving kit in the mail.

Every Hubber I have mentioned my transition to has been genuinely happy for me. Several of them expressed this through Arthur the aardvark, and Monty Python Holy Grail memes (we are geeks after all).

I’ve always been a man, I just needed time and support to live as one and participate as one in society. In most settings I need to explain to people how Ursula Searle became Arthur Searle, but at GitHub I’ve thankfully always been gleeblezoid.

The post Transitioning as a Hubber appeared first on The GitHub Blog.

Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks - The GitHub Blog

2026-06-25T22:59:45.000Z

While the model provides the raw intelligence, the harness shapes how effectively that intelligence is applied. The GitHub Copilot agentic harness is a single shared component of the GitHub Copilot SDK, which powers the GitHub Copilot CLI, GitHub Copilot app, and Copilot code review, along with a wide variety of experiences across GitHub and Microsoft. Improve the harness, and every surface benefits.

The GitHub Copilot agentic harness powers GitHub Copilot experiences.

The tools, context, and workflow are orchestrated by the harness. A harness should be fast, token-efficient, and predictable for developers. That’s what we designed GitHub Copilot’s agentic harness to do.

In this post, we’ll present data showing the efficiency and performance of the GitHub Copilot agentic harness across a wide range of agentic software engineering tasks.

How we iterate with benchmarks

We continuously evaluate the capability and efficiency of the GitHub Copilot agentic harness through a combination of public and internally developed benchmarks. Our public benchmarks include industry standards, while several internal benchmarks are derived from large codebases inside GitHub and Microsoft. We complement this with real-world metrics and online experiments to ensure we understand the harness’s performance in controlled environments and its practical impact on agentic problem solving and task completion. 

We control as many variables as possible to evaluate the performance of GitHub Copilot’s harness compared to the model provider’s harness: use the same model, the same benchmark task, normalized on context window, reasoning efforts, tool selection, and MCP servers.

Below we report our latest results for a subset of the benchmarks we track, across four leading models: Claude Sonnet 4.6, Claude Opus 4.7, GPT‑5.4, and GPT‑5.5:

Benchmark	Domain	Purpose
SWE-bench Verified	500 human-validated bug-fix tasks from open-source Python repositories	Established industry-standard benchmark for coding agents
SWE-bench Pro	More difficult, multi-step engineering tasks requiring deeper reasoning and broader code changes	Better reflects complex, real-world software engineering work
SkillsBench	How effectively an agent uses skills to solve tasks	Evaluates extensibility and skill use and triggering capabilities
TerminalBench	Agent performance on terminal-based tasks	Measures effectiveness in command-line workflows used by developers
Win-Hill	Internal benchmark for tasks running inside Windows containers	Validates that performance generalizes across operating systems and environments

Throughout, we compare GitHub Copilot CLI against the model-vendor harnesses that ship those models natively: Claude Code for Sonnet 4.6 and Opus 4.7, and Codex CLI for GPT‑5.4 and GPT‑5.5.

Token efficiency

Holding the model and task fixed, across multiple benchmark results, the GitHub Copilot harness achieves task completion rates on par with other model-vendor harnesses, while showing lower token consumption across most configurations.

Token efficiency: GitHub Copilot CLI vs. other model-vendor harnesses

Task resolution

Token efficiency only matters if the work actually gets done.

Task resolution rates for the GitHub Copilot agentic harness across these benchmarks are on-par with model-vendor harnesses when used with a fixed model and benchmark task. This ensures that the full potential of the underlying model is available, along with multi-model flexibility, token efficiency, and memory and context capabilities.

Task resolution: GitHub Copilot CLI vs. the model-vendor harnesses

These results reflect effective parity, since the differences in either direction are within the variance due to the stochastic nature of the models, making the cross-harness performance on-par.

TerminalBench: Token efficiency, task completion, and variance

To continuously improve the GitHub Copilot agentic harness on task completion and token efficiency, we regularly perform thorough analyses across benchmarks. Below is an example of variance analysis on TerminalBench 2.0, which not only highlights GitHub Copilot’s strength on task completion and token efficiency, but also shows the run-to-run variance intrinsic to this kind of benchmark.

Resolution rate vs. cost per task. Up and to the left is better: solve more, spend less.

Every marker is one agent-and-model configuration on TerminalBench 2.0, with resolution rate on the vertical axis and dollar cost per task on the horizontal axis. The shaded ellipse around each point shows the ±1σ run-to-run spread, displaying how much each configuration varies between runs.

Three things stand out:

GitHub Copilot’s agentic harness is on par with or ahead of other agents on task completion and cost per task across the configurations we evaluated. Purple (Copilot) markers and their same-model competitors sit within overlapping ellipses on both axes for nearly every model—the differences are inside run-to-run variance. Copilot is never below a competitor on completion or to the right on cost.
Run-to-run variability. We ran each agent-model combination at least five times. The ellipse marks the 1σ spread of those runs; a tighter ellipse in the chart means more reproducible results, while a wider one shows results swinging further from run to run on both cost and task completion.
The benefit of GitHub Copilot’s model choice: The chart shows a real trade-off: GPT models (left) deliver the best value: strong resolution at the lowest cost. Claude Opus (upper right) reaches the highest resolution at a premium. GitHub Copilot puts both on the table, so you can pick efficiency or peak quality per task.

One harness, many models

The GitHub Copilot agentic harness supports 20+ frontier models across the GPT, Claude, Gemini, and MAI families, plus bring your own key for open‑source and local models. You can choose the right model for the capability and cost profile of each task, or let Auto model selection choose for you, balancing task intent and model health to optimize token efficiency.

A multi‑model architecture also unlocks harness‑level capabilities a model-vendor harness simply can’t offer. Rubber Duck, for example, uses cross‑model‑family critique, where one model reviews another’s work to improve outcomes beyond what any single model produces alone.

Conclusion

Benchmarks are just one signal among several. We are constantly working to improve quality across benchmarks, real-world usage metrics, and online experiments, while pushing to efficiently make the most out of every token.

GitHub Copilot delivers task‑resolution on par with leading model-vendor harnesses while using fewer tokens across several configurations, without locking you into a single model through its multi‑model architecture. For developers, this means you can get comparable task completion with lower token cost, while still choosing the model that best fits your task.

Try it yourself

Try GitHub Copilot with the model of your choice, compare approaches on the tasks you run every day, and see how different models and agent strategies perform in your environment.

Learn more about:

The same agentic harness powers these experience. We’re continuing to improve its quality, efficiency, and flexibility.

Methodology

To make the comparison as controlled and reproducible as possible, we run each agent with equivalent settings across models, tasks, and environments.

All runs have a two-hour timeout. All agents run non-interactively single-turn, with web-tools disabled, and all tools allowed.

TerminalBench2 analysis: Default settings enabled for agents with reasoning effort set to medium (e.g. tool search is enabled for Claude Code and Copilot CLI uses github-mcp-server). Codex and Claude Code use direct Anthropic and OpenAI endpoints. To ensure complete and reliable results, any missing data or infrastructure-related failures were re-run until all 89 TerminalBench2 tasks produced results. Model-generated errors were retained and not excluded from the analysis. Each model was evaluated across five independent runs, and Copilot was tested in two separate evaluation batches to enable comparison with Claude Code and Codex.

All benchmarks: All agent model pairs normalized to same context window size, same prompt token limits, reasoning effort (medium) and settings—no tool search, no MCP servers. Keeping the harness’s default built-in tools. Infrastructure-related anomalies and network-access effects are excluded across all agents for a benchmark to ensure fair comparisons. To reduce the impact of run-to-run variability on smaller benchmarks (<100 instances), five independent runs were conducted, and the best scored run is reported. All metrics are presented as pass@1. These normalizations mean results differ from public benchmark submissions, which typically use higher reasoning effort and other tuned settings.

The post Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks appeared first on The GitHub Blog.

Privacy-Aware Infrastructure in the AI-Native Era: An Asset Classification Case Study - Engineering at Meta

2026-06-25T22:30:51.000Z

Privacy controls — systems that enforce retention, access, allowed-purpose, downstream-sharing, or anonymization policies — require a reliable understanding of data to function. Before such a control can operate effectively, it must know exactly what it is looking at. This can be complex, as demonstrated by a field simply named “age“: In one context, it might describe a person and require strict protections, while in another, it could be a cache time-to-live (TTL) numerical value in an infrastructure pipeline.

Figure 1: One column name, two governance outcomes. The identical field age is personal data when it describes a person, but ordinary system metadata when it is a cache TTL. Which is why a name alone cannot determine the privacy requirement.

This is the everyday problem behind privacy-aware infrastructure (PAI): The inputs are noisy and probabilistic, but the outputs need to be precise enough to drive enforcement.

AI-native products make that problem harder. They introduce new data modalities, faster iteration cycles, derived features, embeddings, multimodal inputs, and changing policy interpretations. Manual review remains important for judgment and accountability, but it cannot keep up with the volume and pace of change.

At Meta, we apply a hybrid pattern for asset classification at scale:

Build a rich context before asking a model to reason.
Use LLMs to handle ambiguity, cold start, and novelty.
Keep human-reviewed labels separate from model-generated recommendations.
Distill stable behavior into deterministic, versioned rules for routine enforcement.

The end goal is not “LLMs everywhere.” Instead, it is a system that can learn from ambiguous signals while moving production enforcement toward logic that is low latency, replayable, and easier to audit.

The LLM does not make the production decision in the common case, deterministic rules do. We use LLMs deliberately and narrowly, to interpret novel or ambiguous assets, and then to distill what they learn into versioned human-reviewed deterministic rules, which steadily shrinks the LLM’s role in production over time. Humans stay in the loop where it matters most. People adjudicate the reviewed reference labels, and they review and approve rule promotions that could change how protection is enforced.

PAI addresses four operational concerns:

Understand what data exists and how it is governed.
Discover which data flows are relevant to a policy question.
Enforce retention/access/purpose/sharing constraints.
Demonstrate compliance through verifiable evidence.

Asset classification sits at the understand layer. It provides the foundation that every downstream concern depends on.

Figure 2: The privacy-aware infrastructure stack is a dependency pyramid: each capability rests on the one below it. Understand —classifying what the data actually is — is the load-bearing base. If it is wrong, everything above (discover, enforce, demonstrate) inherits the error.

Why Asset Classification Matters

Asset classification is the foundation for many privacy controls. Before a system can enforce retention, access, allowed-purpose, downstream-sharing, or anonymization policies, it needs a reliable view of what the asset is and how it should be governed.

An asset can be more than a table or column. It can be a nested field inside a payload, a log key, an event parameter, an API field, a machine learning (ML) feature, an embedding, or a derived dataset produced by an intermediate pipeline. That breadth matters because AI-native systems often transform data across many representations. A single source signal can move through pipelines, become a feature, appear in a model-training workflow, or be joined with other derived signals. Classification has to follow the meaning of the data, not just its shape.

There are four recurring challenges:

First, noisy and weak signals: Dozens of context fields are fetched per asset, which forces the model to rediscover what matters each time. High token usage dilutes attention, and decision boundaries get buried in irrelevant or misleading fields. A field called age in a caching pipeline is a concrete example: Without code resolution and lineage analysis, a classifier will trigger false restrictions on the entire pipeline.

Second, the relevant context is distributed. Code, lineage, ownership, semantic annotations, documentation, and usage patterns often live in different systems. A good classifier needs to assemble that context before making a decision.

Third, requirements evolve. Product teams move quickly, and policy interpretation can change as new product capabilities appear. A static rule set or periodic manual review process can leave gaps between reviews.

Fourth, classification is only useful if it feeds enforcement. A false positive can trigger unnecessary restrictions downstream. A false negative can leave a protection gap. The classifier sits near the front of the enforcement pipeline, so its error profile affects every system that depends on it.

This creates the central tension: Classification needs to reason under ambiguity, but enforcement needs decisions that can be explained and reproduced later.

Figure 3: Four distinct difficulties (context dependence, sparse signal, a heavy long tail, and constant schema drift) all collapse into a single tension: Classification wants to reason under ambiguity, while enforcement demands results it can explain and reproduce. The whole design exists to hold these two in balance.

The Pattern

Our approach is built around three principles that emerged from building and operating the system:

First, context beats prompts. Most classification failures were not caused by weak instructions; they were caused by weak or missing evidence. Hours of prompt optimization produced marginal improvement when the model was reasoning over raw, noisy fields. Structuring context into evidence briefs, with supporting signals, contradicting signals, provenance, and masked circular fields, produced much larger accuracy improvements. The practical lesson is simple: Focus on what goes into the model before optimizing how you ask.

Second, decouple evaluation from optimization. LLM outputs are useful recommendations, but they cannot become their own ground truth. The evaluation loop needs to stay independent from the classifier: different models, different prompt strategies, frozen reference sets, human-reviewed labels, and regression gates. If evaluation and optimization share the same loop, the system can end up measuring drift instead of progress.

Third, distill stable behavior into deterministic rules. LLMs are useful for ambiguity, cold start, and new patterns. They are not the right default enforcement mechanism at scale. When the system finds stable, validated patterns, those patterns should become versioned, auditable rules that run without the LLM. Over time, the classifier should progressively shrink its own LLM surface area, leaving model inference for novel or ambiguous assets while routine enforcement becomes deterministic, low-latency, and replayable.

These principles translate into a concrete operating pattern: Define a stable classification contract, build a context mesh, route decisions through a deterministic-first funnel, and keep the learning loop safe with independent evaluation and reviewed labels.

To execute on this pattern, we break the work down into seven practical stages. These stages transform the high-level architecture into a concrete, repeatable process.

The rest of this post walks through those pieces using asset classification as the case study.

Figure 4: The two-lane operating pattern: (1) Most requests (~85%) resolve on the deterministic path in single-digit milliseconds, and within ~40 ms including context assembly; the ~15% LLM fallback is slower (seconds) and budgeted separately; (2-3) a nightly offline lane samples served decisions, adjudicates them against reviewed truth, and re-evaluates; (4) distilled rules are promoted back into the live decision funnel by content-addressed swap. The masking invariant holds on both lanes.

1.) Start With the Contract

A classifier should behave like a platform service. That means its contract should be small, explicit, and stable. For each asset, the classifier receives an identifier and a bundle of context. It returns a structured result with:

A category from the classifier’s taxonomy.
A confidence score – a raw model self-assessment whose calibration we evaluate against reviewed labels (see below).
A decision trace showing which evidence influenced the result.
The rule that matched, if the decision came from deterministic logic.
Version information for the context, rules, and prompt used to make the decision.

The taxonomy is domain-specific. One classifier might distinguish user data from operational data. Another might classify whether an asset is eligible for a particular AI-training use case. We avoid forcing every classifier into one universal taxonomy. Instead, each classifier owns one scoped question, and downstream systems compose the answers when they need multiple facets.

That scoping is important. A narrow classifier is easier to evaluate, easier to debug, and easier to govern. It also makes the decision trace more meaningful because the classifier is explaining one decision, not trying to solve every policy question at once.

Figure 5.:The classifier is a service contract, not a prompt: a fixed request in, a typed result out. Three response fields — matched_rule, decision_trace, and versions — are what make every classification replayable and auditable after the fact.

2.) Build Context Before Prompting

Most classification failures are not prompt failures. They are context failures. If the only signal is a field name, the model has to guess. If the system can also provide code references, lineage, ownership, semantic annotations, and nearby usage, the model can reason from better evidence.

In practice, the context mesh may include:

Source-code resolution, including where a field is defined or used.
Ownership and organizational metadata.
Semantic annotations, such as data type or origin.
Lineage signals that show where data came from and where it flows.
ML heuristic outputs from scanners or embedding-based classifiers.
Code search results that show references, logging declarations, or call sites.

The point is not to pass everything to the LLM. More context is not automatically better. Some fields are redundant. Some are noisy. Some can create circular reasoning if they already encode the label we are trying to predict.

So the system creates an evidence brief – a compact summary of the strongest supporting signals, contradicting signals, and provenance chains. Instead of asking the model to sift through raw context, we ask it to reason over the evidence that is most relevant to the classification decision.

Figure 6: The evidence brief assembled for one asset. Each signal is weighted by reliability (bar length) and split into support versus contra. The pre-existing privacy label is deliberately masked. Feeding it back would let the model grade its own homework and collapse the signal.

Without this structuring, the model receives dozens of raw fields per asset and must rediscover what matters leading to high token consumption, diluted attention, and decision boundaries buried in noise. The evidence brief solves this by pre-ranking signals. For a field like user_payload.email_address, an evidence brief might say:

Supporting signal: Lineage connects the asset to a user-facing logging pipeline (weight 0.8).
Supporting signal: Semantic annotation indicates EMAIL-like data (weight 0.9).
Contradicting signal: Ownership metadata points to an infrastructure team, not a user-facing product (weight 0.3).
Suppressed signal: An existing privacy label was removed to avoid circular reasoning.

That last point matters. A model should not be allowed to “discover” the correct answer by reading a field that already contains the answer. Masking is not just prompt hygiene, it is a system invariant. Fields masked from the LLM are also blocked from learned rule distillation so the model cannot smuggle the answer into a rule by way of a circular field. Deterministic rules that use high-risk fields require explicit review.

Over time, the system can also learn which context fields are useful. Fields that consistently improve classification can be prioritized. Fields that are unstable, redundant, or harmful can be suppressed. This turns signal quality from a matter of intuition into something measurable.

3.) Use a Decision Funnel

Once the context is assembled, the classifier routes the asset through a decision funnel.

The first path is deterministic. If a known, versioned rule matches the asset, the classifier can return a decision quickly and with a clear explanation. Deterministic rules work well for stable patterns – a well-understood namespace, a semantic annotation with high precision, or a combination of signals that has been validated over time.

The second path is LLM-based. If the asset is novel, ambiguous, or outside current rule coverage, the classifier asks the model to reason over the evidence brief. The model returns a candidate label, confidence indicators, a decision path, and cited evidence.

In our production deployment, Figure 7 shows how cheap deterministic rules resolve the large majority of traffic, roughly 85%, in single-digit milliseconds. The LLM is reserved as a fallback for the roughly 15% that is novel or ambiguous. That path is slower — on the order of seconds — and roughly 400 times the compute cost, so it is budgeted separately. Both paths emit the identical result schema. The masking invariant is enforced on each.

Figure 7: Cheap, deterministic rules resolve the large majority of traffic (~85%) in single-digit milliseconds; the LLM is reserved as a fallback for the ~15% that is novel or ambiguous, a path that is slower (on the order of seconds) and roughly 400 times the compute cost, budgeted separately. Both paths emit the identical result schema, and the masking invariant is enforced on each.

That confidence deserves a careful read. The raw score is a model self-assessment, a number the model produces from its own judgment, not an inherent probability of being correct. So we evaluate its calibration against reviewed labels. Raw scores are compared to the correctness rate actually observed on the human-reviewed reference set, which tells us how well a given score tracks a real probability of being right. Confidence-based routing in the funnel, for example, accept automatically versus route to human review, should use calibrated scores where that calibrated path is enabled, rather than the raw number

Both paths emit the same result format. Downstream enforcement systems do not need to know whether a decision came from a rule or from model-based reasoning. They receive a category, confidence, trace, and versioned decision metadata.

This split is what makes the pattern practical. LLMs are useful for ambiguity and cold start. Rules are better for routine enforcement. The more stable behavior we can distill into rules, the less often the serving path needs model inference.

Rule coverage becomes an important operational metric. If coverage rises while quality holds steady, the classifier is moving toward a healthier steady state: fewer routine calls to the model, lower resource use, lower latency, and decisions that are easier to replay.

A critical system invariant: Fields masked from the LLM are also blocked from learned rule distillation, so a masked signal cannot re-enter the decision through an automatically distilled rule. In one production deployment, a subtle bug in how masked context was handled during rule evaluation caused rules to silently fall through to LLM fallback, so rule coverage appeared to plateau even as the rule set grew. Fixing that handling immediately increased rule coverage and cut LLM inference calls significantly.

The lesson: Masking is not a prompt-engineering concern, it is a system invariant. And deterministic rules that rely on high-risk fields require explicit review rather than inheriting masking implicitly.

4.) Solve Cold Start Deliberately

On day zero, a classifier has a hard problem: There may be millions of assets and very few reviewed labels. Random sampling is not enough. The categories that matter most for privacy can be rare, and rare categories are easy to miss if you wait for examples to appear naturally.

Instead, we seed the process with policy-guided examples:

Rare sensitive categories.
Borderline cases where policy interpretation is difficult.
Negative examples that look sensitive but are not.
Assets where context signals disagree.

The goal is not to eliminate human review. It is to focus human attention on the cases where judgment matters most.

5.) Keep the Learning Loop Safe

Once the classifier is live, it needs to improve without grading its own homework.

We separate two loops:

The reference loop produces reviewed labels. These labels are append-only, versioned, and tracked with provenance. If a label changes, the history is preserved rather than overwritten. Model-generated labels are useful recommendations, but they do not become reference labels automatically. Humans adjudicate uncertain or high-risk cases, and those adjudicated labels become the reference set for evaluation.

The optimization loop improves prompts, routing, context usage, and candidate rules. It can evolve quickly, but it is evaluated against the reviewed reference set, not against labels produced by the same model it is trying to optimize. This distinction matters: A classifier that trains or validates itself on its own predictions can appear to improve while drifting away from the policy intent.

For quality control, we use a multi-panel judge – three independent LLM evaluations, each with a different prompt strategy. One classifies directly from evidence. One critiques the reasoning first, then classifies. One focuses exclusively on metadata signals, such as on-call, lineage, and semantic annotations, while ignoring names and descriptions. All three share a single judge model, a larger reasoning model deliberately different from the classifier model.

The three judges share one scaffold and differ only in how they are asked to reason. The skeleton below is illustrative, not the literal production prompts, but it shows the structure. Each judge receives the same masked evidence brief, the masking invariant still holds, and each returns a structured verdict.

# Shared scaffold (all three judges)
INPUT  = masked_evidence_brief   # pre-existing privacy label removed; masking invariant holds
OUTPUT = {label, rationale, confidence}
JUDGE_MODEL = larger reasoning model, deliberately != classifier model
 
# V1 - direct-from-evidence
verdict_1 = judge(brief, instruction="Classify the asset directly from the evidence.")
 
# V2 - critique-then-classify
verdict_2 = judge(brief, instruction="First critique the supporting and contradicting signals, then classify.")
 
# V3 - metadata-only
verdict_3 = judge(brief, instruction="Use ONLY metadata signals (on-call, lineage, semantic annotations). Ignore names and descriptions.")
 
# Aggregate
final_label = majority_vote(verdict_1, verdict_2, verdict_3)
Agreement = cohens_kappa(verdict_1, verdict_2, verdict_3)   # inter-rater reliability

Results aggregate by majority vote. We track panel agreement across the three judge framings as a stability signal, while Cohen’s kappa (κ) compares the judge consensus against the reference labels (or against the classifier output), providing a statistical signal about classification reliability. These kappa scores drive structured loop decisions: Continue when the system is healthy, WidenAudit when label noise is suspected, FreezeAndAudit when quality declines for two or more iterations, and DataProblem when labels or taxonomy appear fundamentally broken and the system should halt and escalate. This prevents the iteration loop from shipping regressions to production.

For imbalanced taxonomies, we use metrics that expose rare-class failures. Accuracy alone can be misleading: A classifier that labels everything as non-sensitive may look accurate if sensitive assets are rare. Matthews correlation coefficient, macro F1, per-class recall, balanced accuracy, and calibration checks give a more complete picture.

We also look for fragile decisions. One useful test is counterfactual masking: Remove one context field at a time and classify again. If the decision flips when a single weak signal disappears, the asset is flagged for review. The original prediction may still be correct, but the reasoning may be too brittle for confident automation.

When quality drops, the system should slow down or stop. That can mean widening the audit sample, freezing optimization, or escalating a taxonomy or labeling problem for human review. A learning system needs brakes, not just accelerators.

6.) Distill Stable Behavior Into Rules

Even a strong LLM classifier should not be the default enforcement path forever. This distillation (not autonomous decision-making) is where we concentrate the model’s value. Any rule that could change how sensitive data is protected is reviewed and approved by a person before it goes live.

As the system collects reviewed labels and decision traces, it can identify patterns that are stable enough to encode as deterministic rules. A rule might capture a high-precision semantic annotation, a reliable ownership and lineage combination, or a repeated pattern across a class of assets.

Candidate rules go through validation before they affect serving decisions. A typical flow looks like this:

Propose a rule from stable context and label patterns.
Test it against a held-out reviewed set.
Run it in shadow mode on production-like traffic without changing serving behavior.
Promote it only if quality, coverage, and regression checks clear the required gates.
Retire or revise it if the pattern becomes stale or quality degrades.

Distillation operates in stages of increasing complexity:

Stage 1: Field-based rules. Extract single-field patterns (exact match, keyword, numeric range, value-set membership, namespace patterns), with a minimum support of two assets and minimum purity of 80%.These are candidate-mining thresholds for surfacing rules to evaluate, not promotion thresholds. Every candidate from any stage still has to clear holdout validation, a higher dev-precision bar, shadow mode, and human review where protection could change before it can serve.

Stage 2: Composite rules. For uncovered categories, search for conjunctions (e.g., “on-call contains X AND semantic type is ACCOUNT_ID”) under stricter gates — 95% purity, 10 examples minimum, and a stability check on 50% subsamples.

Stage 3 (optional): LLM-assisted rule generation. The model proposes custom conditions combining lineage depth with ownership patterns that manual heuristics miss, gated by rollout controls and default-off. Each candidate rule then proceeds through: holdout validation → blacklist if failed (bounded-TTL) → shadow mode (log, don’t apply) → promote to rules.yaml only if quality gates clear. Promoted rules shrink the LLM surface area.

The important principle is that deterministic rules should not quietly reduce protection. Rule promotion needs safeguards that are designed to catch regressions, especially for sensitive classes.

Validated rules are exported to Python, SQL, JSON, or Hack for deployment in production systems with zero LLM dependency. We manage these rollouts using compare-and-swap (CAS) semantics: We write immutable rule and prompt versions, then activate them via a lease-guarded compare-and-swap on the published pointer (atomic within our single-writer model). This ensures the production path remains a deterministic engine, while the LLM is reserved solely for novel assets that lack rule coverage.

This is what makes the hybrid approach sustainable. LLMs help the system learn. Deterministic rules help the system enforce.

7.) Automate the Right Things

Automation is necessary, but the boundary matters.

We automate context acquisition, evidence brief generation, candidate classification, evaluation runs, failure analysis, and candidate rule proposal. These are high-volume tasks where automation can reduce manual toil and make the process more consistent.

We keep human review in the places where judgment matters – ambiguous policy interpretation, reviewed reference labels, high-risk disagreements, and promotion decisions that could materially affect protection. This is a routing policy, not a prompt.

A decision is escalated for human review when any of the following hold:

Low calibrated confidence. The calibrated confidence falls below the auto-accept threshold, so the decision is not safe to ship automatically.
Judge-panel disagreement. The three independent judges produce no clear majority, or inter-rater agreement (Cohen’s kappa) is low, a signal that the case is genuinely ambiguous.
High-cost rare class. The candidate is a rare sensitive category where a false negative is expensive, so the asymmetric error cost warrants a human check even at moderate confidence.
Fragile reasoning. Counterfactual masking flips the label when a single weak signal is removed.The prediction may still be right, but the reasoning is too brittle for confident automation.
Protection-reducing rule promotion. A candidate rule would change enforcement for a sensitive class in a way that could reduce protection. Deterministic rules should not quietly weaken it.
Controller escalation. The tuning controller enters Pausing or Diagnosing, indicating a quality concern or a fundamental labeling or taxonomy problem that a human must resolve.

That balance is deliberate. Privacy-aware infrastructure should not hide uncertainty. If the model, judge, or evaluation loop disagrees, the system should surface that disagreement as a useful signal. Sometimes the right answer is not a better prompt. Sometimes the right answer is clearer policy guidance, better labels, or a narrower taxonomy.

The best automation in this space does not replace people. It concentrates human attention on the hardest cases, records the reasoning, and turns stable learning into repeatable enforcement over time.

What We Learned

Figure 8: Seven principles separate a robust hybrid classifier from a naive “just ask the model” approach. Each row contrasts the failure mode (left) with the design choice that fixes it (right) — favoring richer context, replayable decisions, honest metrics, an uncontaminated reference set, quality-gated coverage, distillation into rules, and a controller that knows when to stop.

Context Quality Beats Prompt Quality

When classification stalls, it is tempting to keep tuning the prompt. In our experience, better context often matters more. Code resolution, lineage, ownership, and semantic annotations can change the decision space in a way prompt edits cannot.

The practical lesson is simple: Before asking whether the model needs a better instruction, ask whether it has the evidence a human reviewer would need. We saw this with a field named age in a caching pipeline. It was a cache TTL, not a person’s age, and prompt-only changes did not fix it reliably, adding code resolution and lineage did. Once the model could see that the field resolved to a TTL, the false positive went away.

Determinism Means Replayability

The goal is not to make an LLM produce the same text every time. The goal is to reproduce a decision later using the same versioned inputs, context, and logic.

That is why versioning matters. A useful decision trace should tell us what evidence was used, which rule or prompt version was active, and how the decision can be replayed during debugging, incident review, or audit support. In one review, we replayed a single past classification from its stored decision trace and the pinned context, rule, and prompt versions, and reconstructed exactly why the asset received the label it did, without rerunning the LLM.

Accuracy Alone Is Not Enough

For imbalanced taxonomies, accuracy can hide the failures that matter most. If a sensitive category is rare, a classifier can look good while missing too many examples of that category.

Balanced metrics, per-class recall, calibration checks, and review of false negatives are all part of the quality picture. No single metric carries the whole story. We saw a classifier that labeled almost everything non-sensitive show a high overall accuracy while its per-class recall on a rare sensitive category stayed low. Matthews correlation coefficient and macro F1 surfaced the gap that accuracy hid, and the misses became the cases we routed back for review.

Keep Recommendation Separate From Truth

Model-generated labels are useful, but they should not automatically become reference labels. The reference set needs reviewed provenance, and holdout evaluation should not be contaminated by the same model outputs being evaluated.

This separation adds friction by design. It is the friction that prevents a self-reinforcing loop from looking better while becoming less grounded. We saw the pattern directly. An optimization run scored against the same model’s earlier labels appeared to improve, but when we re-evaluated it against the frozen human-reviewed reference set, the apparent gains turned out to drift away from policy intent.

Coverage Is Not Correctness

Higher automation coverage is only useful if quality holds. A classifier can auto-resolve more assets while becoming less reliable on the cases that matter.

That is why coverage should be tracked alongside recall, precision, regression checks, and robustness tests. The goal is not to classify more assets automatically at any cost. It is to automate the cases that are stable enough to automate. In one case, promoting a broad rule lifted automation coverage but dropped shadow-mode per-class recall on a sensitive class. Because we track coverage alongside recall, we caught the regression and narrowed the rule before it reached serving.

Distillation Is the Production Model

LLMs are useful for ambiguity, cold start, and new patterns. Deterministic logic is better for the routine path where decisions need to be fast, explainable, and reproducible.

The sustainable model is a funnel: Let LLMs help discover and reason, then distill stable patterns into versioned rules that enforcement systems can run efficiently.

Self-Regulation Is Architectural, Not Operational

A learning system that does not know when to stop is a potential liability. We built a tuning controller that transitions through regimes:

Observing (gathering signal).
Maintaining (healthy iteration).
Conserving (gains slowing).
Pausing (quality concerns).
Diagnosing (halt for fundamental issues).

In practice, the oscillation detector identifies stalled optimization, classifiers cycling between two candidate prompts without improving, and terminates them early, saving thousands of wasted classification calls per stalled run. This self-regulation was designed into the architecture from the start; retrofitting it would have been significantly harder.

Figure 9: The controller is a state machine, not a retry loop. It escalates only as severity demands, Maintaining → Conserving → Pausing, and can recover back down when health returns (dashed). Crucially, Diagnosing is an absorbing state: once the systemic fault repeats, the loop halts and hands off to a human rather than burning budget on more retries.

Upcoming Directions

Three directions follow from this work:

Migrate legacy classifiers to this system, replacing ad-hoc heuristics with the full context-mesh + distillation pipeline.
Expand to other PAI workflows: The same pattern (context → LLM reasoning → distillation → deterministic enforcement) applies to lineage validation, purpose-boundary checking, and retention policy assignment.
Apply beyond privacy: Early experiments suggest these techniques generalize to agent observability and oversight, where the same tension exists between probabilistic reasoning and auditable enforcement.

AI-Native Products Raise the Bar for PAI

AI-native products raise the bar for privacy-aware infrastructure. They create new data modalities, faster iteration cycles, and more ambiguous signals. At the same time, privacy enforcement still needs decisions that are consistent, explainable, and reproducible.

Asset classification shows how to bridge that gap. Start with a clear contract. Build rich context. Use LLMs for novelty and ambiguity. Keep reviewed labels separate from model recommendations. Evaluate with metrics that expose rare-class failures. Distill stable behavior into deterministic, versioned rules.

That pattern lets the system learn from ambiguity without making ambiguity the foundation of enforcement.

The pattern also generalizes beyond our own use. A separate enforcement team compared this pattern against three alternatives head-to-head and chose it for their classification layer, independently of our work. In their evaluation, deterministic-first classification with LLM fallback produced more consistent, debuggable, and auditable decisions than end-to-end LLM approaches. Two teams independently arriving at the same trade-off (reasoning with LLMs, enforcing with rules) suggests a robust pattern.

The broader lesson is that privacy-aware infrastructure is not a tax on engineering. It is a driving force for better architecture: clearer contracts, richer context, stronger evaluation, safer publication, and systems that know when to ask for human judgment.

Acknowledgements

The authors would like to acknowledge the contributions of many members of the Privacy-Aware Infrastructure team who have played a crucial role in the work described here. In particular, we extend special thanks to (in alphabetical order) Fanghao Song, Kartikey Sachdeva, and Loka Potnuru for their foundational contributions to classifier analysis, runtime feature migration, scanner hardening, false-positive reduction, and age-flow precision improvements — as well as the broader PAI team for context enrichment and evaluation.

We are also grateful to Inchara Shivalingaiah, Juemin Wei, Nithya Arumugam, Zhe Wang, Dave Kurtzberg, and team for independently validating the classification pattern within their autonomous remediation pipeline, and to Deborah Davis for editorial guidance throughout.

Shout out to Alex Basiuk whose ideas and good steer made this go from a whiteboard sketch to a working prototype in days.

Special thanks to Jonathan Bergeron for insightful feedback and suggestions.

The post Privacy-Aware Infrastructure in the AI-Native Era: An Asset Classification Case Study appeared first on Engineering at Meta.