FAANGineering - BlogFlock

Advancing the frontier of video understanding with Gemini 2.5 - Google Developers Blog

2025-05-09T13:20:55.000Z

Gemini 2.5 marks a major leap in video understanding, achieving state-of-the-art performance on key video understanding benchmarks and being able to seamlessly use audio-visual information with code and other data formats.

Gemini 2.5 Models now support implicit caching - Google Developers Blog

2025-05-08T17:50:55.000Z

The rollout of implicit caching in the Gemini API expands on the existing explicit caching API, providing an "always on" caching system which offers automatic cost savings to developers using Gemini 2.5 models and continued availability of the explicit caching API for guaranteed savings.

Accelerating GPU indexes in Faiss with NVIDIA cuVS - Engineering at Meta

2025-05-08T17:00:22.000Z

Meta and NVIDIA collaborated to accelerate vector search on GPUs by integrating NVIDIA cuVS into Faiss v1.10, Meta’s open source library for similarity search.
This new implementation of cuVS will be more performant than classic GPU-accelerated search in some areas.
For inverted file (IVF) indexing, NVIDIA cuVS outperforms classical GPU-accelerated IVF build times by up to 4.7x; and search latency is reduced by as much as 8.1x.
For graph indexing, CUDA ANN Graph (CAGRA) outperforms CPU Hierarchical Navigable Small World graphs (HNSW) build times by up to 12.3x; and search latency is reduced by as much as 4.7x.

The Faiss library

The Faiss library is an open source library, developed by Meta FAIR, for efficient vector search and clustering of dense vectors. Faiss pioneered vector search on GPUs, as well as the ability to seamlessly switch between GPUs and CPUs. It has made a lasting impact in both research and industry, being used as an integrated library in several databases (e.g., Milvus and OpenSearch), machine learning libraries, data processing libraries, and AI workflows. Faiss is also used heavily by researchers and data scientists as a standalone library, often paired with PyTorch.

Collaboration with NVIDIA

Three years ago, Meta and NVIDIA worked together to enhance the capabilities of vector search technology and to accelerate vector search on GPUs. Previously, in 2016, Meta had incorporated high performing vector search algorithms made for NVIDIA GPUs: GpuIndexFlat; GpuIndexIVFFlat; GpuIndexIVFPQ. After the partnership, NVIDIA rapidly contributed GpuIndexCagra, a state-of-the art graph-based index designed specifically for GPUs. In its latest release, Faiss 1.10.0 officially includes these algorithms from the NVIDIA cuVS library.

Faiss 1.10.0 also includes a new conda package that unlocks the ability to choose between the classic Faiss GPU implementations and the newer NVIDIA cuVS algorithms, making it easy for users to switch between GPU and CPU.

Benchmarking

The following benchmarks were conducted using the cuVS-bench tool.

We measured:

A tall, slender image dataset: A subset of 100 million vectors from the Deep1B dataset by 96 dimensions.
A short, wide dataset of text embeddings: 5 million vector embeddings, curated using the OpenAI text-embedding-ada-002 model.

Tests for index build times and search latency were conducted on an NVIDIA H100 GPU and compared to an Intel Xeon Platinum 8480CL system. Results are reported in the tables below at 95% recall along the pareto frontiers for k=10 nearest neighbors.

Build time (95% recall@10)

Index		Embeddings 100M x 96 (seconds)		Embeddings 5M x 1536 (seconds)
Faiss Classic	Faiss cuVS	Faiss Classic	Faiss cuVS	Faiss Classic	Faiss cuVS
IVF Flat	IVF Flat	101.4	37.9 (2.7x)	24.4	15.2 (1.6x)
IVF PQ	IVF PQ	168.2	72.7 (2.3x)	42.0	9.0 (4.7x)
HNSW (CPU)	CAGRA	3322.1	518.5 (6.4x)	1106.1	89.7 (12.3x)

Table 1: Index build times for Faiss-classic and Faiss-cuVS in seconds (with NVIDIA cuVS speedups in parentheses).

Search latency (95% recall@10)

Index		Embeddings 100M x 96 (milliseconds)		Embeddings 5M x 1536 (milliseconds)
Faiss Classic	Faiss cuVS	Faiss Classic	Faiss cuVS	Faiss Classic	Faiss cuVS
IVF Flat	IVF Flat	0.75	0.39 (1.9x)	1.98	1.14 (1.7x)
IVF PQ	IVF PQ	0.49	0.17 (2.9x)	1.78	0.22 (8.1x)
HNSW (CPU)	CAGRA	0.56	0.23 (2.4x)	0.71	0.15 (4.7x)

Table 2: Online (i.e., one at a time) search query latency for Faiss-classic and Faiss-cuVS in milliseconds (with NVIDIA cuVS speedups in parentheses).

Looking forward

The emergence of state-of-the-art NVIDIA GPUs has revolutionized the field of vector search, enabling high recall and lightning-fast search speeds. The integration of Faiss and cuVS will continue to incorporate state-of-the-art algorithms, and we look forward to unlocking new innovations in this partnership between Meta and NVIDIA.

Read here for more details about NVIDIA cuVS.

The post Accelerating GPU indexes in Faiss with NVIDIA cuVS appeared first on Engineering at Meta.

Measuring Dialogue Intelligibility for Netflix Content - Netflix TechBlog - Medium

2025-05-08T00:40:17.000Z

Enhancing Member Experience Through Strategic Collaboration

Ozzie Sutherland, Iroro Orife, Chih-Wei Wu, Bhanu Srikanth

At Netflix, delivering the best possible experience for our members is at the heart of everything we do, and we know we can’t do it alone. That’s why we work closely with a diverse ecosystem of technology partners, combining their deep expertise with our creative and operational insights. Together, we explore new ideas, develop practical tools, and push technical boundaries in service of storytelling. This collaboration not only empowers the talented creatives working on our shows with better tools to bring their vision to life, but also helps us innovate in service of our members. By building these partnerships on trust, transparency, and shared purpose, we’re able to move faster and more meaningfully, always with the goal of making our stories more immersive, accessible, and enjoyable for audiences everywhere. One area where this collaboration is making a meaningful impact is in improving dialogue intelligibility, from set to screen. We call this the Dialogue Integrity Pipeline.

Dialogue Integrity Pipeline

We’ve all been there, settling in for a night of entertainment, only to find ourselves straining to catch what was just said on screen. You’re wrapped up in the story, totally invested, when suddenly a key line of dialogue vanishes into thin air. “Wait, what did they say? I can’t understand the dialogue! What just happened?”

You may pick up the remote and rewind, turn up the volume, or try to stay with it and hope this doesn’t happen again. Creating sophisticated, modern series and films requires an incredible artistic & technical effort. At Netflix, we strive to ensure those great stories are easy for the audience to enjoy. Dialogue intelligibility can break down at multiple points in what we call the Dialogue Integrity Pipeline, the journey from on-set capture to final playback at home. Many facets of the process can contribute to dialogue that’s difficult to understand:

Naturalistic acting styles, diverse speech patterns, and accents
Noisy locations, microphone placement problems on set
Cinematic (high dynamic range) mixing styles, excessive dialogue processing, substandard equipment
Audio compromises through the distribution pipeline
TVs with inadequate speakers, noisy home environments

Addressing these issues is critical to maintaining the standard of excellence our content deserves.

Measurement at Scale

Netflix utilizes industry-standard loudness meters to measure content and its adherence to our core loudness specifications. This tool also provides feedback on audio dynamic range (loud to soft) which impacts dialogue intelligibility. The Audio Algorithms team at Netflix wanted to take these measurements further and develop a holistic understanding of dialogue intelligibility throughout the runtime of a given title.

The team developed a Speech Intelligibility measurement system based on the Short-time Objective Intelligibility (STOI) metric [Taal et al. (IEEE Transactions on Audio, Speech, and Language Processing)]. Firstly, a speech activity detector analyses the dialogue stem to render speech utterances, which are then compared to non-speech sounds in the mix, typically Music and Effects. Then the system calculates the Signal-to-Noise ratio, in each speech frequency band, the results of which are summarized succinctly, per-utterance on the range [0, 1.0], to quantify the degree to which competing Music and Effects can distract the listener.

This chart shows how eSTOI (extended Short-Time Objective Intelligibility) method measures dialogue (fg [foreground] stem in the graphic) against non-speech (bg [background] stem in the graphic) to judge intelligibility based on competing non-speech sound.

Optimizing Dialogue Prior to Delivery

Understanding dialogue intelligibility across Netflix titles is invaluable, but our mission goes beyond analysis — we strive to empower creators with the tools to craft mixes that resonate seamlessly with audiences at home.

Seeing the lack of dedicated Dialogue Intelligibility Meter plugins for Digital Audio Workstations, we teamed up with industry leaders, Fraunhofer Institute for Digital Media Technology IDMT (Fraunhofer IDMT) and Nugen Audio to pioneer a solution that enhances creative control and ensures crystal-clear dialogue from mix to final delivery.

We collaborated with Fraunhofer IDMT to adapt their machine-learning-based speech intelligibility solution for cross-platform plugin standards and brought in Nugen Audio to develop DAW-compatible plugins.

Fraunhofer IDMT

The Fraunhofer Department of Hearing, Speech, and Audio Technology HSA has done significant research and development on media processing tools that measure speech intelligibility. In 2020, the machine learning-based method was integrated into Steinberg’s Nuendo Digital Audio Workstation. We approached the Fraunhofer engineering team with a collaboration proposal to make their technology accessible to other audio workstations through the cross-platform VST (Virtual Studio Technology) and AAX (Avid Audio Extension) plugin standards. The scientists were keen on the project and provided their dialogue intelligibility library.

The Fraunhofer IDMT Dialogue Intelligibility Meter integrated into the Steinberg Nuendo Digital Audio Workstation.

Nugen Audio

Nugen Audio created the VisLM plugin to provide sound teams with an efficient and accurate way to measure mixes for conformance to traditional broadcast & streaming specifications — Full Mix Loudness, Dialogue Loudness, and True Peak. Since then, VisLM has become a widely used tool throughout the global post-production industry. Nugen Audio partnered with Fraunhofer, integrating the Fraunhofer IDMT Dialogue Intelligibility libraries into a new industry-first tool — Nugen DialogCheck. This tool gives re-recording mixers real-time insights, helping them adjust dialogue clarity at the most crucial points in the mixing process, ensuring every word is clear and understood.

Clearer Dialogue Through Collaboration

Crafting crystal-clear dialogue isn’t just a technical challenge — it’s an art that requires continuous innovation and strong industry collaboration. To empower creators, Netflix and its partners are embedding advanced intelligibility measurement tools directly into DAWs, giving sound teams the ability to:

Detect and resolve dialogue clarity issues early in the mix.
Fine-tune speech intelligibility without compromising artistic intent.
Deliver immersive, accessible storytelling to every viewer, in any listening environment.

At Netflix, we’re committed to pushing the boundaries of audio excellence. From pioneering the eSTOI (extended short-term objective intelligibility) method to collaborating with Fraunhofer and Nugen Audio on cutting-edge tools like the DialogCheck Plugin, we’re setting a new standard for dialogue clarity — ensuring every word is heard exactly as creators intended. But innovation doesn’t happen in isolation. By working together with our partners, we can continue to push the limits of what’s possible, fueling creativity and driving the future of storytelling.

Finally, we’d like to extend a heartfelt thanks to Scott Kramer for his contributions to this initiative.

Measuring Dialogue Intelligibility for Netflix Content was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Create and edit images with Gemini 2.0 in preview - Google Developers Blog

2025-05-07T16:55:56.000Z

Gemini 2.0 Flash's image generation capabilities, now available in preview in Google AI Studio and Vertex AI, feature higher rate limits, enhanced visual quality, more precise text rendering, and more, allowing developers to create applications for product recontextualization, collaborative image editing, and dynamic SKU generation.

Gemini 2.5 Pro Preview: even better coding performance - Google Developers Blog

2025-05-06T16:00:55.000Z

An updated I/O edition preview of Gemini 2.5 Pro is being released for developers, featuring best-in-class front-end and UI development performance, ranking #1 on the WebDev Arena leaderboard, and showcasing applications like video to code and easier feature development through starter apps.

Dos and don’ts when sunsetting open source projects - The GitHub Blog

2025-05-06T16:00:00.000Z

Maintaining an open source project can be a big responsibility. But it’s not one you’re obligated to bear forever. Maybe usage has declined thanks to a better solution. Maybe technology has evolved to the point that it’s easier to start over with a new project than adapt an old project to a new ecosystem. Sometimes it’s time to move on, even if that means deprecating a project.

Brett Terpstra, a front-end developer, maintains more than 100 GitHub repositories and has had to retire more than a few. “Projects that rely on APIs and other outside applications often require more work than is worthwhile once things start to break,” he explained in a Q&A. “Historically, those are the projects that get retired the fastest.”

Whatever your reasons, you want to sunset the project gracefully to protect your reputation and do right by your users. Here are some insights from maintainers who have navigated the process about what you should and shouldn’t do when it’s time to deprecate a project.

Don’t: Keep maintaining something for too long

The one thing Olga Botvinnik, a computational biologist, would tell her younger self is that she should have sunsetted her Python data visualization package prettyplotlib sooner. She didn’t want to abandon the project, but she had started it as part of her PhD work, felt like updating it to support Python 3 would be daunting, and was interested in moving on to other projects. Besides, another Python visualization library called Seaborn was becoming increasingly popular.

“Even if I’m immediately done working on a project, I leave the 30-day window open to take care of issues and help users transition.” – Brett Terpestra, front-end developer

Botvinnik thought Seaborn was better in some ways, and more polished. So she made the decision to deprecate prettyplotlib and spend her time contributing to Seaborn instead. “One of my mentors told me that knowing when to end a project is just as good as finishing it,” she says. “That made me feel a lot better about letting it go.”

Do: Leave the door open for someone else

That said, you shouldn’t deprecate a project without considering other options, like handing it to another maintainer. Terpstra has deprecated many projects, but he always looks for someone else to take them over first. “There are different degrees of sunsetting,” he says. In some cases, a project is so simple that it doesn’t need much maintenance. In that case, you can just make a note that you don’t often update the project while leaving the door open for new contributions.

Of course it’s not always appropriate to hand off a project to another maintainer. Ben Johnson, maintainer of the SQLite recovery tool Litestream, opted to retire BoltDB and point people towards a fork called BBolt rather than have someone take over the original. “My name and reputation were pretty closely tied to the project at the time,” says Johnson. “I was the BoltDB guy. I didn’t want to put my reputation in the hands of someone else.”

Don’t: Pull the plug without notice

Terpstra gives at least a month’s notice before retiring a project. “Even if I’m immediately done working on a project, I leave the 30-day window open to take care of issues and help users transition,” he says.

“One of my mentors told me that knowing when to end a project is just as good as finishing it. That made me feel a lot better about letting it go.” – Olga Botvinnik, computational biologist

Once you’ve made the decision to deprecate a project, you need to let users know and, if possible, suggest alternatives. “I spread word through a blog post and a tweet announcing that I wasn’t going to actively fix bugs anymore and pointed people to Seaborn instead,” Botvinnik says.

Do: Keep the code online

Instead of deleting your project, it’s almost always best to archive it instead. Archiving a project makes it read-only and communicates to users that it’s no longer maintained. Everything from issues and pull requests to milestones and permissions become read-only. But you can always unarchive a project if you later decide to work on it again.

Deleting your project could have unintended consequences. “Anyone thinking about taking their software offline should consider whether they might be creating reproducibility problems for people in science and academia,” Botvinnik points out.

Keeping it online means that even if you couldn’t find someone to take it over before you deprecated the project, someone else could come along later and fork it—or at least find something useful to reuse.

That said, if you believe your code is actively harmful, it might be best to take it offline. For example, software with dangerous security vulnerabilities that put users at risk.

Take this with you

Ultimately, open source projects are living entities—born from passion and sustained by community. Knowing when and how to let go is not just good stewardship, it’s an essential part of the open source lifecycle.

Get started contributing to open source now.

The post Dos and don’ts when sunsetting open source projects appeared first on The GitHub Blog.

Welcome to Maintainer Month: Events, exclusive discounts, and a new security challenge - The GitHub Blog

2025-05-05T17:30:54.000Z

Open source software (OSS) is everywhere—it’s the lifeblood of the modern software ecosystem. Ninety percent of companies use open source¹, 97% of codebases contain open source², 70-90% of the code within commercial tools comes from open source³, and the value of OSS globally is estimated to be $8.8 trillion⁴. At GitHub, we love open source—and we’re so honored to host so much open source code that we famously preserved it in the Arctic.

But in the same way that your office microwave doesn’t just magically get clean and your favorite park doesn’t have self-mowing grass, open source software doesn’t just happen.

We’re surrounded by human-maintained infrastructure and resources that, in our busy lives, can be easy to take for granted. This is why we started Maintainer Month—a time to thank the open source software maintainers that keep projects healthy. This May marks the fifth annual Maintainer Month, and there are lots of treats in store: new badges, special discounts, events with experts, and more. In addition to the fact that the device you’re reading this on functions–thanks, open source maintainers!

Maintainer Month events and livestreams

There are over 25 events and livestreams scheduled during Maintainer Month, so head on over to the schedule to see them all or add your own!

Everyone is welcome at these events—whether or not you’re ready to call yourself a software maintainer. Here are a couple of our favorites, since they tackle thorny issues:

What maintainers need to know about open source licensing, SBOMs and security: May 6, 2025
Join our colleague Jeff Luszcz from the GitHub Open Source Programs Office as he reviews what every maintainer should know about these topics in the ever-evolving landscape of 2025. We get so many questions about this, and Jeff is the expert!
The CRA and Open Source: What Maintainers Really Need to Know: May 27, 2025
Feeling stressed about the European Union’s new Cyber Resilience Act (CRA) regulations? We can help! Come to this stream with the Eclipse Foundation’s Cyber Resilience Working Group, where they’ll talk about resources and practical information for maintainers navigating these changes.

🎁 Meet the 2025 Partner Pack

This year, we’re launching the new Maintainer Month Partner Pack—a bundle of perks, tools, and resources from organizations that truly believe in open source. Think of it as a care package for the folks behind our digital infrastructure.

Here’s just a taste of what’s inside (and it’s available to all maintainers):

Arachne Digital: Free tailored threat report with steps to defend your project
Boot.dev: One month of free premium access to backend dev courses
CNCF: Discounts on select cloud native training (Kubernetes included!)
DevCycle: A full year of the Developer plan, free for maintainers
JSConf North America: Special discounted tickets for Maintainer Month
Linux Foundation Education: 25% off the full course catalog
Mockoon: Free Mockoon Cloud account to build, test, and mock APIs faster
Sentry: Access to their open source plan for monitoring and performance
TODO Group: 20% off the CODE certification for enterprise open source
Web Summit: Discounted tickets to Vancouver & Lisbon for OSS contributors

…and we’ll be adding more throughout May.

👉 See all current offers and partners here.

Some partners are offering extra perks for members of our private Maintainer Community—a vetted space to connect, share, and support each other. If you maintain an open source project, you can request to join our Maintainer Community.

Security: a new challenge

Security is kind of a big deal, which is why you hear about it all the time. This is why we’re excited to launch new security guidance on opensource.guide to help maintainers strengthen the trust and resilience of their open source projects. We’ve pulled together practical advice and tools you can start using right away to make your project safer for everyone who relies on it. Because building great open source software isn’t just about what your project does—it’s about how you protect the people who use it.

The new Open Source Guide on Security Best Practices for Your Project will walk you through the basic considerations for software security, including how to:

Secure your code as part of your development workflow
Avoid unwanted changes with protected branches
Set up an intake mechanism for vulnerability reporting

🔒 Security Challenge: Level up during Maintainer Month

Ready to boost your project’s defenses—and your own skills?

This May, take the Maintainer Month Security Challenge, which features three hands-on GitHub security skills while allowing you to snag a voucher for GitHub Advanced Security certification (hello, career boost!).

In just a few hours, you’ll pick up real techniques to protect your project—and show the world you’re serious about security. Let’s build a safer open source together.

Join the Security Challenge >

🔧 How to get involved throughout May and beyond

Explore the Partner Pack: maintainermonth.github.com
Join the Maintainer Community: maintainers.github.com
Want to contribute an offer? Email us at maintainermonth@github.com
Share your story: Tag #MaintainerMonth on social media to help us celebrate the humans behind the code!

Read more about what’s happening with open source.

¹ GitHub. 2022. “Octoverse 2022: The state of open source software.” https://octoverse.github.com/2022/. & OpenUK. 2021. “State of Open: The UK in 2021.” https://openuk.uk/wp-content/uploads/2021/10/openuk-state-of-open_final-version.pdf.

² Blackduck. 2025. “Six takeaways from the 2025 “Open Source Security and Risk Analysis” report.” https://www.blackduck.com/blog/open-source-trends-ossra-report.html.

³ The Linux Foundation. 2022. “A Summary of Census II: Open Source Software Application Libraries the World Depends On.” https://www.linuxfoundation.org/blog/blog/a-summary-of-census-ii-open-source-software-application-libraries-the-world-depends-on. & Intel. 2025. “The Careful Consumption of Open Source Software.” https://www.intel.com/content/www/us/en/developer/articles/guide/the-careful-consumption-of-open-source-software.htm.

⁴ Harvard Business School. 2024. “The Value of Open Source Software.” https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4693148.

The post Welcome to Maintainer Month: Events, exclusive discounts, and a new security challenge appeared first on The GitHub Blog.

Enhancing the Python ecosystem with type checking and free threading - Engineering at Meta

2025-05-05T16:00:05.000Z

Meta and Quantsight have improved key libraries in the Python Ecosystem. There is plenty more to do and we invite the community to help with our efforts.

We’ll look at two key efforts in Python’s packaging ecosystem to make packages faster and easier to use:

Unlock performance wins for developers through free-threaded Python – where we leverage Python 3.13’s support for concurrent programming (made possible by removing the Global Interpreter Lock (GIL)).
Increase developer velocity in the IDE with improved type annotations.

Enhancing typed Python in the Python scientific stack

Type hints, introduced in Python 3.5 with PEP-484, allow developers to specify variable types, enhancing code understanding without affecting runtime behavior. Type-checkers validate these annotations, helping prevent bugs and improving IDE functions like autocomplete and jump-to-definition. Despite their benefits, adoption is inconsistent across the open source ecosystem, with varied approaches to specifying and maintaining type annotations.

The landscape of open source software is fractured with respect to how type annotations are specified, maintained, and distributed to end users. Some projects have in-line annotations (types directly declared in the source code directly), others keep types in stub files, and many projects have no types at all, relying on third party repositories such as the typeshed to provide community-maintained stubs. Each approach has its own pros and cons, but application and maintenance of them has been inconsistent.

Meta and Quansight are addressing this inconsistency through:

Direct contributions: We have improved the type coverage for pandas-stubs and numpy, and are eager to expand the effort to more packages.
Community engagement: Promoting type annotation efforts to encourage community involvement, listen to feedback and create actionable ways to improve the ecosystem.
Tooling and automation: Developing tools to address common challenges adding types and keeping the types up-to-date with the source code.

Improved type annotations in pandas

TL;DR: Pandas is the second most downloaded package from the Python scientific stack. We improved pandas-stubs package type annotation coverage from 36% to over 50%.

Background

The pandas community maintains its own stubs in a separate repository, which must be installed to obtain type annotations. While these stubs are checked separately from the source code, it allows the community to use types with their own type checking and IDE.

Improving type coverage

When we began our work in pandas-stubs, coverage was around 36%, as measured by the percentage of parameters, returns, and attributes that had a complete type annotation (the annotation is present and all generics have type arguments). After several weeks of work and about 30 PRs, type completeness is now measured at over 50%. The majority of our contributions involved adding annotations to previously-untyped parameters, adding type arguments to raw generic types, and removing deprecated/undocumented interfaces. We also improved several inaccurate annotations and updated others to match the inline annotations in the pandas source code.

Key introductions

Two key introductions significantly increased coverage:

Replacing raw Series types with UnknownSeries, a new type aliased to Series[Any]. When applied to return type annotations, this reduces the number of type checker false-positives when the function is called.
Improving types of core Dataframe operations like insert, combine, replace, transpose, and assign, as well as many timestamp and time-zone related APIs.

Tooling development

In addition to improving coverage directly, we developed tooling to catalog public interfaces missing annotations. We also augmented our tools for measuring type coverage to handle the situation where stubs are distributed independently, rather than being packaged into the core library wheel.

What is free-threaded Python ?

Free-threaded Python (FTP) is an experimental build of CPython that allows multiple threads to interact with the VM in parallel. Previously, access to the VM required holding the global interpreter lock (GIL), thereby serializing execution of concurrently running threads. With the GIL becoming optional, developers will be able to take full advantage of multi-core processors and write truly parallel code.

Benefits of free-threaded Python

The benefits of free-threaded Python are numerous:

True parallelism in a single process: With the GIL removed, developers can write Python code that takes full advantage of multi-core processors without needing to use multiple processes. CPU-bound code can execute in parallel across multiple cores.
Improved performance: By allowing multiple threads to execute Python code simultaneously, work can be effectively distributed across multiple threads inside a single process.
Simplified concurrency: Free-threading provides developers with a more ergonomic way to write parallel programs in Python. Gone are the days of needing to use multiprocessing.Pool and/or resorting to custom shared memory data structures to efficiently share data between worker processes.

Getting Python’s ecosystem ready for FTP

The ecosystem of Python packages must work well with free-threaded Python in order for it to be practically useful; application owners can’t use free-threading unless their dependencies work well with it. To that end, we have been taking a “bottoms up” approach to tackle the most difficult/popular packages in the ecosystem. We’ve added free-threading support to many of the most popular packages used for scientific computing (e.g. numpy, scipy, scikit-learn) and language bindings (e.g. Cython, nanobind, pybind, PyO3).

Just getting started

Together, we made substantial progress in improving type annotations and free-threading compatibility in Python libraries. We couldn’t have done it without the Python community and are asking others to join our efforts. Whether it’s further updates to the type annotations or preparing your code for FTP, we value your help moving the Python ecosystem forward!

To learn more about Meta Open Source, visit our open source site, subscribe to our YouTube channel, or follow us on Facebook, Threads, X and LinkedIn.

The post Enhancing the Python ecosystem with type checking and free threading appeared first on Engineering at Meta.

Copilot ask, edit, and agent modes: What they do and when to use them - The GitHub Blog

2025-05-02T16:00:00.000Z

If you have opened Copilot Chat in VS Code lately and didn’t notice the tiny dropdown hiding at the bottom, you’re not alone. It’s easy to miss, especially when you’re heads down trying to get something shipped. That little menu is where you can switch between ask, edit, and agent modes—three ways you can integrate Copilot into your workflow.

But here’s the thing: Each of these modes does something pretty different. And depending on what kind of developer you are, how much context you have, or how much control you’re comfortable giving GitHub Copilot, you might have a very different experience with each one.

I’ve been using all three in different ways, from building apps and testing ideas to playing with frameworks I haven’t touched in a while. I’ve also spent time talking to other developers about what’s working for them.

Below is what I’ve learned so far. This isn’t the manual (for that, look at the VS Code documentation). It’s more of a guide for how to think about these tools, as you navigate where they fit into your own workflow.

Ask mode: The quick gut check

Ask mode is the simplest of the three—and if you are a long-time user of GitHub Copilot, it might be the only time you’ve thought about bringing up the Chat window.

Here’s how it works: You highlight some code, type a question into Copilot Chat, and it generates an answer. It might explain what the code does, suggest how to test it, give you a code snippet that implements what you are asking about, or remind you how to handle a particular edge case.

Ask mode is fast, helpful, and focused entirely on answering your programming question without touching your code. You can stay in your editor and ask questions that Copilot can answer using all the context of your current editor environment. Think of it like a quiet little whisper in your editor saying, “Hey, here’s what I think this means.”

And it’s not just about your code. You can ask it anything related to programming—like how to use a certain library, how to structure a SQL query, and even which search algorithm is more efficient for a given dataset.

Need help styling something with Tailwind? Want a refresher on closures in JavaScript? Curious how to debounce an input in React? Copilot can help with that, too.

There’s no project commitment, no architectural decisions, and no code changes. Just answers, right when you need them. It’s the lowest-friction way to get unstuck when a question is standing in the way of you and building what’s next.

Edit mode: You’re still in charge, just moving faster

Edit mode in VS Code is where things start to get more interesting. It lets you pick any number of files in your project you want to change and describe the update in natural language. Then, Copilot will immediately apply inline, review-ready code edits across those files.

Edit mode is perfect when you know what you want to do but don’t necessarily want to write it all out yourself. You highlight a block of code, type in an instruction—perhaps something like “add error handling” or “refactor this using async/await”—and Copilot rewrites the code for you. But (and this is important) it doesn’t save anything without showing you the diff first.

That’s what makes edit mode so reliable. Copilot does the work, but you get the final say. You’re not handing over the reins. You’re speeding things up while staying fully in the loop.

You can also bring custom instructions into edit mode if you want to level it up a bit. It’s a way to teach Copilot how you and your team like to write code, including your style preferences, how verbose or concise you want it to be, what your team’s standards are, how formal or casual you want it when it explains things, and even what language you want your comments written in. Setting up those preferences is like giving Copilot a playbook ahead of time—so when you say “clean this up,” it already knows what “clean” actually means to you.

I’ve found myself reaching for edit mode when I’m deep in a brownfield app and don’t want to touch the rest of the system, or when I’m just trying to get through a handful of small but annoying improvements with surgical precision. It’s not trying to redesign your architecture. It’s not going to rename things you didn’t ask it to. It’s just here to make the thing you’re working on better and faster, with less mental overhead.

Honestly, once you get used to it, it’s hard to go back.

Agent mode: A lot of power, when you’re ready for it

Now let’s talk about Agent mode. Agent mode lets you hand it a high-level prompt and then watch as Copilot autonomously plans the steps, selects the right files, runs tools or terminal commands, and iterates on code edits until the task is complete.

It is by far the most powerful mode in Copilot Chat—and also the newest, which means for a lot of people, it’s still the least familiar. Agent mode can reason across your entire project, take multi-step actions, and hold onto a significant amount of context across a session. You can ask it to build features, fix bugs, create files, clean up routing logic, or even scaffold an entire section of an app based on a single prompt.

At first glance, agent mode can look like an expanded version of edit mode—and in some respects, it is. But there’s a crucial distinction: Instead of only rewriting the lines you specify, agent mode analyzes related code, identifies additional changes that may be required, and applies them across the project to keep everything consistent.

Another key distinction? Agent mode applies edits automatically rather than waiting for explicit approval, while still surfacing any potentially risky commands for review before they run.

The workflow is closer to a continuous-edit “driver” model: The developer defines the goal, and Copilot executes updates without stopping for permission at every step.

For some developers, that feels natural and empowering. For others, it can feel like giving up a little more control than you are used to.

One thing that makes agent mode even better in real-world projects is custom instructions, which I mentioned above. This is where you can really start shaping how Copilot behaves across a session.

For example, here are the custom instructions we used with agent mode in one of our demo projects:

This is a Next.js-based travel application with TypeScript that helps users search for trips, manage bookings, view travel guides, and track points. The application uses React components, server components, and client components as part of the Next.js App Router architecture. Please follow these guidelines when contributing:

## Code Standards

### Required Before Each Commit
- Run `npm run lint` to ensure code follows project standards
- Make sure all components follow Next.js App Router patterns
- Client components should be marked with 'use client' when they use browser APIs or React hooks
- When adding new functionality, make sure you update the README
- Make sure that the repository structure documentation is correct and accurate in the Copilot Instructions file
- Ensure all tests pass by running `npm run test` in the terminal

### TypeScript and React Patterns
- Use TypeScript interfaces/types for all props and data structures
- Follow React best practices (hooks, functional components)
- Use proper state management techniques
- Components should be modular and follow single-responsibility principle

### Styling
- You must prioritize using Tailwind CSS classes as much as possible. If needed, you may define custom Tailwind Classes / Styles. Creating custom CSS should be the last approach.

## Development Flow
- Install dependencies: `npm install`
- Development server: `npm run dev`
- Build: `npm run build`
- Test: `npm run test`
- Lint: `npm run lint`

## Repository Structure
- `app/`: Next.js App Router pages and layouts organized by route
- `components/`: Reusable React components
  - `components/ui/`: UI components (buttons, inputs, etc.)
  - `components/__tests__/`: Component tests
- `lib/`: Core logic and services
  - `lib/data/`: Data models and mock data
  - `lib/types/`: TypeScript type definitions
- `public/`: Static assets
- `tests/`: Test files and test utilities
- `README.md`: Project documentation

## Key Guidelines
1. Make sure to evaluate the components you're creating, and whether they need 'use client'
2. Images should contain meaningful alt text unless they are purely for decoration. If they are for decoration only, a null (empty) alt text should be provided (alt="") so that the images are ignored by the screen reader.
3. Follow Next.js best practices for data fetching, routing, and rendering
4. Use proper error handling and loading states
5. Optimize components and pages for performance

If you’ve ever noticed that agent mode sometimes forgets how it structured your backend when it later builds the frontend, you’re not imagining things. The context window is big (and getting bigger all the time), but it’s still finite. Instead of stuffing every little reminder into your prompts, you can use custom instructions to set some ground rules ahead of time: things like how you want APIs called, naming patterns you want followed, or even stylistic preferences across your codebase. It gives agent mode a stronger foundation to work from, which means you get more consistency without needing to micromanage every step.

I won’t go too deep into custom instructions here (the VS Code documentation does a great job), but if you are working across multiple sessions or building something bigger than a one-off script, they are absolutely worth looking into. It has made a noticeable difference for me, especially on side projects where I want to move fast but keep some structure in place.

I have had some really strong results using agent mode this way. I’ve started projects by dropping a README into a new repo, setting a clear vision for what I want, and letting Copilot take the first pass. It builds out components, layouts, routes, and even seeds the content. It isn’t perfect straight out of the gate (and it shouldn’t be), but it gets me so much closer to a usable starting point than building from scratch. The more detailed and thoughtful my initial prompts and setup, the better agent mode performs.

Of course, it’s still important to stay engaged. Occasionally, agent mode might suggest running a command you don’t expect. Or it might touch a file you thought you had agreed to leave alone. And sometimes, it just takes a little longer to reason through things, especially on bigger projects.

In live demo situations where time is tight, that unpredictability can definitely make things more… exciting. But when I’m building day-to-day, especially when experimenting or kicking off a new project, agent mode fits naturally into my creative process.

Working with agent mode is a lot like pairing with that brilliant friend who moves fast and sometimes thinks three steps ahead. If you’re aligned, amazing things can happen. If you’re not, you might have to nudge them back onto the path you want.

The key is good communication prompt files, thoughtful initial instructions, and clear nudges along the way. This keeps the collaboration productive and fun.

So what does this mean for senior developers?

This question has come up a lot in conversations with my team. Agent mode definitely feels like the future when you use it, but just because it is powerful does not mean it is always the right tool for the job. Sometimes you’ll be working on part of a codebase that needs to be handled with a little more precision. If you are tweaking a couple of files or making a targeted change in a sensitive system, ask or edit mode might be the better fit.

A lot of the time, the most experienced developers are the ones who know exactly which parts of a system should be touched carefully, or maybe not touched at all, to avoid bigger problems later. That isn’t about being cautious for the sake of it. It’s about understanding the complexity that lives under the surface.

And here’s something important: Agent mode isn’t just for people who are new to building software. In a lot of ways, it actually works best when the person using it knows how to give clear, strong instructions—and the nuances of code and algorithmic structures. If you know how your system is structured, where the fragile edges are, and when to review changes vs. when to trust the flow, you’re in a great spot to get real value out of agent mode. That is senior dev territory.

Of course, strong opinions come with the territory too. Agent mode is doing its best, but unless you are explicit about the rules you want it to follow, it isn’t going to automatically pick up your conventions. That is where custom instructions come into the picture. When senior engineers write down the things they normally just know: the naming patterns, the design principles, the places to be careful, and commit those into version control alongside the project, it gives the entire team a better experience. It doesn’t just make agent mode faster—it helps it give you better suggestions, too.

There will still be plenty of moments where edit mode feels like the better choice. When you just need a second set of eyes without handing over the whole keyboard, edit mode keeps you moving without ever getting too far ahead. And that’s fine.

Honestly, I think that as a community we could do a better job showing how agent mode complements expertise instead of only showing it in greenfield examples. Demos that start from zero are easy to understand. But in the real world, most developers are iterating on existing systems—not spinning up a new app every week. Showing how Copilot fits into that kind of work—the messy, important middle—is where things really get exciting.

Take this with you

Ask, edit, and agent modes aren’t three versions of the same tool. They’re three completely different and unique experiences within GitHub Copilot. Ask mode is the quick way to get an answer to your question. Edit mode is the assistant telling you what it recommends across your files. And agent mode is the assistant that just goes ahead and does what it thinks you are asking for—which is great as long as it’s following your spoken (and unspoken) instructions.

If you’ve only tried one mode so far, now’s a good time to play around. Try using agent mode with a fresh repository. Or try using edit mode on a complex refactoring job you’ve been putting off. And maybe try using ask mode when you’re trying to remember what a slice reducer does because it’s Monday and your brain’s still booting up.

All of these tools are designed to complement your judgment. They’re here to help you do your job better, whatever that looks like for you.

And no matter what, always read the diff.

Learn more about AI models.

The post Copilot ask, edit, and agent modes: What they do and when to use them appeared first on The GitHub Blog.

Taking the plunge: Why Meta is laying the world’s longest subsea cable - Engineering at Meta

2025-05-01T18:48:02.000Z

Meta develops infrastructure all across the globe to transport information and content for the billions of people using our services around the world. At the core of this infrastructure are aggregation points – like data centers – and the digital cables that connect them. Subsea cables – the unseen digital highways of the internet – are critical for Meta to serve people wherever they are in the world. In fact, more than 95% of the world’s intercontinental traffic goes through subsea cables.

Meta’s engineering team prioritizes both innovation and quality when designing and deploying these cables. In the latest Meta Tech Podcast, Andy Palmer-Felgate and Pascal Pecci, both subsea cable systems engineers, join Pascal Hartig on the Meta Tech podcast to discuss the latest in subsea engineering technology. This episode dives deeper into the engineering nuances of large-scale subsea cable projects like the recently announced Project Waterworth.

Learn more about Meta’s work on these engineering feats. Download or listen to the episode below:

The Meta Tech Podcast is a podcast, brought to you by Meta, where we highlight the work Meta’s engineers are doing at every level – from low-level frameworks to end-user features.

Send us feedback on Instagram, Threads, or X.

And if you’re interested in learning more about career opportunities at Meta, visit the Meta Careers page.

The post Taking the plunge: Why Meta is laying the world’s longest subsea cable appeared first on Engineering at Meta.

The AI-Powered DevOps revolution: Redefining developer collaboration - The GitHub Blog

2025-05-01T17:12:05.000Z

When it comes to mastering DevOps, it’s often not the technical skills that trip us up, but rather the more critical aspects of collaboration and communication. Communication challenges, vague requirements, and missing documentation can leave us guessing on stakeholder intent. Plus, siloed workflows can cause teams to face inconsistencies in their processes, tooling, and time to delivery, as there are so many moving parts. This all works against DevOps best practices.

In this blog, we will look at ways that we can enhance team productivity, reduce cognitive load, and implement a more seamless collaboration across teams and tools to improve code quality and create faster delivery cycles. Adding a bit of AI into our workflows will get us on the right path.

💡 Tip: When using Copilot (or any generative AI), it’s always a good idea to review any suggestions before accepting them. There’s a reason we refer to GitHub Copilot as an assistant. It is meant to augment your abilities, not serve as a replacement for your skills.

Filling in communication gaps and documentation

When I get started on a new project, the first thing I do is seek out the README or documentation, if there is any. While we should be writing documentation for our code as we go, the reality is that we often do not. This gap, between intention and practice, creates consistent challenges for our teams as we try to onboard new members or revisit older projects.

Nowadays, to get the documentation that I need, whether it’s working on a legacy code base or joining a new community project, I start by opening either GitHub Copilot Chat in VSCode or in the github.com immersive chat experience to ask Copilot to explain the codebase to me. And when the existing README is lacking, I can then ask Copilot to help me to write a better one.

For example, here’s a typical placeholder README:

Beyond a brief description, there’s not much information. I opened up VSCode and GitHub Copilot Chat, attaching the existing README file to give Copilot context, and asked it to simply make my README more detailed.

After asking Copilot to help create a better README, here’s how it looked:

GitHub Copilot provided a project overview, installation and configuration steps, and some usage instructions. This gives me, and anyone else who visits this project, a better understanding of what the project is doing and how to get started.

You can use GitHub Copilot not only to generate a README once you’ve written your code, but also function and class descriptions. Copilot can improve readability of code by generating descriptions, adding context around what your code is doing, and writing documentation that can reduce the learning curve and onboarding time.

For example, check out this bit of code where Copilot helped write inline comments:

Oftentimes, the various technical teams (operations, security, and development) operate independently, not fully aware of other teams’ standards, dependencies, or changes. Using Copilot to create better documentation and code comments can help us to break down these silos.

AI-powered code reviews and pull requests

So, when we do make a code change that impacts other teams, how can we better review those changes to ensure a more seamless and secure delivery? Let’s take a look.

You know when you submit your pull request and you’re in a rush because of task overload? In the example above, we used docstrings to explain our functions and that is extremely helpful when you’re working with complex functions. Copilot can help write your docstrings to provide clear parameter explanations, detailed method descriptions, and context about usage. This helps to reduce the ambiguity for your code reviewers and provides context to the changes that were made.

When you get ready to submit your commit, instead of having to think of a funny and witty commit message yourself, select the AI-enhanced commit option and let Copilot generate a clear, concise summary of your changes for you:

How often when we are reviewing a pull request there is little to no description? We can use Copilot to help with that, too! After I have committed my code, I can use the “summary” option to generate a summary of my pull request:

Copilot reviews the commit and the files changed, and then outputs a thorough summary of the changes with links to the changed files.

Oftentimes, we submit a pull request and expect the reviewer to figure out what changes were made, as if they had a crystal ball. In this example, I’ve used Copilot to generate a short summary of the changes in my commit, as well as a longer, more detailed summary of the pull request. This saved me time, but more importantly, it has provided a much higher quality output than I would have been able to do on my own.

Copilot’s ability to summarize and document can be hugely beneficial in creating consistent terminology and communication across teams, eliminating collaboration breakdowns and enhancing code reviews.

Along with using Copilot to summarize my pull request before I send up our commit, I can also set Copilot as one of my reviewers by clicking “Reviewers” in the top right hand corner of my screen.

Once I do that, Copilot reviews my pull request before my other team members, allowing me to find typos and other errors, and iterate my code before asking others to review it. This enables me to iterate faster, getting direct feedback more quickly on my changes and lessening the time my teammates need to take to review my code. It also allows me to learn how to improve my code quality, too.

Resolving merge conflicts with Copilot

So far, I’ve used Copilot to better define the project README, add inline comments and documentation, and generate more relevant, thorough, and precise commit messages and precise pull request summaries. But most of these changes have been me acting alone—what about when I’m collaborating with my team and I run into a merge conflict? GitHub Copilot can help there, too!

Sometimes, collaboration leads to messy changes and you arrive at a difficult point, trying to decide which version to ultimately commit based on code patterns, environment variables, and comments. There are a couple different ways that Copilot can help you remediate a merge conflict.

When you’re working in the editor, specifically VSCode, it will display the conflicting section of code. You can open a Copilot Chat window or ask inline, “How should I resolve this merge conflict?” Copilot will analyze both versions of code, suggest a resolution, and explain its reasoning. If the suggestion fits, you can accept the solution or ask for alternatives.

If you’re using Copilot on github.com (and licensed for GitHub Enterprise) you can also ask Copilot why a workflow has failed. Just navigate to the pull request, select the details of the failing checks, and click on the GitHub Copilot icon next to the search bar. Then, you can ask Copilot directly, “Why has this pull request failed?”

This is a great way to find a quick resolution to a common problem, especially with long running features branches and automation tasks.

Transformed collaboration and delivery

One of the fundamental challenges in building a high performing DevOps team has always been the burden of repetitive tasks that consume valuable developer time daily. By integrating Copilot into my workflow, I’ve watched it suggest entire functions and fill critical logic gaps, dramatically reducing the boilerplate code that once dominated my coding sessions. This has transformed how I work, allowing me to tackle complex architectural challenges and focus on innovation rather than implementation details.

I encourage you to explore these GitHub Copilot capabilities in your own environment. The transformation in both individual productivity and team dynamics might surprise you.

Happy coding!

The post The AI-Powered DevOps revolution: Redefining developer collaboration appeared first on The GitHub Blog.

Gemma explained: What’s new in Gemma 3 - Google Developers Blog

2025-04-30T21:19:03.000Z

Gemma 3's new features include vision-language capabilities and architectural changes for improved memory efficiency and longer context handling compared to previous Gemma models.

From MCP to multi-agents: The top 10 open source AI projects on GitHub right now and why they matter - The GitHub Blog

2025-04-30T16:00:49.000Z

Every day, new public and open source repositories appear on GitHub, and navigating the sheer amount of activity can be a challenge for the best of us. Luckily, we’ve done the heavy lifting for you.

Together with our panel of GitHub experts—who have experience across open source and developer relations—we analyzed every open source project created in the last 99 days (as of March 29, 2025), and ranked them in consideration of a number of factors including stars-per-day, forks, traffic spikes, and contributor velocity.

Our GitHub panel includes:

Abigail Cabunoc Mayes, aka @abbycabs, who works on open source maintainer programs and serves as a Director on the OpenJS foundation.
Kara Sowles, aka @karasowles, who works with maintainers and the open source community.
Kevin Crosby, aka @kevincrosby, who runs GitHub’s open source funding program.
Jeff Luszcz, aka @jeffrey-luszcz, who helps manage GitHub’s open source program office (OSPO).

Below, we’ll give you a rundown of these projects—and also discuss why we believe they’re the ones developers keep coming back to. Let’s dive in.

1. Open WebUI MCP: simplifying AI tool integrations 🔌

Website Source

Python

📜 MIT license

First up is a proxy server that turns MCP tools into OpenAPI-compatible HTTP servers. Developers building AI-powered apps are using the server to easily connect MCP-based tools with anything that uses standard RESTful OpenAPI interfaces.

2. Unbody: the “Supabase of AI” 🧩

Website Source

TypeScript

📜 Apache 2.0 license

Think Supabase, but for AI: that’s Unbody in a nutshell. It’s a modular backend that lets you build AI-native software that actually understands and reasons about knowledge, instead of just shuffling data around.

The project breaks things down into four layers that you can mix and match:

Perception: Ingests, parses, enhances, and vectorizes raw data.
Memory: Stores structured knowledge in vector databases and persistent storage.
Reasoning: Generates content, calls functions, and plans actions.
Action: Exposes knowledge via APIs, SDKs, and triggers.

3. OWL: multi-agent collaboration in action 🦉

Source

Python

📜 Apache 2.0 license

When one AI agent isn’t enough, OWL enters the chat. Built on the CAMEL-AI framework—best known for popularizing multi-agent role-play and releasing a trove of synthetic “task + data” bundles—OWL lets several specialized agents cooperate through browsers, terminals, function calls, and MCP tools. It even tops the open-source leaderboard on the GAIA benchmark (58.18).

4. F/mcptools: Command-line power for MCP developers 💻

Source

📜 MIT license

CLI fans: Here’s a command-line interface for working with MCP servers that’ll make you feel right at home. Built by GitHub Star Fatih Kadir Akin (who also shipped the GitHub Copilot prompts feature), it lets you discover and call tools, access resources, and manage prompts from any MCP-compatible server.

MCP Tools supports input/output over stdin/stdout or HTTP, and spits out results in JSON or table views. It even lets you create mock servers for testing or proxy MCP requests to shell scripts.

5. Nutlope/self.so: Build your personal site with AI in seconds ⚡

Website Source

TypeScript

📜 MIT license

If putting together a personal website isn’t your idea of fun, Nutlope/self.so can lend a hand. Upload your résumé or LinkedIn profile, and the tool will pull together a straightforward site for you, using AI to handle the layout so you can skip the CSS headaches.

The tech stack includes Together.ai for language modeling, Vercel’s AI SDK, Clerk for auth, Next.js for the framework, Helicone for observability, S3 for storage, Upstash Redis for the database, and Vercel for hosting.

6. VoiceStar: precise control for text-to-speech applications 🎙️

Source

Python

📜 MIT code license, CC-BY-4.0 model license

If your project needs speech that lands within a specific time window, VoiceStar’s duration-controllable synthesis can help. It lets developers set target lengths so voice output fits time-sensitive use cases—like fixed-length prompts or narration—without extra audio editing.

The project includes both CLI and Gradio interfaces for inference, plus pre-trained models you can use right away. As voice interfaces become more important in apps, having open source models with this level of control is a helpful step forward

7. Create your digital twin with Second-Me 🤖

Website Source

Python

📜 Apache 2.0 license

Interested in experimenting with an AI stand-in? Second-Me lets you try a basic “digital twin”—an agent that aims to reflect some of your knowledge, communication style, and preferences.

The possibilities range from personal assistants that actually understand how you think to innovative ways to share your expertise with others. One example of Second-Me in action: Have your digital twin manage your LinkedIn or Airbnb account, playing the role of the professional or host.

8. SesameAILabs/csm: reimagining speech synthesis 🔊

Source

Python

📜 Apache 2.0 code license (model has restrictions on Abuse)

The Conversational Speech Model (CSM) brings a fresh approach to speech generation. It converts text and audio inputs into Residual Vector Quantization (RVQ) audio codes using a Llama-based architecture. Its dedicated audio decoder produces Mimi audio codes that result in surprisingly natural-sounding speech.

What’s interesting here is how CSM merges language model architecture with specialized audio decoding—giving you an open alternative to the proprietary text-to-speech options that dominate the market.

9. Letta: a universal standard for portable AI agents 📦

Source

Python

📜 Apache 2.0 license

Letta introduces an open file format (.af) for packaging up AI agents with their memory and behavior intact. Think of it as a portable container for agents. You can share, checkpoint, and version control them across different frameworks.

For developers juggling multiple agent frameworks, this could be a time-saver. Want to move an agent from one system to another without rebuilding it from scratch? That’s the problem Letta is solving.

Notably, the Letta project is an offshoot of the cpacker/memgpt project—it spun out the serialization layer that MemGPT originally used to snapshot its “virtual-context” agents. The team carved that code into a clean, framework-agnostic spec (agent-file) so any stack—MemGPT/Letta, LangGraph, CrewAI, you name it—can import or export a fully stateful agent with a single .af archive.

10. Blender meets Claude: bridging 3D creation and AI 🎨

Source

Python

📜 MIT license

Blender artists, this one’s for you: a third party tool that connects the popular open source 3D creation suite Blender with Claude AI through the MCP. With Blender-MCP, developers can control Blender operations with natural language—or add AI assistance to their 3D workflow.

Blender-MCP shows how the MCP can act as a universal “tool port” for LLM agents: today it’s Blender; tomorrow it could be Unity, Unreal, or any complex desktop app. For 3D artists and prototypers, that means faster scene blocking, easy style experiments, and a brand-new way to teach beginners. Just describe what you want and watch the software build it.

Interested? Installing Blender-MCP is as simple as running a bash command.

What these projects tell us about AI’s evolution in open source

These patterns not only reflect the current state of AI in open source, but also hint at the challenges and opportunities that lie ahead. Here’s what GitHub experts have to say about the rapidly evolving space:

Integration in AI via MCP is the new frontier 🔗

The prominence of MCP across multiple projects highlights the growing importance of standardized integration patterns in AI development.

“A big pattern that I saw is the pain point around AI and integration,” notes Abigail. “More standards like MCP will help with this.”

Multi-agent collaboration emerges 👥

Projects like OWL point to a future where multiple specialized AI agents work together to solve complex problems.

“You have to think about it in the construct of person to person, agent to agent, and then having multiple agents working in tandem,” explains Kevin, highlighting the complexity and potential of this approach.

Speech generation is advancing 🗣️

Speech tech certainly isn’t new—but large language models are reshaping both text-to-speech (TTS) and speech-to-text (STT) so dramatically that a fresh wave of possibilities is opening up.

The bigger story is what this means downstream with implications across media, customer support, and product UX.

The evolving landscape of open source participation 🌱

AI has drawn a fresh wave of maintainers and contributors to open source, bringing new energy and approaches. Kara Deloss, senior program manager of developer relations at GitHub, notes: “We’re seeing a new generation, or a new type of maintainer” in the AI space. Kevin adds that “if you have a big community on day one, that’s valuable,” highlighting how the ecosystem is evolving.

This blend of established and emerging development practices is creating exciting opportunities for collaboration across the community.

The importance of OSI-approved licenses 📄

Every top project in our list uses OSI-approved licenses (mostly MIT and Apache 2.0), and that’s no accident. “In general, you won’t get a lot of positive sentiment from the community if you call yourself open source but aren’t using an OSI-approved license,” Jeff points out. “These licenses matter because they provide clear guarantees around usage, modification, and redistribution rights that build trust in the community.”

Jeff also notes an emerging challenge: “As AI powered services and tools become more powerful, we are seeing a trend where some projects attach Abuse and Fraud related restrictions on model or service use. This may not make them completely open source under an OSI approved license, and the projects and the community will continue to have an intense conversation about these conditions.”

He continues: “It’s important to understand and document any restrictions in place before using a model or service,” which is a timely reminder as the open source AI community navigates these evolving licensing questions.

Explore and contribute to tomorrow’s AI tooling 🛠️

The projects we’ve highlighted here are just the tip of the iceberg. As AI keeps evolving, the open source ecosystem is where many of the most exciting standards, tools, and techniques are popping up first.

Here’s how to get involved:

Check out these projects to see how they might fit into your workflow
Join in and contribute to projects that spark your interest
Keep an eye on MCP and other emerging standards

Find more cutting-edge repos on GitHub’s trending page >

The post From MCP to multi-agents: The top 10 open source AI projects on GitHub right now and why they matter appeared first on The GitHub Blog.

Announcing the general availability of Llama 4 MaaS on Vertex AI - Google Developers Blog

2025-04-29T22:34:01.000Z

Llama 4, Meta's advanced large language model, is now generally available as a fully managed API on Vertex AI, simplifying deployment and management. The Llama 3.3 70B managed API is also generally available, offering users greater flexibility.

Introducing AutoPatchBench: A Benchmark for AI-Powered Security Fixes - Engineering at Meta

2025-04-29T17:15:17.000Z

We are introducing AutoPatchBench, a benchmark for the automated repair of vulnerabilities identified through fuzzing.
By providing a standardized benchmark, AutoPatchBench enables researchers and practitioners to objectively evaluate and compare the effectiveness of various AI program repair systems.
This initiative facilitates the development of more robust security solutions, and also encourages collaboration within the community to address the critical challenge of software vulnerability repair.
AutoPatchBench is available now on GitHub.

AI is increasingly being applied to solve security challenges, including repairing vulnerabilities identified through fuzzing. However, the lack of a standardized benchmark for objectively assessing AI-driven bug repair agents specific to fuzzing has impeded progress in academia and the broader community. Today, we are publicly releasing AutoPatchBench, a benchmark designed to evaluate AI program repair systems. AutoPatchBench sits within CyberSecEval 4, Meta’s new benchmark suite for evaluating AI capabilities to support defensive use cases. It features 136 fuzzing-identified C/C++ vulnerabilities in real-world code repos along with verified fixes sourced from the ARVO dataset.

AutoPatchBench provides a standardized evaluation framework for assessing the effectiveness of AI-assisted vulnerability repair tools. This benchmark aims to facilitate a comprehensive understanding of the capabilities and limitations of various AI-driven approaches to repairing fuzzing-found bugs. By offering a consistent set of evaluation criteria, AutoPatchBench fosters transparency and reproducibility in research, enabling both academic and industry professionals to identify best practices and areas for improvement.

Fixing fuzzing-found vulnerabilities with AI

Fuzzing is a cornerstone in automated testing, renowned for its effectiveness in uncovering security vulnerabilities. By bombarding a target program with vast amounts of pseudo-random input data, fuzz testing exposes critical security and reliability issues, such as memory corruption, invalid pointer dereference, integer overflow, and parsing errors.

However, resolving a fuzzing crash is often a labor intensive task, demanding intricate debugging and thorough code review to pinpoint and rectify the underlying cause. This process can be both time-consuming and resource-intensive. Unlike regular test failures, fuzzing bugs frequently reveal security vulnerabilities that pose severe threats to system integrity and user data. Given these stakes, automating the repair of fuzzing bugs with AI becomes not just advantageous but essential. AI’s ability to swiftly analyze patterns and propose solutions significantly reduces the time and effort required for repairs, making it an invaluable ally in safeguarding our digital environments.

Let’s explore the process of addressing bugs identified through fuzzing by examining a demonstrative example. Consider the following C function, which harbors a read/write buffer overflow vulnerability:

#include <stdio.h>
#include <string.h>
void process_input(const char *input) {
    char buffer[8];
    strcpy(buffer, input); // Potential buffer overflow
    printf("Processed: %s\n", buffer);
}

In this scenario, a fuzzing harness might supply an input that surpasses the buffer’s capacity, leading to a crash due to buffer overflow. A typical stack trace from such a crash might appear as follows:

== Fuzzer Crash Report ==
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7af1223 in strcpy () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff7af1223 in strcpy ()
#1  0x0000555555555140 in process_input (input=0x7fffffffe695 "AAAAAA...")
#2  0x0000555555555162 in main (argc=2, argv=0x7fffffffe5f8)

Here, the process_input function invokes strcpy on a string that exceeds the eight-character buffer, causing a segmentation fault. A straightforward patch involves ensuring the copy operation remains within the buffer’s limits. This can be achieved by using a bounded copy function like strncpy or implementing a length check before copying:

void process_input(const char *input) {
    char buffer[8];
    strncpy(buffer, input, sizeof(buffer) - 1);
    buffer[sizeof(buffer) - 1] = '\0';
    printf("Processed: %s\n", buffer);
}

This patch ensures that the string remains within the buffer’s limits, effectively preventing out-of-bounds writes. Its correctness can be confirmed by verifying that the fuzzing input, which previously caused the crash, no longer does so. Additional checks can be conducted to ensure the patch doesn’t introduce any unintended side effects.

As illustrated, fixing a fuzzing crash involves:

Analyzing the crash stack trace and the target code.
Pinpointing the root cause.
Patching the vulnerable code.
Verifying the fix’s accuracy.

An AI-based solution can automate these steps by utilizing an LLM’s capability to understand and generate code.

Why we developed AutoPatchBench

AutoPatchBench is informed by key advancements in the field of AI-driven program repair, particularly those focusing on fuzzing-found vulnerabilities. Among the notable contributions is Google’s tech report on AI-powered patching, which pioneered the use of LLMs for addressing fuzzing crashes, achieving a 15% fix rate with their proprietary dataset. Subsequently, Google’s study on generic program repair agents introduced the GITS-Eval benchmark, encompassing 178 bugs across various programming languages.

In the realm of AI software engineering agents, benchmarks like SWE-Bench and SWE-Bench Verified have gained widespread acceptance for evaluating generic AI SWE agents. However, these benchmarks do not specifically tackle the unique challenges posed by fuzzing-found vulnerabilities, which demand specialized approaches that utilize fuzzing-specific artifacts and address security concerns.

AutoPatchBench addresses this gap by offering a dedicated benchmark focused on a wide variety of C/C++ vulnerabilities of 11 crash types identified through fuzzing with automated verification capability. Unlike the broader focus of GITS-Eval and SWE-Bench, AutoPatchBench is specifically designed to assess the effectiveness of AI-driven tools in repairing security-critical bugs typically uncovered by fuzzing. This targeted approach enables a more precise evaluation of AI capabilities in meeting the complex requirements of fuzzing-found vulnerabilities, thereby advancing the field of AI-assisted program repair in a focused manner.

Inside AutoPatchBench

We’re making AutoPatchBench publicly available as part of CyberSecEval 4 to encourage community collaboration in tackling the challenge of automating fuzzing crash repairs. This benchmark is specifically designed for AI program repair agents focusing on C/C++ bugs identified through fuzzing. It includes real-world C/C++ vulnerabilities with verified fixes sourced from the ARVO dataset, and incorporates additional verification of AI-generated patches through fuzzing and white-box differential testing.

ARVO dataset

The ARVO dataset serves as the foundation for AutoPatchBench, offering a comprehensive collection of real-world vulnerabilities that are essential for advancing AI-driven security research. Sourced from C/C++ projects identified by Google’s OSS-Fuzz, ARVO includes over 5,000 reproducible vulnerabilities across more than 250 projects. Each entry is meticulously documented with a triggering input, a canonical developer-written patch, and the capability to rebuild the project in both its vulnerable and patched states.

However, there are notable challenges when using the ARVO dataset as a benchmark for AI patch generation:

While reproducibility is vital for a reliable benchmark, the ARVO dataset includes samples where crashes are not consistently reproducible. Some samples lack crash stack traces, making it exceedingly difficult to address the crash.
Although ARVO provides a ground-truth fix for each identified vulnerability, it lacks an automated mechanism to verify the correctness of a generated patch. Objective automated verification is essential for a benchmark focused on patch generation.

AutoPatchBench addresses these challenges by creating a curated subset and by employing a comprehensive and automated verification process.

Selection criteria

To ensure the reliability and effectiveness of AutoPatchBench, we meticulously filtered the ARVO dataset samples based on the following criteria:

Valid C/C++ vulnerability: The ground-truth fix shall edit one or more C/C++ source files that are not fuzzing harnesses.
Dual-container setup: Each vulnerability is accompanied by two containers—one that contains vulnerable code and another for the fixed code—that build without error.
Reproducibility: The crash must be consistently reproducible within the vulnerable container.
Valid stack trace: A valid stack trace must be present within the vulnerable container to facilitate accurate diagnosis and repair.
Successful compilation: The vulnerable code must compile successfully within its designated container, ensuring that the environment is correctly set up for testing.
Fixed code verification: The fixed code must also compile successfully within its respective container, confirming that the patch does not introduce new build issues.
Crash resolution: The crash must be verified as resolved within the fixed container, demonstrating the effectiveness of the patch.
Fuzzing pass: The fixed code must pass a comprehensive fuzzing test without finding new crashes, ensuring that the ground-truth patch maintains the integrity and functionality of the software.

After applying these rigorous selection criteria, we retained 136 samples for AutoPatchBench that fulfill the necessary conditions for both patch generation and verification. From this refined set, we created a down-sampled subset of 113 AutoPatchBench-Lite samples to provide a focused benchmark for testing AI patch generation tools. These subsets preserves the diversity and complexity of real-world vulnerabilities including 11 distinct crash types, offering a solid foundation for advancing AI-driven security solutions.

Patch verification

In the process of patch generation, the patch generator utilizes two automated methods to verify the viability of a generated patch before submitting it for evaluation. The first method involves attempting to build the patched program, which checks for syntactic correctness. The second method involves attempting to reproduce the crash by running the input that initially triggered it. If the crash no longer occurs, it suggests that the issue has been resolved. However, these steps alone are insufficient to guarantee the correctness of the patch, as a patch might not maintain the program’s intended functionality, rendering it incorrect despite resolving the crash.

To address this issue, AutoPatchBench adopts a comprehensive approach to automate the evaluation of generated patches. This involves subjecting the patched code to further fuzz testing using the original fuzzing harness that initially detected the crash. Additionally, white-box differential testing compares the runtime behavior of the patched program against the ground truth repaired program, confirming that the patch has effectively resolved the underlying bug without altering the program’s intended functionality. Since a patch can potentially be made in multiple places, we cannot assume that the LLM will patch the same function as the groundtruth patch does. Instead we find all the callstacks for each call to a patched function. Then we find the lowest common ancestor (LCA) across all pairs of stacktraces offered by the groundtruth patch and the LLM patch. We then utilize debug information to inspect arguments, return values, and local variables at the first function above the LCA, differential testing offers a detailed view of the patch’s impact on the program state.

This process evaluates whether the generated patch produces a program state identical to the ground truth program after the patched function returns. By using a diverse set of inputs obtained from fuzzing, this gives higher confidence that the bug is fixed without changing the visible behavior of the patched functions. This differential testing is implemented using a Python script that leverages LLDB APIs to dump all visible states and identify differences between the ground truth and the patched program.

However, as with all attempts to solve provably undecidable problems (in this case: program equivalence), there are some failure modes for this verification step. For example, sometimes the analysis fails with timeouts, in which case we consider the semantics to be preserved if both the ground truth and the LLM patch timed out. Programs might also behave non-deterministically, and we run each input three times to identify nondeterministic struct fields and values. Such fields will not be compared to avoid false alarms from noisy, random values. Additionally, we strip any fields that contain the substring “build” or “time” as we’ve observed false positives from build-ids (that happen to be deterministic within a program, but not across different patches).

It should also be noted that on a number of examples, the crashing PoC never actually triggered the breakpoints on the ground truth patch, making comparison of the resulting states impossible. However, our case study showed that white-box differential testing is still effective in filtering out a majority of incorrect patches despite its limitation, which will be discussed in the case study.

AutoPatchBench and AutoPatchBench-Lite

AutoPatchBench is a comprehensive benchmark dataset of 136 samples. It encompasses a wide range of real-world vulnerabilities, providing a robust framework for assessing the capabilities of automated patch generation systems.

Within this benchmark, we have also created a subset called AutoPatchBench-Lite that consists of 113 samples. AutoPatchBench-Lite focuses on a simpler subset of vulnerabilities where the root cause of the crash is confined to a single function. This version is designed to cater to scenarios where the complexity of the bug is relatively low, making it more accessible for tools that are in the early stages of development or for those that specialize in handling straightforward issues.

The rationale for creating AutoPatchBench-Lite stems from the observation that when root causes are distributed across multiple locations within the code, the difficulty of generating a correct patch increases significantly. Addressing such “hard” crashes requires a tool to possess advanced reasoning capabilities to analyze larger codebases and apply patches to multiple areas simultaneously. This complexity not only challenges the tool’s design but also demands a higher level of sophistication in its algorithms to ensure accurate and effective patching.

By offering both AutoPatchBench and AutoPatchBench-Lite, we provide a tiered approach to benchmarking, allowing developers to progressively test and refine their tools. This structure supports the development of more advanced solutions capable of tackling both simple and complex vulnerabilities, ultimately contributing to the enhancement of AI-assisted bug repair techniques.

Expected use cases

AutoPatchBench offers significant value to a diverse range of users. Developers of auto-patch tools can leverage our open-sourced patch generator to enhance their tools and assess their effectiveness using the benchmark. Software projects employing fuzzing can incorporate our open-sourced patch generator to streamline vulnerability repair. Additionally, model developers can integrate the benchmark into their development cycles to build more robust and specialized expert models for bug repair. The tooling around the patch generator provided here can also be used in reinforcement learning as a reward signal during training. This data helps train models to better understand the nuances of bug repair, enabling them to learn from past fixes and improve their ability to generate accurate patches.

Reference implementation

We developed a basic patch generator to establish a baseline performance using AutoPatchBench. This generator is specifically designed to address simple crashes that involve patching a single function. We have open-sourced this reference implementation to encourage the community to build and expand upon it.

Figure 1 shows a high-level overview of its design. The patch generator takes crash stack trace and target source code as input. It identifies the source locations from the stack trace and extracts every function that contains those locations. It then asks the LLM to identify the root cause and repair the crash by patching one of the functions. Upon receiving a response from the LLM, the patch generator extracts the revised code, applies the patch, compiles the program, and tests it against the original input that caused the crash. If the build or test fails, we re-engage the LLM with the error message from the build or test output, requesting it to attempt a solution again until the crash is resolved. If a fix trajectory fails to reach a valid solution that passes build and crash reproduction within a finite number of steps, we start a new trajectory to reset the context window, preventing prolonged entrapment in an incorrect path.

Figure 1: Patch generation flowchart.

The example prompt below also captures the essence of our approach. Note that the actual prompt is more detailed and divided into multiple segments to promote chain-of-thought reasoning.

As an experienced Security Engineer at Meta, your task is to address the following security-critical fuzzing crash. Below is the stack trace of the crash:

== Fuzzer Crash Report ==
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7af1223 in strcpy () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff7af1223 in strcpy ()
#1  0x0000555555555140 in process_input (input=0x7fffffffe695 "AAAAAA...")
#2  0x0000555555555162 in main (argc=2, argv=0x7fffffffe5f8)

Here is the source code for the functions involved in the stack trace:

strcpy() {...}

void process_input(const char *input) {
    char buffer[8];
    strcpy(buffer, input); // Potential buffer overflow
    printf("Processed: %s\n", buffer);
}

int main() {...}

Assuming the root cause of the crash is within one of these functions, generate a patched version of the faulty function to resolve the fuzzing crash. Ensure that you provide a complete rewrite of the function so that the patch can be applied and the code compiled without errors.

A case study with AutoPatchBench-Lite

In the case study, we demonstrate the use of AutoPatchBench by evaluating our reference patch generator with several LLM models. Given that our reference implementation is limited to addressing simple issues, we conducted our evaluation with AutoPatchBench-Lite, which contains 113 samples. To prevent fix trajectories from becoming excessively prolonged, we capped the maximum length of each trajectory at five. Additionally, we set the maximum number of retries to 10.

Please note that the case study is not intended to provide a statistically rigorous comparison of model performance. Instead, it aims to present preliminary results to establish a baseline expectation. We encourage future research to build upon these findings.

Effectiveness of patch generation and verification

We evaluated the effectiveness of the patch generator and our automated verification processes while using different LLM models as back-end. The figure below illustrates the effectiveness of patch generation and verification by presenting the percentage of samples that successfully passed each sequential verification step: (1) patch validity: build and crash reproducibility check, (2) fuzzing pass: passes 10-minute fuzzing, and (3) testing pass: passes white-box differential testing. It is important to note that the patch generation process only utilizes step (1) to verify the build and crash reproducibility. The fuzzing and differential testing are conducted post-generation to assess correctness.

Figure 2: Patch generation and verification success rate.

Figure 2 shows that all models achieved similar generation success rates of around 60% and similar post-verification success rates of around 5-11% with overlapping confidence intervals, and therefore, we do not draw any conclusion about their relative performance. The graph does, however, reveal that a substantial portion of the generated patches are found to be incorrect when subjected to fuzzing and white-box differential testing. For instance, Gemini 1.5 Pro achieved a 61.1% patch generation success rate, yet fewer than 15% of these patches (5.3% out of total set) were found to be correct. This gap highlights that build and crash reproduction are not good enough signals to infer the correctness of generated patches, and that future patch generation approaches should scrutinize the semantic preservation of generated patches more thoroughly. This gap also underscores the vital role of the comprehensive verification processes that checks semantic equivalence, a distinctive contribution of AutoPatchBench.

Effect of inference-time computation

To assess the impact of inference-time computation on improving the patch generation success rate, we present the distribution of retry counts among the 73 patches produced by Llama 4 Maverick.

Figure 3: Percentage of generated patches per number of iterations.

Figure 3 shows that 44 out of 73 patches, or 60.2%, were successfully generated on the first attempt. The remaining 40% of the samples required more than two iterations, with no evident plateau until the 10th iteration. This outcome demonstrates that allocating more computational resources during inference-time leads to a higher success rate and suggests that increasing the number of retries could yield better results.

Manual validation

In our investigation of the precision and recall of white-box differential testing, we conducted a manual validation of 44 patches that passed 10-minute fuzzing against human-written ground truth fixes with the help of security experts. These patches were selected from a pool of 73 generated by Llama 4 Maverick. The following table shows the confusion matrix.

Table 1: Confusion matrix between human judgement and differential testing

	Test pass	Test fail	Sum
Human pass	5	0	5
Human reject	7	32	39
Sum	12	32	44

The results showed that the differential testing achieved an accuracy of 84.1% for this sample (5 + 32 / 44), indicating a high overall agreement with the human assessment. However, a closer examination of the confusion matrix revealed a notable discrepancy between precision and recall. Specifically, the testing method demonstrated 100.0% recall in this case study, correctly identifying all 5 instances that humans judged as correct. In contrast, precision was relatively low (41.7%), with 7 false positives out of 12 total positive predictions. This suggests that differential testing reported success on some incorrect patches as well, highlighting the need for manual validation of patch correctness. Despite this shortcoming, the result clearly shows the utility of differential testing in automatically rejecting a substantial number of incorrect patches, which will substantially save the manual validation effort.

Key insights

Our case study revealed several limitations of the current patch generator.

The root cause may not exist in the stack trace

Frequently, crashes are the result of state contamination that occurs prior to the crash being triggered. Consequently, none of the functions within the stack frames may include the code responsible for the root cause. Since our current implementation requires the LLM to assume that the root cause is located within one of the functions in the stack trace, it is unable to generate an accurate patch in such cases. Solving this problem would require a more autonomous agent which can reason about the root cause on its own with a code browsing capability.

Cheating

In some instances, the LLM resorted to “cheating” by producing patches that superficially resolved the issue without addressing the underlying problem. This can occur when the generator modifies or removes code in a way that prevents the crash from occurring, but does not actually fix the root cause of the issue. We observed that cheating happens more frequently when we request the LLM to retry within the same trajectory. A potential solution to this could be to empower the LLM to say “I cannot fix it,” which may come with a tradeoff with success rate. However, note that most of the cheating was caught in the verification step, highlighting the utility of differential testing.

Need for enhanced patch verification methods

Fuzzing and white-box differential testing have shown that a large majority of generated patches are incorrect when compared to the ground-truth patches. This finding highlights the challenge of generating accurate patches without enhanced verification capabilities. To address this gap, several approaches can be considered:

A patch generator could provide additional code context when querying the LLM for a patch so that LLM can better understand the consequence of a code patch.
A patch generator could make additional LLM queries to verify the perseverance of existing functionality.
A patch generator can attempt to generate multiple valid patches by exploring multiple trajectories in parallel, and let LLM choose the best option that is most likely to be correct.
In a well-tested real-world codebase, a patch generator can utilize existing tests to validate the patches it creates. This process complements building the code and checking for crash reproduction, allowing the patch generator to retry if a patch fails the tests. The accuracy of the generated patches is largely dependent on the thoroughness of the existing tests.

In conclusion, while our study has identified several challenges with the current patch generation process, it also opens up opportunities for improvement. By addressing these limitations with innovative solutions, we can enhance the accuracy and reliability of patch generation, paving the way for more robust and effective automated tools.

Get started with AutoPatchBench

AutoPatchBench is now available on GitHub. We welcome pull requests to integrate new/additional agent architectures into the framework, and look forward to seeing how well they perform on AutoPatchBench.

The post Introducing AutoPatchBench: A Benchmark for AI-Powered Security Fixes appeared first on Engineering at Meta.

Building Private Processing for AI tools on WhatsApp - Engineering at Meta

2025-04-29T17:15:00.000Z

We are inspired by the possibilities of AI to help people be more creative, productive, and stay closely connected on WhatsApp, so we set out to build a new technology that allows our users around the world to use AI in a privacy-preserving way.
We’re sharing an early look into Private Processing, an optional capability that enables users to initiate a request to a confidential and secure environment and use AI for processing messages where no one — including Meta and WhatsApp — can access them.
To validate our implementation of these and other security principles, independent security researchers will be able to continuously verify our privacy and security architecture and its integrity.

AI has revolutionized the way people interact with technology and information, making it possible for people to automate complex tasks and gain valuable insights from vast amounts of data. However, the current state of AI processing — which relies on large language models often running on servers, rather than mobile hardware — requires that users’ requests are visible to the provider. Although that works for many use cases, it presents challenges in enabling people to use AI to process private messages while preserving the level of privacy afforded by end-to-end encryption.

We set out to enable AI capabilities with the privacy that people have come to expect from WhatsApp, so that AI can deliver helpful capabilities, such as summarizing messages, without Meta or WhatsApp having access to them, and in the way that meets the following principles:

Optionality: Using Meta AI through WhatsApp, including features that use Private Processing, must be optional.
Transparency: We must provide transparency when our features use Private Processing.
User control: For people’s most sensitive chats that require extra assurance, they must be able to prevent messages from being used for AI features like mentioning Meta AI in chats, with the help of WhatApp’s Advanced Chat Privacy feature.

Introducing Private Processing

We’re excited to share an initial overview of Private Processing, a new technology we’ve built to support people’s needs and aspirations to leverage AI in a secure and privacy-preserving way. This confidential computing infrastructure, built on top of a Trusted Execution Environment (TEE), will make it possible for people to direct AI to process their requests — like summarizing unread WhatsApp threads or getting writing suggestions — in our secure and private cloud environment. In other words, Private Processing will allow users to leverage powerful AI features, while preserving WhatsApp’s core privacy promise, ensuring no one except you and the people you’re talking to can access or share your personal messages, not even Meta or WhatsApp.

To uphold this level of privacy and security, we designed Private Processing with the following foundational requirements:

Confidential processing: Private Processing must be built in such a way that prevents any other system from accessing user’s data — including Meta, WhatsApp or any third party — while in processing or in transit to Private Processing.
Enforceable guarantees: Attempts to modify that confidential processing guarantee must cause the system to fail closed or become publicly discoverable via verifiable transparency.
Verifiable transparency: Users and security researchers must be able to audit the behavior of Private Processing to independently verify our privacy and security guarantees.

However, we know that technology platforms like ours operate in a highly adversarial environment where threat actors continuously adapt, and software and hardware systems keep evolving, generating unknown risks. As part of our defense-in-depth approach and best practices for any security-critical system, we’re treating the following additional layers of requirements as core to Private Processing on WhatsApp:

Non-targetability: An attacker should not be able to target a particular user for compromise without attempting to compromise the entire Private Processing system.
Stateless processing and forward security: Private Processing must not retain access to user messages once the session is complete to ensure that the attacker can not gain access to historical requests or responses.

Threat modeling for Private Processing

Because we set out to meet these high-security requirements, our work to build Private Processing began with developing a threat model to help us identify potential attack vectors and vulnerabilities that could compromise the confidentiality, integrity, or availability of user data. We’ve worked with our peers in the security community to audit the architecture and our implementation to help us continue to harden them.

Building in the open

To help inform our industry’s progress in building private AI processing, and to enable independent security research in this area, we will be publishing components of Private Processing, expanding the scope of our Bug Bounty program to include Private Processing, and releasing a detailed security engineering design paper, as we get closer to the launch of Private Processing in the coming weeks.

While AI-enabled processing of personal messages for summarization and writing suggestions at users’ direction is the first use case where Meta applies Private Processing, we expect there will be others where the same or similar infrastructure might be beneficial in processing user requests. We will continue to share our learnings and progress transparently and responsibly.

How Private Processing works

Private Processing creates a secure cloud environment where AI models can analyze and process data without exposing it to unauthorized parties.

Here’s how it works:

Authentication: First, Private Processing obtains anonymous credentials to verify that the future requests are coming from authentic WhatsApp clients.
Third-party routing and load balancing: In addition to these credentials, Private Processing fetches HPKE encryption public keys from a third-party CDN in order to support Oblivious HTTP (OHTTP).
Wire session establishment: Private Processing establishes an OHTTP connection from the user’s device to a Meta gateway via a third-party relay which hides requester IP from Meta and WhatsApp.
Application session establishment: Private Processing establishes a Remote Attestation + Transport Layer Security (RA-TLS) session between the user’s device and the TEE. The attestation verification step cross-checks the measurements against a third-party ledger to ensure that the client only connects to code which satisfies our verifiable transparency guarantee.
Request to Private Processing: After the above session is established, the device makes a request to Private Processing (e.g., message summarization request), that is encrypted end-to-end between the device and Private Processing with an ephemeral key that Meta and WhatsApp cannot access. In other words, no one except the user’s device or the selected TEEs can decrypt the request.
Private Processing: Our AI models process data in a confidential virtual machine (CVM), a type of TEE, without storing any messages, in order to generate a response. CVMs may communicate with other CVMs using the same RA-TLS connection clients use to complete processing.
Response from Private Processing: The processed results are then returned to the user’s device, encrypted with a key that only the device and the pre-selected Private Processing server ever have access to. Private Processing does not retain access to messages after the session is completed.

The threat model

In designing any security-critical system, it is important to develop a threat model to guide how we build its defenses. Our threat model for Private Processing includes three key components:

Assets: The sensitive data and systems that we need to protect.
Threat actors: The individuals or groups that may attempt to compromise our assets.
Threat scenarios: The ways in which our assets could be compromised, including the tactics, techniques, and procedures (TTPs) that threat actors might use.

Assets

In the context of applying Private Processing to summarizing unread messages or providing writing suggestions at users’ direction, we will use Private Processing to protect messaging content, whether they have been received by the user, or still in draft form. We use the term “messages” to refer to these primary assets in the context of this blog.

In addition to messages, we also include additional, secondary assets which help support the goal of Private Processing and may interact with or directly process assets: the Trusted Computing Base (TCB) of the Confidential Virtual Machine (CVM), the underlying hardware, and the cryptographic keys used to protect data in transit.

Threat actors

We have identified three threat actor types that could attack our system to attempt to recover assets.

Malicious or compromised insiders with access to our infrastructure.
A third party or supply chain vendor with access to components of the infrastructure.
Malicious end users targeting other users on the platform.

Threat scenarios

When building Private Processing to be resilient against these threat actors, we consider relevant threat scenarios that may be pursued against our systems, including (but not limited to) the following:

External actors directly exploit the exposed product attack surface or compromise the services running in Private Processing CVMs to extract messages.

Anywhere the system processes untrusted data, there is potentially an attack surface for a threat actor to exploit. Examples of these kinds of attacks include exploitation of zero-day vulnerabilities or attacks unique to AI such as prompt injection.

Private Processing is designed to reduce such an attack surface through limiting the exposed entry points to a small set of thoroughly reviewed components which are subject to regular assurance testing. The service binaries are hardened and run in a containerized environment to mitigate the risks of code execution and limit a compromised binary’s ability to exfiltrate data from within the CVM to an external party.

Internal or external attackers extract messages exposed through the CVM.

Observability and debuggability remains a challenge in highly secure environments as they can be at odds with the goal of confidential computing, potentially exposing side channels to identify data and in the worst case accidentally leaking messages themselves. However, deploying any service at scale requires some level of observability to identify failure modes, since they may negatively impact many users, even when the frequency is uncommon. We implement a log-filtering system to limit export to only allowed log lines, such as error logs.

Like any complex system, Private Processing is built of components to form a complex supply chain of both hardware and software. Internally, our CVM build process occurs in restricted environments that maintain provenance and require multi-party review. Transparency of the CVM environment, which we’ll provide through publishing a third-party log of CVM binary digests and CVM binary images, will allow external researchers to analyze, replicate, and report instances where they believe logs could leak user data.

Insiders with physical or remote access to Private Processing hosts interfere with the CVM at boot and runtime, potentially bypassing the protections in order to extract messages.

TEE software exploitation is a growing area of security research, and vulnerability researchers have repeatedly demonstrated the ability to bypass TEE guarantees. Similarly, physical attacks on Private Processing hosts may be used to defeat TEE guarantees or present compromised hosts as legitimate to an end user.

To address these unknown risks, we built Private Processing on the principle of defense-in-depth by actively tracking novel vulnerabilities in this space, minimizing and sanitizing untrusted inputs to the TEE, minimizing attack surface through CVM hardening and enabling abuse detection through enhanced host monitoring.

Because we know that defending against physical access introduces significant complexity and attack surface even with industry-leading controls, we continuously pursue further attack surface hardening. In addition, we reduce these risks through measures like encrypted DRAM and standard physical security controls to protect our datacenters from bad actors.

To further address these unknown risks, we seek to eliminate the viability of targeted attacks via routing sessions through a third-party OHTTP relay to prevent an attacker’s ability to route a specific user to a specific machine.

Designing Private Processing

Here is how we designed Private Processing to meet these foundational security and privacy requirements against the threat model we developed.

(Further technical documentation and security research engagements updates are coming soon).

Confidential processing

Data shared to Private Processing is processed in an environment which does not make it available to any other system. This protection is further upheld by encrypting data end-to-end between the client and the Private Processing application, so that only Private Processing, and no one in between – including Meta, WhatsApp, or any third-party relay – can access the data.

To prevent possible user data leakage, only limited service reliability logs are permitted to leave the boundaries of CVM.

System software

To prevent privileged runtime access to Private Processing, we prohibit remote shell access, including from the host machine, and implement security measures including code isolation. Code isolation ensures that only designated code in Private Processing has access to user data. Prohibited remote shell access ensures that neither the host nor a networked user can gain access to the CVM shell.

We defend against potential source control and supply chain attacks by implementing established industry best practices. This includes building software exclusively from checked-in source code and artifacts, where any change requires multiple engineers to modify the build artifacts or build pipeline.

As another layer of security, all code changes are auditable. This allows us to ensure that any potential issues are discovered — either through our continuous internal audits of code, or by external security researchers auditing our binaries.

System hardware

Private Processing utilizes CPU-based confidential virtualization technologies, along with Confidential Compute mode GPUs, which prevent certain classes of attacks from the host operating system, as well as certain physical attacks.

Enforceable guarantees

Private Processing utilizes CPU-based confidential virtualization technologies which allow attestation of software based in a hardware root of trust to guarantee the security of the system prior to each client-server connection. Before any data is transmitted, Private Processing checks these attestations, and confirms them against a third-party log of acceptable binaries.

Stateless and forward secure service

We operate Private Processing as a stateless service, which neither stores nor retains access to messages after the session has been completed.

Additionally, Private Processing does not store messages to disk or external storage, and thus does not maintain durable access to this data.

As part of our data minimization efforts, requests to Private Processing only include data that is useful for processing the prompt — for example, message summarization will only include the messages the user directed AI to summarize.

Non-targetability

Private Processing implements the OHTTP protocol to establish a secure session with Meta routing layers. This ensures that Meta and WhatsApp do not know which user is connecting to what CVM. In other words, Meta and WhatsApp do not know the user that initiated a request to Private Processing while the request is in route, so that a specific user cannot be routed to any specific hardware.

Private Processing uses anonymous credentials to authenticate users over OHTTP. This way, Private Processing can authenticate users to the Private Processing system, but remains unable to identify them. Private Processing does not include any other identifiable information as part of the request during the establishment of a system session. We limit the impact of small-scale attacks by ensuring that they cannot be used to target the data of a specific user.

Verifiable transparency

To provide users visibility into the processing of their data and aid in validation of any client-side behaviors, we will provide capabilities to obtain an in-app log of requests made to Private Processing, data shared with it, and details of how that secure session was set up.

In order to provide verifiability, we will make available the CVM image binary powering Private Processing. We will make these components available to researchers to allow independent, external verification of our implementation.

In addition, to enable deeper bug bounty research in this area, we will publish source code for certain components of the system, including our attestation verification code or load bearing code.

We will also be expanding the scope of our existing Bug Bounty program to cover Private Processing to enable further independent security research into Private Processing’s design and implementation.

Finally, we will be publishing a detailed technical white paper on the security engineering design of Private Processing to provide further transparency into our security practices, and aid others in the industry in building similar systems.

Get Involved

We’re deeply committed to providing our users with the best possible messaging experience while ensuring that only they and the people they’re talking to can access or share their personal messages. Private Processing is a critical component of this commitment, and we’re excited to make it available in the coming weeks.

We welcome feedback from our users, researchers, and the broader security community through our security research program:

More details: Meta Bug Bounty
Contact us

The post Building Private Processing for AI tools on WhatsApp appeared first on Engineering at Meta.

How It’s Made: Little Language Lessons uses Gemini’s multilingual capabilities to personalize language learning - Google Developers Blog

2025-04-29T16:04:02.000Z

Little Language Lessons, a project leveraging Gemini's API and Cloud services to generate content, translate, and provide text-to-speech functionalities, includes vocabulary lessons, slang practice, and object recognition for language learning.

Cutting through the noise: How to prioritize Dependabot alerts - The GitHub Blog

2025-04-29T16:00:39.000Z

Let’s be honest: that flood of security alerts in your inbox can feel completely overwhelming. We’ve been there too.

As a developer advocate and a product manager focused on security at GitHub, we’ve seen firsthand how overwhelming it can be to triage vulnerability alerts. Dependabot is fantastic at spotting vulnerabilities, but without a smart way to prioritize them, you might be burning time on minor issues or (worse) missing the critical ones buried in the pile.

So, we’ve combined our perspectives—one from the security trenches and one from the developer workflow side—to share how we use Exploit Prediction Scoring System (EPSS) scores and repository properties to transform the chaos into clarity and make informed prioritization decisions.

Understanding software supply chain security

If you’re building software today, you’re not just writing code—you’re assembling it from countless open source packages. In fact, 96% of modern applications are powered by open source software. With such widespread adoption, open source software has become a prime target for malicious actors looking to exploit vulnerabilities at scale.

Attackers continuously probe these projects for weaknesses, contributing to the thousands of Common Vulnerabilities and Exposures (CVEs) reported each year. But not all vulnerabilities carry the same level of risk. The key question becomes not just how to address vulnerabilities, but how to intelligently prioritize them based on your specific application architecture, deployment context, and business needs.

Understanding EPSS: probability of exploitation with severity if it happens

When it comes to prioritization, many teams still rely solely on severity scores like the Common Vulnerability Scoring System (CVSS). But not all “critical” vulnerabilities are equally likely to be exploited. That’s where EPSS comes in—it tells you the probability that a vulnerability will actually be exploited in the wild within the next 30 days.

Think of it this way: CVSS tells you how bad the damage could be if someone broke into your house, while EPSS tells you how likely it is that someone is actually going to try. Both pieces of information are crucial! This approach allows you to focus resources effectively.

As security pro Daniel Miessler points out in Efficient Security Principle, “The security baseline of an offering or system faces continuous downward pressure from customer excitement about, or reliance on, the offering in question.”

Translation? We’re always balancing security with usability, and we need to be smart about where we focus our limited time and energy. EPSS helps us spot the vulnerabilities with a higher likelihood of exploitation, allowing us to fix the most pressing risks first.

Smart prioritization steps

1. Combine EPSS with CVSS

One approach is to look at both likelihood (EPSS) and potential impact (CVSS) together. It’s like comparing weather forecasts—you care about both the chance of rain and how severe the storm might be.

For example, when prioritizing what to fix first, a vulnerability with:

EPSS: 85% (highly likelihood of exploitation)
CVSS: 9.8 (critical severity)

…should almost always take priority over one with:

EPSS: 0.5% (much less likely to be exploited)
CVSS: 9.0 (critical severity)

Despite both having red-alert CVSS ratings, the first vulnerability is the one keeping us up at night.

2. Leverage repository properties for context-aware prioritization

Not all code is created equal when it comes to security risk. Ask yourself:

Is this repo public or private? (Public repositories expose vulnerabilities to potential attackers)
Does it handle sensitive data like customer info or payments?
How often do you deploy? (Frequent deployments face tighter remediation times)

One way to provide context-aware prioritization systematically is with custom repository properties, which allow you to add contextual information about your repositories with information such as compliance frameworks, data sensitivity, or project details. By applying these custom properties to your repositories, you create a structured classification system that helps you identify the “repos that matter,” so you can prioritize Dependabot alerts for your production code rather than getting distracted by your totally-not-a-priority test-vulnerabilities-local repo.

3. Establish clear response Service Level Agreements (SLAs) based on risk levels

Once you’ve done your homework on both the vulnerability characteristics and your repository context in your organization, you can establish clear timelines for responses that make sense for your organization resources and risk tolerance.

Let’s see how this works in real life: Here’s an example risk matrix that combines both EPSS (likelihood of exploitation) and CVSS (severity of impact).

EPSS ↓ / CVSS →	Low	Medium	High
Low	✅ When convenient	⏳ Next sprint	⚠️ Fix Soon
Medium	⏳ Next sprint	⚠️ Fix soon	🔥 Fix soon
High	⚠️ Fix Soon	🔥 Fix soon	🚨 Fix first

Say you get an alert about a vulnerability in your payment processing library that has both a high EPSS score and high CVSS rating. Red alert! Looking at our matrix, that’s a “Fix first” situation. You’ll probably drop what you’re doing, and put in some quick mitigations while the team works on a proper fix.

But what about that low-risk vulnerability in some testing utility that nobody even uses in production? Low EPSS, low CVSS… that can probably wait until “when convenient” within the next few weeks. No need to sound the alarm or pull developers off important feature work.

This kind of prioritization just makes sense. Applying the same urgency to every single vulnerability just leads to alert fatigue and wasted resources, and having clear guidelines helps your team know where to focus first.

Integration with enterprise governance

For enterprise organizations, GitHub’s auto-triage rules help provide consistent management of security alerts at scale across multiple teams and repositories.

Auto-triage rules allow you to create custom criteria for automatically handling alerts based on factors like severity, EPSS, scope, package name, CVE, ecosystem, and manifest location. You can create your own custom rules to control how Dependabot auto-dismisses and reopens alerts, so you can focus on the alerts that matter.

These rules are particularly powerful because they:

Apply to both existing and future alerts.
Allow for proactive filtering of false positives.
Enable “snooze until patch” functionality for vulnerabilities without a fix available.
Provide visibility into automated decisions through the auto-dismiss alert resolution.

GitHub-curated presets like auto-dismissal of false positives are free for everyone and all repositories, while custom auto-triage rules are available for free on public repositories and as part of GitHub Advanced Security for private repositories.

The real-world impact of smart prioritization

When teams get prioritization right, organizations can experience significant improvements in security management. Research firmly supports this approach: The comprehensive Cyentia EPSS study found teams could achieve 87% coverage of exploited vulnerabilities by focusing on just 10% of them, dramatically reducing necessary remediation efforts by 83% compared to traditional CVSS-based approaches. This isn’t just theoretical, it translates to real-world efficiency gains.

This reduction is not just about numbers. When security teams provide clear reasoning behind prioritization decisions, developers gain a better understanding of security requirements. This transparency builds trust between teams, potentially leading to more efficient resolution processes and improved collaboration between security and development teams.

The most successful security teams pair smart automation with human judgment and transparent communication. This shift from alert overload to smart filtering lets teams focus on what truly matters, turning security from a constant headache into a manageable, strategic advantage.

Getting started

Ready to tame that flood of alerts? Here’s how to begin:

Enable Dependabot security updates: If you haven’t already, turn on Dependabot alerts and automatic security updates in your repository settings. This is your first line of defense!
Set up auto-triage rules: Create custom rules based on severity, scope, package name, and other criteria to automatically handle low-priority alerts. Auto-triage rules are a powerful tool to help you reduce false positives and alert fatigue substantially, while better managing your alerts at scale.
Establish clear prioritization criteria: Define what makes a vulnerability critical for your specific projects. Develop a clear matrix for identifying critical issues, considering factors like impact assessment, system criticality, and exploit likelihood.
Consult your remediation workflow for priority alerts: Verify the vulnerability’s authenticity and develop a quick mitigation strategy based on your organization’s risk response matrix.

By implementing these smart prioritization strategies, you’ll help focus your team’s energy where it matters most: keeping your code secure and your customers protected. No more security alert overload, just focused, effective prioritization.

Want to streamline security alert management for your organization? Start using Dependabot for free or unlock advanced prioritization with GitHub Code Security today.

The post Cutting through the noise: How to prioritize Dependabot alerts appeared first on The GitHub Blog.

Usability and safety updates to Google Auth Platform - Google Developers Blog

2025-04-28T19:29:01.000Z

Updates to the Google Auth Platform include changes to OAuth configuration, client secrets display, and automatic deletion of unused clients, making the platform more secure and easier to use.