Use GitHub's AI-powered code security and code quality features responsibly by understanding their purposes, capabilities, and limitations.
GitHub’s application and platform cards are intended to help you understand how our AI technology works, the choices application owners can make that influence application performance and behavior, and the importance of considering the whole application, including the technology, the people, and the environment. Application cards are created for AI applications and platform cards are created for AI platform services. These resources can support the development or deployment of your own applications and can be shared with users or stakeholders impacted by them.
As part of its commitment to responsible AI, GitHub adheres to Microsoft's six core principles: fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability. These principles are embedded in the Responsible AI Standard, which guides teams in designing, building, and testing AI applications. Application and Platform Cards play a key role in operationalizing these principles by offering transparency around capabilities, intended uses, and limitations. For further insight, readers are encouraged to explore Microsoft’s Responsible AI Transparency Report and GitHub Terms.
GitHub's security and quality platform includes several AI-powered capabilities that help developers find and fix security vulnerabilities, detect leaked secrets, and improve code quality. This application card covers the following experiences:
Copilot Autofix for code scanning: Automatically generates fix suggestions for CodeQL alerts on pull requests and the default branch.
Generic secret detection: Uses a model to identify unstructured secrets in source code that deterministic pattern matching cannot find.
Custom pattern regex generator: Uses AI to generate regular expressions for custom secret scanning patterns from natural language descriptions.
GitHub Code Quality: Surfaces code quality issues and offers LLM-powered fix suggestions on pull requests and the default branch. Also provides AI-powered automatic generation of code coverage workflows.
Copilot Autofix is an expansion of code scanning that provides users with targeted recommendations to help them fix code scanning alerts, avoiding the introduction of new security vulnerabilities. Potential fixes are generated automatically by large language models (LLMs) using data from the codebase and from code scanning analysis. Copilot Autofix is available for CodeQL analysis and does not require a GitHub Copilot subscription.
Code scanning users can already see security alerts on their pull requests. However, developers often have little training in secure coding, so fixing these alerts requires substantial effort. Copilot Autofix lowers the barrier of entry by combining information on best practices with details of the codebase and alert to suggest a potential fix. Instead of starting with a search for information about the vulnerability, the developer starts with a code suggestion that demonstrates a potential solution for their codebase. The developer evaluates the potential fix to determine whether it is the best solution for their codebase and to ensure that it maintains the intended behavior.
Secret scanning's generic secret detection is an AI-powered expansion of secret scanning that identifies unstructured secrets in source code or other GitHub surfaces and generates an alert. GitHub Secret Protection and GitHub Advanced Security users can already receive secret scanning alerts for partner or custom patterns found in their source code, but unstructured secrets are not easily discoverable. Secret scanning uses models to identify these secrets. When a finding is detected, an alert is displayed in the "Generic" list of secret scanning alerts (under the Security and quality tab of the repository, organization, or enterprise), so that maintainers and security managers can review the alert and, where necessary, remove the credential or implement a fix. Generic secret detection does not require a GitHub Copilot subscription.
Secret scanning's custom pattern regular expression generator makes it possible to define custom secret scanning patterns without knowledge of regular expressions. Users input a natural language description of what they want to detect, along with optional example strings, and the generator produces up to three candidate regular expressions. These patterns can then be validated via the dry-run mechanism before being deployed as custom patterns. The regular expression generator does not require a GitHub Copilot subscription.
GitHub Code Quality helps users improve code reliability, maintainability, and overall project health by surfacing actionable feedback and offering automatic fixes for findings in pull requests and on the default branch. When Code Quality is enabled, two types of analysis run: CodeQL quality queries identify problems with the maintainability, reliability, or style of code, and LLM-powered analysis provides additional insights beyond what deterministic engines can find. When a quality issue is detected, Copilot Autofix suggests a relevant fix. On pull requests, results are displayed as comments left by the github-code-quality bot. On the default branch, LLM-powered findings are displayed in the AI findings dashboard under the Security and quality tab. Additionally, Code Quality can automatically generate a code coverage CI workflow for your repository using an AI-powered agent, reducing the manual effort required to set up coverage tracking.
The primary supported language for GitHub Code Security AI features is English.
The following list provides a glossary of key terms related to GitHub Code Security AI features:
CodeQL: GitHub's semantic code analysis engine for identifying security vulnerabilities in source code.
Copilot Autofix: GitHub's LLM-powered feature that automatically generates fix suggestions for code scanning alerts. Copilot Autofix is available for CodeQL analysis and does not require a GitHub Copilot subscription.
Large language model (LLM): A type of neural network trained on a large body of text data that can generate, analyze, and transform natural language and code. Copilot Autofix uses one or more LLMs to process code scanning alerts and produce fix suggestions.
AI detection for secret scanning: AI-powered capabilities that extend secret scanning, including generic secret detection. Does not require a GitHub Copilot subscription.
Generic secret detection: AI identification of unstructured secrets (such as passwords) that are not covered by partner or custom patterns. Generic secret detection uses models to scan for password-like strings in source code.
Custom pattern: A user-defined regular expression used by secret scanning to detect secrets that match a specific format. The custom pattern regular expression generator helps create these patterns from natural language descriptions.
SARIF: Static Analysis Results Interchange Format—the standard format CodeQL uses to report code scanning findings, including alert locations and descriptions.
GitHub Code Quality: A feature that surfaces code quality issues and offers LLM-powered fixes. Code Quality combines CodeQL quality queries with LLM-powered analysis to identify maintainability, reliability, and style issues.
AI findings: The dashboard under the Security and quality tab where LLM-powered Code Quality findings for the default branch are displayed.
The key features and capabilities outlined here describe what GitHub Code Security AI features are designed to do and how they perform across supported tasks.
Automated fix suggestions for security alerts: Copilot Autofix automatically generates code change suggestions for CodeQL alerts found on pull requests and on the default branch. Each suggestion includes both the proposed code change and a natural language explanation of the fix.
Alert-to-fix translation: Copilot Autofix translates the description and location of a code scanning alert into actionable code changes that may resolve the underlying security vulnerability. The system uses CodeQL alert data in SARIF format, surrounding code snippets, and query help text to generate relevant fixes.
Multi-language support: Copilot Autofix supports fix generation for a subset of queries included in the default and security-extended CodeQL query suites for C#, C/C++, Go, Java/Kotlin, Swift, JavaScript/TypeScript, Python, Ruby, and Rust. For more information on these query suites, see CodeQL query suites.
AI-powered password detection: Secret scanning's generic secret detection scans repository content using AI to identify unstructured secrets (like passwords) that deterministic pattern matching cannot find. Detected secrets are surfaced as alerts in the secret scanning alert list under the Security and quality tab.
AI-powered regular expression generation: Secret scanning's regular expression generator takes a natural language description of the pattern you want to detect, along with optional example strings, and produces up to three candidate regular expressions. Each result includes an AI-generated plain language description, and you can validate patterns via a dry run before deployment.
Code quality issue detection: GitHub Code Quality runs CodeQL quality queries on changed code in pull requests and periodically on the full default branch. These queries identify maintainability, reliability, and style issues.
LLM-powered code quality analysis: After each push to the default branch, an LLM analyzes recently changed files for quality issues beyond what deterministic engines can find. Findings are displayed in the AI findings dashboard.
Automated fix suggestions for quality findings: When a quality issue is detected by either type of analysis, Copilot Autofix generates a fix suggestion. On pull requests, the github-code-quality bot posts a comment with the suggested change.
GitHub Code Security AI features can be used in multiple scenarios across a variety of industries. Some examples of use cases include:
Accelerating remediation of security vulnerabilities: Use Copilot Autofix to quickly generate fix suggestions for CodeQL alerts, reducing the time and expertise required to address security issues found during code scanning.
Reducing the barrier to secure coding: Copilot Autofix helps developers with limited secure-coding training. Instead of researching vulnerabilities independently, developers start with a code suggestion that demonstrates a potential solution for their codebase.
Streamlining pull request review: When code scanning finds alerts on a pull request, Copilot Autofix provides suggested fixes inline, helping developers resolve security issues before merging.
Fixing alerts on the default branch: Copilot Autofix can also generate fix suggestions for existing alerts on the default branch, helping teams reduce their backlog of security findings.
Detecting leaked passwords in source code: Use generic secret detection to find unstructured secrets in repositories that fall outside the coverage of partner and custom secret scanning patterns.
Triaging credentials with contextual alerts: When a password is detected, an alert with AI-detection context is displayed in the alerts list, enabling maintainers and security managers to review the finding and take action.
Creating custom secret scanning patterns without regex expertise: Use the regular expression generator to define custom patterns by describing what you want to detect in natural language, removing the need to write regular expressions manually.
Validating generated patterns before deployment: After generating regular expressions, use the dry-run mechanism to test patterns across your repository or organization before deploying them as custom patterns.
Surfacing code quality issues across a repository: Use GitHub Code Quality to identify maintainability, reliability, and style issues so developers and administrators can quickly prioritize areas of risk.
Accelerating remediation of code quality findings: Copilot Autofix suggests fixes for quality findings, combining information on best practices with details of the codebase to propose a potential fix directly on the pull request or in the AI findings dashboard.
Providing actionable feedback on pull requests: The github-code-quality bot posts comments with suggested fixes on pull requests, helping developers address quality issues before merging.
Copilot Autofix uses internal GitHub Copilot APIs interfacing with the large language models, which produce both suggested fixes in code and explanatory text for those fixes.
Generic secret detection uses models to scan for unstructured secrets.
The custom pattern regular expression generator uses LLMs and the GitHub Copilot API to generate regular expressions that match user-provided descriptions and examples.
GitHub Code Quality's LLM-powered analysis uses Copilot language models to analyze recently changed files for quality issues. The CodeQL quality queries component does not use an LLM. Copilot Autofix for Code Quality findings uses the same LLM pipeline as Copilot Autofix for code scanning.
For a comparison of the models available for Copilot, see AI model comparison. For the full list of supported models, see Supported AI models in GitHub Copilot. For information on where models are hosted, see Hosting of models for GitHub Copilot. To learn more about the data used to train the foundation models behind GitHub security and quality, see What data has GitHub Copilot been trained on? in the GitHub Copilot FAQ.
Data handled by Copilot Autofix is not employed for LLM training purposes. The use of this feature is governed by the existing terms and conditions associated with GitHub Advanced Security. For more information, see GitHub Terms for Additional Products and Features.
When Copilot Autofix is enabled for a repository, code scanning alerts are processed through the following pipeline:
AI secret detection processes input and produces output as follows:
Custom pattern regex generator processes input and produces output as follows:
GitHub Code Quality LLM-powered analysis processes input and produces output as follows:
Copilot Autofix for Code Quality findings on pull requests:
github-code-quality bot posts a comment on the pull request with the suggested change. Users can also request autofix generation for results on the default branch.Understanding GitHub Code Security AI features' limitations is crucial to determine if it is used within safe and effective boundaries. While we encourage customers to leverage GitHub Code Security AI features in their innovative solutions or applications, it's important to note that GitHub Code Security AI features was not designed for every possible scenario. We encourage users to refer to GitHub Terms as well as the following considerations when choosing a use case:
.cs, .go, .java, .js, .kt, .php, .py, .rb, .scala, .swift, or .ts.Performance and safety evaluations assess whether AI applications are operating reliably and securely by examining factors like groundedness, relevance, and coherence while identifying the risks of generating harmful content. The following evaluations were conducted with safety components already in place, which are also described in 9. Safety components and mitigations.
GitHub Security AI features are evaluated across its supported surfaces using a combination of industry-standard benchmarks (e.g., SWE-Bench) and internally developed evaluation suites. Benchmark tasks are sourced from public open-source repositories and synthetic scenarios; no real user queries or customer code are used without permission. Each evaluation includes multiple independent runs to account for nondeterminism in model outputs. Key metrics include resolution rate (percentage of tasks successfully completed), token efficiency, latency, and tool call reliability. Models are re-evaluated when updates are made and monitored continuously in production via error rates, response latency, and aggregate usage patterns.
New models undergo a staged evaluation process before deployment to Code Security, Code Quality, and Secret Protection. Integrator teams run benchmark suites specific to their surface, testing the model on representative coding tasks such as bug fixes, code generation, and multi-file refactoring. Results are reviewed against established baselines and existing production models. Models must meet or exceed baseline performance across key metrics like resolution rate, token efficiency, and latency, before advancing to the next stage.
Evaluating potential risks associated with AI-generated content is essential for safeguarding against content risks with varying degrees of severity. This includes evaluating an AI application's predisposition towards generating harmful content or testing vulnerabilities to jailbreak attacks. For GitHub, we conduct performance evaluations, including those which are adapted for coding purposes from Microsoft Foundry:
Our evaluation data is custom-built to assess AI application performance across key areas of safety and quality, simulating real-world scenarios and risks. We begin by identifying relevant evaluation aspects of concern based on multi-disciplinary research and expert input. These concerns are translated into targeted evaluation objectives and guide formulation of evaluation metrics. For safety, we create adversarial prompts to elicit undesirable or edge-case responses, which are then scored using AI-assisted annotators trained to assess alignment with GitHub’s standards. For quality, we craft rubric-based prompts relevant to scenarios including evaluating retrieval-augmented generation (RAG) applications and agents. Datasets are curated from diverse sources including synthetic and public datasets to simulate real-world user scenarios. Using the curated datasets, both evaluations undergo iterative refinement and human alignment to improve metric efficacy and reliability. This methodology forms the foundation of repeatable, rigorous assessments that reflect how customers use evaluations to build better AI.
GitHub uses an automated test harness to continuously monitor the quality of Copilot Autofix suggestions. The test harness includes a set of over 2,300 alerts from a diverse set of public repositories where the highlighted code has test coverage. Suggestions for these alerts are tested to determine how much a developer would need to edit them before committing them to the codebase. For many of the test alerts, suggestions generated by the LLM could be committed as-is to fix the alert while continuing to successfully pass all existing CI tests.
GitHub tests the effectiveness of suggestions by merging all suggested changes, unedited, before running code scanning and the repository's unit tests on the resulting code:
In addition, GitHub spot-checks many successful suggestions and verifies that they fix the alert without introducing new problems. When one or more of these checks fail, manual triage showed that in many cases the proposed fix was nearly correct but needed some minor modifications that a user could identify and manually perform.
The system is also stress-tested to check for potential harm (red teaming), and a filtering system on the LLM helps prevent potentially harmful suggestions from being displayed to users.
AI secret detection has been subject to Responsible AI Red Teaming and GitHub continues to monitor the efficacy and safety of the feature over time.
Custom pattern regex generator results are validated through the dry-run mechanism, which allows users to test generated patterns across their repository or organization before deploying them as custom patterns. This built-in validation step helps ensure that generated regular expressions perform as expected before they are used in production.
GitHub Code Quality's LLM-powered analysis shares the evaluation framework of Copilot code review. Copilot Autofix suggestions for Code Quality findings follow the same test harness as Copilot Autofix for code scanning.
Human-in-the-loop review: Copilot Autofix presents all suggestions as proposed code changes that require explicit developer review and acceptance before being applied. Developers must evaluate each suggestion and verify it maintains the codebase's intended behavior.
Content filtering: A filtering system on the LLM detects and prevents potentially harmful suggestions from being displayed to users. The system is stress-tested through red teaming to identify potential vulnerabilities.
Internal quality testing: Suggestions that fail internal testing are not displayed to users. Fix generation is only shown when the system has sufficient confidence in the suggestion's quality.
Opt-in/opt-out controls: Copilot Autofix is allowed by default and enabled for every repository using CodeQL, but administrators can disable Copilot Autofix at the enterprise, organization, and repository levels. For more information, see Disabling Copilot Autofix for code scanning.
No training on customer data: Data handled by Copilot Autofix is not employed for LLM training purposes. The use of this feature is governed by the existing terms and conditions associated with GitHub Advanced Security.
False positive feedback loop: When users close a generic secret detection alert and mark the reason as "False positive," GitHub uses the false positive volume to improve the model. GitHub does not have access to the secret literals themselves.
Dry-run validation for generated patterns: Generated regular expressions from the custom pattern regex generator must go through a dry-run validation step before deployment. Users explicitly import a result into the custom pattern form and test it across their repository or organization, ensuring patterns perform as expected before they are used in production.
Explicit user action required: The regex generator does not automatically deploy patterns. Users must click Use result to copy a generated expression into the custom pattern form, then manually save and enable the pattern.
Feedback mechanism for Code Quality: Users can provide feedback on Code Quality suggestions using the thumbs up and thumbs down buttons on the github-code-quality bot's comments, helping GitHub improve suggestion quality.
Responsible AI is a shared commitment between GitHub and its customers. While GitHub builds AI applications with safety, fairness, and transparency at the core, customers play a critical role in deploying and using these technologies responsibly within their own contexts. To support this partnership, we offer the following best practices for deployers and end users to help customers implement responsible AI effectively.
Exercise caution and evaluate outcomes when using GitHub Security AI features for consequential decisions or in sensitive domains:
Consequential decisions are those that may have a legal or significant impact on a person's access to education, employment, financial platforms, government benefits, healthcare, housing, insurance, legal platforms, or that could result in physical, psychological, or financial harm. Sensitive domains—such as financial platforms, healthcare, and housing—require particular care due to the potential for disproportionate impact on different groups of people. When using AI for decisions in these areas, make sure that impacted stakeholders can understand how decisions are made, appeal decisions, and update any relevant input data.
Evaluate legal and regulatory considerations:
Customers need to evaluate potential specific legal and regulatory obligations when using any AI platforms and solutions, which may not be appropriate for use in every industry or scenario. Additionally, AI platforms or solutions are not designed for and may not be used in ways prohibited in applicable terms of service and relevant codes of conduct.
Always review suggestions before accepting: Evaluate the proposed code change to ensure it correctly fixes the security vulnerability without changing the intended behavior of your code. Having good test coverage helps verify that a fix does not change the behavior of the codebase.
Verify CI tests pass: After committing a suggested fix or modified fix, always verify that continuous integration testing (CI) for the codebase continues to pass and that the alert is shown as resolved before merging your pull request.
Review dependency changes carefully: If a suggested fix includes changes to dependencies, verify that any added or updated dependencies are secure, supported, and maintain the intended behavior of the codebase. Use dependency management solutions, such as the dependency review API and action, to evaluate changes. For more information, see Dependency review.
Close false positive alerts appropriately: Since AI secret detection may generate more false positives than partner pattern detection, review the accuracy of each alert. When you verify an alert to be a false positive, close the alert and mark the reason as "False positive" in the GitHub UI. This feedback helps improve the model.
Validate generated regex patterns with a dry run: When using the custom pattern regex generator, always perform a dry run across representative repositories before deploying a generated pattern organization-wide.
Be specific with descriptions: To improve the quality of generated regular expressions, be as specific as possible with your natural language descriptions and include diverse example strings that represent the patterns you want to detect.
Review all generated patterns: Carefully review each of the generated regular expressions, including the AI-generated plain language descriptions, and consider modifying results to more fully meet your needs. You remain ultimately responsible for any custom patterns you decide to use.
Review Code Quality findings before applying fixes: Always verify the accuracy and applicability of Code Quality findings and Autofix suggestions to your codebase before accepting them.
Provide feedback on Code Quality suggestions: Use the thumbs up and thumbs down buttons on the github-code-quality bot's comments to help improve the tool and address any concerns or limitations.
Exercise human oversight when appropriate: Human oversight is an important safeguard when interacting with AI applications. While we continuously improve our AI applications, AI might still make mistakes. The outputs generated may be inaccurate, incomplete, biased, misaligned, or irrelevant to your intended goals. This could happen due to various reasons, such as ambiguity in the inputs or limitations of the underlying models. As such, users should review the responses generated by GitHub Code Security AI features and verify that they match their expectations and requirements.
Be aware of the risk of overreliance:
Overreliance on AI happens when users accept incorrect or incomplete AI outputs, mainly because mistakes in AI outputs may be hard to detect. For the end-user, overreliance could result in decreased productivity, loss of trust, application abandonment, financial loss, psychological harm, physical harm, among others. (e.g. a doctor accepts an incorrect AI output).
Exercise caution when designing agentic AI in sensitive domains:
Users should exercise caution when designing and/or deploying agentic AI applications in sensitive domains where agent actions are irreversible or highly consequential. Additional precautions should also be taken when creating autonomous agentic AI as described further in the GitHub Terms.
Enable CI testing on pull requests: Ensure continuous integration testing is in place before enabling Copilot Autofix, so that functional requirements are verified after developers apply fixes.
Use dependency management solutions: Enable dependency review on pull requests to catch potentially risky dependency changes introduced by Autofix suggestions.
Review security overview metrics: Use your organization's security overview dashboard to view the total number of Copilot Autofix suggestions generated on open and closed pull requests for a given time period. For more information, see Security overview dashboard metrics.
Evaluate false-positive volume for secret detection: Evaluate the false-positive volume and establish triage processes for the alerts list.
Monitor Code Quality suggestion volume and quality: Evaluate the volume and quality of Code Quality suggestions and adjust enablement as appropriate for your organization.
For additional guidance on the responsible use of GitHub Security AI features, we recommend reviewing the following documentation: