Nick Silverman is a Senior Infrastructure Engineer on the Edge Infrastructure team at Shopify, where he maintains the systems that provision, renew, and publish SSL certificates for millions of merchants’ custom domains. He is also a contributor to the Ruby acme-client gem.

The challenge

Shopify’s automated certificate management system relied on a static renewal threshold: 30 days before the end of the 90-day lifetime. To spread the load of provisioning and renewing certificates, we implemented a random 0–72 hour delay for each. While this helps evenly distribute certificate management over time, it did not take into account the Certificate Authority’s (CA) load. It was also incapable of reacting to a dynamic renewal window based on information provided by the CA.

However, this approach needed greater resilience to solve what is, in the end, a distributed coordination problem. The weaknesses are:

• No rapid revocation response: The static logic is not aware of revocations at all.

• Brittleness to lifetime changes: The static 30-day threshold is not resilient to changes in certificate lifetime, such as Let’s Encrypt’s announced plan to move to 45-day certificates.

• Imperfect load distribution: Despite the random jitter, massive renewal bursts could still occur.

Shopify needed to develop a global coordination system to balance the load and handle regular and urgent renewals. Thankfully, Let’s Encrypt has led the charge on a solution for this and other very important aspects of the certificate lifecycle.

The journey

Let’s Encrypt and the Internet Engineering Task Force (IETF) published the ACME Renewal Information (ARI) standard which makes an endpoint available that provides a recommended window of time for the renewal to occur. The endpoint returns a payload that looks something like this:

GET /renewal-info/ACME_KEY_IDENTIFIER
{
  "suggestedWindow": {
    "start": "2026-02-03T04:00:00Z",
    "end": "2026-02-04T04:00:00Z"
  }
}

Shopify’s certificate management system uses the acme-client Ruby gem originally authored by another Shopify employee. A growing number of ACME clients, including certbot, have enabled support for ARI, but the Ruby gem did not yet support this feature. Rather than building a custom solution, we decided to enable support for the ARI extension directly in the client.

Let’s Encrypt’s guide to integrating ARI provided the necessary roadmap, and the implementation was completed with one PR. This contribution means that not only Shopify, but also the wider Ruby community, can benefit from the ARI extension.

Deployment and ARI at scale

Once we shipped the gem support, integrating ARI into our certificate management system was straightforward. Instead of checking a static 30-day threshold, we now query the ARI endpoint and use the suggested renewal window as the gate for initiating renewals. Those dates are stored alongside the certificate upon its initial provisioning.

The updated Ruby gem provides a method for fetching renewal information:

renewal_info = client.renewal_info(certificate: existing_certificate_pem)

This method generates an ARI certificate identifier that can be used when making the API call. The client also includes a helper method, suggested_renewal_time, which chooses a random time between the returned start and end dates. The certificate identifier can be passed to the new_order method via the replaces key, which can grant a higher priority or bypass rate limits for renewals occurring during the window, depending on the CA’s policies.

Critically, Shopify also regularly polls the ARI endpoint for updated renewal timestamps. This allows our systems to rely on those timestamps as the primary renewal timing logic and removes the need for inflexible hard-coded expiry thresholds. This becomes the mechanism that LetsEncrypt uses to dynamically change the renewal time due to a revocation event.

Results and rewards

Since enabling the use of the ARI extension, our certificate management system has become significantly more robust. Shopify now delegates the responsibility of determining renewal timing to Let’s Encrypt. The ARI extension has proven to be an impactful infrastructure improvement and the benefits gained are immediate. These benefits, alongside fewer manual interventions, are the operational success story:

• Future-proofing: We gained resilience against any future certificate lifetime changes and mass revocation events without needing code updates—ensuring our renewal logic is flexible.

• Optimized load: We directly benefit from the CA’s coordinated load balancing provided by the suggested renewal window, eliminating local randomness issues and the need for complex global coordination.

• Revocation readiness: ARI allows systems to quickly detect and respond to revocation events when an urgent renewal is necessary, well before certificates get close to their due dates.

• Simple implementation: The extension is mature (RFC 9773) and the implementation is straightforward, providing simplified renewal logic and CA-optimized timing.

• Good citizenship: Anyone using ARI helps the CA optimize its infrastructure, and contributes to better aggregate behavior across the entire ecosystem.

If you’re still relying on static renewal thresholds, give ARI a look—Shopify wholeheartedly encourages all ACME users and client developers to adopt the ARI extension.

This was also posted on EFF’s blog.

As we announced earlier this year, Let’s Encrypt now issues IP address and six-day certificates to the general public. The Certbot team at the Electronic Frontier Foundation has been working on two improvements to support these features: the --preferred-profile flag released last year in Certbot 4.0, and the --ip-address flag, new in Certbot 5.3. With these improvements together, you can now use Certbot to get those IP address certificates!

If you want to try getting an IP address certificate using Certbot, install version 5.4 or higher (for webroot support with IP addresses), and run this command:

sudo certbot certonly --staging \
  --preferred-profile shortlived \
  --webroot \
  --webroot-path <filesystem path to webserver root> \
  --ip-address <your ip address>

Two things of note:

• This will request a non-trusted certificate from the Let’s Encrypt staging server. Once you’ve got things working the way you want, run without the --staging flag to get a publicly trusted certificate.

• This requests a certificate with Let’s Encrypt’s “shortlived” profile, which will be good for 6 days. This is a Let’s Encrypt requirement for IP address certificates.

As of right now, Certbot only supports getting IP address certificates, not yet installing them in your web server. There’s work to come on that front. In the meantime, edit your webserver configuration to load the newly issued certificate from /etc/letsencrypt/live/<ip address>/fullchain.pem and /etc/letsencrypt/live/<ip address>/privkey.pem.

The command line above uses Certbot’s “webroot” mode, which places a challenge response file in a location where your already-running webserver can serve it. This is nice since you don’t have to temporarily take down your server.

There are two other plugins that support IP address certificates today: --manual and --standalone. The manual plugin is like webroot, except Certbot pauses while you place the challenge response file manually (or runs a user-provided hook to place the file). The standalone plugin runs a simple web server that serves a challenge response. It has the advantage of being very easy to configure, but has the disadvantage that any running webserver on port 80 has to be temporarily taken down so Certbot can listen on that port. The nginx and apache plugins don’t yet support IP addresses.

You should also be sure that Certbot is set up for automatic renewal. Most installation methods for Certbot set up automatic renewal for you. However, since the webserver-specific installers don’t yet support IP address certificates, you’ll have to set a --deploy-hook that tells your webserver to load the most up-to-date certificates from disk. You can provide this --deploy-hook through the certbot reconfigure command using the rest of the flags above.

We hope you enjoy using IP address certificates with Let’s Encrypt and Certbot, and as always if you get stuck you can ask for help in our Community Forum.

The good news for subscribers is that you don’t need any changes to your rate limits, whether you are using our default limits or have requested an override. Our rate limits affect issuance for new domain names (or groups of domain names), but renewals are exempt. So, for instance, if you are managing a set of 15,000 certificates that you continually renew, and create 250 new certificates (with new domain names) each day, you will be well within our limits both before and after the transition. The 250 new certificates daily will still be well under our New Orders per Account limit of 300 per day. And the 15,000 existing certificates will continue to be unaffected by rate limits, whether your ACME client is renewing them every sixty days or every thirty.

We are implementing support for a new ACME challenge type, DNS-PERSIST-01, based on a new IETF draft specification. As the name implies, it uses DNS as the validation mechanism, but replaces repeated demonstrations of control with a persistent authorization record bound to a specific ACME account and CA. The draft describes this method as being “particularly suited for environments where traditional challenge methods are impractical, such as IoT deployments, multi-tenant platforms, and scenarios requiring batch certificate operations”.

DNS-01 Proves Control Repeatedly

With DNS-01, validation relies on a one-time token generated by us. Your ACME client publishes a TXT record containing that token at _acme-challenge.<YOUR_DOMAIN>, and we query DNS to confirm that it matches the expected value. Because each authorization requires a new token, DNS updates become part of the issuance workflow. The benefit is that each successful validation provides fresh proof that you currently control DNS for the name being issued.

In practice, this often means DNS API credentials live somewhere in your issuance pipeline, validation attempts involve waiting for DNS propagation, and DNS changes happen frequently — sometimes many times per day in large deployments. Many subscribers accept these tradeoffs, but others would prefer to keep DNS updates and sensitive credentials out of their issuance path.

DNS-PERSIST-01 Authorizes Persistently

DNS-PERSIST-01 approaches validation differently. Instead of publishing a new challenge record for each issuance, you publish a standing authorization in the form of a TXT record that identifies both the CA and the specific ACME account you authorize to issue for this domain.

For the hostname example.com, the record would live at _validation-persist.example.com:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890"
)

Once this record exists, it can be reused for new issuance and all subsequent renewals. Operationally, this removes DNS changes from the critical path.

Security and Operational Tradeoffs

With DNS-01, the sensitive asset is DNS write access. In many deployments, DNS API credentials are distributed throughout issuance and renewal pipelines, increasing the number of places an attacker might compromise them. DNS-PERSIST-01 instead binds authorization directly to an ACME account, allowing DNS write access to remain more tightly controlled after initial setup. The tradeoff is that, because the authorization record persists over time, protecting the ACME account key becomes the central concern.

Controlling Scope and Lifetime

DNS-PERSIST-01 also introduces explicit scope controls. Without additional parameters, authorization applies only to the validated Fully Qualified Domain Name (FQDN) and remains valid indefinitely.

Wildcard Certificates

Adding policy=wildcard broadens the authorization scope to include the validated FQDN, wildcard certificates such as *.example.com, and subdomains whose suffix matches the validated FQDN:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890;"
  " policy=wildcard"
)

Optional Expiration

Subscribers who aren’t comfortable with authorization persisting indefinitely can include an optional persistUntil timestamp. This limits how long the record may be used for new validations, but also means it must be updated or replaced before it expires. Anyone using this feature should ensure they have adequate reminders or monitoring in place so that authorization does not expire unexpectedly. The timestamp is expressed as UTC seconds since 1970-01-01:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890;"
  " persistUntil=1767225600"
)

Authorizing Multiple CAs

Multiple CAs can be simultaneously authorized by publishing multiple TXT records at _validation-persist.<YOUR_DOMAIN>, each containing the issuer-domain-name of the CA you intend to authorize. During validation, each CA queries the same DNS label and evaluates only the records that match its own issuer-domain-name.

Rollout Timeline

The CA/Browser Forum ballot SC-088v3, defining “3.2.2.4.22 DNS TXT Record with Persistent Value”, passed unanimously in October 2025, and the IETF ACME working group adopted the draft that same month. While the document remains an active IETF draft, the core mechanisms described here are not expected to change substantially.

Support for the draft specification is available now in Pebble, a miniature version of Boulder, our production CA software. Work is also in progress on a lego-cli client implementation to make it easier for subscribers to experiment with and adopt. Staging rollout is planned for late Q1 2026, with a production rollout targeted for some time in Q2 2026.

In a recent conversation with a Let’s Encrypt subscriber, we asked them to guess how many people work at ISRG, the nonprofit behind Let’s Encrypt (and Prossimo and Divvi Up). Their guess was about 100; they’d overestimated by 72.5 people. We’re a pretty small team, and we get a lot done, but most of that work is entirely remote, distributed, and automated. 

That is a big part of what makes FOSDEM special. For the last few years, we’ve had a stand at this annual conference in Belgium, where a few folks from our team have the opportunity to speak directly with thousands of conference-goers. We continue to learn so much from these conversations! 

That’s where the “Hello” part of this blog post comes in. At this year’s FOSDEM, we met so many Let’s Encrypt subscribers, and each of them has a unique relationship to Let’s Encrypt. We were pleasantly surprised by how many people told us they were using IP-address certificates, a new option we just made generally available in December. We had a lot of conversations about our plans to shorten certificate lifetimes. There were a few folks who asked about S/MIME (still no plans to do that). We invited people to continue to stay in touch by signing up for our newsletter. 

The most meaningful part of FOSDEM is being able to say “thank you”. Our goal in starting Let’s Encrypt was to improve security and privacy for people using the Internet, but that could not be achieved without the now millions of folks who decided to get a certificate. Our impact is predicated on this symbiotic exchange. While we were only able to directly express our gratitude to a few thousand people at FOSDEM, it was a reminder of how important the community is.

Update: March 11, 2026

If you use Certbot, see Six-Day and IP Address Certificates Available in Certbot for details on requesting these certificates.

Short-lived and IP address certificates are now generally available from Let’s Encrypt. These certificates are valid for 160 hours, just over six days. In order to get a short-lived certificate subscribers simply need to select the ‘shortlived’ certificate profile in their ACME client.

Short-lived certificates improve security by requiring more frequent validation and reducing reliance on unreliable revocation mechanisms. If a certificate’s private key is exposed or compromised, revocation has historically been the way to mitigate damage prior to the certificate’s expiration. Unfortunately, revocation is an unreliable system so many relying parties continue to be vulnerable until the certificate expires, a period as long as 90 days. With short-lived certificates that vulnerability window is greatly reduced.

Short-lived certificates are opt-in and we have no plan to make them the default at this time. Subscribers that have fully automated their renewal process should be able to switch to short-lived certificates easily if they wish, but we understand that not everyone is in that position and generally comfortable with this significantly shorter lifetime. We hope that over time everyone moves to automated solutions and we can demonstrate that short-lived certificates work well.

Our default certificate lifetimes will be going from 90 days down to 45 days over the next few years, as previously announced.

IP address certificates allow server operators to authenticate TLS connections to IP addresses rather than domain names. Let’s Encrypt supports both IPv4 and IPv6. IP address certificates must be short-lived certificates, a decision we made because IP addresses are more transient than domain names, so validating more frequently is important. You can learn more about our IP address certificates and the use cases for them from our post announcing our first IP Certificate.

We’d like to thank the Open Technology Fund and Sovereign Tech Agency, along with our Sponsors and Donors, for supporting the development of this work.

This letter was originally published in our 2025 Annual Report.

This year was the 10th anniversary of Let’s Encrypt. We’ve come a long way! Today we’re serving more than 700 million websites, issuing ten million certificates on some days. Most importantly, when we started 39% of page loads on the Internet were encrypted. Today, in many parts of the world, over 95% of all page loads are encrypted. We can’t claim all the credit for that, but we’re proud of the leading role we played. Being able to help ISRG and Let’s Encrypt get to where we are today has been the opportunity of a lifetime for me.

There’s more I could talk about from the past ten years, but this 10th year was about as good as any before it so I want to focus on our most recent work. I’ll get the headline for 2025 out right away: over the past year we went from serving 492 million websites to 762 million. That’s a 50% increase in a single year, equivalent to the growth we saw over our first six years of existence combined. Our staff did an amazing job accommodating the additional traffic.

I’m also particularly proud of the things we did to improve privacy this year, across all of our projects.

At the start of 2025 we were serving over four billion Online Certificate Status Protocol (OCSP) requests per day. That’s 180 million per hour, or 50,000 per second. OCSP has been an important mechanism for providing certificate revocation information for a long time, but the way it works is bad for privacy. It requires browsers to check with certificate authorities for every website they visit, which is basically providing your browsing history to third parties. Let’s Encrypt never held onto that data; it got dropped immediately. However, there is no way to know if that was standard practice across the industry, and even well-intentioned CAs could make a mistake or be compelled to save that data. It was a system ripe for abuse, so we decided to become the first major CA to turn off our OCSP service. We couldn’t be sure what the full impact would be, but this was a way in which the Internet needed to get better. In August of 2025 we turned off our OCSP service. There was no major fallout and we haven’t looked back.

Another big privacy-focused change we made to Let’s Encrypt in 2025 was no longer storing subscriber email addresses in our CA database, associated with issuance data. In June of this year we stopped adding the optional email addresses that subscribers send to our database, and we deleted the millions of email addresses that had accumulated over the years. Making this change was not an easy thing to decide to do—it limits our ability to contact subscribers and we had to turn off our expiration reminder email service—but we feel the ecosystem has grown enough over the past ten years that the privacy implications of holding onto the email addresses outweighed the utility.

Privacy was at the forefront for the folks at ISRG researching human digital identity as well. They have been hard at work on an implementation of the Anonymous Credentials from ECDSA scheme, also known as Longfellow. This is a cryptographic library that can be used in digital identity management, including things like digital wallets, in order to improve privacy when sharing credentials. Digital identity systems should have strong privacy and compatibility requirements, but such requirements pose challenges that existing digital credential technologies are going to struggle to meet. New schemes such as Longfellow aim to address these challenges, bringing privacy improvements to systems that need to work with existing cryptographic hardware. This is exciting stuff, but not easy to build (so much math!)—watching our talented engineers make progress has been thrilling.

The last example of great privacy work I want to highlight from 2025 is our Prossimo project’s work towards encrypted recursive-to-authoritative DNS. Prossimo is focused on bringing memory safety to critical software infrastructure, but sometimes that dovetails nicely with other initiatives. DNS queries are fundamental to the operation of the Internet. Without getting into the details here too much, there are basically two types of DNS queries: stub-to-recursive and recursive-to-authoritative. A lot of work has gone into encrypting stub queries over the past decade, mostly through DNS over HTTPS (DoH) initiatives. Authoritative queries, however, remain almost entirely unencrypted. This is a particular problem for Certificate Authorities like Let’s Encrypt. During 2025, our Prossimo project started work on changing that, investing heavily in encrypted authoritative resolution by implementing RFC 9539 Unilateral Opportunistic Deployment of Encrypted Recursive‑to‑Authoritative DNS and other related improvements in Hickory DNS. Once this is ready, early in 2026, Hickory DNS will be a high performance and memory safe option that DNS operators can use to start making and receiving encrypted authoritative DNS queries. It can also be used for integration testing with other DNS implementations.

It’s wonderful, and a real responsibility, to be able to have this kind of positive impact on the lives of everyone using the Internet. Charitable contributions from people like you and organizations around the world make what we do possible. We are particularly grateful to Jeff Atwood, Betsy Burton, and Stina Ehrensvärd for their special gifts this year. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations. If you’re one of those people, thank you. If you’re considering becoming a supporter, I hope this annual report will make the case that we’re making every dollar count.

Every year we aim to make the dollars entrusted to us go as far as possible, and next year will be no exception.

In 2023, we marked the tenth anniversary of the creation of our nonprofit, Internet Security Research Group, which continues to host Let’s Encrypt and other public benefit infrastructure projects. Now, in honor of the tenth anniversary of Let’s Encrypt’s public certificate issuance and the start of the general availability of our services, we’re looking back at a few milestones and factors that contributed to our success.

Growth

A conspicuous part of Let’s Encrypt’s history is how thoroughly our vision of scalability through automation has succeeded.

In March 2016, we issued our one millionth certificate. Just two years later, in September 2018, we were issuing a million certificates every day. In 2020 we reached a billion total certificates issued and as of late 2025 we’re frequently issuing ten million certificates per day. We’re now on track to reach a billion active sites, probably sometime in the coming year. (The “certificates issued” and “certificates active” metrics are quite different because our certificates regularly expire and get replaced.)

The steady growth of our issuance volume shows the strength of our architecture, the validity of our vision, and the great efforts of our engineering team to scale up our own infrastructure. It also reminds us of the confidence that the Internet community is placing in us, making the use of a Let’s Encrypt certificate a normal and, dare we say, boring choice. But I often point out that our ever-growing issuance volumes are only an indirect measure of value. What ultimately matters is improving the security of people’s use of the web, which, as far as Let’s Encrypt’s contribution goes, is not measured by issuance volumes so much as by the prevalence of HTTPS encryption. For that reason, we’ve always emphasized the graph of the percentage of encrypted connections that web users make (here represented by statistics from Firefox).

(These graphs are snapshots as of the date of this post; a dynamically updated version is found on our stats page.) Our biggest goal was to make a concrete, measurable security impact on the web by getting HTTPS connection prevalence to increase—and it’s worked. It took five years or so to get the global percentage from below 30% to around 80%, where it’s remained ever since. In the U.S. it has been close to 95% for a while now.

A good amount of the remaining unencrypted traffic probably comes from internal or private organizational sites (intranets), but other than that we don’t know much about it; this would be a great topic for Internet security researchers to look into.

We believe our present growth in certificate issuance volume is essentially coming from growth in the web as a whole. In other words, if we protect 20% more sites over some time period, it’s because the web itself grew by 20%.

A few milestones

We’ve blogged about most of Let’s Encrypt’s most significant milestones as they’ve happened, and I invite everyone in our community to look over those blog posts to see how far we’ve come. We’ve also published annual reports for the past seven years, which offer elegant and concise summaries of our work.

As I personally think back on the past decade, just a few of the many events that come to mind include:

• Telling the world about the project in November 2014

• Our first certificate issuance in September 2015

• Our one millionth certificate in March 2016, then our 100 millionth certificate in June 2017, and then our billionth certificate in 2020

• Along the way, first issuing one million certificates in a single day (in September 2018), significantly contributed to by the SquareSpace and Shopify Let’s Encrypt integrations

• Just at the end of September 2025, we issued more than ten million certificates in a day for the first time.

We’ve also periodically rolled out new features such as internationalized domain name support (2016), wildcard support (2018), and short-lived and IP address (2025) certificates. We’re always working on more new features for the future.

There are many technical milestones like our database server upgrades in 2021, where we found we needed a serious server infrastructure boost because of the tremendous volumes of data we were dealing with. Similarly, our original infrastructure was using Gigabit Ethernet internally, and, with the growth of our issuance volume and logging, we found that our Gigabit Ethernet network eventually became too slow to synchronize database instances! (Today we’re using 25-gig Ethernet.) More recently, we’ve experimented with architectural upgrades to our ever-growing Certificate Transparency logs, and decided to go ahead with deploying those upgrades—to help us not just keep up with, but get ahead of, our continuing growth.

These kinds of growing pains and successful responses to them are nice to remember because they point to the inexorable increase in demands on our infrastructure as we’ve become a more and more essential part of the Internet. I’m proud of our technical teams which have handled those increased demands capably and professionally.

I also recall the ongoing work involved in making sure our certificates would be as widely accepted as possible, which has meant managing the original cross-signature from IdenTrust, and subsequently creating and propagating our own root CA certificates. This process has required PKI engineering, key ceremonies, root program interactions, documentation, and community support associated with certificate migrations. Most users never have reason to look behind the scenes at our chains of trust, but our engineers update it as root and intermediate certificates have been replaced. We’ve engaged at the CA/B Forum, IETF, and in other venues with the browser root programs to help shape the web PKI as a technical leader.

As I wrote in 2020, our ideal of complete automation of the web PKI aims at a world where most site owners wouldn’t even need to think about certificates at all. We continue to get closer and closer to that world, which creates a risk that people will take us and our services for granted, as the details of certificate renewal occupy less of site operators’ mental energy. As I said at the time,

When your strategy as a nonprofit is to get out of the way, to offer services that people don’t need to think about, you’re running a real risk that you’ll eventually be taken for granted. There is a tension between wanting your work to be invisible and the need for recognition of its value. If people aren’t aware of how valuable our services are then we may not get the support we need to continue providing them.

I’m also grateful to our communications and fundraising staff who help make clear what we’re doing every day and how we’re making the Internet safer.

Recognition of Let’s Encrypt

Our community continually recognizes our work in tangible ways by using our certificates—now by the tens of millions per day—and by sponsoring us.

We were honored to be recognized with awards including the 2022 Levchin Prize for Real-World Cryptography and the 2019 O’Reilly Open Source Award. In October of this year some of the individuals who got Let’s Encrypt started were honored to receive the IEEE Cybersecurity Award for Practice.

We documented the history, design, and goals of the project in an academic paper at the ACM CCS ‘19 conference, which has subsequently been cited hundreds of times in academic research.

Our initial sponsors

Ten years later, I’m still deeply grateful to the five initial sponsors that got Let’s Encrypt off the ground - Mozilla, EFF, Cisco, Akamai, and IdenTrust. When they committed significant resources to the project, it was just an ambitious idea. They saw the potential and believed in our team, and because of that we were able to build the service we operate today.

IdenTrust: A critical technical partner

I’d like to particularly recognize IdenTrust, a PKI company that worked as a partner from the outset and enabled us to issue publicly-trusted certificates via a cross-signature from one of their roots. We would simply not have been able to launch our publicly-trusted certificate service without them. Back when I first told them that we were starting a new nonprofit certificate authority that would give away millions of certificates for free, there wasn’t any precedent for this arrangement, and there wasn’t necessarily much reason for IdenTrust to pay attention to our proposal. But the company really understood what we were trying to do and was willing to engage from the beginning. Ultimately, IdenTrust’s support made our original issuance model a reality.

Conclusion

I’m proud of what we have achieved with our staff, partners, and donors over the past ten years. I hope to be even more proud of the next ten years, as we use our strong footing to continue to pursue our mission to protect Internet users by lowering monetary, technological, and informational barriers to a more secure and privacy-respecting Internet.

Let’s Encrypt is a project of the nonprofit Internet Security Research Group, a 501(c)(3) nonprofit. You can help us make the next ten years great as well by donating or becoming a sponsor.

This change is being made along with the rest of the industry, as required by the CA/Browser Forum Baseline Requirements, which set the technical requirements that we must follow. All publicly-trusted Certificate Authorities like Let’s Encrypt will be making similar changes. Reducing how long certificates are valid for helps improve the security of the internet, by limiting the scope of compromise, and making certificate revocation technologies more efficient.

We are also reducing the authorization reuse period, which is the length of time after validating domain control that we allow certificates to be issued for that domain. It is currently 30 days, which will be reduced to 7 hours by 2028.

Timeline of Changes

To minimize disruption, Let’s Encrypt will roll this change out in multiple stages. We will use ACME Profiles to allow you control over when these changes take effect. They are configured in your ACME client. For more information, see our blog post announcing them.

Changes will be deployed to our staging environment approximately one month before the production dates below.

• May 13, 2026: Let’s Encrypt will switch our tlsserver ACME profile to issue 45-day certificates. This profile is opt-in and can be used by early adopters and for testing.
• February 10, 2027: Let’s Encrypt will switch our default classic ACME profile to issuing 64-day certificates with a 10-day authorization reuse period. This will affect all users who have not opted into the tlsserver or shortlived (6-day) profiles.
• February 16, 2028: We will further update the classic profile to issue 45-day certificates with a 7 hour authorization reuse period.

These dates are when the change takes effect for new certificates, so Let’s Encrypt users will see the reduced certificate validity period at their next renewal after these dates.

Action Required

Most users of Let’s Encrypt who automatically issue certificates will not have to make any changes. However, you should verify that your automation is compatible with certificates that have shorter validity periods.

To ensure your ACME client renews on time, we recommend using ACME Renewal Information (ARI). ARI is a feature we’ve introduced to help clients know when they need to renew their certificates. Consult your ACME client’s documentation on how to enable ARI, as it differs from client to client. If you are a client developer, check out this integration guide.

If your client doesn’t support ARI yet, ensure it runs on a schedule that is compatible with 45-day certificates. For example, renewing at a hardcoded interval of 60 days will no longer be sufficient. Acceptable behavior includes renewing certificates at approximately two thirds of the way through the current certificate’s lifetime.

Manually renewing certificates is not recommended, as it will need to be done more frequently with shorter certificate lifetimes.

We also recommend that you make sure your systems have sufficient monitoring in place to alert appropriately if certificates aren’t renewed when expected. There are many available options, some of which are documented on our Monitoring Service Options page.

Making Automation Easier with a new DNS Challenge Type

For many of our users, the hardest part of automatically issuing certificates is proving domain control. Reducing certificate lifetimes and the authorization reuse period will make users need to demonstrate control more often.

All validation methods today require that the ACME client have live access to your infrastructure, either to serve the correct HTTP-01 token, perform the right TLS-ALPN-01 handshake, or update the right DNS-01 TXT record. For a long time, people have wanted a way to run an ACME client without granting it access to these sensitive systems.

These challenges are why we are working with our partners at the CA/Browser Forum and IETF to standardize a new validation method called DNS-PERSIST-01. The key advantage of this new method is that the DNS TXT entry used to demonstrate control does not have to change every renewal.

This means you can set up the DNS entry once and begin automatically renewing certificates without needing a way to automatically update DNS. This should allow even more people to automate their certificate renewals. It will also reduce reliance on authorization reuse, since the DNS records can stay unchanged without any further ACME client involvement.

We expect DNS-PERSIST-01 to be available in 2026, and will have more to announce soon.

Keep Up to Date

Additional updates, reminders, and other changes will be shared on our technical updates mailing list. Subscribe to keep up-to-date with these and all other upcoming changes. If you have any questions, please ask on our community forum. If you want to read more about the work happening at Let’s Encrypt and our other projects, check out our Annual Report, which was published today.

The two new roots look very similar to our existing roots. The new ISRG Root YR has an RSA 4096 key and is valid for twenty years, just like ISRG Root X1. Similarly, the new ISRG Root YE has an ECDSA P-384 key, just like ISRG Root X2. We’ve made a few adjustments (for example, replacing “Internet Security Research Group” with “ISRG” to save a few bytes), but nothing major. Each of these new roots is intended to eventually replace its corresponding predecessor, and to that end we have cross-signed the new roots from the old ones.

The six new intermediates consist of three intermediates each issued from each of the two new roots. ISRG Root YE has issued the YE1, YE2, and YE3 intermediates, while ISRG Root YR has issued the YR1, YR2, and YR3 intermediates. These have two key differences from our current crop of issuing intermediates. First, their names: we’re now numbering intermediates under each root sequentially, rather than using the same numbering sequence across all intermediates. This makes it slightly easier to keep track of which intermediates are currently in use, and prepares for a possible post-quantum future where we have additional key types.

Second and more importantly: these intermediates do not contain the “TLS Web Client Authentication” Extended Key Usage. This means that these intermediates cannot issue end-entity certificates containing that EKU. As we’ve already announced, we will be phasing out issuance of tlsClientAuth certificates in 2026 due to a root program requirement. Until that time, we will only be using the new hierarchy to issue certificates under the “tlsserver” and “shortlived” profiles, which already omit that EKU. After the tlsClientAuth deprecation is complete, we will shift to using the new intermediates for all issuance.

If you’re requesting the tlsserver or shortlived profile, you can expect to see issuance from (the Staging equivalent of) the new hierarchy as of today. We expect to make the same change in our Production environment next month. As before, each issuance will choose which intermediate to use at random, to discourage intermediate key pinning.

We’ll be submitting the new roots for inclusion in the Apple, Chrome, Microsoft, Mozilla, and other root programs shortly thereafter. We look forward to updating you again when the new hierarchy is officially included in those trust stores!

Seth Schoen was an early contributor to Let's Encrypt through his work at the Electronic Frontier Foundation. He's also one of the longest standing participants in the Let's Encrypt community support forum, so we asked him to offer his thoughts on the role and impact of the forum as a resource for our users. Thank you for your many years of expertise and participation, Seth!

Along with the tenth anniversary of Let’s Encrypt’s first certificate, we’re also celebrating ten years of the Let’s Encrypt Community Forum, which has played a vital role in the Let’s Encrypt community.

It’s been the first stop for end users with technical questions. It’s been the main way that client developers got help with ACME and debugged compatibility issues. It’s been the place where Let’s Encrypt staff made technical announcements and got immediate feedback from affected parties.

It’s happened in many different languages (including official French, Spanish, and Portuguese categories, use of numerous volunteers’ native languages, as well as many successful conversations via machine translation). For example, people have gotten help in Dutch, Russian, German, and Chinese.

Thousands of volunteers have provided help and successfully helped tens of thousands of users get their certificates. Occasionally, they’ve also reported bugs in client software, documentation, or even the Let’s Encrypt service itself. Many times a responsible developer was there to interact directly with the bug reporter.

Here are the monthly pageviews from the creation of the Community Forum until the present day:

Seeing the results of one’s efforts

The most common kind of interaction on the forum is one in which a Let’s Encrypt end user shows up with some kind of problem, usually an inability to get or renew a certificate. In most cases, if the user is willing to answer some questions, the community is ultimately able to resolve the problem.

I’ve often compared the satisfaction of helping users on the Let’s Encrypt forum to what I felt while installing bike lights at a local cycling organization’s bike light giveaway event. In both cases, one could engage for a few minutes with someone, maybe deal with some unanticipated oddities (extra-thick handlebars? an unusual seat post or cargo rack? a strange DNS setup? an unusual Apache configuration?). Usually, this would lead to a concrete practical improvement in safety afterward (the blinking red tail light freshly installed on someone’s bike, or the lovely https:// prefix or padlock icon newly visible in the browser bar when browsing to a visitor’s website). It’s common in the Internet security world not to be able to see or appreciate how what we do helps people, so “we just helped make connections to your specific web site more secure” is especially satisfying.

Tangible safety upgrades!

A channel between Let’s Encrypt staff and the community

Let’s Encrypt describes itself as “free, automated, and open”; part of that openness consists of its use of open standards (ACME) and open source CA software (Boulder). Part of it is also about how much of the CA’s thinking happens in public! One example (of dozens) is that the September 2025 Let’s Encrypt root ceremony was discussed ahead of time on the forum, starting back in July, with the plans and details all open for discussion and review. Let’s Encrypt staff have even asked the community for feedback on how production and testing certificates ought to be named!

In other cases, like when there was new functionality announced, or substantive technical changes affecting certificate issuance, or proposed rate limit changes, or problems requiring mass revocation, or expiring root certificates, Let’s Encrypt staff were available talking about all the details and directly answering end users’ questions. Again, there are lots of other examples, where changes large or small got announced, proposed, or discussed on the forum, with Let’s Encrypt’s own experts engaging with the community.

Final thoughts and thanks

The forum runs on Discourse, which has continued to be an effective choice of forum software for the community. Discourse has nice technical and user interface features, but it’s also pleasantly unobtrusive. The Discourse company has also generously been donating pro bono hosting for the forum for many years, and, of course, it uses a Let’s Encrypt certificate.

The volunteers on the Let’s Encrypt forum have made a huge contribution to Let’s Encrypt’s success. It’s easy to imagine that many users might have given up on Let’s Encrypt in frustration were it not for the efforts of dedicated volunteers to draw out the necessary details, notice the relevant issues, and patiently explain concepts that were confusing people. There are also volunteer moderators who’ve worked hard to keep the forum on track, stop spam, and defuse distracting conflicts. Thanks to all of you.

Several software projects have been informed by discussions and issues on the forum, as developers there found opportunities to help large numbers of users. I would particularly highlight Alex Zorin’s Let’s Debug and Jonathan Griffin’s CertSage as examples in this category. Let’s Debug runs a series of practical live tests on a specified site to help users figure out why certificate issuance is failing, giving useful explanations of many of the most common failure reasons. CertSage is a client meant for users who have hosting plans without native support for Let’s Encrypt, and without administrative access—but where they can run PHP scripts. These projects grew out of Alex’s and Jonathan’s experiences helping users on the forum and seeing the kinds of issues that came up repeatedly there. Joona Hoikkala’s helpful acme-dns, which helps subscribers complete the ACME DNS challenge with a dedicated service instead of using an existing DNS server, also helped respond to a common issue that brought many people to the forum.

I would also like to thank Jacob Hoffman-Andrews for his early efforts to set a positive and welcoming tone on the forum. Jacob and other forum administrators always reminded the community to be patient and welcoming to each visitor, emphasizing that the forum was many users’ first interaction with Let’s Encrypt, and that users ought to be welcomed regardless of their expertise or background (and regardless of whether their questions had been asked before by others).

Since the ACME protocol was first published as RFC 8555, the IETF ACME working group has remained active, defining various extensions to the original ACME protocol, initiated either by Let’s Encrypt or by colleagues from other organizations. For example, ACME WG documents have specified how to validate kinds of identifiers other than domain names, making it possible to use ACME to issue certificates for IP addresses, or even in PKIs other than the web PKI.

The publication of RFC 9773 is the culmination of a process that began in September 2021 with the first ARI draft. Along the way, numerous colleagues from Let’s Encrypt and elsewhere (thanked individually at the end of this post) contributed to the ARI specification and helped improve it.

Why implement ARI?

This is a good opportunity to remind our community about ARI and how implementing it can help users. If you’re an ACME client user, you may want to check the documentation for your client to see if it has implemented ARI yet. New functionality like this is a great reason to make sure you’re using up-to-date ACME client software. If you’re a client developer, questions about ARI implementation are welcome in the Community Forum’s Client Dev category.

Sometimes certificate authorities, including Let’s Encrypt, may perform mass revocations of an entire group or category of certificates. This most often happens when someone discovers that a certificate authority has made a mistake in how it validates or issues certificates, or has made a misstatement in how it describes its policies and procedures. In this case, the CA is required to revoke the affected certificates. This may happen through absolutely no fault of the subscribers. For example, in January 2022, we had to revoke approximately two million certificates due to a technical error in our validation processes.

When we have to revoke certificates, we want to make sure that the websites using those certificates don’t experience issues. That means those websites need to re-request issuance and install new certificates. Since CAs are sometimes required to revoke certificates on a 24 hour timeline or a 5 day timeline (depending on the nature of the incident), a process that relies on manual intervention from system administrators won’t reach most websites in time.

ARI allows a certificate authority to advise a client to perform an early renewal of a certificate that the client would have anticipated did not need to be renewed yet, broadly because the CA knows that an early renewal is helpful, or necessary, in particular circumstances. In the mass revocation scenario, this allows ARI-aware clients to avoid outages due to certificate invalidity, because they can replace their certificates even before the revocation occurs.

Of course, we and other certificate authorities work diligently to prevent mass revocation events. We’re encouraging ARI implementation as a form of emergency preparedness that can significantly mitigate the impact of this kind of problem, if and when it happens.

ARI also provides features to reduce the impact of load spikes where too many clients request certificates in a short period of time. Let’s Encrypt doesn’t need to use ARI for this today, because other improvements in popular clients’ renewal practices have already sufficiently smoothed out our load spikes. Even so, this will be a valuable ability for all ACME CAs to have available in the long term to better manage emergencies and disruptions.

On the server side, we added support for the ARI draft specification to our Boulder CA software in late 2021, so the Let’s Encrypt CA has supported ARI for some time. If you are implementing ARI in your own client, the Pebble ACME test-bed also supports ARI so you can test against that implementation.

Thanks

Thanks to all of the people who contributed to this process at the ACME WG and elsewhere, including:  Roland Shoemaker and Jacob Hoffman-Andrews for coming up with the initial idea of ARI and for helping me learn the IETF process; Samantha Frank, Matt Holt, Ilari Liusvaara, and Wouter Tinus for contributing client implementations; Freddy Zhang for contributing an independent server implementation; and Rob Stradling, Andrew Ayer, and J.C. Jones for providing meaningful feedback and suggestions that significantly improved this specification.

Finally, our congratulations also to Q Misell for the recent publication of RFC 9799, another ACME WG document that went through the standards process alongside ARI.

NGINX and Let's Encrypt share a common vision of an open and secure web. Now, with built-in support for ACME, the world's most popular web server, reverse proxy and ingress controller for Kubernetes can simplify certificate management for everyone. From the home lab to scaled-out, mission-critical enterprise deployments.

Our ideal has always been that server software could get and renew Let’s Encrypt certificates automatically, with minimal human intervention.

Over time, more and more web servers and hosting environments have become capable of that, often via native ACME and Let’s Encrypt integrations that allow users to manage certificates without third-party tools. On August 12, the popular open source web server NGINX announced support for ACME with their official ngx_http_acme module (implemented with memory safe Rust code!).

NGINX is one of the most widely used pieces of software for operating a web server or proxy. In directly supporting ACME, NGINX joins other web servers like Traefik, Caddy and Apache httpd that can directly take advantage of certificates from Let’s Encrypt and other ACME Certificate Authorities. NGINX’s new support for ACME, together with other servers, means a significant majority of sites can now have native ACME support. Many other software environments, hosting plans, and devices also offer built-in official support for ACME.

Users have a wide range of choices to achieve integrations for their particular hosting environments. Native support in web servers is an option alongside third-party clients that can integrate with many of those same web servers. Native support typically provides more seamless integration, and it’s less work for operators since they don’t have to manage a separate ACME client. Having more tools that take care of certificates automatically helps us achieve our goal of encrypting more and more of the web, while reducing the amount of time and energy site operators have to spend.

Other project developers interested in integrating ACME more directly can read about the ACME protocol, find existing ACME library implementations and other reusable software components, and join the Client Dev conversation on our Community Forum.

We’d like to thank NGINX and their parent company, F5, for their sponsorship of Let’s Encrypt. This financial support helps us provide a trusted and reliable service to nearly 700 million websites.

Update, August 18, 2025

We have updated the read-only and shutdown dates to ensure that our new Static CT API logs are fully trusted by browsers before switching Oak to read-only in order to avoid any disruption.

Let’s Encrypt operates two types of Certificate Transparency (“CT”) logs—some implement the original RFC 6962 API, and some that implement the newer Static CT API. Today we are announcing that on November 30, 2025, we will make our RFC 6962 logs read-only. Past that date, we will write only to our Static CT logs. On February 28, 2026, we will entirely shut down our RFC 6962 logs.

End users (consumers or relying parties) of Web PKI certificates do not need to take any action. The work that needs to be done to make this transition will be handled by Let’s Encrypt and the browsers.

RFC 6962 is from June of 2013 and describes the original version of CT. It was a revolutionary upgrade for transparency in the Web PKI, ultimately allowing anyone to monitor issuance from all certificate authorities. Over time, though, growth in certificate issuance volume has revealed that the original CT design doesn’t scale well enough. Let’s Encrypt currently issues more publicly trusted certificates in a single day than existed in total during 2013.

What are the issues with RFC 6962 logs?

Cost

The first issue with RFC 6962 logs is the high cost of running them, particularly at Web scale, which has significantly limited the number of entities willing to operate them. Annual cloud costs for our logs are approaching seven figures.

The biggest contributor to this is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.

The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 7 and 10 terabytes. That’s big enough to be concerning for a single database: we previously had a test log fail when we ran into a 16 TiB limit in MySQL.

Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates. Ideally, clients should be able to obtain copies of the whole log in a relatively short time, but the traditional API has made that impractical.

Availability

The second issue with RFC 6962 logs is the potential for problems and non-compliance when the period called a Maximum Merge Delay (“MMD”) is exceeded.

One of the goals of CT was to have limited latency for submission to the logs. The Merge Delay design feature was added to guarantee that property. When receiving a new certificate submission, a CT log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s MMD, conventionally 24 hours. While this seems like a good tradeoff to avoid the alternative of slowing down certificate issuance, there have been multiple incidents in which important logs have exceeded their maximum merge delay, breaking that promise.

If the log does not integrate the certificate within the MMD window, the log is out of compliance and can be distrusted. If a log is distrusted, it’s disruptive for the operators and those who depend on it, and there are fewer logs for the ecosystem to rely on.

How does the new type of log resolve these issues?

In 2023 Filippo Valsorda suggested a new API for CT logs that avoids both of these issues—the Static CT API. The Static CT API for submitting certificates to logs is the same as RFC 6962, but the API for retrieving certificate information is quite different and the MMD is eliminated. The result is logs that are much more cost effective to operate and have better availability. We previously discussed our experiences testing out the new design in “Reflections on a Year of Sunlight.”

Serving Tiles

Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.

Static CT tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Static CT specification.

Unlike the dynamic endpoints in the RFC 6962 API, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash, which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!

The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.

Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.

No More Merge Delay

Static CT takes a different approach to adding certificates to the log while maintaining the same external submission API as RFC 6962. Static CT logs hold submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.

It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.

Going Forward

Let’s Encrypt has submitted new Static CT API logs for inclusion in certificate transparency programs. We expect these logs to be included and trusted prior to the read-only date for our RFC 6962 logs, and we will begin submitting our certificates to them as soon as possible.

We may stop submitting our own certificates to our RFC 6962 logs prior to the read-only date, but other CAs will be able to write certificates to our RFC 6962 logs until the read-only date.

Conclusion

The Static CT API has proven to be operationally better and more scalable than the RFC 6962 design in almost every way. With the substantial increase in certificate volume over time, we need that improved efficiency to make running CT logs cost-effective. As a result, we’re switching fully to the new log architecture in order to make the best use of our resources. The Static CT API logs we operate will provide exactly the same security and transparency benefits as our old RFC 6962 log did.

As we explained above, this requires no changes at all for end users. It may require configuration changes on the part of certificate authorities that have been submitting certs to our old log (they can start submitting to one or both of our new logs instead). It also requires software updates for ecosystem participants who monitor logs; they’ll need to ensure that they have client software that’s compatible with the new API. Overall, this change should help ensure that our CT logging continues to be able to grow with the Web PKI.

We ended support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, it could accidentally be retained or CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

At the height of our OCSP service’s traffic earlier this year, we handled approximately 340 billion OCSP requests per month. That’s more than 140,000 requests per second handled by our CDN, with 15,000 requests per second handled by our origin. We’d like to thank Akamai for generously donating CDN services for OCSP to Let’s Encrypt for the past ten years.

Update: January 15, 2026

Six-day and IP address certificates are now generally available. See 6-day and IP Address Certificates are Generally Available for details.

Since Let’s Encrypt started issuing certificates in 2015, people have repeatedly requested the ability to get certificates for IP addresses, an option that only a few certificate authorities have offered. Until now, they’ve had to look elsewhere, because we haven’t provided that feature.

Today, we’ve issued our first certificate for an IP address, as we announced we would in January. As with other new certificate features on our engineering roadmap, we’ll now start gradually rolling out this option to more and more of our subscribers.

Some Background on IP Address Certs

IP addresses are the underlying numerical addresses used on the Internet. Every device on the Internet has one (though, in modern practice, it might be shared with other devices, like when an entire home network shares a single public IP address). The Internet infrastructure uses them to route communications to their proper destination. IP addresses come in two forms, IPv4 and IPv6, and generally look like 54.215.62.21 (IPv4) or 2600:1f1c:446:4900::65 (IPv6).

Most Internet users rarely see or refer to IP addresses directly. Instead, they almost always use domain names like letsencrypt.org to refer to Internet services. The domain name system (DNS) is a part of the Internet infrastructure that’s responsible for allowing software to find the IP addresses associated with a particular domain name. For instance, your web browser can use DNS to find out that the service https://letsencrypt.org/ (Let’s Encrypt’s own website) is provided from the IP addresses 54.215.62.21 and 2600:1f1c:446:4900::65, among several others. This probably happened behind the scenes before you started reading this article! Your web browser needed to know our IP address in order to actually connect to our site and fetch this article.

Because we overwhelmingly tend to think and talk about Internet services in terms of domain names, those are the identifiers that are normally listed in certificates like those that Let’s Encrypt provides to our subscribers. Since you know us as “letsencrypt.org” and not as, say, “54.215.62.21,” it makes the most sense for our domain name to be on our certificate. After all, that’s what you’ll want your web browser to check against. This also gives Internet services more flexibility to be hosted in multiple locations, or to change where they’re hosted, without necessarily needing separate certificates for each server.

In principle, there’s no reason that a certificate couldn’t be issued for an IP address rather than a domain name, and in fact the technical and policy standards for certificates have always allowed this, with a handful of certificate authorities offering this service on a small scale. In Let’s Encrypt’s case, we’ve preferred to wait until some other pieces, like short-lived certs, were in place before we made this option available for our subscribers.

Why IP Address Certs Are Less Common

First and foremost, it’s because Internet users usually know services by domain names, not by IP addresses, and because IP addresses can easily change “behind the scenes” with no prior notice. For instance, a popular site could switch from one cloud hosting company to a different one, and update its DNS records to point at the new host. Most users wouldn’t ever notice the change at all, even though the site’s underlying IP addresses would be completely different.

Second, because IP addresses can change so easily, the sense of “ownership” one might have for them—or that a certificate authority might be able to attest to—tends to be weaker than for a domain name. If you’re hosting something in your house on a residential broadband connection, your Internet service provider most likely doesn’t guarantee that your IP address will stay the same over time. (That is, most home Internet users have a “dynamic IP address” from their ISPs, rather than a “static IP address.”) In that case, you have to contend with the possibility that that address may change often, possibly without warning, and that your old address may be assigned to somebody else.

Third, most Internet service operators don’t expect that end users will ever intentionally connect to their sites directly by IP address. In some cases, when an IP address is shared by different websites or different devices, connecting by IP address alone wouldn’t even work properly. In that case, there’s not much benefit to obtaining a certificate for the IP address!

How Let’s Encrypt Subscribers May Use IP Address Certs

Most current subscribers should be fine with their existing domain name certs and won’t need IP address certs. Subscribers who have a use for an IP address cert are typically already aware of that. A few use cases that we’re aware of include:

• A default page for hosting providers, in case someone pastes a server’s IP address into a browser instead of an individual site name (right now, this normally produces an error in the browser).

• A way to access your website if you don’t have a domain name at all (at some cost in reliability and convenience compared to getting a domain name).

• Securing DNS over HTTPS (DoH) or other infrastructure services. Having a certificate makes it much easier for DoH servers to prove their identities to clients. That could make it more feasible for DoH users or clients to enforce a requirement for a valid publicly-trusted certificate when connecting to DoH servers.

• Securing remote access to some home devices (like network-attached storage servers and Internet-of-things devices) even without a domain name.

• Securing ephemeral connections within cloud hosting infrastructure, like connections between one back-end cloud server and another, or ephemeral connections to administer new or short-lived back-end servers via HTTPS—as long as those servers have at least one public IP address available.

How To Get an IP Address Cert

IP address certificates are available right now in Staging. They should be generally available in Prod later in 2025, at the same time that short-lived certificates become generally available. Prior to general availability we may allow list issuance for a limited number of partners who can provide us with feedback.

Many Let’s Encrypt client applications should already be able to request certificates for IP addresses, although there can be minor technical changes required to support this in some client software.

As a matter of policy, Let’s Encrypt certificates that cover IP addresses must be short-lived certs, valid for only about six days. As such, your ACME client must support the draft ACME Profiles specification, and you must configure it to request the shortlived profile. And, probably not surprisingly, you can’t use the DNS challenge method to prove your control over an IP address; only the http-01 and tls-alpn-01 methods can be used.

If your client software requests an IP address cert with details that aren’t compatible with these policies, the order will be rejected by the ACME server. In this case, your client application may need to be updated or reconfigured. Feel free to ask for help on the Let’s Encrypt community forum if you encounter any problems, either as a client application developer or as an end user.

1. Over the past 10 years more and more of our subscribers have been able to put reliable automation into place for certificate renewal.
2. Providing expiration notification emails means that we have to retain millions of email addresses connected to issuance records. As an organization that values privacy, removing this requirement is important to us.
3. Providing expiration notifications costs Let’s Encrypt tens of thousands of dollars per year, money that we believe can be better spent on other aspects of our infrastructure.
4. Providing expiration notifications adds complexity to our infrastructure, which takes time and attention to manage and increases the likelihood of mistakes being made. Over the long term, particularly as we add support for new service components, we need to manage overall complexity by phasing out system components that can no longer be justified.

For those who would like to continue receiving expiration notifications, we recommend using a third party service such as Red Sift Certificates Lite (formerly Hardenize). Red Sift’s monitoring service providing expiration emails is free of charge for up to 250 certificates. More monitoring options can be found here.

We have deleted the email addresses provided to Let’s Encrypt via the ACME API that were stored in our CA database in association with issuance data. This doesn’t affect addresses signed up to mailing lists and other systems. They are managed in a separate ISRG system unassociated with issuance data.

Going forward, if an email address is provided to Let’s Encrypt via the ACME API, Let’s Encrypt will not store the address but will instead forward it to the general ISRG mailing list system unassociated with any account data. If the email address has not been seen before, that system may send an onboarding email with information about how to subscribe to various sources of updates.

If you’d like to stay informed about technical updates and other news about Let’s Encrypt and our parent nonprofit, ISRG, based on the preferences you choose, you can sign up for our email lists below:

Let’s Encrypt participates in CT both as a certificate issuer and as a log operator. For the past year, we’ve also been running an experiment to help validate a next-generation design for Certificate Transparency logs. That experiment is now nearing a successful conclusion. We’ve demonstrated that the new architecture (called the “Static CT API”) works well, providing greater efficiency and making it easier to run huge and reliable CT log services with comparatively modest resources. The Static CT API also makes it easier to download and share data from CT logs.

The Sunlight log implementation, alongside other Static CT API log implementations, is now on a path to production use. Browsers are now officially accepting Static CT API logs into their log programs as a means to help guarantee that the contents of CA-issued certificates are all publicly disclosed and publicly accessible (see Safari’s and Chrome’s recent announcements), although the browsers also require the continued use of a traditional RFC 6962 log alongside the new type.

All of this is good news for everyone who runs, submits certificates to, or monitors a CT log: as the new architecture gets adopted, we can expect to see more organizations running more logs, at lower cost, and with greater overall capacity to keep up with the large volume of publicly-trusted certificates.

Certificate Transparency

Certificate Transparency (CT) was introduced in 2013 in response to concerns about how Internet users could detect misbehavior and compromise of certificate authorities. Prior to CT, it was possible for a CA to issue an inaccurate or malicious certificate that could be used to attack a relatively small number of users, and that might never come to wider attention. A team led by Google responded to this by creating a transparency log mechanism, where certificate authorities (like Let’s Encrypt) must disclose all of the certificates that we issue by submitting them to public log services. Web browsers now generally reject certificates unless the certificates include cryptographic proof (“Signed Certificate Timestamps”, or SCTs) demonstrating that they were submitted to and accepted by such logs.

The CT logs themselves use a cryptographic append-only ledger to prove that they haven’t deleted or modified their records. There are currently over a dozen CT log services, most of them also run by certificate authorities, including Let’s Encrypt’s own Oak log.

The Static CT API

The original 2013 CT log design has been used with relatively few technical changes since it was first introduced, but several other transparency logging systems have been created in other areas, such as sumdb for Golang, which helps ensure that the contents of Golang package updates are publicly recorded. While they were originally inspired by CT, more-recently invented transparency logs have improved on its design.

The current major evolution of CT was led by Filippo Valsorda, a cryptographer with an interest in transparency log mechanisms, with help from others in the CT ecosystem. Portions of the new design are directly based on sumdb. In addition to designing the new architecture, Valsorda also wrote the implementation that we’ve been using, called Sunlight, with support from Let’s Encrypt. We’re excited to see that there are now at least three other compatible implementations: Google’s trillian-tessera, Cloudflare’s Azul, and an independent project called Itko.

The biggest change for the Static CT API is that logs are now represented, and downloaded by verifiers, as simple collections of flat files (called “tiles,” so some implementers have also been referring to these as “tiled logs” or “tlogs”). Anyone who wants to download log data can do so just by downloading these files. This is great for log operators because these simple file downloads can be distributed in various ways, including caching by a CDN, which was less practical and efficient for the classic CT API.

The new design is also simpler and more efficient from the log operator’s perspective, making it cheaper to run logs. As we said last year, this may enable us and other operators to increase reliability and availability by running several separate logs, likely with lower overall resource requirements than a single traditional log.

Our Sunlight experiment

For the past year, we’ve run three Sunlight logs, called Twig, Willow, and Sycamore. We’ve been logging all of our own issued certificates, which represent a majority of the total volume of all publicly-trusted certificates, into our Sunlight logs. Sunlight logged these certificates quickly and correctly on relatively modest server hardware. Notably, each log’s write side was handled comfortably by just a single server. We also achieved high availability for these log services throughout the course of this experiment. (Because our Sunlight logs are not yet trusted by web browsers, we didn’t include the SCT proofs that they returned to us in the actual certificates we gave out to our subscribers; those proofs wouldn’t have been of use to our subscribers yet and would just have taken up space.)

A potential failure mode of traditional CT logs is that they could be unacceptably slow in incorporating newly-submitted certificates (known as missing the maximum merge delay), which can result in a log becoming distrusted. This isn’t a possibility for our new Sunlight-based logs: they always completely incorporate newly-submitted certificates before returning an SCT to the submitter, so the effective merge delay is zero! Of course, any log can suffer outages for a variety of reasons, but this feature of Sunlight makes it less likely that any outages will be fatal to a log’s continued operation.

We’ve demonstrated that Sunlight and the Static CT API work in practice, and this demonstration has helped to confirm the browser developers’ hope that Static CT API logs can become an officially-supported part of CT. As a result, the major browsers that enforce CT have now permitted Static CT API logs to apply for inclusion in browsers as publicly-trusted logs, and we’re preparing to apply for this status for our Willow and Sycamore logs with the Chrome and Safari CT log programs.

Let’s Encrypt will run at least these two logs, and possibly others over time, for the foreseeable future. Once they’re trusted by browsers, we’ll encourage other CAs to submit to them as well, and we’ll begin including SCTs from these logs in our own certificates (alongside SCTs from traditional CT logs).

How to participate

The new Static CT API and the rollout of tile-based logs will bring various changes and opportunities for community members.

New Certificate Transparency log operators

Companies and non-profit organizations could help support the web PKI by running a CT log and applying for it to be publicly trusted. Implementations like Sunlight will have substantially lower resource requirements than first-generation CT logs, particularly when cached behind a CDN. The biggest resource demands for a log operator will be storage and upstream bandwidth. A publicly-trusted log is also expected to maintain relatively high availability, because CAs need logs to be available in order to continue issuing certificates.

We don’t have statistics to share about the exact resource requirements for such a log yet, but after we have practical experience running a fully publicly-trusted Sunlight log, we should be able to make this more concrete. As noted above, the compute side of the log can be handled by a single server. Sunlight author Filippo Valsorda has recently started running a Sunlight log—also on just a single server—and offered more detailed cost breakdowns for that log’s setup, with an estimated total cost around $10,000 per year. The costs for our production Static CT API logs may be higher than those for Filippo’s log, but still far less than the costs for our traditional RFC 6962 logs.

As with trust decisions about CAs, browser developers are the authorities about which CT logs become publicly trusted. Although any person or organization can run a log, browser developers will generally prefer to trust logs whose continued availability they’re confident of—typically those run by stable organizations with experience running some form of public Internet services. Unlike becoming a certificate authority, running a log does not require a formal audit, as the validation of the log’s availability and correctness can be performed purely by observation.

Certificate authorities

Once the Willow and Sycamore logs are trusted by browsers, our fellow certificate authorities can choose to start logging certificates to them as part of their issuance processes. (Initially, you should still include at least one SCT from a traditional CT log in each certificate.) The details, including the log API endpoints and keys, are available at our CT log page. You can start submitting to these logs right away if you prefer; just bear in mind that the SCTs they return aren’t useful to subscribers yet, and won’t be useful until browsers are updated to trust the new logs.

CT data users

You can monitor CT in order to watch for certificate issuances for your own domain names, or as part of monitoring or security products or services, or for Internet security research purposes. Many of our colleagues have been doing this for some time as a part of various tools they maintain. The Static CT API should make this easier, because you’ll be able to download and share log tiles as sets of ordinary files.

If you already run such monitoring tools, please note that you’ll need to update your data pipeline in order to access Static CT API logs; since the read API is not backwards-compatible, CT API clients will need to be modified to support the new API. Without updated tools, your view of the CT system will become partial!

Also note that getting a complete view of all of CT will still require downloading data from traditional logs, which will probably continue to be true for several years.

Software developers

As logs based on the new API enter production use, it will be important to have tools to interact with and search these logs. We can all benefit from more software that understands how to do this. Since file downloads are such a familiar piece of software functionality, it will probably be easier for developers to develop against the new API compared to the original one.

We’ve also continued to see greater integration of transparency logging tools into other kinds of services, such as software updates. There’s a growing transparency log ecosystem that’s always in need of more tools and integrations. As we mentioned above, transparency logs are increasingly learning from one another, and there are also mechanisms for more direct integration between different kinds of transparency logs (known as “witnessing”). Software developers can help improve different aspects of Internet security by contributing to this active and growing area.

Conclusion

The Certificate Transparency community and larger transparency logging community have experienced a virtuous cycle of innovation, sharing ideas and implementation code between different systems and demonstrating the feasibility of new mechanisms and functionality. With the advent of tile-based logging in CT, the state of the art has moved forward in a way that helps log operators run our logs much more efficiently without compromising security.

We’re proud to have participated in this experiment and the engineering conversation around the evolution of logging architectures. Now that we’ve shown how well the new API really works at scale, we look forward to having publicly-trusted Sunlight logs later this year!

The zombies in question are abandoned or misconfigured Internet servers and ACME clients that have been set to request certificates from Let’s Encrypt. As our certificates last for at most 90 days, these zombie clients’ software knows that their certificates are out-of-date and need to be replaced. What they don’t realize is that their quest for new certificates is doomed! These devices are cursed to seek certificates again and again, never receiving them.

But they do use up a lot of certificate authority resources in the process.

The Zombie Client Problem

Unlike a human being, software doesn’t give up in frustration, or try to modify its approach, when it repeatedly fails at the same task. Our emphasis on automation means that the vast majority of Let’s Encrypt certificate renewals are performed by automated software. This is great when those renewals succeed, but it also means that forgotten clients and devices can continue requesting renewals unsuccessfully for months, or even years.

How might that happen? Most often, it happens when a device no longer has a domain name pointed to it. The device itself doesn’t know that this has changed, so it treats renewal failures as transient even though they are actually permanent. For instance:

• An organization may have allowed a domain name registration to lapse because it is no longer needed, but its servers are still configured to request certs for it.
• Or, a home user stopped using a particular dynamic-DNS domain with a network-attached storage device, but is still using that device at home. The device doesn’t realize that the user no longer expects to use the name, so it keeps requesting certs for it.
• Or, a web hosting or CDN customer migrated to a different service provider, but never informed the old service provider. The old service provider’s servers keep requesting certs unsuccessfully. If the customer was in a free service tier, there might not be invoices or charges reminding the customer to cancel the service.
• Or any number of other, subtler changes in a subscriber’s infrastructure, such as changing a firewall rule or some webserver configuration.

At the scale of Let’s Encrypt, which now covers hundreds of millions of names, scenarios like these have become common, and their impact has become substantial. In 2024, we noticed that about half of all certificate requests to the Let’s Encrypt ACME API came from about a million accounts that never successfully complete any validations. Many of these had completed validations and issued certificates sometime in the past, but nowadays every single one of their validation attempts fails, and they show no signs that this will change anytime soon.

Unfortunately, trying to validate those futile requests still uses resources. Our CA software has to generate challenges, reach out and attempt to validate them over the Internet, detect and report failures, and record all of the associated information in our databases and audit logs. And over time, we’ve seen more and more recurring failures: accounts that always fail their issuance requests have been growing at around 18% per year.

In January, we mentioned that we had been addressing the zombie client problem through our rate limit system. This post provides more detail on that progress. 

Our Rate Limit Philosophy

If you’ve used Let’s Encrypt as a subscriber, you may have run into one of our rate limits at some point, maybe during your initial setup process. We have eight different kinds of rate limits in place now; as our January post describes, they’ve become more algorithmically sophisticated and grown to address a wider range of problems. A key principle for Let’s Encrypt is that our rate limiting is not a punishment. We don’t think of rate limits as a way of retaliating against a client for misbehavior. Rate limits are simply a tool to maximize the efficient use of our limited resources and prevent people and programs from using up those resources for no constructive purpose.

We’ve consistently tried to design our rate limit mechanisms in line with that philosophy. So if a misconfiguration or misunderstanding has caused excessive requests in the past, we’re still happy to welcome the user in question back and start issuing them certificates again—once the problem has been addressed. We want the rate limits to put a brake on wasteful use of our systems, but not to frustrate users who are actively trying to make Let’s Encrypt work for them.

In addition, we’ve always implemented our rate limits to err on the side of permissiveness. For example, if the Redis instances where rate limits are tracked have an outage or lose data, the system is designed to permit more issuance rather than less issuance as a result.

We wanted to create additional limits that would target zombie clients, but in a correspondingly non-punitive way that would avoid any disruption to valid issuance, and welcome subscribers back quickly if they happened to notice and fix a long-time problem with their setups.

Our Zombie-Related Rate Limits and Their Impact

In planning a new zombie-specific response, we decided on a “pausing” approach, which can temporarily limit an account’s ability to proceed with certificate requests. The core idea is that, if a particular account consistently fails to complete validation for a particular hostname, we’ll pause that account-hostname pair. The pause means that any new order requests from that account for that hostname will be rejected immediately, before we get to the resource-intensive validation phase.

This approach is more finely targeted than pausing an entire account. Pausing account-hostname pairs means that your ability to issue certs for a specific name could be paused due to repeated failures, but you can still get all of your other certs like normal. So a large hosting provider doesn’t have to fear that its certificate issuance on behalf of one customer will be affected by renewal failures related to a problem with a different customer’s domain name. The account-specificity of the pause, in turn, means that validation failures from one subscriber or device won’t prevent a different subscriber or device from attempting to validate the same name, as long as the devices in question don’t share a single Let’s Encrypt account.

In September 2024, we began applying our zombie rate limits manually by pausing about 21,000 of the most recurrently-failing account-hostname pairs, those which were consistently repeating the same failed requests many times per day, every day. After implementing that first round of pauses, we immediately saw a significant impact on our failed request rates. As we announced at that time, we also began using a formula to automatically pause other zombie client account-hostname pairs from December 2024 onward. The associated new rate limit is called “Consecutive Authorization Failures per Hostname Per Account” (and is independent of the existing “Authorization Failures per Hostname Per Account” limit, which resets every hour).

This formula relates to the frequency of successive failed issuance requests for the same domain name by the same Let’s Encrypt account. It applies only to failures that happen again and again, with no successful issuances at all in between: a single successful validation immediately resets the rate limit all the way to zero. Like all of our rate limits, this is not a punitive measure but is simply intended to reduce the waste of resources. So, we decided to set the thresholds rather high in the expectation that we would catch only the most disruptive zombie clients, and ultimately only those clients that were extremely unlikely to succeed in the future based on their substantial history of failed requests. We don’t hurry to block requesters as zombies: according to our current formula, client software following the default established by EFF’s Certbot (two renewal attempts per day) would be paused as a zombie only after about ten years of constant failures. More aggressive failed issuance attempts will get a client paused sooner, but clients will generally have to fail hundreds or thousands of attempts in a row before they are paused.

Most subscribers using mainstream client applications with default configurations will never encounter this rate limit, even if they forget to deactivate renewal attempts for domains that are no longer pointed at their servers. As described below, our current limit is already providing noticeable benefits with minimal disruption, and we’re likely to tighten it a bit in the near future, so it will trigger after somewhat fewer consecutive failures.

Self-Service Unpausing

A key feature in our zombie issuance pausing mechanism is self-service unpausing. Whenever an account-hostname pair is paused, any new certificate requests for that hostname submitted by that account are immediately rejected. But this means that the “one successful validation immediately resets the rate limit counter” feature can no longer come into effect: once they’re paused, they can’t even attempt validation anymore.

So every rejection comes with an error message explaining what has happened and a custom link that can be used to immediately unpause that account-hostname pair and remove any other pauses on the same account at the same time. The point of this is that subscribers who notice at some point that issuance is failing and want to intervene to get it working again have a straightforward option to let Let’s Encrypt know that they’re aware of the recurring failures and are still planning to use a particular account. As soon as subscribers notify us via the self-service link, they’ll be able to issue certificates again.

Currently, the user interface for an affected subscriber looks like this:

This link would be provided via an ACME error message in response to any request that was blocked due to a paused account-hostname pair.

As it’s turned out, the unpause option shown above has only been used by about 3% of affected accounts! This goes to show that most of the zombies we’ve paused were, in fact, well and truly forgotten about.

However, the unpause feature is there for whenever it’s needed, and there may be cases when it will become more important. A very large integration could trigger the zombie-related rate limits if a newly-introduced software bug causes what looks like a very high volume of zombie requests in a very short time. In that case, once that bug has been noticed and fixed, an integrator may need to unpause its issuance on behalf of lots of customers at once. Our unpause feature permits unpausing 50,000 domain names on a single account at a time, so even the largest integrators can get themselves unpaused expeditiously in this situation.

Conclusion

We’ve been very happy with the results of our zombie mitigation measures, and, as far as we can tell, there’s been almost no impact for subscribers! Our statistics indicate that we’ve managed to reduce the load on our infrastructure while causing no detectable harm or inconvenience to subscribers’ valid issuance requests.

Since implementing the manual pauses in September and the automated pauses in December, we’ve seen:

• Over 100,000 account-hostname pairs have been paused for excessive failures.
• We received zero (!) associated complaints or support requests.
• About 3,200 people manually unpaused issuance.
• Failed certificate orders fell by about 30% so far, and should continue to fall over time as we fine-tune the rate limit formula and catch more zombie clients.

The new rate limit and the self-service unpause system are also ready to deal with circumstances that might produce more zombie clients in the future. For instance, we’ve announced that we’re going to be discontinuing renewal reminder emails soon. If some subscribers overlook failed renewals in the future, we might see more paused clients that result from unintentional renewal failures. We think taking advantage of the existing self-service unpause feature will be straightforward in that case. But it’s much better to notice problems and get them fixed up front, so please remember to set up your own monitoring to avoid unnoticed renewal failures in the future.

If you’re a subscriber who’s had occasion to use the self-service unpause feature, we’d love your feedback on the Community Forum about your experience using the feature and the circumstances that surrounded your account’s getting paused.

Also, if you’re a Let’s Encrypt client developer, please remember to make renewal requests at a random time (not precisely at midnight) so that the load on our infrastructure is smoothed out. You can also reduce the impact of zombie renewals by repeating failed requests somewhat less frequently over time (a “back-off” strategy), especially if the failure reason makes it look like a domain name may no longer be in use at all.

Recurring donations are more than just financial support; they allow us to plan, innovate, and keep improving with confidence, knowing that month after month, year after year, our supporters are there. This consistent backing empowers us to maintain a secure, privacy-respecting Internet for all.

Our tenth anniversary tagline, Encryption for Everybody, highlights this vision. It’s both a technical goal and a fundamental belief that secure communication should be available to everyone, everywhere.

When we asked our recurring donors why they give, their responses affirmed how essential this commitment is. One longtime supporter shared:

For some, it’s about paying it forward, helping future users benefit as they once did:

The next decade of Let’s Encrypt will likely be about maintaining that commitment to encryption for everybody. It’s about ensuring that our work remains reliable, accessible, and—most importantly—supported by people who believe in what we do. To everyone who’s been part of this journey, thank you. We couldn’t do it without you.

During Let’s Encrypt’s 10th Anniversary Year, we’re celebrating our community and reflecting on our journey. We’d be thrilled to hear from you. Connect with us on LinkedIn, our community forum, or email us at outreach@letsencrypt.org. Let’s keep building a secure Internet together!

Let’s Encrypt is a project of the nonprofit Internet Security Research Group, a 501(c)(3) nonprofit. To support our work, visit letsencrypt.org/donate.

Update March 16, 2026: Thanks to some timeline changes in the root program requirements, we have been able to push back the removal of the tlsclient profile slightly. If you are already using the tlsclient profile before May 13, 2026, now you will be able to continue to do so through July 8, 2026.

Let’s Encrypt will no longer include the “TLS Client Authentication” Extended Key Usage (EKU) in our certificates beginning in 2026. Most users who use Let’s Encrypt to secure websites won’t be affected and won’t need to take any action. However, if you use Let’s Encrypt certificates as client certificates to authenticate to a server, this change may impact you.

To minimize disruption, Let’s Encrypt will roll this change out in multiple stages, using ACME Profiles:

• Today: Let’s Encrypt already excludes the Client Authentication EKU on our tlsserver ACME profile. You can verify compatibility by issuing certificates with this profile now.
• October 1, 2025: Let’s Encrypt will launch a new tlsclient ACME profile which will retain the TLS Client Authentication EKU. Users who need additional time to migrate can opt-in to this profile.
• February 11, 2026: the default classic ACME profile will no longer contain the Client Authentication EKU.
• July 8, 2026: the tlsclient ACME profile will no longer be available and no further certificates with the Client Authentication EKU will be issued.

Once this is completed, Let’s Encrypt will switch to issuing with new intermediate Certificate Authorities which also do not contain the TLS Client Authentication EKU.

For some background information, all certificates include a list of intended uses, known as Extended Key Usages (EKU). Let’s Encrypt certificates have included two EKUs: TLS Server Authentication and TLS Client Authentication.

• TLS Server Authentication is used to authenticate connections to TLS Servers, like websites.
• TLS Client Authentication is used by clients to authenticate themselves to a server. This feature is not typically used on the web, and is not required on the certificates used on a website.

After this change is complete, only TLS Server Authentication will be available from Let’s Encrypt.

This change is prompted by changes to Google Chrome’s root program requirements, which impose a June 2026 deadline to split TLS Client and Server Authentication into separate PKIs. Many uses of client authentication are better served by a private certificate authority, and so Let’s Encrypt is discontinuing support for TLS Client Authentication ahead of this deadline.

Together with the IETF community, we created the ACME standard to support completely automated certificate issuance. This open standard is now supported by dozens of clients. On the server side, did you know that we have not one but two open-source ACME server implementations?

The big implementation, which we use ourselves in production, is called Boulder. Boulder handles all of the facets and details needed for a production certificate authority, including policy compliance, database interfaces, challenge verifications, and logging. You can adapt and use Boulder yourself if you need to run a real certificate authority, including an internal, non-publicly-trusted ACME certificate authority within an organization.

The small implementation is called Pebble. It’s meant entirely for testing, not for use as a real certificate authority, and we and ACME client developers use it for various automated and manual testing purposes. For example, Certbot has used Pebble in its development process for years in order to perform a series of basic but realistic checks of the ability to request and obtain certificates from an ACME server.

Pebble is Easy to Use for ACME Client Testing

For any developer or team creating an ACME client application, Pebble solves a range of problems along the lines of “how do I check whether I’ve implemented ACME correctly, so that I could actually get certificates from a CA, without necessarily using a real domain name, and without running into CA rate limits during my routine testing?” Pebble is quick and easy to set up if you need to test an ACME client’s functionality.

It runs in RAM without dependencies or persistence; you won’t need to set up a database or a configuration for it. You can get Pebble running with a single golang command in just a few seconds, and immediately start making local ACME requests. That’s suitable for inclusion in a client’s integration test suite, making much more realistic integration tests possible without needing to worry about real domains, CA rate limits, or network outages.

We see Pebble getting used in the official test suites for ACME clients including getssl, Lego, Certbot, simp_le, and others. In many cases, every change committed to the ACME client’s code base is automatically tested against Pebble.

Pebble is Intentionally Different From Boulder

Pebble is also deliberately different from Boulder in some places in order to provide clients with an opportunity to interoperate with slightly different ACME implementations. The Pebble code explains that

For instance, the Let’s Encrypt service currently offers its newAccount resource at the path /acme/new-acct, whereas Pebble uses a different name /sign-me-up, so clients will be reminded to check the directory rather than assuming a specific path. Other substantive differences include:

• Pebble rejects 5% of all requests as having an invalid nonce, even if the nonce was otherwise valid, so clients can test how they respond to this error condition
• Pebble only reuses valid authorizations 50% of the time, so clients can check their ability to perform validations when they might not have expected to
• Pebble truncates timestamps to a different degree of precision than Boulder
• Unlike Boulder, Pebble respects the notBefore and notAfter fields of new-order requests

The ability of ACME clients to work with both versions is a good test of their conformance to the ACME specification, rather than making assumptions about the current behavior of the Let’s Encrypt service in particular. This helps ensure that clients will work properly with other ACME CAs, and also with future versions of Let’s Encrypt’s own API.

Pebble is Useful to Both Let’s Encrypt and Client Developers as ACME Evolves

We often test out new ACME features by implementing them, at least in a simplified form, in Pebble before Boulder. This lets us and client developers experiment with support for those features even before they get rolled out in our staging service. We can do this quickly because a Pebble feature implementation doesn’t have to work with a full-scale CA backend.

We continue to encourage ACME client developers to use a copy of Pebble to test their clients’ functionality and ACME interoperability. It’s convenient and it’s likely to increase the correctness and robustness of their client applications.

Try Out Pebble Yourself

Want to try Pebble with your ACME client right now? On a Unix-like system, you can run

git clone https://github.com/letsencrypt/pebble/
cd pebble
go run ./cmd/pebble

Wait a few seconds; now you have a working ACME CA directory available at https://localhost:14000/dir! Your local ACME Server can immediately receive requests and issue certificates, though not publicly-trusted ones, of course. (If you prefer, we also offer other options for installing Pebble, like a Docker image.)

We welcome code contributions to Pebble. For example, ACME client developers may want to add simple versions of an ACME feature that’s not currently tested in Pebble in order to make their test suites more comprehensive. Also, if you notice a possibly unintended divergence between Pebble and Boulder or Pebble and the ACME specification, we’d love for you to let us know.

With that sense of gratitude top of mind, we are proud to announce a contribution of $1,000,000 from Jeff Atwood. Jeff has been a longtime supporter of our work, beginning many years ago with Discourse providing our community forum pro bono; something Discourse still provides to this day. As best we can tell, our forum has helped hundreds of thousands of people get up and running with Let’s Encrypt—an impact that has helped billions of people use an Internet that’s more secure and privacy-respecting thanks to widely adopted TLS.

When we first spoke with Jeff about the road ahead for Let’s Encrypt back in 2023, we knew a few things wouldn’t change no matter how the Internet changes over the next decade:

1. Free TLS is the only way to ensure it is and remains accessible to as many people as possible.
2. Let’s Encrypt is here to provide a reliable, trusted, and sound service no matter the scale.
3. Generosity from our global community of supporters will be how we sustain our work.

We’re proud that Jeff not only agrees, but has chosen to support us in such a meaningful way. In discussing how Jeff might want us to best celebrate his generosity and recognize his commitment to our work, he shared:

Indeed, this contribution is significant because of its scale, but more importantly because of its signal: a signal that supporting the not-so-glamorous but oh-so-nerdy work of encryption at scale matters to the lives of billions of people every day; a signal that supporting free privacy and security afforded by TLS for all of the Internet’s five billion users just makes sense.

Ten years ago we set out to build a better Internet through easy to use TLS. If you or your organization have supported us throughout the years, thank you for joining Jeff in believing in the work of Let’s Encrypt. For a deeper dive into the impact of Let’s Encrypt and ISRG’s other projects, take a look at our most recent annual report.

Let’s Encrypt is a project of the nonprofit Internet Security Research Group, a 501(c)(3) nonprofit committed to protecting Internet users by lowering monetary, technological, and informational barriers to a more secure and privacy-respecting Internet. For more, visit abetterinternet.org. Press inquiries can be sent to press@abetterinternet.org

The next step is for us to make short-lived certificates available to a small set of our subscribers so we can make sure our systems scale as expected prior to general availability. We expect this next phase to begin during Q2 of this year.

We expect short-lived certificates to be generally available by the end of this year.

How To Get Six-Day Certificates

Once short-lived certificates are an option for you, you’ll need to use an ACME client that supports ACME certificate profiles and select the short-lived certificate profile (“shortlived”). The lego client recently added this functionality.

In the meantime, the best way to prepare to take advantage of short-lived certificates is to make sure your ACME client is reliably renewing certificates in an automated fashion. If that’s working well then there should be no costs to switching to short-lived certificates.

You’ll also want to be sure your ACME client is running frequently - both for the sake of renewing short-lived certificates and so as to take advantage of ACME Renewal Information (ARI). ARI allows Let’s Encrypt to notify your client if it should renew early for some reason. ARI checks should happen at least once per day, and short-lived certificates should be renewed every two to three days, so we recommend having your client run at least once per day.

Shorter Certificate Lifetimes Are Good for Security

When the private key associated with a certificate is compromised, the recommendation has always been to have the certificate revoked so that people will know not to use it. Unfortunately, certificate revocation doesn’t work very well. This means that certificates with compromised keys (or other issues) may continue to be used until they expire. The longer the lifetime of the certificate, the longer the potential for use of a problematic certificate.

The primary advantage of short-lived certificates is that they greatly reduce the potential compromise window because they expire relatively quickly. This reduces the need for certificate revocation, which has historically been unreliable. Our six-day certificates will not include OCSP or CRL URLs. Additionally, short-lived certificates practically require automation, and we believe that automating certificate issuance is important for security.

Questions

If you have questions or comments about our plans, feel free to let us know on our community forums.

We’d like to thank Open Technology Fund for supporting this work.

Our First 6-Day Certificate

PEM format:

-----BEGIN CERTIFICATE-----
MIIDSzCCAtGgAwIBAgISA7CwFcGk4mQWEXMacRtxHeDvMAoGCCqGSM49BAMDMDIx
CzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJF
NjAeFw0yNTAyMTkxNzMwMDFaFw0yNTAyMjYwOTMwMDBaMAAwWTATBgcqhkjOPQIB
BggqhkjOPQMBBwNCAAQoSItt2V1aocI5dxrKR8iLfmm0KiVvOhiwKByzu2kLeC7C
0BdfAgtwdICdkuEhAXokhXLq6DNZZgmh5T4flVwZo4IB9zCCAfMwDgYDVR0PAQH/
BAQDAgeAMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHwYDVR0j
BBgwFoAUkydGmAOpUWiOmNbEQkjbI79YlNIwVQYIKwYBBQUHAQEESTBHMCEGCCsG
AQUFBzABhhVodHRwOi8vZTYuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6
Ly9lNi5pLmxlbmNyLm9yZy8wKAYDVR0RAQH/BB4wHIIaaGVsbG93b3JsZC5sZXRz
ZW5jcnlwdC5vcmcwEwYDVR0gBAwwCjAIBgZngQwBAgEwggEFBgorBgEEAdZ5AgQC
BIH2BIHzAPEAdgDM+w9qhXEJZf6Vm1PO6bJ8IumFXA2XjbapflTA/kwNsAAAAZUf
d/zOAAAEAwBHMEUCIFNd51TfSNiJrO+294t49C5ANc4oC7gTUzf7xnlNlhKsAiEA
wi5hfiC9SsKLxlTQ0sctUxhLmdYh40r6ECWQS/yWw2AAdwDgkrP8DB3I52g2H95h
uZZNClJ4GYpy1nLEsE2lbW9UBAAAAZUfd/0TAAAEAwBIMEYCIQCs2NuZIUIloOaH
1t9eXDKb8bjoWESBPsK4i2BxMvEIswIhAOMNaQNyr1YkzrcNUz15qGV0oVLg5BJN
+ikWxXOdcRHFMAoGCCqGSM49BAMDA2gAMGUCMDANqy7G09AIwzXcd7SNl7uFwhC+
xlfduvp1PeEDHc/FA9K3mRYkGXuKtzNdOh7wcAIxALjEMDmBQiwXbB447oGkaZAe
0rqxA3EtNV5wj0obeObluj/NgUsVEG9OqiBIoggFRw==
-----END CERTIFICATE-----

openssl x509 -text output:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:b0:b0:15:c1:a4:e2:64:16:11:73:1a:71:1b:71:1d:e0:ef
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C=US, O=Let's Encrypt, CN=E6
        Validity
            Not Before: Feb 19 17:30:01 2025 GMT
            Not After : Feb 26 09:30:00 2025 GMT
        Subject:
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:28:48:8b:6d:d9:5d:5a:a1:c2:39:77:1a:ca:47:
                    c8:8b:7e:69:b4:2a:25:6f:3a:18:b0:28:1c:b3:bb:
                    69:0b:78:2e:c2:d0:17:5f:02:0b:70:74:80:9d:92:
                    e1:21:01:7a:24:85:72:ea:e8:33:59:66:09:a1:e5:
                    3e:1f:95:5c:19
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                93:27:46:98:03:A9:51:68:8E:98:D6:C4:42:48:DB:23:BF:58:94:D2
            Authority Information Access:
                OCSP - URI:http://e6.o.lencr.org
                CA Issuers - URI:http://e6.i.lencr.org/
            X509v3 Subject Alternative Name: critical
                DNS:helloworld.letsencrypt.org
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : CC:FB:0F:6A:85:71:09:65:FE:95:9B:53:CE:E9:B2:7C:
                                22:E9:85:5C:0D:97:8D:B6:A9:7E:54:C0:FE:4C:0D:B0
                    Timestamp : Feb 19 18:28:32.078 2025 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:53:5D:E7:54:DF:48:D8:89:AC:EF:B6:F7:
                                8B:78:F4:2E:40:35:CE:28:0B:B8:13:53:37:FB:C6:79:
                                4D:96:12:AC:02:21:00:C2:2E:61:7E:20:BD:4A:C2:8B:
                                C6:54:D0:D2:C7:2D:53:18:4B:99:D6:21:E3:4A:FA:10:
                                25:90:4B:FC:96:C3:60
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : E0:92:B3:FC:0C:1D:C8:E7:68:36:1F:DE:61:B9:96:4D:
                                0A:52:78:19:8A:72:D6:72:C4:B0:4D:A5:6D:6F:54:04
                    Timestamp : Feb 19 18:28:32.147 2025 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:AC:D8:DB:99:21:42:25:A0:E6:87:D6:
                                DF:5E:5C:32:9B:F1:B8:E8:58:44:81:3E:C2:B8:8B:60:
                                71:32:F1:08:B3:02:21:00:E3:0D:69:03:72:AF:56:24:
                                CE:B7:0D:53:3D:79:A8:65:74:A1:52:E0:E4:12:4D:FA:
                                29:16:C5:73:9D:71:11:C5
    Signature Algorithm: ecdsa-with-SHA384
    Signature Value:
        30:65:02:30:30:0d:ab:2e:c6:d3:d0:08:c3:35:dc:77:b4:8d:
        97:bb:85:c2:10:be:c6:57:dd:ba:fa:75:3d:e1:03:1d:cf:c5:
        03:d2:b7:99:16:24:19:7b:8a:b7:33:5d:3a:1e:f0:70:02:31:
        00:b8:c4:30:39:81:42:2c:17:6c:1e:38:ee:81:a4:69:90:1e:
        d2:ba:b1:03:71:2d:35:5e:70:8f:4a:1b:78:e6:e5:ba:3f:cd:
        81:4b:15:10:6f:4e:aa:20:48:a2:08:05:47

2025 marks ten years of Let’s Encrypt. Already this year we’ve taken steps to continue to deliver on our values of user privacy, efficiency, and innovation, all with the intent of continuing to deliver free TLS certificates to as many people as possible; to deliver encryption for everybody.

And while we’re excited about the technical progress we’ll make this year, we’re also going to celebrate this tenth anniversary by highlighting the people around the world who make our impact possible. It’s no small village.

From a community forum that has provided free technical support, to our roster of sponsors who provide vital funding, to the thousands of individual supporters who contribute financially to Let’s Encrypt each year, free TLS at Internet scale works because people have supported it year in, year out, for ten years.

Each month we’ll highlight a different set of people behind our “everybody.” Who do you want to see us highlight? What use cases of Let’s Encrypt have you seen that amazed you? What about our work do you hope we’ll continue or improve as we go forward? Let us know on LinkedIn, or drop a note to outreach@letsencrypt.org.

Encryption for Everybody is our unofficial tagline for this tenth anniversary year. What we love about it is that, yes, it captures our commitment to ensuring anyone around the world can easily get a cert for free. But more importantly, it captures the reality that technical innovation won’t work without people believing in it and supporting it. We’re grateful that, for ten years (and counting!), our community of supporters has made an impact on the lives of billions of Internet users—an impact that’s made the Web more secure and privacy-respecting for everybody, everywhere.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

We needed a solution built for the future—one that could scale with demand, reduce the load on MariaDB, and adapt to real-world subscriber request patterns. The result was a new rate limiting system powered by Redis and a proven virtual scheduling algorithm from the mid-90s. Efficient and scalable, and capable of handling over a billion active certificates.

Rate Limiting a Free Service is Hard

In 2015, Let’s Encrypt was in early preview, and we faced a unique challenge. We were poised to become incredibly popular, offering certificates freely and without requiring contact information or email verification. Ensuring fair usage and preventing abuse without traditional safeguards demanded an atypical approach to rate limiting.

We decided to limit the number of certificates issued—per week—for each registered domain. Registered domains are a limited resource with real costs, making them a natural and effective basis for rate limiting—one that mirrors the structure of the Web itself. Specifically, this approach targets the effective Top-Level Domain (eTLD), as defined by the Public Suffix List (PSL), plus one additional label to the left. For example, in new.blog.example.co.uk, the eTLD is .co.uk, making example.co.uk the eTLD+1.

Counting Events Was Easy

For each successfully issued certificate, we logged an entry in a table that recorded the registered domain, the issuance date, and other relevant details. To enforce rate limits, the system scanned this table, counted the rows matching a given registered domain within a specific time window, and compared the total to a configured threshold. This simple design formed the basis for all future rate limits.

Counting a Lot of Events Got Expensive

By 2019, we had added six new rate limits to protect our infrastructure as demand for certificates surged. Enforcing these limits required frequent scans of database tables to count recent matching events. These operations, especially on our heavily-used authorizations table, caused significant overhead, with reads outpacing all other tables—often by an order of magnitude.

Rate limit calculations were performed early in request processing and often. Counting rows in MariaDB, particularly for accounts with rate limit overrides, was inherently expensive and quickly became a scaling bottleneck.

Adding new limits required careful trade-offs. Decisions about whether to reuse existing schema, optimize indexes, or design purpose-built tables helped balance performance, complexity, and long-term maintainability.

Buying Runway — Offloading Reads

In late 2021, we updated our control plane and Boulder—our in-house CA software—to route most API reads, including rate limit checks, to database replicas. This reduced the load on the primary database and improved its overall health. At the same time, however, latency of rate limit checks during peak hours continued to rise, highlighting the limitations of scaling reads alone.

Sliding Windows Got Frustrating

Subscribers were frequently hitting rate limits unexpectedly, leaving them unable to request certificates for days. This issue stemmed from our use of relatively large rate limiting windows—most spanning a week. Subscribers could deplete their entire limit in just a few moments by repeating the same request, and find themselves locked out for the remainder of the week. This approach was inflexible and disruptive, causing unnecessary frustration and delays.

In early 2022, we patched the Duplicate Certificate limit to address this rigidity. Using a naive token-bucket approach, we allowed users to “earn back” requests incrementally, cutting the wait time—once rate limited—to about 1.4 days. The patch worked by fetching recent issuance timestamps and calculating the time between them to grant requests based on the time waited. This change also allowed us to include a Retry-After timestamp in rate limited responses. While this improved the user experience for this one limit, we understood it to be a temporary fix for a system in need of a larger overhaul.

When a Problem Grows Large Enough, It Finds the Time for You

Setting aside time for a complete overhaul of our rate-limiting system wasn’t easy. Our development team, composed of just three permanent engineers, typically juggles several competing priorities. Yet by 2023, our flagging rate limits code had begun to endanger the reliability of our MariaDB databases.

Our authorizations table was now regularly read an order of magnitude more than any other. Individually identifying and deleting unnecessary rows—or specific values—had proved unworkable due to poor MariaDB delete performance. Storage engines like InnoDB must maintain indexes, foreign key constraints, and transaction logs for every deletion, which significantly increases overhead for concurrent transactions and leads to gruelingly slow deletes.

Our SRE team automated the cleanup of old rows for many tables using the PARTITION command, which worked well for bookkeeping and compliance data. Unfortunately, we couldn’t apply it to most of our purpose-built rate limit tables. These tables depend on ON DUPLICATE KEY UPDATE, a mechanism that requires the targeted column to be a unique index or primary key, while partitioning demands that the primary key be included in the partitioning key.

Indexes on these tables—such as those tracking requested hostnames—often grew larger than the tables themselves and, in some cases, exceeded the memory of our smaller staging environment databases, eventually forcing us to periodically wipe them entirely.

By late 2023, this cascading confluence of complexities required a reckoning. We set out to design a rate limiting system built for the future.

The Solution: Redis + GCRA

We designed a system from the ground up that combines Redis for storage and the Generic Cell Rate Algorithm (GCRA) for managing request flow.

Why Redis?

Our engineers were already familiar with Redis, having recently deployed it to cache and serve OCSP responses. Its high throughput and low latency made it a candidate for tracking rate limit state as well.

By moving this data from MariaDB to Redis, we could eliminate the need for ever-expanding, purpose-built tables and indexes, significantly reducing read and write pressure. Redis’s feature set made it a perfect fit for the task. Most rate limit data is ephemeral—after a few days, or sometimes just minutes, it becomes irrelevant unless the subscriber calls us again. Redis’s per-key Time-To-Live would allow us to expire this data the moment it was no longer needed.

Redis also supports atomic integer operations, enabling fast, reliable counter updates, even when increments occur concurrently. Its “set if not exist” functionality ensures efficient initialization of keys, while pipeline support allows us to get and set multiple keys in bulk. This combination of familiarity, speed, simplicity, and flexibility made Redis the natural choice.

Why GCRA?

The Generic Cell Rate Algorithm (GCRA) is a virtual scheduling algorithm originally designed for telecommunication networks to regulate traffic and prevent congestion. Unlike traditional sliding window approaches that work in fixed time blocks, GCRA enforces rate limits continuously, making it well-suited to our goals.

A rate limit in GCRA is defined by two parameters: the emission interval and the burst tolerance. The emission interval specifies the minimum time that must pass between consecutive requests to maintain a steady rate. For example, an emission interval of one second allows one request per second on average. The burst tolerance determines how much unused capacity can be drawn on to allow short bursts of requests beyond the steady rate.

When a request is received, GCRA compares the current time to the Theoretical Arrival Time (TAT), which indicates when the next request is allowed under the steady rate. If the current time is greater than or equal to the TAT, the request is permitted, and the TAT is updated by adding the emission interval. If the current time plus the burst tolerance is greater than or equal to the TAT, the request is also permitted. In this case, the TAT is updated by adding the emission interval, reducing the remaining burst capacity.

However, if the current time plus the burst tolerance is less than the TAT, the request exceeds the rate limit and is denied. Conveniently, the difference between the TAT and the current time can then be returned to the subscriber in a Retry-After header, informing their client exactly how long to wait before trying again.

To illustrate, consider a rate limit of one request per second (emission interval = 1s) with a burst tolerance of three requests. Up to three requests can arrive back-to-back, but subsequent requests will be delayed until “now” catches up to the TAT, ensuring that the average rate over time remains one request per second.

What sets GCRA apart is its ability to automatically refill capacity gradually and continuously. Unlike sliding windows, where users must wait for an entire time block to reset, GCRA allows users to retry as soon as enough time has passed to maintain the steady rate. This dynamic pacing reduces frustration and provides a smoother, more predictable experience for subscribers.

GCRA is also storage and computationally efficient. It requires tracking only the TAT—stored as a single Unix timestamp—and performing simple arithmetic to enforce limits. This lightweight design allows it to scale to handle billions of requests, with minimal computational and memory overhead.

The Results: Faster, Smoother, and More Scalable

The transition to Redis and GCRA brought immediate, measurable improvements. We cut database load, improved response times, and delivered consistent performance even during periods of peak traffic. Subscribers now experience smoother, more predictable behavior, while the system’s increased permissiveness allows for certificates that the previous approach would have delayed—all achieved without sacrificing scalability or fairness.

Rate Limit Check Latency

Check latency is the extra time added to each request while verifying rate limit compliance. Under the old MariaDB-based system, these checks slowed noticeably during peak traffic, when database contention caused significant delays. Our new Redis-based system dramatically reduced this overhead. The high-traffic “new-order” endpoint saw the greatest improvement, while the “new-account” endpoint—though considerably lighter in traffic—also benefited, especially callers with IPv6 addresses. These results show that our subscribers now experience consistent response times, even under peak load.

Database Health

Our once strained database servers are now operating with ample headroom. In total, MariaDB operations have dropped by 80%, improving responsiveness, reducing contention, and freeing up resources for mission-critical issuance workflows.

Buffer pool requests have decreased by more than 50%, improving caching efficiency and reducing overall memory pressure.

Reads of the authorizations table—a notorious bottleneck—have dropped by over 99%. Previously, this table outpaced all others by more than two orders of magnitude; now it ranks second (the green line below), just narrowly surpassing our third most-read table.

Tracking Zombie Clients

In late 2024, we turned our new rate limiting system toward a longstanding challenge: “zombie clients.” These requesters repeatedly attempt to issue certificates but fail, often because of expired domains or misconfigured DNS records. Together, they generate nearly half of all order attempts yet almost never succeed. We were able to build on this new infrastructure to record consecutive ACME challenge failures by account/domain pair and automatically “pause” this problematic issuance. The result has been a considerable reduction in resource consumption, freeing database and network capacity without disrupting legitimate traffic.

Scalability on Redis

Before deploying the limits to track zombie clients, we maintained just over 12.6 million unique TATs across several Redis databases. Within 24 hours, that number more than doubled to 26 million, and by the end of the week, it peaked at over 30 million. Yet, even with this sharp increase, there was no noticeable impact on rate limit responsiveness. That’s all we’ll share for now about zombie clients—there’s plenty more to unpack, but we’ll save those insights and figures for a future blog post.

What’s Next?

Scaling our rate limits to keep pace with the growth of the Web is a huge achievement, but there’s still more to do. In the near term, many of our other ACME endpoints rely on load balancers to enforce per-IP limits, which works but gives us little control over the feedback provided to subscribers. We’re looking to deploy this new infrastructure across those endpoints as well. Looking further ahead, we’re exploring how we might redefine our rate limits now that we’re no longer constrained by a system that simply counts events between two points in time.

By adopting Redis and GCRA, we’ve built a flexible, efficient rate limit system that promotes fair usage and enables our infrastructure to handle ever-growing demand. We’ll keep adapting to the ever-evolving Web while honoring our primary goal: giving people the certificates they need, for free, in the most user-friendly way we can.

1. Over the past 10 years more and more of our subscribers have been able to put reliable automation into place for certificate renewal.

2. Providing expiration notification emails means that we have to retain millions of email addresses connected to issuance records. As an organization that values privacy, removing this requirement is important to us.

3. Providing expiration notifications costs Let’s Encrypt tens of thousands of dollars per year, money that we believe can be better spent on other aspects of our infrastructure.

4. Providing expiration notifications adds complexity to our infrastructure, which takes time and attention to manage and increases the likelihood of mistakes being made. Over the long term, particularly as we add support for new service components, we need to manage overall complexity by phasing out system components that can no longer be justified.

For those who would like to continue receiving expiration notifications, we recommend using a third party service such as Red Sift Certificates Lite (formerly Hardenize). Red Sift’s monitoring service providing expiration emails is free of charge for up to 250 certificates. More monitoring options can be found here.

While we will be minimizing the email addresses we retain connected to issuance data, you can opt in to receive other emails. We’ll keep you informed about technical updates, and other news about Let’s Encrypt and our parent nonprofit, ISRG, based on the preferences you choose. You can sign up for our email lists below:

Update: January 15, 2026

Six-day and IP address certificates are now generally available. See 6-day and IP Address Certificates are Generally Available for details.

This year we will continue to pursue our commitment to improving the security of the Web PKI by introducing the option to get certificates with six-day lifetimes (“short-lived certificates”). We will also add support for IP addresses in addition to domain names. Our longer-lived certificates, which currently have a lifetime of 90 days, will continue to be available alongside our six-day offering. Subscribers will be able to opt in to short-lived certificates via a certificate profile mechanism being added to our ACME API.

Shorter Certificate Lifetimes Are Good for Security

When the private key associated with a certificate is compromised, the recommendation has always been to have the certificate revoked so that people will know not to use it. Unfortunately, certificate revocation doesn’t work very well. This means that certificates with compromised keys (or other issues) may continue to be used until they expire. The longer the lifetime of the certificate, the longer the potential for use of a problematic certificate.

The primary advantage of short-lived certificates is that they greatly reduce the potential compromise window because they expire relatively quickly. This reduces the need for certificate revocation, which has historically been unreliable. Our six-day certificates will not include OCSP or CRL URLs. Additionally, short-lived certificates practically require automation, and we believe that automating certificate issuance is important for security.

IP Address Support For Securing Additional Use Cases

We will support including IP addresses as Subject Alternative Names in our six-day certificates. This will enable secure TLS connections, with publicly trusted certificates, to services made available via IP address, without the need for a domain name.

Validation for IP addresses will work much the same as validation for domain names, though validation will be restricted to the http-01 and tls-alpn-01 challenge types. The dns-01 challenge type will not be available because the DNS is not involved in validating IP addresses. Additionally, there is no mechanism to check CAA records for IP addresses.

Timeline

We expect to issue the first valid short-lived certificates to ourselves in February of this year. Around April we will enable short-lived certificates for a small set of early adopting subscribers. We hope to make short-lived certificates generally available by the end of 2025.

The earliest short-lived certificates we issue may not support IP addresses, but we intend to enable IP address support by the time short-lived certificates reach general availability.

How To Get Six-Day and IP Address Certificates

Once short-lived certificates are an option for you, you’ll need to use an ACME client that supports ACME certificate profiles and select the short-lived certificate profile (the name of which will be published at a later date).

Once IP address support is an option for you, requesting an IP address in a certificate will automatically select a short-lived certificate profile.

Looking Ahead

The best way to prepare to take advantage of short-lived certificates is to make sure your ACME client is reliably renewing certificates in an automated fashion. If that’s working well then there should be no costs to switching to short-lived certificates.

If you have questions or comments about our plans, feel free to let us know on our community forums.

As of today, the staging environment is advertising a new field in its directory resource:

GET /directory HTTP/1.1
HTTP/1.1 200 OK
Content-Type: application/json

{
    ...
    "meta": {
        "profiles": {
            "classic": "The same profile you're accustomed to",
            "tlsserver": "https://letsencrypt.org/2025/01/09/acme-profiles/"
        }
    }
}

Here, the keys are the names of new “profiles”, and the values are human-readable descriptions of those profiles. A profile describes a collection of attributes about the certificate that will be issued, such as what extensions it will contain, how long it will be valid for, and more.

For example, the “classic” profile is exactly what it sounds like: certificates issued under the classic profile will look exactly the same as those that we have always issued, valid for 90 days.

But certificates issued under the “tlsserver” profile will have a number of differences tailored specifically towards TLS server usage:

• No Common Name field (including a CN has been NOT RECOMMENDED by the Baseline Requirements for several years now)
• No Subject Key Identifier (including a SKID is NOT RECOMMENDED by the Baseline Requirements)
• No TLS Client Auth Extended Key Usage (root programs are moving towards requiring “single-purpose” issuance hierarchies, where every certificate has only a single EKU)
• No Key Encipherment Key Usage for certificates with RSA public keys (this KU was used by older RSA-based TLS cipher suites, but is fully unnecessary in TLS 1.3)

Additionally, in the near future we will offer a “shortlived” profile which will be identical to the “tlsserver” profile but with a validity period of only 6 days. This profile isn’t available in Staging just yet, so keep an eye out for further announcements regarding short-lived certificates and why we think they’re exciting.

An ACME client can supply a desired profile name in a new-order request:

POST /acme/new-order HTTP/1.1
Host: example.com
Content-Type: application/jose+json

{
    "protected": base64url(...),
    "payload": base64url({
        "profile": "tlsserver",
        "identifiers": [
            { "type": "dns", "value": "www.example.org" },
            { "type": "dns", "value": "example.org" }
        ],
    }),
    "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

If the new-order request is accepted, then the selected profile name will be reflected in the Order object when it is returned, and the resulting certificate after finalization will be issued with the selected profile. If the new-order request does not specify a profile, then the server will select one for it.

Guidance for ACME clients and users

If you are an ACME client author, we encourage you to introduce support for this new field in your client. Start by taking a look at the draft specification in the IETF ACME Working Group. A simple implementation might allow the user to configure a static profile name and include that name in all new-order requests. For a better user experience, check the configured name against the list of profiles advertised in the directory, to ensure that changes to the available profiles don’t result in invalid new-order requests. For clients with a user interface, such as a control panel or interactive command line interface, an implementation could fetch the list of profiles and their descriptions to prompt the user to select one on first run. It could also use a notification mechanism to inform the user of changes to the list of available profiles. We’d also love to hear from you about your experience implementing and deploying this new extension.

If you are a site operator or ACME client user, we encourage you to keep an eye on your ACME client of choice to see when they adopt this new feature, and update your client when they do. We also encourage you to try out the modern “tlsserver” profile in Staging, and let us know what you think of the changes we’ve made to the certificates issued under that profile.

What’s next?

Obviously there is more work to be done here. The draft standard will go through multiple rounds of review and tweaks before becoming an IETF RFC, and our implementation will evolve along with it if necessary. Over the coming weeks and months we will also be providing more information about when we enable profile selection in our production environment, and what our production profile options will be.

Thank you for coming along with us on this journey into the future of the Web PKI. We look forward to your testing and feedback!

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

• January 30, 2025
  • OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
• May 7, 2025
  • Prior to this date we will have added CRL URLs to certificates
  • On this date we will drop OCSP URLs from certificates
  • On this date all requests including the OCSP Must Staple extension will fail
• August 6, 2025
  • On this date we will turn off our OCSP responders

Additionally, a very small percentage of our subscribers request certificates with the OCSP Must Staple Extension. If you have manually configured your ACME client to request that extension, action is required before May 7. See “Must Staple” below for details.

OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL.

Must Staple

Because of the privacy issues with OCSP, browsers and servers implement a feature called “OCSP Stapling”, where the web server sends a copy of the appropriate OCSP response during the TLS handshake, and the browser skips making a request to the CA, thus better preserving privacy.

In addition to OCSP Stapling (a TLS feature negotiated at handshake time), there’s an extension that can be added to certificates at issuance time, colloquially called “OCSP Must Staple.” This tells browsers that, if they see that extension in a certificate, they should never contact the CA about it and should instead expect to see a stapled copy in the handshake. Failing that, browsers should refuse to connect. This was designed to solve some security problems with revocation.

Let’s Encrypt has supported OCSP Must Staple for a long time, because of the potential to improve both privacy and security. However, Must Staple has failed to get wide browser support after many years. And popular web servers still implement OCSP Stapling in ways that create serious risks of downtime.

As part of removing OCSP, we’ll also be removing support for OCSP Must Staple. CRLs have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration. Thanks to all our subscribers who have helped with the OCSP Must Staple experiment.

If you are not certain whether you are using OCSP Must Staple, you can check this list of hostnames and certificate serials (11.1 MB, .zip).

As of January 30, 2025, issuance requests that include the OCSP Must Staple extension will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension.

As of May 7, all issuance requests that include the OCSP Must Staple extension will fail, including renewals. Please change your ACME client configuration to not request the extension.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

RFC 9773 provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pin intermediate certificates (more on that later).

The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.

New Intermediate Certificates

Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.

All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.

The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.

It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.

A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.

Removing DST Root CA X3 Cross-sign

The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.

ECDSA Intermediates as Default for ECDSA Certificates

Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.

Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.

We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.

Risks of Pinning or Hard-Coding Intermediates

We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).

Questions?

We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.

We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.

Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.

The New Certificates

We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.

We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.

You can see details of all of the certificates on our updated hierarchy page.

Rotating Issuance

Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.

Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.

We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.

These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.

Providing Smaller Chains

When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.

ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.

This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.

Other Minor Changes

We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:

• We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.

• We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.

Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.

Deployment

We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.

Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.

We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.

CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.

In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.

Bottlenecks from the Database

Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.

The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.

Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.

Serving Tiles

Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.

Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.

Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.

Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!

The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.

Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.

Running More Logs

The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.

The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective required by CT log programs.

The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.

No More Merge Delay

One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.

Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.

It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.

A Sunny Future

Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.

We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”

If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack.

We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.

This letter was originally published in our 2023 Annual Report.

We typically open our annual report with a letter from our Executive Director and co-founder, Josh Aas, but he’s on parental leave so I’ll be filling in. I’ve run the Brand & Donor Development team at ISRG since 2016, so I’ve had the pleasure of watching our work mature, our impact grow, and I’ve had the opportunity to get to know many great people who care deeply about security and privacy on the Internet.

One of the biggest observations I’ve made during Josh’s absence is that all 23 people who work at ISRG fall into that class of folks. Of course I was a bit nervous as Josh embarked on his leave to discover just how many balls he has been keeping in the air for the last decade. Answer: it’s a lot. But the roster of staff that we’ve built up made it pretty seamless for us to keep moving forward.

Let’s Encrypt is supporting 40 million more websites than a year ago, bringing the total to over 360 million. The engineering team has grown to 12 people who are responsible for our continued reliability and ability to scale. But they’re not maintaining the status quo. Let’s Encrypt engineers are pushing forward our expectations for ourselves and for the WebPKI community. We’ve added shorter-lived certificates to our 2024 roadmap. We’re committing to this work because sub-10 day certificates significantly reduce the impact of key compromise and it broadens the universe of people who can use our certs. In addition, the team started an ambitious project to develop a new Certificate Transparency implementation because the only existing option cannot scale for the future and is prone to operational fragility. These projects are led by two excellent technical leads, Aaron Gable and James Renken, who balance our ambition with our desire for a good quality of life for our teams.

Prossimo continues to deliver highly performant and memory safe software and components in a world that is increasingly eager to address the memory safety problem. This was evidenced by participation at Tectonics, a gathering we hosted which drew industry leaders for invigorated conversation. Meanwhile, initiatives like our memory safe AV1 decoder are in line to replace a C version in Google Chrome. This change would improve security for billions of people. We’re grateful to the community that helps to guide and implement our efforts in this area, including Dirkjan Ochtman, the firms Tweede golf and Ferrous Systems, and the maintainers of the many projects we are involved with.

Our newest project, Divvi Up, brought on our first two subscribers in 2023. Horizontal, a small international nonprofit serving Human Rights Defenders, will be collecting privacy-preserving telemetry metrics about the users of their Tella app, which people use to document human rights violations. Mozilla is using Divvi Up to gain insight into aspects of user behavior in the Firefox browser. It took a combination of focus and determination to get us to a production-ready state and our technical lead, Brandon Pitman played a big role in getting us there.

We hired Kristin Berdan to fill a new role as General Counsel and her impact is already apparent within our organization. She joins Sarah Heil, our CFO, Josh, and me in ISRG leadership.

Collectively, we operate three impactful and growing projects for $7 million a year. This is possible because of the amazing leadership assembled across our teams and the ongoing commitment from our community to validate the usefulness of our work. As we look toward 2024 and the challenges and opportunities that face us, I ask that you join us in building a more secure and privacy respecting Internet by sponsoring us, making a donation or gift through your DAF, or sharing with the folks you know why security and privacy matter to them.

Beyond being a big number, what does that signify? What’s the importance of having TLS being widely adopted anyways? We’ll take a closer look at these questions through the lens of one group of Subscribers we can relate to particularly well: nonprofits.

Serving .org at Internet scale

Let’s Encrypt serves 57% of all websites using the .org top level domain (TLD), which is commonly used by nonprofits. In the US alone there are 1.8M registered nonprofit organizations. And while the focus of these organizations is varied, all of them rely on the Internet in some capacity.

When a nonprofit uses a TLS certificate on their website, it protects their visitors and stakeholders from snoopers, MITM attacks, and surveillance. Without TLS, nonprofits’ content could be changed without their knowledge or their visitors’ private information could be compromised. Access to free and automated TLS via Let’s Encrypt means these nonprofits face as few barriers as possible to adopting TLS.

In short, something as fundamental as security and privacy should be as easy to access as possible. For nonprofits both large and small, Let’s Encrypt makes it easy to provide security and privacy for users of their websites, enabling them to remain focused on their missions.

Zooming in on three nonprofits we serve

The American Civil Liberties Union (ACLU) uses Let’s Encrypt as it works to realize its focus of being a “guardian of liberty” for US citizens. Using Let’s Encrypt protects ACLU’s constituents when they’re trying to know their rights or take action. With more than 4 million page views per month, ACLU’s website is a critical part of their mission.

Human Rights Watch (HRW) is an international nonprofit organization. With more than 500 individuals on staff around the world, HRW’s website is a trove of information empowering individuals and organizations alike to be informed and take action with a global perspective. Nearly 70% of HRW’s web traffic comes from people outside of the United States; that’s millions of page views per month secured by Let’s Encrypt—and by extension, millions of people around the world benefitting from a more secure and privacy-respecting Web.

The Center for Democracy & Technology (CDT) uses Let’s Encrypt to advance its mission to promote democratic values by shaping technology policy and architecture, with a focus on the rights of the individual. The CDT website offers updated and insightful information into the ways policy and innovation impact the digital space. Without a TLS certificate, the content of these pages could be intercepted and changed. What’s more, for those looking to financially support CDT, using TLS on their donation page encrypts the transaction protecting user details such as credit card and other personal information. Mallory Knodel, CTO at CDT and longtime digital rights defender and advocate commented, “Billions of people in over 60 countries access the internet with less censorship and surveillance because Let’s Encrypt hastened the adoption of web security measures by making certificates easy to obtain.”

Serving philanthropic foundations

In the United States, the work of nonprofits is made possible in large part through philanthropic foundations and organizations. When it comes to philanthropy’s web presence, Let’s Encrypt is there, too.

We provide TLS to billion dollar philanthropic organizations like the Hewlett Foundation, the Silicon Valley Community Foundation, Yield Giving, and many others. Taking a look at the top 50 philanthropic organizations around the world, Let’s Encrypt serves 36% of them. For large philanthropies, their website is the primary tool they have to communicate their focus areas for future funding as well as the impact they’ve made with past giving.

One of the leading philanthropists in the US, Craig Newmark, uses Let’s Encrypt and Digital Ocean for his website, craig newmark philanthropies. Commenting on our work, Craig recently shared, “The people at ISRG have been helping protect the Internet for over ten years, and continue to protect us all. They’re a necessary part of Cyber Civil Defense and national security.”

Overall, while Let’s Encrypt aims to build a better Internet, we’re particularly proud that our impact protects those seeking to build a better world.

Internet Security Research Group (ISRG) is the parent organization of Prossimo, Let’s Encrypt, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

What is CAA?

CAA is a type of DNS record that allows site owners to specify which Certificate Authorities are allowed to issue certificates containing their domain names. Using CAA is a proactive way to ensure that your domain(s) and subdomain(s) are under your control—you’re able to add a layer of security to your DNS governance. (By contrast, Certificate Transparency (CT) logs are a reactive way to monitor your DNS governance—by publicly publishing certificates issued to domains, Subscribers can verify that their domain(s) are using the intended CA(s).)

We think CAA is important for every Subscriber, but it’s all the more important if you’re handling TLS at scale. This is particularly true if a team or multiple teams have access to your integration.

Account and Method Binding is another layer of CAA that can improve your security even further. Method binding allows Subscribers to limit the sets of domain control validation methods—DNS-01, HTTP-01, or TLS-ALPN-01— which can be used to demonstrate control over their domain. Account binding allows a Subscriber to limit issuance to a specific ACME account. For further technical details, review our community post or take a look at RFC 8657.

CAA Adoption

Famedly, a German healthcare company, set up CAA as part of updating their overall ACME setup. In hearing more about why they chose to turn on CAA now, the answer was simple: because it was easy to do and low hanging fruit to enhance their security.

“The biggest benefit of using CAA along with account and method binding is closing the DV loophole,” said Jan Christian Grünhage, Famedly’s Head of Infrastructure. “By using DNSSEC with DNS-01 challenges, we’ve got cryptographic signatures all the way through the stack.”

The team at Famedly set up CAA over the course of a few days. “The larger project was to transition our issuance to a single ACME account ID, so adopting CAA as part of that work only added marginal effort,” remarked Jan. “The added security benefit was absolutely worth the effort.”

Getting started with CAA

If you manage TLS at scale, consider adopting CAA and Account and Method Binding. To get started, review our documentation, RFC 8659 and RFC 8657, and check out the community forum for more from Subscribers who’ve set up or are using CAA.

As with anything with DNS, there are some potential hiccups to avoid. The most important to highlight is that CAA will always respect the CAA record closest to the domain name it is issuing a certificate for. For more on this, check out this section of the CAA documentation. You’ll also want to ensure that your DNS provider supports setting CAA records.

Thanks to Famedly

We’re grateful for Famedly taking the time to share with us more about their experience in setting up CAA. What’s more, Famedly financially supported ISRG this year as part of our tenth anniversary campaign.

As a project of the Internet Security Research Group (ISRG), 100% of the funding for Let’s Encrypt comes from contributions from our community of users and supporters. We depend on their support in order to provide our public benefit services. If your company or organization would like to sponsor Let’s Encrypt, please email us at sponsor@letsencrypt.org. If you or your organization can support us with a donation of any size, we ask that you consider a contribution.

Come late 2021, our cross-signed intermediates and DST Root CA X3 itself were expiring. And while all up-to-date browsers at that time trusted our root, over a third of Android devices were still running old versions of the OS which would suddenly stop trusting websites using our certificates. That breakage would have been too widespread, so we arranged for a new cross-sign – this time directly onto our root rather than our intermediates – which would outlive DST Root CA X3 itself. This stopgap allowed those old Android devices to continue trusting our certificates for three more years.

On September 30th, 2024, that cross-sign too will expire.

In the last three years, the percentage of Android devices which trust our ISRG Root X1 has risen from 66% to 93.9%. That percentage will increase further over the next year, especially as Android releases version 14, which has the ability to update its trust store without a full OS update. In addition, dropping the cross-sign will reduce the number of certificate bytes sent in a TLS handshake by over 40%. Finally, it will significantly reduce our operating costs, allowing us to focus our funding on continuing to improve your privacy and security.

For these reasons, we will not be getting a new cross-sign to extend compatibility any further.

The transition will roll out as follows:

• On Thursday, Feb 8th, 2024, we stopped providing the cross-sign by default in requests made to our /acme/certificate API endpoint. For most Subscribers, this means that your ACME client will configure a chain which terminates at ISRG Root X1, and your webserver will begin providing this shorter chain in all TLS handshakes. The longer chain, terminating at the soon-to-expire cross-sign, will still be available as an alternate chain which you can configure your client to request.

• On Thursday, June 6th, 2024, we will stop providing the longer cross-signed chain entirely. This is just over 90 days (the lifetime of one certificate) before the cross-sign expires, and we need to make sure subscribers have had at least one full issuance cycle to migrate off of the cross-signed chain.

• On Monday, September 30th, 2024, the cross-signed certificate will expire. This should be a non-event for most people, as any client breakages should have occurred over the preceding six months.

If you use Android 7.0 or earlier, you may need to take action to ensure you can still access websites secured by Let’s Encrypt certificates. We recommend installing and using Firefox Mobile, which uses its own trust store instead of the Android OS trust store, and therefore trusts ISRG Root X1.

If you are a site operator, you should keep an eye on your website usage statistics and active user-agent strings during Q2 and Q3 of 2024. If you see a sudden drop in visits from Android, it is likely because you have a significant population of users on Android 7.0 or earlier. We encourage you to provide the same advice to them as we provided above.

If you are an ACME client author, please make sure that your client correctly downloads and installs the certificate chain provided by our API during every certificate issuance, including renewals. Failure modes we have seen in the past include a) never downloading the chain at all and only serving the end-entity certificate; b) never downloading the chain and instead serving a hard-coded chain; and c) only downloading the chain at first issuance and not re-downloading during renewals. Please ensure that your client does not fall into any of these buckets.

We appreciate your understanding and support, both now and in the years to come as we provide safe and secure communication to everyone who uses the web. If you have any questions about this transition or any of the other work we do, please ask on our community forum.

We’d like to thank IdenTrust for their years of partnership. They played an important role in helping Let’s Encrypt get to where we are today and their willingness to arrange a stopgap cross sign in 2021 demonstrated a true commitment to creating a secure Web. 

We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

It’s hard to believe 10 years have passed since Eric Rescorla, Alex Halderman, Peter Eckersley and I founded ISRG as a nonprofit home for public benefit digital infrastructure. We had an ambitious vision, but we couldn’t have known then the extent to which that vision would become shared and leveraged by so much of the Internet.

Since its founding in 2013, ISRG’s Let’s Encrypt certificate authority has come to serve hundreds of millions of websites and protect just about everyone who uses the Web. Our Prossimo project has brought the urgent issue of memory safety to the fore, and Divvi Up is set to revolutionize the way apps collect metrics while preserving user privacy. I’ve tried to comprehend how much data about peoples’ lives our work has and will protect, and tried even harder to comprehend what that means if one could quantify privacy. It’s simply beyond my ability.

Some of the highlights from the past ten years include:

• May 24, 2013: ISRG is incorporated, intending to build Let’s Encrypt

• November 18, 2014: The Let’s Encrypt project is announced publicly

• September 14, 2015: Let’s Encrypt issues its first certificate

• October 19, 2015: Let’s Encrypt becomes publicly trusted

• December 3, 2015: Let’s Encrypt becomes generally available

• March 8, 2016: Let’s Encrypt issues its millionth certificate

• June 28, 2017: Let’s Encrypt issues its 100 millionth certificate

• March 11, 2019: The ACME protocol becomes an IETF standard

• February 27, 2020: Let’s Encrypt issues its billionth certificate

• October 26, 2020: ISRG board approves a privacy preserving metrics project, now Divvi Up

• December 9, 2020: ISRG board approves a memory safety project, now Prossimo

• December 18, 2020: Divvi Up starts servicing COVID exposure notification

• October 3, 2022: Support for Rust is merged into the Linux kernel

All this wouldn’t be possible without our staff, community, donors, funders, and other partners, all of whom I’d like to thank wholeheartedly.

I feel so fortunate that we’ve been able to thrive. We’re fortunate primarily because great people got involved and funders stepped up, but there’s also just a bit of good fortune involved in any success story. The world is a complicated place, there is complex context that one can’t control around every effort. Despite our best efforts, fortune has a role to play in terms of the degree to which the context swirling around us helps or hinders. We have been fortunate in every sense of the word and for that I am grateful.

Our work is far from over. Each of our three projects has challenges and opportunities ahead.

For Let’s Encrypt, which is more critical than ever and relatively mature, our focus over the next few years will be on long-term sustainability. More and more people working with certificates can’t recall a time when Let’s Encrypt didn’t exist, and most people who benefit from our service don’t need to know it exists at all (by design!). Let’s Encrypt is just part of how the Internet works now, which is great for many reasons, but it also means it’s at risk of being taken for granted. We are making sure that doesn’t happen so we can keep Let’s Encrypt running reliably and make investments in its future.

Prossimo is making a huge amount of progress moving critical software infrastructure to memory safe code, from the Linux kernel to NTP, TLS, media codecs, and even sudo/su. We have two major challenges ahead of us here. The first is to raise the money we need to complete development work. The second is to get the safer software we’ve been building adopted widely. We feel pretty good about our plans but it’s not going to be easy. Things worth doing rarely are.

Divvi Up is exciting technology with a bright future. Our biggest challenge here, like most things involving cryptography, is to make it easy to use. We also need to make sure we can provide the service at a cost that will allow for widespread adoption, so we’ll be doing a lot of optimization. Our hope is that over the next decade we can make privacy respecting metrics the norm, just like we did for HTTPS.

The Internet wasn’t built with security or privacy in mind, so there is a bountiful opportunity for us to improve its infrastructure. The Internet is also constantly growing and changing, so it is also our job to look into the future and prepare for the next set of threats and challenges as best we can.

Thanks to our supporters, we’ll continue adapting and responding to help ensure the Web is more secure long into the future. Please consider becoming a sponsor or making a donation in support of our work.

With ARI, Let’s Encrypt can signal to ACME clients when they should renew certificates. In the normal case of a certificate with a 90 day lifetime, ARI might signal for renewal at 60 days. If Let’s Encrypt needs to revoke a certificate for some reason, ARI can signal that renewal needs to happen prior to the revocation. This means that even in extenuating circumstances, renewal can happen in an entirely automated way without disrupting subscriber services.

Without ARI, an unexpected revocation event might mean that Let’s Encrypt would have to send emails to affected subscribers, maybe those emails are read in time to avoid a service disruption, maybe they aren’t, and engineers have to manually take action to trigger early renewals, possibly in the middle of the night. We can’t wait for ARI to make this scenario a thing of the past.

ARI has a couple of additional benefits for Let’s Encrypt and our subscribers. First, we can use ARI to help modulate renewals as needed to avoid load spikes on the Let’s Encrypt infrastructure (of course subscribers can still renew whenever they want or need, as ARI is merely a signal or suggestion). Second, ARI can be used to set subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers even shorter-lived certificates in the future.

ARI has been standardized in the IETF, a process that started with an email from Let’s Encrypt engineer Roland Shoemaker in March of 2020. In September of 2021 Let’s Encrypt engineer Aaron Gable submitted the first draft to the IETF’s ACME working group, and now ARI is in production. The next step is for ACME clients to start supporting ARI, a process we plan to help with as best we can in the coming months.

ARI is a huge step forward for agility and resiliency in the TLS certificate ecosystem and we’re excited to see it gain widespread adoption!

Supporting Let’s Encrypt

As a project of the Internet Security Research Group (ISRG), 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our public benefit services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. If you can support us with a donation, we ask that you make an individual contribution.

With us since the beginning

In 2015, the work ISRG set out to do was viewed by many as audacious, if not incredible. Back then, SSL/TLS was used by less than 40% of page loads. Getting a certificate was costly and complicated. Our aim was to make access to SSL certificates easy, free, and automated. That same year nineteen sponsors came on board to help us realize this mission. Today, seventeen of those sponsors and grantmakers have stayed on board every year! Now in their eighth year as a sponsor, Hostpoint has renewed their support for 2023. Their Co-founder and CEO, Markus Gebert, commented: 

“We have supported Let’s Encrypt since the very beginning. It is very valuable and important that nowadays any website can be equipped with an SSL certificate free of charge.”

Our thanks to Akamai, Cisco, Mozilla, Google, OVHcloud, Internet Society, Shopify, Hostpoint, SiteGround, Cyon, IdenTrust, Vultr, Automattic, Electronic Frontier Foundation, infomaniak, PlanetHoster, and Discourse for their eight years of support.

Committed to a better Internet

We know that finding sponsorship dollars is often anything but a straightforward path. That’s why we approach sponsorship as an ongoing conversation, not a one-time transactional interaction. As a result, we’re proud that each year we see on average 80% of our sponsors renew their support. From large organizations with thousands of staff to one-person shops, our sponsors come in all shapes and sizes—but all share a common goal of helping to make our work happen. 

We are grateful to the 70 sponsors renewing their support for 2023 who combined provide close to 60% of our operating budget. Their continued support means we begin 2023 well on our way towards our fundraising need for the year. Shopify, a sponsor since 2015 has renewed their Gold sponsorship for 2023. Their Founder and CEO, Tobi Lütke, commented:

“Let’s Encrypt makes it easy for everyone to do the right thing to secure the Internet. We couldn’t be happier to give our support to such a great effort.”

Together, these organizations make the mission of ISRG, and its impact for billions of people around the world, possible. Powered by their support, we look forward to continuing to build a Web that works for everyone, everywhere.

Supporting Let’s Encrypt

As a project of the Internet Security Research Group (ISRG), 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our public benefit services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. If you can support us with a donation, we ask that you make an individual contribution.

1. Aligning Growth Aspirations

It happens again and again: a solid engineer grows and moves up the ranks and is then promoted to team manager, where they are supposed to juggle individual contributions, technical oversight, and people management. Many times, the engineer may not even have growth aspirations in people management, and yet they are put in a position where they are expected to know how to manage people and do it well. This could lead to the employee feeling like they cannot do their job sufficiently, feelings of imposter syndrome, burnout, or unreasonable expectations for everyone else put in a similar position.

To address this issue, our engineering career ladder intentionally does not include management requirements. This enables engineers to continually grow as individual contributors without having to be forced into responsibilities they may not be interested in or skilled at.

Many of our Site Reliability Engineers (SREs) have a background in operations work. To support their growth, we run a job rotation cycle where SREs spend 12-18 months on our Developer team to foster coding, architecture, and design skills. Some extra benefits to this are the strengthening of mentorship amongst team members as well as the connection between the two teams for better alignment in priorities and understanding.

It is essential to support employees in their growth and goal setting consistently so that workforce planning can be done with the employees’ best interests in mind. This is done through cultivating a psychologically safe environment where employees feel comfortable asking questions, making mistakes, and are encouraged to reflect and be open about their aspirations. Processes that help this along are regularly structured check-ins, performance reviews, blameless post-incident debriefs, and open feedback and communication with their peers and leaders.

2. Mitigate the Management SPOF (Single Point of Failure)

Every engineering team at ISRG is led by a Technical Lead and a People Manager. This separation of technical and people oversight allows for the work of leading an engineering team to be broken up so that it is not all resting on one person. The Technical Lead can focus on being in charge of the technical viability, structures, and processes while the People Manager can focus on things such as individual and team goal setting, growth opportunities, and conflict resolution.

The Technical Lead and People Manager come together when it comes to process development, visibility, and recognition. They also work together to address things for each other while not playing the other’s role, thus mitigating the “who manages the manager” quandary. There are more instances where collaboration is needed between the two positions and that crossover lends to more perspective and opinion on what could be a complex issue.

3. Intentional Scalability

It is easy to dive straight into action items and deadlines, and then before you know it, things are rapidly scaling in efforts to keep up. The analogy of “building the plane while it’s flying” comes to mind. Later down the line those scaled systems show flaws that are far more difficult to repair.

Much like designing a reliable and scalable engineering system, our goal is to create a workforce system that can handle increases in load while maintaining effective performance without redesigning the whole thing or “rebuilding the plane.”

Our dual leadership approach sets up our management with increased load and changing priorities in mind. Both people have more wiggle room to anticipate and adjust. It may seem superfluous to have Engineering People Managers in a small organization, however this prepares for future growth with a relatively lean solution without the extra complexity.

Like all scalable solutions, there is the upfront investment of time and money. However, the benefits will far outweigh the costs in the long run since building on scalable systems is typically less expensive than trying to adapt or redesign less agile systems.

While reflecting on our engineering workplace systems and how they came to be, we recognized that many were organically built out of having a remote workplace, autonomous teams, and the driving values of flexibility and inclusion. We will continue to design practices with these things in mind.

All in all, when looked at with a holistic lens, building an engineering workplace culture has several considerations that are similar to those we focus on when designing software systems. The obvious difference is that instead of functions and data, we are dealing with actual people with feelings and ever changing wants and needs. That is why it is important to once again acknowledge that no two workplaces are the same and there are no perfect solutions, but we hope that these few points lead to thoughtful reflection on how organizations can improve their engineer workplace experience.

If this sounds like a culture you’d like to be a part of, check out our open jobs!

Supporting Let’s Encrypt

As a project of the Internet Security Research Group (ISRG), 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our public benefit services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. If you can support us with a donation, we ask that you make an individual contribution.

About OCSP Responses

OCSP is used to communicate the revocation status of TLS certificates. When an ACME agent signs a request to revoke a certificate, our Let’s Encrypt Certificate Authority (CA) verifies whether or not the request is authorized and if it is, we begin publishing a ‘revoked’ OCSP response for that certificate. Each time a relying party, such as a browser, visits a domain with a Let’s Encrypt certificate, they can request information about whether the certificate has been revoked and we serve a reply containing ‘good’ or ‘revoked’, signed by our CA, which we call an OCSP response.

An Enormous OCSP Response Load: 100,000 Every Second

Let’s Encrypt currently serves over 300 million domains, which means we receive an enormous number of certificate revocation status requests — fielding around 100,000 OCSP responses every second!

Normally 98-99% of our OCSP responses are handled by our Content Delivery Network (CDN). But there are times when our CDN has an issue resulting in Let’s Encrypt being required to directly accept a larger number of requests. Historically, we could effectively respond to a maximum of 6% of our OCSP response traffic on our own. Should the need arise for us to accept much higher than that, some of our systems might begin to take too long to return results, return significant numbers of errors, or even stop accepting new requests. Not an ideal situation for us, or the Internet.

Our inability to serve OCSP responses during an issue with one of our CDNs could result in a slowdown in users’ browsing speed or not being able to connect to a website — or worse, Internet users unintentionally visiting domains for which a certificate has been revoked. Browsers react differently to unresponsive OCSP, but one thing was clear, our systems needed to handle these occasions much better.

Increasing our Reliability

After working on this throughout most of 2022, our engineers have dramatically improved our ability to independently serve OCSP responses. We did that by deploying Redis as an in-memory caching layer that helps protect our database by absorbing traffic spikes, whether due to CDN issues or our own actions, such as CDN cache clearing.

Pivot in Design

Our team developed a system architecture design to organize/change all of the various interconnected systems needed to make Redis trusted to serve our OCSP responses. Amidst the fervor of developing this design, our engineers identified a resource we could depend upon more heavily to simplify the overall architecture and still realize incredible reliability gains. Rather than pre-signing OCSP status responses at regular intervals, storing the results in a relational database, and asking Redis to keep copies—we could keep simple but authoritative certificate status information in our database. We could then leverage fast, concurrent signing power from our HSMs to Just-in-Time sign a fresh OCSP response, cache it in Redis, and return it to the requester. Thanks to this, the demands on the relational database became much lighter (especially total table-writes and write-contention), the speed was impressive, and Redis wasn’t holding anything that couldn’t be (very very quickly) regenerated.

Testing our Systems

The first test was to directly accept 1/16 of the requests by dropping a segment of our CDN cache. In that initial test we handled ~12,500 requests per second. Successive tests ratcheted up to 1/8th CDN cache drop, then 1/4th, then 1/2, then a 100% cache drop. With each ratcheting up of the test load we were able to monitor and glean insights as to how our deployment could handle the traffic. In the final test of 100% of requests, our systems remained responsive. This means that if we experience a spike in the number of OCSP responses we need to accept moving forward, we are equipped to handle them, dramatically reducing the risks to Internet users.

Supporting Let’s Encrypt

As a project of the Internet Security Research Group (ISRG), 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our public benefit services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. If you can support us with a donation, we ask that you make an individual contribution.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability.

A particularly big moment was when Let’s Encrypt surpassed 300,000,000 websites served. When I was informed that we had reached that milestone, my first reaction was to be excited and happy about how many people we’ve been able to help. My second reaction, following on quickly after the first, was to take a deep breath and reflect on the magnitude of the responsibility we have here.

The way ISRG is translating that sense of responsibility to action today is probably best described as a focus on agility and resilience. We need to assume that, despite our best efforts trying to prevent issues, unexpected and unfortunate events will happen and we need to position ourselves to handle them.

Back in March of 2020 Let’s Encrypt needed to respond to a compliance incident that affected nearly three million certificates. That meant we needed to get our subscribers to renew those three million certificates in a very short period of time or the sites might have availability issues. We dealt with that incident pretty well considering the remediation options available, but it was clear that incremental improvements would not make enough of a difference for events like this in the future. We needed to introduce systems that would allow us to be significantly more agile and resilient going forward.

Since then we’ve developed a specification for automating certificate renewal signals so that our subscribers can handle revocation/renewal events as easily as they can get certificates in the first place (it just happens automatically in the background!). That specification is making its way through the IETF standards process so that the whole ecosystem can benefit, and we plan to deploy it in production at Let’s Encrypt shortly. Combined with other steps we’ve taken in order to more easily handle renewal traffic surges, Let’s Encrypt should be able to respond on a whole different level the next time we need to ask significant numbers of subscribers to renew early.

This kind of work on agility and resilience is critical if we’re going to improve security and privacy at scale on the Web.

Our Divvi Up team has made a huge amount of progress implementing a new service that will bring privacy respecting metrics to millions of people. Applications collect all kinds of metrics: some of them are sensitive, some of them aren’t, and some of them seem innocuous but could reveal private information about a person. We’re making it possible for apps to get aggregated, anonymized metrics that give insight at a population level while protecting the privacy of the people who are using those apps. Everybody wins - users get great privacy and apps get the metrics they need without handling individual user data. As we move into 2023, we’ll continue to grow our roster of beta testers and partners.

Our Prossimo project started in 2020 with a clear goal: move security sensitive software infrastructure to memory safe code. Since then, we’ve gotten a lot of code written to improve memory safety on the Internet.

We’re ending the year with Rust support being merged into the Linux kernel and the completion of a memory safe NTP client and server implementation. We’re thrilled about the potential for a more memory safe kernel, but now we need to see the development of drivers in Rust. We’re particularly excited about an NVMe driver that shows excellent initial performance metrics while coming with the benefit of never producing a memory safety bug. We are actively working to make similar progress on Rustls, a high-performance TLS library, and Trust-DNS, a fully recursive DNS resolver.

All of this is made possible by charitable contributions from people like you and organizations around the world. Since 2015, tens of thousands of people have given to our work. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations. That’s all added up to $17M that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and support us financially if you can.

Artwork by Hugh D’Andrade

Peter Eckersley, a Let’s Encrypt co-founder, passed away unexpectedly on September 2nd from complications of cancer treatment. As an incredibly kind, bright, and energetic person, he was a beloved member of the community of people working to make the Internet a better place. He played an important role in the founding of Let’s Encrypt and his loss is felt deeply by many in our organization.

Peter met Alex Halderman at the RSA Conference in 2012 and the two of them started to make plans for technology to automate the process of acquiring HTTPS certificates. This work included early designs for what would become the ACME protocol. Peter and Alex later teamed up with a parallel effort by Josh Aas and Eric Rescorla at Mozilla, and the four of us worked together to create a new automated public benefit CA. The result was Let’s Encrypt, which began service in 2015.

Peter also led the development of the initial ACME client, which would eventually become Certbot. In a reflection of Peter’s vision for making the Internet secure by default, Certbot aims to fully automate HTTPS deployment, rather than simply procure a certificate. Today, Certbot is among the most popular ACME clients, and it is developed and maintained by Peter’s former team at the Electronic Frontier Foundation (EFF).

Peter was a member of our Board of Directors for several years. We greatly valued his contributions as a Director, but one of the memories from that time that makes us smile the most is Peter’s habit of showing up to board meetings with a messenger bag over his shoulder, helmet hair, and rosy cheeks from arriving by bike.

Making change at scale on the Internet is not easy. One way to get it done is to be both a dreamer and someone who possesses the deep technical knowledge necessary to bring dreams to reality. Peter was one of those people, and we’re grateful to have been able to work with him.

We hope to honor Peter’s life by letting the qualities we admired so much in him - his energy, optimism, kindness, and pursuit of knowledge - inspire our efforts going forward.

Peter's longtime friend and colleague Seth Schoen, who was among the earliest contributors to Let’s Encrypt and Certbot, further memorializes Peter in a post on our community forum.

A Brief History of Revocation

When a certificate becomes untrustworthy (for instance because its private key was compromised), that certificate must be revoked and that information publicized so that no one relies upon it in the future. However, it’s a well-worn adage in the world of the Web Public Key Infrastructure (the Web PKI) that revocation is broken. Over the history of the Web PKI, there have been two primary mechanisms for declaring that a TLS/SSL certificate should no longer be trusted: Certificate Revocation Lists (CRLs) and the Online Certificate Status Protocol (OCSP). Unfortunately, both have major drawbacks.

CRLs are basically just lists of all of the certificates that a given Certificate Authority (CA) has issued which have been revoked. This means that they’re often very large – easily the size of a whole movie. It’s inefficient for your browser to download a giant list of revoked certificates just to check if the single certificate for the site you’re visiting right now is revoked. These slow downloads and checks made web page loads slow, so OCSP was developed as an alternative.

OCSP is sort of like “what if there were a separate CRL for every single certificate”: when you want to check whether a given certificate has been revoked, your browser can check the status for just that one certificate by contacting the CA’s OCSP service. But because OCSP infrastructure has to be running constantly and can suffer downtime just like any other web service, most browsers treat getting no response at all as equivalent to getting a “not revoked” response. This means that attackers can prevent you from discovering that a certificate has been revoked simply by blocking all of your requests for OCSP information. To help reduce load on a CA’s OCSP services, OCSP responses are valid and can be cached for about a week. But this means that clients don’t retrieve updates very frequently, and often continue to trust certificates for a week after they’re revoked. And perhaps worst of all: because your browser makes an OCSP request for every website you visit, a malicious (or legally compelled) CA could track your browsing behavior by keeping track of what sites you request OCSP for.

So both of the existing solutions don’t really work: CRLs are so inefficient that most browsers don’t check them, and OCSP is so unreliable that most browsers don’t check it. We need something better.

Browser-Summarized CRLs

One possible solution that has been making headway recently is the idea of proprietary, browser-specific CRLs. Although different browsers are implementing this differently (e.g. Mozilla calls theirs CRLite, and Chrome’s are CRLSets), the basic idea is the same.

Rather than having each user’s browser download large CRLs when they want to check revocation, the browser vendor downloads the CRLs centrally. They process the CRLs into a smaller format such as a Bloom filter, then push the new compressed object to all of the installed browser instances using pre-existing rapid update mechanisms. Firefox, for example, is pushing updates as quickly as every 6 hours.

This means that browsers can download revocation lists ahead of time, keeping page loads fast and mitigating the worst problems of vanilla CRLs. It keeps revocation checks local, and the pushed updates can take immediate effect without waiting for a potentially days-long OCSP cache to expire, preventing all of the worst problems with OCSP.

Thanks to the promise of these browser-summarized CRLs, both the Apple and Mozilla root programs are requiring that all CAs begin issuing CRLs before October 1st, 2022. Specifically, they are requiring that CAs begin issuing one or more CRLs which together cover all certificates issued by that CA, and that the list of URLs pointing to those CRLs be disclosed in the Common CA Database (CCADB). This will allow Safari and Firefox to switch to using browser-summarized CRL checking for revocation.

Our New Infrastructure

When Let’s Encrypt was founded, we made an explicit decision to only support OCSP and not produce CRLs at all. This was because the root program requirements at the time only mandated OCSP, and maintaining both revocation mechanisms would have increased the number of places where a bug could lead to a compliance incident.

When we set out to develop CRL infrastructure, we knew we needed to build for scale, and do so in a way that reflects our emphasis on efficiency and simplicity. Over the last few months we have developed a few new pieces of infrastructure to enable us to publish CRLs in compliance with the upcoming requirements. Each component is lightweight, dedicated to doing a single task and doing it well, and will be able to scale well past our current needs.

Let’s Encrypt currently has over 200 million active certificates on any given day. If we had an incident where we needed to revoke every single one of those certificates at the same time, the resulting CRL would be over 8 gigabytes. In order to make things less unwieldy, we will be dividing our CRLs into 128 shards, each topping out at a worst-case maximum of 70 megabytes. We use some carefully constructed math to ensure that – as long as the number of shards doesn’t change – all certificates will remain within their same shards when the CRLs are re-issued, so that each shard can be treated as a mini-CRL with a consistent scope.

In line with the same best practices that we follow for our certificate issuance, all of our CRLs will be checked for compliance with RFC 5280 and the Baseline Requirements before they are signed by our issuing intermediates. Although the popular linting library zlint does not yet support linting CRLs, we have written our own collection of checks and hope to upstream them to zlint in the future. These checks will help prevent compliance incidents and ensure a seamless issuance and renewal cycle.

As part of developing these new capabilities, we have also made several improvements to the Go standard library’s implementation of CRL generation and parsing. We look forward to contributing more improvements as we and the rest of the Go community work with CRLs more frequently in the future.

Although we will be producing CRLs which cover all certificates that we issue, we will not be including those URLs in the CRL Distribution Point extension of our certificates. For now, as required by the Baseline Requirements, our certificates will continue to include an OCSP URL which can be used by anyone to obtain revocation information for each certificate. Our new CRL URLs will be disclosed only in CCADB, so that the Apple and Mozilla root programs can consume them without exposing them to potentially large download traffic from the rest of the internet at large.

The Future of Revocation

There’s still a long way to go before revocation in the Web PKI is truly fixed. The privacy concerns around OCSP will only be mitigated once all clients have stopped relying on it, and we still need to develop good ways for non-browser clients to reliably check revocation information.

We look forward to continuing to work with the rest of the Web PKI community to make revocation checking private, reliable, and efficient for everyone.

If you’re excited about our work developing more robust and private revocation mechanisms, you can support us with a donation, or encourage your company or organization to sponsor our work. As a nonprofit project, 100% of our funding comes from contributions from our community and supporters, and we depend on your support.

Our Oak log is the only qualified CT log that runs on an entirely open source stack ³. In the interest of lowering the barrier for other organizations to join the CT ecosystem, we want to cover a few recent changes to Oak that might be helpful to anyone else planning to launch a log based on Google’s Trillian backed by MariaDB:

• The disk I/O workload of Trillian atop MariaDB is easily mediated by front-end rate limits, and

• It’s worth the complexity to split each new annual CT log into its own Trillian/MariaDB stack.

This post will update some of the information from the previous post How Let’s Encrypt Runs CT Logs.

Growing Oak While Staying Open Source

Oak runs on a free and open source stack: Google’s Trillian data store, backed by MariaDB, running at Amazon Web Services (AWS) via Amazon’s Relational Database Service (RDS). To our knowledge, Oak is the only trusted CT log without closed-source components ³.

Other operators of Trillian have opted to use different databases which segment data differently, but the provided MySQL-compatible datastore has successfully kept up with Let’s Encrypt’s CT log volume (currently above 400 GB per month). The story for scaling Oak atop MariaDB is quite typical for any relational database, though the performance requirements are stringent.

Keeping Oak Qualified

The policies that Certificate Transparency Log operators follow require there to be no significant downtime, in addition to the more absolute and difficult requirement that the logs themselves make no mistakes: Given the append-only nature of Certificate Transparency, seemingly minor data corruption prompts permanent disqualification of the log ⁴. To minimize the impacts of corruption, as well as for scalability reasons, it’s become normal for CT logs to distribute the certificates they contain in different, smaller individual CT logs, called shards.

Splitting Many Years Of Data Among Many Trees

The Let’s Encrypt Oak CT log is actually made up of many individual CT log shards each named after a period of time: Oak 2020 contains certificates which expired in 2020; Oak 2022 contains certificates which expire in 2022. For ease of reference, we refer to these as “temporal log shards,” though in truth each is an individual CT log sharing the Oak family name.

It is straightforward to configure a single Trillian installation to support multiple CT log shards. Each log shard is allocated storage within the backing database, and the Trillian Log Server can then service requests for all configured logs.

The Trillian database schema is quite compact and easy to understand:

• Each configured log gets a Tree ID, with metadata in several tables.

• All log entries – certificates in our case – get a row in LeafData.

• Entries that haven’t been sequenced yet get a row in the table Unsequenced, which is normally kept empty by the Trillian Log Signer service.

• Once sequenced, entries are removed from the Unsequenced table and added as a row in SequencedLeafData.

In a nutshell: No matter how many different certificate transparency trees and subtrees you set up for a given copy of Trillian, all of them will store the lion’s share of their data, particularly the DER-encoded certificates themselves, interwoven into the one LeafData table. Since Trillian Log Server can only be configured with a single MySQL connection URI, limiting it to a single database, that single table can get quite big.

For Oak, the database currently grows at a rate of about 400 GB per month; that rate is ever-increasing as the use of TLS grows and more Certificate Authorities submit their certificates to our logs.

Amazon RDS Size Limitations

In March 2021 we discovered that Amazon RDS has a 16TB limit per tablespace when RDS is configured to use one file-per-table, as we were doing for all of our CT log shards. Luckily, we reached this limit first in our testing environment, the Testflume log.

Part of Testflume’s purpose was to grow ahead of the production logs in total size, as well as test growth with more aggressive configuration options than the production Oak log had, and in these ways it was highly successful.

Revisiting Database Design

In our blog post, How Let’s Encrypt Runs CT Logs, we wrote that each year we planned “to freeze the previous year’s shard and move it to a less expensive serving infrastructure, reclaiming its storage for our live shards.” However, that is not practical while continuing to serve traffic from the same database instance. Deleting terabytes of rows from an InnoDB table that is in-use is not feasible. Trillian’s MySQL-compatible storage backend agrees: as implemented, Trillian’s built-in Tree Deletion mechanism marks a tree as “soft deleted,” and leaves the removal of data from the LeafData table (and others) as an exercise for the administrator.

Since Trillian’s MySQL-compatible backend does not support splitting the LeafData among multiple tables by itself, and since deleting stale data from those tables yields slow performance across the whole database server, to continue to scale the Oak CT log we have to instead prune out the prior seasons’ data another way.

Single RDS Instance with Distinct Schema per Log Shard

We considered adding new database schemas to our existing MariaDB-backed Amazon RDS instance. In this design, we would run a Trillian CT Front-End (CTFE) instance per temporal log shard, each pointing to individual Trillian Log Server and Log Signer instances, which themselves point to a specific temporally-identified database schema name and tablespace. This is cost-effective, and it gives us ample room to avoid the 16 TB limit.

However, if heavy maintenance is required on any part of the underlying database, it would affect every log shard contained within. In particular, we know from using MariaDB with InnoDB inside the Let’s Encrypt CA infrastructure that truncating and deleting a multi-terabyte table causes performance issues for the whole database while the operation runs. Inside the CA infrastructure we mitigate that performance issue by deleting table data only on database replicas; this is more complicated in a more hands-off managed hosting environment like RDS.

Since we wish to clear out old data regularly as a matter of data hygiene, and the performance requirements for a CT log are strict, this option wasn’t feasible.

Distinct RDS Instance per Log Shard

While it increases the number of managed system components, it is much cleaner to give each temporal log shard its own database instance. Like the Distinct Schema per Log Shard model, we now run Trillian CTFE, Log Server, and Log Signer instances for each temporal log shard. However, each log shard gets its own RDS instance for the active life of the log ⁵. At log shutdown, the RDS instance is simply deprovisioned.

With the original specifications for the Oak log, this would require allocating a significant amount of data I/O resources. However, years of experience running the Testflume log showed that Trillian in AWS did not require the highest possible disk performance.

Tuning IOPS

We launched Oak using the highest performance AWS Elastic Block Storage available at the time: Provisioned IOPS SSDs (type io1). Because of the strict performance requirements on CT logs, we worried that without the best possible performance for disk I/O that latency issues might crop up that could lead to disqualification. As we called out in our blog post How Let’s Encrypt Runs CT Logs, we hoped that we could use a simpler storage type in the future.

To test that, we used General Purpose SSD storage type (type gp2) for our testing CT log, Testflume, and obtained nominal results over the lifespan of the log. In practice higher performance was unnecessary because Trillian makes good use of database indices. Downloading the whole log tree from the first leaf entry is the most significant demand of disk I/O, and that manner of operation is easily managed via rate limits at the load balancer layer.

Our 2022 and 2023 Oak shards now use type gp2 storage and are performing well.

Synergistically, the earlier change to run a distinct RDS instance for each temporal log shard has also further reduced Trillian’s I/O load: A larger percentage of the trimmed-down data fits in MariaDB’s in-memory buffer pool.

More Future Improvements

It’s clear that CT logs will continue to accelerate their rate of growth. Eventually, if we remain on this architecture, even a single year’s CT log will exceed the 16 TB table size limit. In advance of that, we’ll have to take further actions. Some of those might be:

• Change our temporal log sharding strategy to shorter-than-year intervals, perhaps every 3 or 6 months.

• Reduce the absolute storage requirements for Trillian’s MySQL-compatible storage backend by de-duplicating intermediate certificates.

• Contribute a patch to add table sharding to Trillian’s MySQL-compatible storage backend.

• Change storage backends entirely, perhaps to a sharding-aware middleware, or another more horizontally-scalable open-source system.

We’ve also uprooted our current Testflume CT log and brought online a replacement which we’ve named Sapling. As before, this test-only log will evaluate more aggressive configurations that might bear fruit in the future.

As Always, Scaling Data Is The Hard Part

Though the performance requirements for CT logs are strict, the bulk of the scalability difficulty has to do with the large amount of data and the high and ever-increasing rate of growth; this is the way of relational databases. Horizontal scaling continues to be the solution, and is straightforward to apply to the open source Trillian and MariaDB stack.

Supporting Let’s Encrypt

As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. If you can support us with a donation, we ask that you make an individual contribution.

MongoDB’s managed multicloud database service, called Atlas, uses Let’s Encrypt certificates to secure the connection between customers’ applications and MongoDB databases, and between service points inside the platform. We spoke with Kenn White, Security Principal at MongoDB, about how his team uses Let’s Encrypt certificates for over two million databases, across 200 datacenters and three cloud providers.

"Let’s Encrypt has become a core part of our infrastructure stack," said Kenn. Interestingly, our relationship didn’t start out that way. MongoDB became a financial sponsor of Let’s Encrypt years earlier simply to support our mission to pursue security and privacy. MongoDB Atlas began to take off and it became clear that TLS would continue to be a priority as they brought on customers like currency exchanges, treasury platforms and retail payment networks. "The whole notion of high automation and no human touch all really appealed to us," said Kenn of MongoDB’s decision to use Let’s Encrypt.

MongoDB’s diverse customer roster means they support a wide variety of languages, libraries, and operating systems. Consequently, their monitoring is quite robust. Over the years, MongoDB has become a helpful resource for Let’s Encrypt engineers to identify edge case implementation bugs. Their ability to accurately identify issues early helps us respond efficiently; this is a benefit that ripples out across our diverse subscribers all over the Web.

The open sharing of information is a core part of how Let’s Encrypt operates. In fact, "transparency" is one of our key operating principles. The ability to see and understand how Let’s Encrypt is changing helped MongoDB gain trust and confidence in our operations. "I don’t think you can really put a price on the experience we’ve had working with the Let’s Encrypt engineering team," said Kenn. "One thing that I appreciate about Let’s Encrypt is that you’ve always been extremely transparent on your priorities and your roadmap vision. In terms of the technology and your telemetry, this is an evolution; where you are today is far better than where you were two years ago. And two years ago you were already head and shoulders above almost every peer in the industry."

Check out other blog posts in this series about how other large subscribers use Let’s Encrypt certificates.

TLS Simply and Automatically for Europe’s Largest Cloud Customers

Speed at scale: Let’s Encrypt serving Shopify’s 4.5 million domains

Supporting Let’s Encrypt

As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. If you can support us with a donation, we ask that you make an individual contribution.

ISRG's pragmatic, public-interest approach to Internet security has fundamentally changed the web at an astonishing scale and pace.

The Internet has considerable potential to help build a more just, equitable, and sustainable world for all people. Yet for everyone online—and indeed the billions not yet online—barriers to secure and privacy-respecting communication remain pervasive.

ISRG was founded in 2013 to find and eliminate these barriers. Today, we’re proud to announce a $1M grant from the Ford Foundation to continue our efforts.

Our first project, Let’s Encrypt, leverages technology whose foundation has existed for nearly three decades—TLS certificates for securely communicating information via HTTP. Yet even for people well-versed in technology, adopting TLS proved daunting.

Before Let’s Encrypt, the growth rate for HTTPS page loads merely puttered along. As recently as 2013, just 25% of websites used HTTPS. In order for the Internet to reach its full potential, this glaring risk to people’s security and privacy needed to be mitigated.

Let’s Encrypt changed the paradigm. Today 81% of website page loads use HTTPS. That means that you and the other 4.9 billion people online can leverage the Internet for your own pursuits with a greater degree of security and privacy than ever before.

But TLS adoption was just one hurdle. Much can be done to further improve the Internet’s most critical pieces of technology to be more secure; much can be done to further improve the privacy of everyone using the Internet today.

Building our efforts thanks to transformational support

Ford Foundation’s commitment recognizes that the Internet can be a technological tool to build a more just, equitable, and sustainable world, but that it will take organizations like ISRG to help build it.

“Ford Foundation is one of the most respected grantmaking institutions in the world,” Josh Aas, ISRG Executive Director, said. “We are proud that Ford believes in the impact we’ve created and the potential of our efforts to continue benefiting everyone using the Internet.”

This support, which began in 2021, will help ISRG continue to invest in Let’s Encrypt and our other projects, Prossimo and Divvi Up.

Launched in late 2020, Prossimo intends to move the Internet's most critical security-sensitive software infrastructure to memory safe code. Society pays the price for these vulnerabilities with privacy violations, staggering financial losses, denial of public services (e.g., hospitals, power grids), and human rights violations. Meaningful effort will be required to bring about such change, but the Internet will be around for a long time. There is time for ambitious efforts to pay off.

Divvi Up is a system for privacy-preserving metrics analysis. With Divvi Up, organizations can analyze and share data to further their aims without sacrificing their users’ privacy. Divvi Up is currently used for COVID-19 Exposure Notification apps and has processed over 14 billion metrics to aid Public Health Authorities to better hone their app to be responsive to their local populations.

"ISRG's pragmatic, public-interest approach to Internet security has fundamentally changed the web at an astonishing scale and pace,” Michael Brennan of the Ford Foundation said. "I believe their new projects have the same potential and I am eager to see what they turn their sights to next."

We’re grateful to Ford for their support of our efforts, and to all of you who have contributed time and resources to our projects. For more information on ISRG and our projects, take a read through our 2021 Annual Report. 100% of ISRG’s funding comes from contributed sources. If you or your organization are interested in helping advance our mission, consider becoming a sponsor, making a one-time contribution, or reaching out with your idea on how you can help financially support our mission at sponsor@abetterinternet.org.

We can do a lot to improve security and privacy on the Internet by taking existing ideas and applying them in ways that benefit the general public at scale. Our work certainly does involve some research, as our name implies, but the success that we’ve had in pursuing our mission largely comes from our ability to go from ideas to implementations that improve the lives of billions of people around the world.

Our first major project, Let’s Encrypt, now helps to protect more than 260 million websites by offering free and fully automated TLS certificate issuance and management. Since it launched in 2015, encrypted page loads have gone from under 40% to 92% in the U.S. and 83% globally.

We didn’t invent certificate authorities. We didn’t invent automated issuance and management. We refined those ideas and applied them in ways that benefit the general public at scale.

We launched our Prossimo project in late 2020. Our hope is that this project will greatly improve security and privacy on the Internet by making memory safety vulnerabilities in the Internet’s most critical software a thing of the past. We’re bringing a healthy dose of ambition to the table and we’re backing it up with effective strategies and strong partnerships.

Again, we didn’t invent any memory safe languages or techniques, and we certainly didn’t invent memory safety itself. We’re simply taking existing ideas and applying them in ways that benefit the general public at scale. We’re getting the work done.

With our latest project, Divvi Up for Privacy Preserving Metrics (PPM), the core ideas are a bit newer than the ideas behind our other projects, but we didn’t invent them either. Over the past decade or so some bright people have come up with a way to resolve the tension between wanting to collect metrics about populations and needing to collect data about individuals.

We believe those ideas have matured enough that it’s time to deploy them to the public’s benefit. We started by building and deploying a PPM service for Covid-19 Exposure Notification applications in late 2020, in partnership with Apple, Google, the Bill & Melinda Gates Foundation and the Linux Foundation. We’re expanding that service so any application can collect metrics in a privacy-preserving way.

Being ready to bring ideas to life means a few different things.

We need to have an excellent engineering team that knows how to build services at scale. It’s not enough to just build something that works - the quality and reliability of our work needs to inspire confidence. People need to be able to rely on us.

We also need to have the experience, perspective, and capacity to effectively consider ideas. We are not an organization that “throws things at the wall to see what sticks.” Between our staff, our board of directors, our partners, and our community, we’re able to do a great job evaluating opportunities to understand technical feasibility, potential impact, and alignment with our public benefit mission—to reduce financial, technological, and educational barriers to secure communication over the Internet.

Administrative and communications capabilities are essential. From fundraising and accounting to legal and social media, our administrative teams exist in order to support and amplify the critical work that we do. We’re proud to run a financially efficient organization that provides services for billions of people on only a few million dollars each year.

Finally, it means having the financial resources we need to function. As a nonprofit, 100% of our funding comes from charitable contributions from people like you and organizations around the world. But global impact doesn’t necessarily require million dollar checks: since 2015 tens of thousands of people have given to our work. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to $17M that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and support us financially if you can.

OVHcloud first started looking into using Let’s Encrypt certificates because the team saw a need for the protection provided by TLS for every customer (remember, way back five years ago, when that wasn’t just a thing everybody did?). “Our goal was to deliver TLS simply. We didn’t want to have to write a tutorial for our customers to upload a cert, but instead just click and it works,” said Guillaume Marchand, OVHcloud’s Technical Team Lead.

They considered building their own CA but determined the cost and complexity of doing so would be impractical. Instead, they built an ACME client to prepare for using Let’s Encrypt. It took about six months, “we simply followed the RFC and did a bit of reverse engineering of Certbot,” said Guillaume. In addition to a custom client, OVHcloud automated their Certificate Signing Request (CSR) process and certificate installation process.

Getting a TLS certificate is on the critical path to onboarding a shared hosting client, so monitoring is a big part of OVHcloud’s success with Let’s Encrypt. They set up monitoring at every step in the delivery process: requesting the certificate, asking for challenges, waiting for validation, and requesting certificate creation. They also keep an eye on how long it takes to get a certificate (“it’s really fast”). OVHcloud also monitors our status page to stay apprised of our operational status.

Over 10,000 certificates are issued from Let’s Encrypt to OVHcloud every day. As the company continues to expand into North America, they predict that number will grow. The initial and ongoing work done by the OVHcloud team ensures that TLS will be a simple and reliable aspect of their service.

OVHcloud is a longtime sponsor of ISRG so we’d like to close by thanking them for not just being great technical collaborators, but also financial supporters.

Check out our blog post about how Shopify uses Let’s Encrypt certificates for another example of how our certificates are used in the enterprise.

Supporting Let’s Encrypt

As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. If you can support us with a donation, we ask that you make an individual contribution.

The Internet Society has supported our work toward a 100% encrypted Web since before we’d even issued our first certificate. Their commitment to helping us execute our vision has been a substantial help over the years. Today, I’m excited to invite Christine Runnegar, Senior Director at The Internet Society and member of ISRG’s Board of Directors, to share her thoughts.

-Josh Aas, Executive Director, ISRG & Let’s Encrypt

Today, across the world, communities, organizations, and individuals are celebrating Global Encryption Day. Organized by the Global Encryption Coalition (GEC), it’s a day to take stock of the crucial role that encryption plays in securing our communications on the Internet.

The Internet Society is a GEC Steering Committee member because access to encryption is a key tool for us to realize our mission of keeping the Internet a force for good. That’s why the Internet Society is also a proud financial sponsor of Internet Security Research Group (ISRG), which founded and operates Let’s Encrypt. Let’s Encrypt provides digital certificates to more than 260 million websites, making a more secure and privacy-respecting Web for users all over the world. In just five years, the percentage of Web pages loaded over HTTPS has risen from under 50% to more than 85% and climbing, principally because of the community that has coalesced around the importance of encryption everywhere. Encrypted Web traffic protects the confidentiality and integrity of information users share with, or learn from, websites. It makes us all safer online.

Let’s Encrypt is a great success story, and an outstanding example of how supporting public interest infrastructure, such as a certificate authority operated for the public’s benefit, helps ensure everyone has access to the benefits of encryption.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

For most websites, it was just another day on the Internet, but inevitably with such a big change some sites and configurations have issues. Our overview of the planned expiration is here. You can read about what we’ve done to make the process smoother. Most problems can be solved by updating the software on the machine that is having trouble.

You may also find these links helpful:

Our certificate compatibility page.

Workarounds for OpenSSL 1.0.2.

Whenever there is a significant change to our API, we post in the API Announcements category in our community forum. Sign in and click the bell for notifications to be sent to your email! If you want to hear even more from Let’s Encrypt and the nonprofit team behind it, subscribe to our newsletter. You’ll only receive a handful of emails each year.

We (and our community) are here for you! If you have any questions about this change, search on our community forum or post on the thread we have to help you with this very topic.

Supporting Let’s Encrypt

As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. If you can support us with a donation, we ask that you make an individual contribution.

Today, Let’s Encrypt provides TLS for 4.5 million Shopify domains. We sat down with Charles Barbier, Development Manager at Shopify, to hear why Let’s Encrypt is their choice for reliable, free, and automated TLS at scale.

“In 2016, the TLS team started transitioning all of our merchants’ stores to HTTPS through Let’s Encrypt,” Charles said. “And when we started exploring the concept a few years earlier, it was a daunting task.” Implementing TLS for 680,000+ domains wasn’t just daunting, Charles and the team needed automated management, something that simply didn’t exist. “We didn’t want to have TLS be the merchant’s responsibility,” Charles said.

Back in 2016, although Let’s Encrypt had been making noise, it wasn’t Shopify’s first choice for a CA. “We ended up going with a different option that turned out to be problematic because the API was so slow,” Charles said. “We did some napkin math and realized it was going to take us around 100 days to provision all of our certs for our merchants. If this solution had been just for regular issuance, it would have been fine, but an emergency would be very problematic.”

That realization led Charles and the team to give Let’s Encrypt a try, making them one of the first single Let’s Encrypt subscribers to request and provision certs at a X00,000 scale. “We were able to roll out all of our domains in a couple of hours,” Charles said. “And to be frank, I think it was our ordering process that caused issuance to take even that long. It was very encouraging.”

The speed of Let’s Encrypt helped Shopify realize their goal of provisioning certs for all of their domains and automating management. Since Let’s Encrypt uses the IETF-standardized ACME protocol, Shopify felt confident that if they needed to, they could roll over to a different ACME CA. “We knew in the future, if things went well with the ACME standard, we’d be able to add a different ACME provider with the exact same implementation,” Charles said.

Of course, “things going well” doesn’t just mean technically. It means ensuring the nonprofit behind Let’s Encrypt is sound as well—which is why Shopify has financially supported Let’s Encrypt since they began using it in 2016. This year, they increased their support. “For us, using Let’s Encrypt has been a great experience,” Charles said.

Today, Let’s Encrypt certificates cover 4.5 million domains for Shopify. That means a more secure and privacy-respecting Web for all of Shopify’s merchants who, in 2020, created $307 billion in economic impact around the world. And it means a more secure Web for everyone visiting and engaging with a Shopify merchant.

We’re proud to serve Shopify with a reliable, speedy, and free service, and grateful for their longtime support of our work by being a sponsor. Together, we’re helping bolster a Web that’s free, open and more secure for everyone, everywhere.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

We recently completed most of the work and investments needed to issue 200 million certificates in a day and we thought we’d let people know what was involved. All of this was made possible by our sponsors and funders, including major hardware contributions from Cisco, Thales, and Fortinet.

Scenarios

Security and compliance events are a certainty in our industry. We obviously try to minimize them, but since we anticipate they will happen, we spend a lot of time preparing to respond in the best ways possible.

In February of 2020 a bug affecting our compliance caused us to need to revoke and replace three million active certificates. That was approximately 2.6% of all active certificates.

What if that bug had affected all of our certificates? That’s more than 150 million certificates covering more than 240 million domains. What if it had also been a more serious bug, requiring us to revoke and replace all certificates within 24 hours? That’s the kind of worst case scenario we need to be prepared for.

To make matters worse, during an incident it takes some time to evaluate the problem and make decisions, so we would be starting revocation and reissuance somewhat after the beginning of the 24-hour clock. That means we would actually have less time once the momentous decision has been made to revoke and replace 150 million or more certificates.

Infrastructure Improvements

After reviewing our systems, we determined that there were four primary bottlenecks that would prevent us from replacing 200 million certificates in a day. Database performance, internal networking speed, cryptographic signing module (HSM) performance, and bandwidth.

Database Performance

Let’s Encrypt has a primary certificate authority database at the heart of the service we offer. This tracks the status of certificate issuance and the associated accounts. It is write heavy, with plenty of reads as well. At any given time a single database server is the writer, with some reads directed at identical machines with replicas. The single writer non-clustered strategy helps with consistency and reduces complexity, but it means that writes and some reads must operate within the performance constraints of a single machine.

Our previous generation of database servers had dual Intel Xeon E5-2650 v4 CPUs, 24 physical cores in total. They had 1TB memory with 24 3.8TB SSDs connected via SATA in a RAID 10 configuration. These worked fine for daily issuance but would not be able to handle replacing all of our certificates in a single day.

We have replaced them with a new generation of database server from Dell featuring dual AMD EPYC 7542 CPUs, 64 physical cores in total. These machines have 2TB of faster RAM. Much faster CPUs and double the memory is great, but the really interesting thing about these machines is that the EPYC CPUs provide 128 PCIe4 lanes each. This means we could pack in 24 6.4TB NVME drives for massive I/O performance. There is no viable hardware RAID for NVME, so we’ve switched to ZFS to provide the data protection we need.

Internal Networking

Let’s Encrypt infrastructure was originally built with 1G copper networking. Between bonding multiple connections for 2G performance, and use of the very limited number of 10G ports available on our switches, it served us pretty well until 2020.

By 2020 the volume of data we were moving around internally was too much to handle efficiently with 1G copper. Some normal operations were significantly slower than we’d like (e.g. backups, replicas), and during incidents the 1G network could cause significant delays in our response.

We originally looked into upgrading to 10G, but learned that upgrading to 25G fiber wasn’t much more expensive. Cisco ended up generously donating most of the switches and equipment we needed for this upgrade, and after replacing a lot of server NICs Let’s Encrypt is now running on a 25G fiber network!

Funny story - back in 2014 Cisco had donated really nice 10G fiber switches to use in building the original Let’s Encrypt infrastructure. Our cabinets at the time had an unusually short depth though, and the 10G switches didn’t physically fit. We had to return them for physically smaller 1G switches. 1G seemed like plenty at the time. We have since moved to new cabinets with standard depth!

HSM Performance

Each Let’s Encrypt data center has a pair of Luna HSMs that sign all certificates and their OCSP responses. If we want to revoke and reissue 200 million certificates, we need the Luna HSMs to perform the following cryptographic signing operations:

• 200 million OCSP response signatures for revocation
• 200 million certificate signatures for replacement certificates
• 200 million OCSP response signatures for the new certificates

That means we need to perform 600 million cryptographic signatures in 24 hours or less, with some performance overhead to account for request clustering.

We need to assume that during incidents we might only have one data center online, so when we’re thinking about HSM performance we’re thinking about what we can do with just one pair. Our previous HSMs could perform approximately 1,100 signing operations each per second, 2,200 between the pair. That’s 190,080,000 signatures in 24 hours, working at full capacity. That isn’t enough.

In order to get to where we need to be Thales generously donated new HSMs with about 10x the performance - approximately 10,000 signing operations per second, 20,000 between the pair. That means we can now perform 864,000,000 signing operations in 24 hours from a single data center.

Bandwidth

Issuing a certificate is not particularly bandwidth intensive, but during an incident we typically use a lot more bandwidth for systems recovery and analysis. We move large volumes of logs and other files between data centers and the cloud for analysis and forensic purposes. We may need to synchronize large databases. We create copies and additional back-ups. If the connections in our data centers are slow, it slows down our response.

After determining that data center connection speed could add significantly to our response time, we increased bandwidth accordingly. Fortinet helped provide hardware that helps us protect and manage these higher capacity connections.

API Extension

In order to get all those certificates replaced, we need an efficient and automated way to notify ACME clients that they should perform early renewal. Normally ACME clients renew their certificates when one third of their lifetime is remaining, and don’t contact our servers otherwise. We published a draft extension to ACME last year that describes a way for clients to regularly poll ACME servers to find out about early-renewal events. We plan to polish up that draft, implement, and collaborate with clients and large integrators to get it implemented on the client side.

Supporting Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

What exactly are we doing with these servers?

Our CA software, Boulder, uses MySQL-style schemas and queries to manage subscriber accounts and the entire certificate issuance process. It’s designed to work with a single MySQL, MariaDB, or Percona database. We currently use MariaDB, with the InnoDB database engine.

We run the CA against a single database in order to minimize complexity. Minimizing complexity is good for security, reliability, and reducing maintenance burden. We have a number of replicas of the database active at any given time, and we direct some read operations to replica database servers to reduce load on the primary.

One consequence of this design is that our database machines need to be pretty powerful. Eventually we may need to shard or break the single database into multiple databases, but hardware advancements have allowed us to avoid that so far.

Hardware Specifications

The previous generation of database hardware was powerful but it was regularly being pushed to its limits. For the next generation, we wanted to more than double almost every performance metric in the same 2U form factor. In order to pull that off, we needed AMD EPYC chips and Dell’s PowerEdge R7525 was ideal. Here are the specifications:

By going with AMD EPYC, we were able to get 64 physical CPU cores while keeping clock speeds high: 2.9GHz base with 3.4GHz boost. More importantly, EPYC provides 128 PCIe v4.0 lanes, which allows us to put 24 NVMe drives in a single machine. NVMe is incredibly fast (~5.7x faster than the SATA SSDs in our previous-gen database servers) because it uses PCIe instead of SATA. However, PCIe lanes are typically very limited: modern consumer chips typically have only 16 lanes, and Intel’s Xeon chips have 48. By providing 128 PCI lanes per chip (v4.0, no less), AMD EPYC has made it possible to pack large numbers of NVMe drives into a single machine. We’ll talk more about NVMe later.

Performance Impact

We’ll start by looking at our median time to process a request because it best reflects subscribers’ experience. Before the upgrade, we turned around the median API request in ~90 ms. The upgrade decimated that metric to ~9 ms!

We can clearly see how our old CPUs were reaching their limit. In the week before we upgraded our primary database server, its CPU usage (from /proc/stat) averaged over 90%:

The new AMD EPYC CPUs sit at about 25%. You can see in this graph where we promoted the new database server from replica (read-only) to primary (read/write) on September 15.

The upgrade greatly reduced our overall database latency. The average query response time (from INFORMATION_SCHEMA) used to be ~0.45ms.

Queries now average three times faster, about 0.15ms.

OpenZFS and NVMe

NVMe drives are becoming increasingly popular because of their incredible performance. Up until recently, though, it was nearly impossible to get many of them in a single machine because NVMe uses PCIe lanes. Those were very limited: Intel’s Xeon processors come with just 48 PCIe v3 lanes, and a number of those are used up by the chipset and add-on cards such as network adapters and GPUs. You can’t fit many NVMe drives in the remaining lanes.

AMD’s latest generation of EPYC processors come with 128 PCIe lanes - more than double what Intel offers - and they’re PCIe v4! This is enough to pack a 2U server full of NVMe drives (24 in our case).

Once you have a server full of NVMe drives, you have to decide how to manage them. Our previous generation of database servers used hardware RAID in a RAID-10 configuration, but there is no effective hardware RAID for NVMe, so we needed another solution. One option was software RAID (Linux mdraid), but we got several recommendations for OpenZFS and decided to give it a shot. We’ve been very happy with it!

There wasn’t a lot of information out there about how best to set up and optimize OpenZFS for a pool of NVMe drives and a database workload, so we want to share what we learned. You can find detailed information about our setup in this GitHub repository.

Conclusion

This database upgrade was necessary as more people rely on Let’s Encrypt for the security and privacy that TLS/SSL provides. The equipment is quite expensive and it was a sizable undertaking for our SRE team to plan and execute the transition, but we gained a lot through the process.

Support Let’s Encrypt

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

This letter was originally published in our 2020 annual report.

ISRG’s first project, Let’s Encrypt, has been wildly successful. We’re now helping to secure more than 225 million websites and the Web is making great progress towards 100% HTTPS. We’ve put in a lot of hard work and dealt with some challenges along the way, but at a high level the outlook is quite sunny. I’m incredibly proud to share some of what our organization has accomplished in 2020.

While I’m deeply appreciative of being in this position today, I don’t let it distract me, or our fantastic Board of Directors, from thinking diligently about the risks on the road ahead. A big part of our job is to look into the future, see threats and challenges, and prepare to face them as best we can. I’m sometimes asked what I view as the biggest threat to our organization and our ability to pursue our mission and my answer is simple: being taken for granted.

When digital security and privacy is your goal, ease of use has to be your focus. When we examine why real world systems aren’t secure, it usually isn’t because we don’t have the technological means to secure them. The problem is almost always that the solutions are not easy enough to use, either for implementers or consumers.

HTTPS has been around since the mid-90s but uptake was abysmally slow because SSL/TLS certificates weren’t easy to get or manage. Let’s Encrypt made getting and managing certificates easy and as a result HTTPS adoption rates shot up. Critically, the answer wasn’t to get people to think more about their certificates—we needed to make it possible for people to spend much less time thinking about certificates. Ideally we’d be invisible—server software should just get and manage certificates automatically.

Our next project after Let’s Encrypt is going live shortly: Divvi Up. It’s a system for collecting digital metrics that allows organizations to collect the information they need without any entity having the ability to access any individual user’s data. Much like Let’s Encrypt, it protects people without them having to know anything about it.

Despite 2020 being a year of unprecedented, global challenges, ISRG is well positioned for the years ahead. Our current momentum is possible through new major in-kind donations, nearly 90% of our existing sponsors renewing their support for 2020, funding from the Ford Foundation and the Bill & Melinda Gates Foundation, and by welcoming new major sponsors, including AWS, Thales, and Avast.

When your strategy as a nonprofit is to get out of the way, to offer services that people don’t need to think about, you’re running a real risk that you’ll eventually be taken for granted. There is a tension between wanting your work to be invisible and the need for recognition of its value. If people aren’t aware of how valuable our services are then we may not get the support we need to continue providing them.

How are we going to mitigate this risk? The most important thing we can do is continue to communicate effectively with people who are in a position to understand our work and support it. The most important things you can do as a supporter include being an advocate for your company sponsoring us, making an individual donation, or going over this annual report with a few people that you think should know more about us.

On behalf of the hundreds of millions of people benefiting from Let’s Encrypt around the world and our team of sixteen dedicated to this work, thank you for your support.

Update, May 13, 2021

Please visit this post on our community forum for the latest information about chain changes as some information about the changes and dates in this blog post are outdated.

We’re happy to announce that we have developed a way for older Android devices to retain their ability to visit sites that use Let’s Encrypt certificates after our cross-signed intermediates expire. We are no longer planning any changes in January that may cause compatibility issues for Let’s Encrypt subscribers.

A recurring theme in our posts about our upcoming chain switch has been our concern over the effects on users of Android operating systems prior to 7.1.1, whose devices don’t trust our ISRG Root X1. Thanks to some innovative thinking from our community and our wonderful partners at IdenTrust, we now have a solution that allows us to maintain wide compatibility. Critical to our mission as a nonprofit is to help create a more secure and privacy-respecting Web for as many people as possible. This work brings us closer to that goal.

IdenTrust has agreed to issue a 3-year cross-sign for our ISRG Root X1 from their DST Root CA X3. The new cross-sign will be somewhat novel because it extends beyond the expiration of DST Root CA X3. This solution works because Android intentionally does not enforce the expiration dates of certificates used as trust anchors. ISRG and IdenTrust reached out to our auditors and root programs to review this plan and ensure there weren’t any compliance concerns.

As such, we will be able to provide subscribers with a chain which contains both ISRG Root X1 and DST Root CA X3, ensuring uninterrupted service to all users and avoiding the potential breakage we have been concerned about.

We will not be performing our previously-planned chain switch on January 11th, 2021. Instead, we will be switching to provide this new chain by default in late January or early February. The transition should have no impact on Let’s Encrypt subscribers, much like our switch to our R3 intermediate earlier this month.

Some additional technical details follow.

I’m an ACME client developer, what do I need to do? If your client handled the X3 to R3 transition smoothly, then you shouldn’t need to take action. Ensure that your client correctly uses the intermediate certificate provided by the ACME API at the end of issuance, and doesn’t retrieve intermediates by other means (e.g. hardcoding them, reusing what is on disk already, or fetching from AIA URLs).

I’m a Let’s Encrypt subscriber, what do I need to do? In the vast majority of cases, nothing. If you want to double check, please ensure that your ACME client is up-to-date.

I use an old Android device, what do I need to do? Nothing! We’re trying to ensure that this change is completely invisible to end-users.

What exactly is changing? Today, when a subscriber fetches their certificate from our API, we provide the following chain by default: Subscriber Certificate < – R3 < – DST Root CA X3 After this change, we will instead provide this chain by default: Subscriber Certificate < – R3 < – ISRG Root X1 < – DST Root CA X3 This does mean that the default chain we provide to subscribers will be larger than before, because it contains two intermediates instead of just one. This will make TLS handshakes larger and slightly less efficient, but the tradeoff is worthwhile for the extra compatibility.

But isn’t DST Root CA X3 expiring? The self-signed certificate which represents the DST Root CA X3 keypair is expiring. But browser and OS root stores don’t contain certificates per se, they contain “trust anchors”, and the standards for verifying certificates allow implementations to choose whether or not to use fields on trust anchors. Android has intentionally chosen not to use the notAfter field of trust anchors. Just as our ISRG Root X1 hasn’t been added to older Android trust stores, DST Root CA X3 hasn’t been removed. So it can issue a cross-sign whose validity extends beyond the expiration of its own self-signed certificate without any issues.

What happens when the new cross-sign expires? This new cross-sign will expire in early 2024. Prior to that, perhaps as early as June 2021, we will be making a similar change to what we intended to make this January. When we make that change, subscribers will have the option to continue using DST Root CA X3 by configuring their ACME client to specifically request it. If you are a Let’s Encrypt subscriber who is already working to mitigate the chain switch (e.g. by configuring alternate chains or encouraging your users on old Android devices to install Firefox), continue that work so you’ll be well-positioned in the future.

What about the alternate chain? Today, some ACME clients are able to instead request an alternate chain, if their user has configured it. We currently provide the option of getting the chain: Subscriber Certificate < – R3 < – ISRG Root X1 We will continue to offer this same chain as an alternate. However, note that most ACME clients don’t yet have a way to select this alternate chain (for example, Certbot selects chains by looking to see if they contain a given Issuer Name, but this chain doesn’t contain any Issuer Names which the high compatibility chain above doesn’t). We’ll be working with ACME client developers to create more flexible chain selection mechanisms going forward.

What does this have to do with the upcoming ECDSA chain? This change is unrelated to our upcoming issuance from the ECDSA-based ISRG Root X2 and E1 intermediate. We look forward to using those to provide smaller, more efficient certificates and chains, but that future offering is not related to this change.

I have more questions, how can I get them answered? Please refer your questions to our community forum.

In the future we hope to share more technical details about how and why this plan works, demonstrate why the expiration of DST Root CA X3 doesn’t affect the security of the web, and what this plan means long-term for our subscribers and end-users on the web.

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Update, July 10, 2023

See our new blog post for details on the September 2024 expiration of the newer ISRG Root X1 cross-sign from IdenTrust.

Update, December 21, 2020

Thanks to community feedback and our wonderful partners at IdenTrust, we will be able to continue to offer service without interruption to people using older Android devices. We flagged the content of this blog post that is no longer accurate. Please visit this post on our community forum for the latest information about chain changes.

When a new Certificate Authority (CA) comes on the scene, it faces a conundrum: In order to be useful to people, it needs its root certificate to be trusted by a wide variety of operating systems (OSes) and browsers. However, it can take years for the OSes and browsers to accept the new root certificate, and even longer for people to upgrade their devices to the newer versions that include that change. The common solution: a new CA will often ask an existing, trusted CA for a cross-signature, to quickly get it into being trusted by lots of devices.

Five years ago, when Let’s Encrypt launched, that’s exactly what we did. We got a cross-signature from IdenTrust. Their “DST Root X3” had been around for a long time, and all the major software platforms trusted it already: Windows, Firefox, macOS, Android, iOS, and a variety of Linux distributions. That cross-signature allowed us to start issuing certificates right away, and have them be useful to a lot of people. Without IdenTrust, Let’s Encrypt may have never happened and we are grateful to them for their partnership. Meanwhile, we issued our own root certificate (“ISRG Root X1”) and applied for it to be trusted by the major software platforms.

[This section out of date] Now, those software platforms have trusted our root certificate for years. And the DST Root X3 root certificate that we relied on to get us off the ground is going to expire - on September 1, 2021. Fortunately, we’re ready to stand on our own, and rely solely on our own root certificate.

However, this does introduce some compatibility woes. Some software that hasn’t been updated since 2016 (approximately when our root was accepted to many root programs) still doesn’t trust our root certificate, ISRG Root X1. Most notably, this includes versions of Android prior to 7.1.1. That means those older versions of Android will no longer trust certificates issued by Let’s Encrypt.

Android has a long-standing and well known issue with operating system updates. There are lots of Android devices in the world running out-of-date operating systems. The causes are complex and hard to fix: for each phone, the core Android operating system is commonly modified by both the manufacturer and a mobile carrier before an end-user receives it. When there’s an update to Android, both the manufacturer and the mobile carrier have to incorporate those changes into their customized version before sending it out. Often manufacturers decide that’s not worth the effort. The result is bad for the people who buy these devices: many are stuck on operating systems that are years out of date.

Google no longer provides version numbers on its Distribution Dashboard, but you can still get some data by downloading Android Studio. Here’s what the numbers looked like as of September 2020:

Currently, 66.2% of Android devices are running version 7.1 or above. The remaining 33.8% of Android devices will eventually start getting certificate errors when users visit sites that have a Let’s Encrypt certificate. In our communications with large integrators, we have found that this represents around 1-5% of traffic to their sites. Hopefully these numbers will be lower by the time DST Root X3 expires next year, but the change may not be very significant.

What can we do about this? Well, while we’d love to improve the Android update situation, there’s not much we can do there. We also can’t afford to buy the world a new phone. Can we get another cross-signature? We’ve explored this option and it seems unlikely. It’s a big risk for a CA to cross-sign another CA’s certificate, since they become responsible for everything that CA does. That also means the recipient of the cross-signature has to follow all the procedures laid out by the cross-signing CA. It’s important for us to be able to stand on our own. Also, the Android update problem doesn’t seem to be going away. If we commit ourselves to supporting old Android versions, we would commit ourselves to seeking cross-signatures from other CAs indefinitely.

It’s quite a bind. We’re committed to everybody on the planet having secure and privacy-respecting communications. And we know that the people most affected by the Android update problem are those we most want to help - people who may not be able to buy a new phone every four years. Unfortunately, we don’t expect the Android usage numbers to change much prior to DST Root X3’s expiration. By raising awareness of this change now, we hope to help our community to find the best path forward.

[This section out of date] If You Are a Site Owner

As of January 11, 2021, we’re planning to make a change to our API so that ACME clients will, by default, serve a certificate chain that leads to ISRG Root X1. However, it will also be possible to serve an alternate certificate chain for the same certificate that leads to DST Root X3 and offers broader compatibility. This is implemented via the ACME “alternate” link relation. This is supported by Certbot from version 1.6.0 onwards. If you use a different ACME client, please check your client’s documentation to see if the “alternate” link relation is supported.

There will be site owners that receive complaints from users and we are empathetic to that being not ideal. We’re working hard to alert site owners so you can plan and prepare. We encourage site owners to deploy a temporary fix (switching to the alternate certificate chain) to keep your site working while you evaluate what you need for a long-term solution: whether you need to run a banner asking your Android users on older OSes to install Firefox, stop supporting older Android versions, drop back to HTTP for older Android versions, or switch to a CA that is installed on those older versions.

[This section out of date] If You Get Let’s Encrypt Certificates Through Your Hosting Provider

Your hosting provider may be serving the DST Root X3 until September 2021, or they may decide to switch to the certificate chain that leads to ISRG Root X1 after January 11, 2021. Please contact them if you have any questions!

If You Use an Older Version of Android

If you’re on an older version of Android, we recommend you install Firefox Mobile, which supports Android 5.0 and above as of the time of writing.

Why does installing Firefox help? For an Android phone’s built-in browser, the list of trusted root certificates comes from the operating system - which is out of date on these older phones. However, Firefox is currently unique among browsers - it ships with its own list of trusted root certificates. So anyone who installs the latest Firefox version gets the benefit of an up-to-date list of trusted certificate authorities, even if their operating system is out of date.

We appreciate your understanding and support both now and over the years as we continue to grow as a CA, making sure people everywhere have access to encryption. We will provide any future updates on how this root transition affects Android devices via our community forum post. Our community is always ready to help should you have any questions about this change: community.letsencrypt.org.

We depend on contributions from our supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

If You Are an App Developer

If you develop an Android app, you can ship an update that adds ISRG Root X1 as a trusted root within the context of your app. There is a discussion about ways to do so in this forum thread, and this GitHub issue (on a third-party repository).

Given that we issue 1.5 million certificates every day, what makes these ones special? Why did we issue them? How did we issue them? Let’s answer these questions, and in the process take a tour of how Certificate Authorities think and work.

The Backstory

Every publicly-trusted Certificate Authority (such as Let’s Encrypt) has at least one root certificate which is incorporated into various browser and OS vendors’ (e.g. Mozilla, Google) trusted root stores. This is what allows users who receive a certificate from a website to confirm that the certificate was issued by an organization that their browser trusts. But root certificates, by virtue of their widespread trust and long lives, must have their corresponding private key carefully protected and stored offline, and therefore can’t be used to sign things all the time. So every Certificate Authority (CA) also has some number of “intermediates”, certificates which are able to issue additional certificates but are not roots, which they use for day-to-day issuance.

For the last five years, Let’s Encrypt has had one root: the ISRG Root X1, which has a 4096-bit RSA key and is valid until 2035.

Over that same time, we’ve had four intermediates: the Let’s Encrypt Authorities X1, X2, X3, and X4. The first two were issued when Let’s Encrypt first began operations in 2015, and were valid for 5 years. The latter two were issued about a year later, in 2016, and are also valid for 5 years, expiring about this time next year. All of these intermediates use 2048-bit RSA keys. In addition, all of these intermediates are cross-signed by IdenTrust’s DST Root CA X3, another root certificate controlled by a different certificate authority which is trusted by most root stores.

Finally, we also have the ISRG Root OCSP X1 certificate. This one is a little different – it doesn’t issue any certificates. Instead, it signs Online Certificate Status Protocol (OCSP) responses that indicate the intermediate certificates have not been revoked. This is important because the only other thing capable of signing such statements is our root itself, and as mentioned above, the root needs to stay offline and safely secured.

The New Certificates

For starters, we’ve issued two new 2048-bit RSA intermediates which we’re calling R3 and R4. These are both issued by ISRG Root X1, and have 5-year lifetimes. They will also be cross-signed by IdenTrust. They’re basically direct replacements for our current X3 and X4, which are expiring in a year. We expect to switch our primary issuance pipeline to use R3 later this year, which won’t have any real effect on issuance or renewal.

The other new certificates are more interesting. First up, we have the new ISRG Root X2, which has an ECDSA P-384 key instead of RSA, and is valid until 2040. Issued from that, we have two new intermediates, E1 and E2, which are both also ECDSA and are valid for 5 years.

Notably, these ECDSA intermediates are not cross-signed by IdenTrust’s DST Root CA X3. Instead, the ISRG Root X2 itself is cross-signed by our existing ISRG Root X1. An astute observer might also notice that we have not issued an OCSP Signing Certificate from ISRG Root X2.

Now that we have the technical details out of the way, let’s dive in to why the new hierarchy looks the way it does.

Why We Issued an ECDSA Root and Intermediates

There are lots of other articles you can read about the benefits of ECDSA (smaller key sizes for the same level of security; correspondingly faster encryption, decryption, signing, and verification operations; and more). But for us, the big benefit comes from their smaller certificate sizes.

Every connection to a remote domain over https:// requires a TLS handshake. Every TLS handshake requires that the server provide its certificate. Validating that certificate requires a certificate chain (the list of all intermediates up to but not including a trusted root), which is also usually provided by the server. This means that every connection – and a page covered in ads and tracking pixels might have dozens or hundreds – ends up transmitting a large amount of certificate data. And every certificate contains both its own public key and a signature provided by its issuer.

While a 2048-bit RSA public key is about 256 bytes long, an ECDSA P-384 public key is only about 48 bytes. Similarly, the RSA signature will be another 256 bytes, while the ECDSA signature will only be 96 bytes. Factoring in some additional overhead, that’s a savings of nearly 400 bytes per certificate. Multiply that by how many certificates are in your chain, and how many connections you get in a day, and the bandwidth savings add up fast.

These savings are a public benefit both for our subscribers – who can be sites for which bandwidth can be a meaningful cost every month – and for end-users, who may have limited or metered connections. Bringing privacy to the whole Web doesn’t just mean making certificates available, it means making them efficient, too.

As an aside: since we’re concerned about certificate sizes, we’ve also taken a few other measures to save bytes in our new certificates. We’ve shortened their Subject Common Names from “Let’s Encrypt Authority X3” to just “R3”, relying on the previously-redundant Organization Name field to supply the words “Let’s Encrypt”. We’ve shortened their Authority Information Access Issuer and CRL Distribution Point URLs, and we’ve dropped their CPS and OCSP URLs entirely. All of this adds up to another approximately 120 bytes of savings without making any substantive change to the useful information in the certificate.

Why We Cross-Signed the ECDSA Root

Cross-signing is an important step, bridging the gap between when a new root certificate is issued and when that root is incorporated into various trust stores. We know that it is going to take 5 years or so for our new ISRG Root X2 to be widely trusted itself, so in order for certificates issued by the E1 intermediate to be trusted, there needs to be a cross-sign somewhere in the chain.

We had basically two options: we could cross-sign the new ISRG Root X2 from our existing ISRG Root X1, or we could cross-sign the new E1 and E2 intermediates from ISRG Root X1. Let’s examine the pros and cons of each.

Cross-signing the new ISRG Root X2 certificate means that, if a user has ISRG Root X2 in their trust store, then their full certificate chain will be 100% ECDSA, giving them fast validation, as discussed above. And over the next few years, as ISRG Root X2 is incorporated into more and more trust stores, validation of ECDSA end-entity certificates will get faster without users or websites having to change anything. The tradeoff though is that, as long as X2 isn’t in trust stores, user agents will have to validate a chain with two intermediates: both E1 and X2 chaining up to the X1 root. This takes more time during certificate validation.

Cross-signing the intermediates directly has the opposite tradeoff. On the one hand, all of our chains will be the same length, with just one intermediate between the subscriber certificate and the widely-trusted ISRG Root X1. But on the other hand, when the ISRG Root X2 does become widely trusted, we’d have to go through another chain switch in order for anyone to gain the benefits of an all-ECDSA chain.

In the end, we decided that providing the option of all-ECDSA chains was more important, and so opted to go with the first option, and cross-sign the ISRG Root X2 itself.

Why We Didn’t Issue an OCSP Responder

The Online Certificate Status Protocol is a way for user agents to discover, in real time, whether or not a certificate they’re validating has been revoked. Whenever a browser wants to know if a certificate is still valid, it can simply hit a URL contained within the certificate itself and get a yes or no response, which is signed by another certificate and can be similarly validated. This is great for end-entity certificates, because the responses are small and fast, and any given user might care about (and therefore have to fetch) the validity of wildly different sets of certificates, depending on what sites they visit.

But intermediate certificates are a tiny subset of all certificates in the wild, are generally well-known, and are rarely revoked. Because of this, it can be much more efficient to simply maintain a Certificate Revocation List (CRL) containing validity information for all well-known intermediates. Our intermediate certificates all contain a URL from which a browser can fetch their CRL, and in fact some browsers even aggregate these into their own CRLs which they distribute with each update. This means that checking the revocation status of intermediates doesn’t require an extra network round trip before you can load a site, resulting in a better experience for everyone.

In fact, a recent change (ballot SC31) to the Baseline Requirements, which govern CAs, has made it so intermediate certificates are no longer required to include an OCSP URL; they can now have their revocation status served solely by CRL. And as noted above, we have removed the OCSP URL from our new intermediates. As a result, we didn’t need to issue an OCSP responder signed by ISRG Root X2.

Putting It All Together

Now that we’ve shared why our new certificates look the way they do, there’s one last thing we’d like to mention: how we actually went about issuing them.

The creation of new root and intermediate certificates is a big deal, since their contents are so regulated and their private keys have to be so carefully protected. So much so that the act of issuing new ones is called a “ceremony”. Let’s Encrypt believes strongly in automation, so we wanted our ceremony to require as little human involvement as possible.

Over the last few months we’ve built a ceremony tool which, given appropriate configuration, can produce all of the desired keys, certificates, and requests for cross-signs. We also built a demo of our ceremony, showing what our configuration files would be, and allowing anyone to run it themselves and examine the resulting output. Our SREs put together a replica network, complete with Hardware Security Modules, and practiced the ceremony multiple times to ensure it would work flawlessly. We shared this demo with our technical advisory board, our community, and various mailing lists, and in the process received valuable feedback that actually influenced some of the decisions we’ve talked about above! Finally, on September 3rd, our Executive Director met with SREs at a secure datacenter to execute the whole ceremony, and record it for future audits.

And now the ceremony is complete. We’ve updated our certificates page to include details about all of our new certificates, and are beginning the process of requesting that our new root be incorporated into various trust stores. We intend to begin issuing with our new intermediates over the coming weeks, and will post further announcements in our community forum when we do.

We hope that this has been an interesting and informative tour around our hierarchy, and we look forward to continuing to improve the internet one certificate at a time. We’d like to thank IdenTrust for their early and ongoing support of our vision to change security on the Web for the better.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

One thing that’s different now is that the Web is much more encrypted than it was. In June of 2017 approximately 58% of page loads used HTTPS globally, 64% in the United States. Today 81% of page loads use HTTPS globally, and we’re at 91% in the United States! This is an incredible achievement. That’s a lot more privacy and security for everybody.

Another thing that’s different is that our organization has grown a bit, but not by much! In June of 2017 we were serving approximately 46M websites, and we did so with 11 full time staff and an annual budget of $2.61M. Today we serve nearly 192M websites with 13 full time staff and an annual budget of approximately $3.35M. This means we’re serving more than 4x the websites with only two additional staff and a 28% increase in budget. The additional staff and budget did more than just improve our ability to scale though - we’ve made improvements across the board to provide even more secure and reliable service.

Nothing drives adoption like ease of use, and the foundation for ease of use in the certificate space is our ACME protocol. ACME allows for extensive automation, which means computers can do most of the work. It was also standardized as RFC 8555 in 2019, which allows the Web community to confidently build an even richer ecosystem of software around it. Today, thanks to our incredible community, there is an ACME client for just about every deployment environment. Certbot is one of our favorites, and they’ve been working hard to make it even easier for people to use.

When you combine ease of use with incentives, that’s when adoption really takes off. Since 2017 browsers have started requiring HTTPS for more features, and they’ve greatly improved the ways in which they communicate to their users about the risks of not using HTTPS. When websites put their users at risk by not using HTTPS, major browsers now show stronger warnings. Many sites have responded by deploying HTTPS.

Thanks for taking the time to reflect on this milestone with us. As a community we’ve done incredible things to protect people on the Web. Having issued one billion certificates is affirmation of all the progress we’ve made as a community, and we’re excited to keep working with you to create an even more secure and privacy-respecting Web for everyone.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Domain validation is a process that all CAs use to ensure that a certificate applicant actually controls the domain they want a certificate for. Typically the domain validation process involves asking the applicant to place a particular file or token at a controlled location for the domain, such as a particular path or a DNS entry. Then the CA will check that the applicant was able to do so. Historically it looks something like this:

A potential issue with this process is that if a network attacker can hijack or redirect network traffic along the validation path (for the challenge request, or associated DNS queries), then the attacker can trick a CA into incorrectly issuing a certificate. This is precisely what a research team from Princeton demonstrated can be done with an attack on BGP. Such attacks are rare today, but we are concerned that these attacks will become more numerous in the future.

The Border Gateway Protocol (BGP) and most deployments of it are not secure. While there are ongoing efforts to secure BGP, such as RPKI and BGPsec, it may be a long time until BGP hijacking is a thing of the past. We don’t want to wait until we can depend on BGP being secure, so we’ve worked with the research team from Princeton to devise a way to make such attacks more difficult. Instead of validating from one network perspective, we now validate from multiple perspectives as well as from our own data centers:

Today we are validating from multiple regions within a single cloud provider. We plan to diversify network perspectives to other cloud providers in the future.

This makes the kind of attack described earlier more difficult because an attacker must successfully compromise three different network paths at the same time (the primary path from our data center, and at least two of the three remote paths). It also increases the likelihood that such an attack will be detected by the Internet topology community.

We’d like to thank the research groups of Prof. Prateek Mittal and Prof. Jennifer Rexford at Princeton University for their partnership in developing this work. We will continue to work with them to refine the effectiveness of our multiple perspective validation design and implementation. We’d also like to thank Open Technology Fund for supporting this work.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Sectigo and Amazon Web Services have generously provided support to cover a significant portion of the cost of running our CT log. “Sectigo is proud to sponsor the Let’s Encrypt CT Log. We believe this initiative will provide much-needed reinforcement of the CT ecosystem,” said Ed Giaquinto, Sectigo’s CIO.

For more background information about CT and how it works, we recommend reading “How Certificate Transparency Works.”

If you have questions about any of what we’ve written here, feel free to ask on our community forums.

Objectives

1. Scale: Let’s Encrypt issues over 1 million certificates per day, and that number grows each month. We want our log to consume our certificates as well as those from other CAs, so we need to be able to handle as many as 2 million or more certificates per day. To support this ever-increasing number of certificates, CT software and infrastructure need to be architected for scale.
2. Stability and Compliance: We target 99% uptime, with no outage lasting longer than 24 hours, in compliance with the Chromium and Apple CT policies.
3. Sharding: Best practice for a CT log is to break it into several temporal shards. For more information on temporal sharding, check out these blog posts.
4. Low Maintenance: Staff time is expensive, we want to minimize the amount of time spent maintaining infrastructure.

System Architecture

Staging and Production Logs

We run two equivalent logs, one for staging and one for production. Any changes we plan to make to the production log are first deployed to the staging log. This is critical for making sure that updates and upgrades don’t cause problems before being deployed to production. You can find access details for these logs in our documentation.

We keep the staging log continually under production-level load so that any scale-related problems manifest there first. We also use the staging CT log to submit certificates from our staging CA environment, and make it available for use by other CAs’ staging environments.

As a point of clarification, we consider a log to be comprised of several temporal shards. While each shard is technically a separate log, it makes sense to conceptualize the shards as belonging to a single log.

Amazon Web Services (AWS)

We decided to run our CT logs on AWS for two reasons.

One consideration for us was cloud provider diversity. Since there are relatively few trusted logs in the ecosystem, we don’t want multiple logs to go down due to a single cloud provider outage. At the time we made the decision there were logs running on Google and Digital Ocean infrastructure, as well as self-hosted. We were not aware of any on AWS (in hindsight we may have missed the fact that Digicert had started using AWS for logs). If you’re thinking about setting up a trusted log for CAs to use, please consider cloud provider diversity.

Additionally, AWS provides a solid set of features and our team has experience using it for other purposes. We had little doubt that AWS was up to the task.

Terraform

Let’s Encrypt uses Hashicorp Terraform for a number of cloud-based projects. We were able to bootstrap our CT log infrastructure by reusing our existing Terraform code. There are roughly 50 components in our CT deployments; including EC2, RDS, EKS, IAM, security groups, and routing. Centrally managing this code allows our small team to reproduce a CT infrastructure in any Amazon region of the globe, prevent configuration drift, and easily test infrastructure changes.

Database

We chose to use MariaDB for our CT log database because we have extensive experience using it to run our certificate authority. MariaDB has scaled well on our journey to becoming the largest publicly trusted certificate authority.

We chose to have our MariaDB instances managed by Amazon RDS because RDS provides synchronous writes to standby cluster members. This allows for automatic database failover and ensures database consistency. Synchronous writes to database replicas are essential for a CT log. One missed write during a database failover can mean a certificate was not included as promised, and could lead to the log being disqualified. Having RDS manage this for us reduces complexity and saves staff time. We are still responsible for managing the database performance, tuning, and monitoring.

It’s important to calculate the necessary amount of storage for a CT log database carefully. Too little storage can result in needing to undertake time-consuming and potentially risky storage migrations. Too much storage can result in unnecessarily high costs.

A back of the napkin storage estimation is 1TB per 100 million entries. We expect to need to store 1 billion certificates and precertificates per annual temporal shard, for which we would need 10TB. We considered having separate database storage per annual temporal shard, with approximately 10TB allocated to each, but that was cost prohibitive. We decided to create a 12TB storage block per log (10TB plus some breathing room), which is duplicated for redundancy by RDS. Each year we plan to freeze the previous year’s shard and move it to a less expensive serving infrastructure, reclaiming its storage for our live shards.

We use 2x db.r5.4xlarge instances for RDS for each CT log. Each of these instances contains 8 CPU cores and 128GB of RAM.

Kubernetes

After trying a few different strategies for managing application instances, we decided to use Kubernetes. There is a hefty learning curve for Kubernetes and the decision was not made lightly. This was our first project making use of Kubernetes, and part of the reason we went with it was to gain experience and possibly apply that knowledge to other parts of our infrastructure in the future.

Kubernetes provides abstractions for operators such as deployments, scaling, and service discovery that we would not have to build ourselves. We utilized the example Kubernetes deployment manifests in the Trillian repository to assist with our deployment.

A Kubernetes cluster is comprised of two main components: the control plane which handles the Kubernetes APIs, and worker nodes where containerized applications run. We chose to have Amazon EKS manage our Kubernetes control plane.

We use 4x c5.2xlarge EC2 instances for the worker node pool for each CT log. Each of these instances contains 8 CPU cores and 16GB of RAM.

Application Software

There are three main CT components that we run in a Kubernetes cluster.

The certificate transparency front end, or CTFE, provides RFC 6962 endpoints and translates them to gRPC API requests for the Trillian backend.

Trillian describes itself as a “transparent, highly scalable and cryptographically verifiable data store.” Essentially, Trillian implements a generalized verifiable data store via a Merkle tree that can be used as the back-end for a CT log via the CTFE. Trillian consists of two components; the log signer and log server. The log signer’s function is to periodically process incoming leaf data (certificates in the case of CT) and incorporate them into a Merkle tree. The log server retrieves objects from a Merkle tree in order to fulfill CT API monitoring requests.

Load Balancing

Traffic enters the CT log through an Amazon ELB which is mapped to a Kubernetes Nginx ingress service. The ingress service balances traffic amongst multiple Nginx pods. The Nginx pods proxy traffic to the CTFE service which balance that traffic to CTFE pods.

We employ IP and user agent based rate limiting at this Nginx layer.

Logging and Monitoring

Trillian and the CTFE expose Prometheus metrics which we transform into monitoring dashboards and alerts. It is essential to set a Service Level Objective for the CT log endpoints above the 99% uptime dictated by CT policy to ensure that your log is trusted. A FluentD pod running in a DaemonSet ships logs to centralized storage for further analysis.

We developed a free and open source tool named ct-woodpecker that is used to monitor various aspects of log stability and correctness. This tool is an important part of how we ensure we’re meeting our service level objectives. Each ct-woodpecker instance runs externally from Amazon VPCs containing CT logs.

Future Efficiency Improvements

Here are some ways we may be able to improve the efficiency of our system in the future:

• Trillian stores a copy of each certificate chain, including many duplicate copies of the same intermediate certificates. Being able to de-duplicate these in Trillian would significantly reduce storage costs. We’re planning to look into whether this is possible and reasonable.
• See if we can successfully use a cheaper form of storage than IO1 block storage and provisioned IOPs.
• See if we can reduce the Kubernetes worker EC2 instance size or use fewer EC2 instances.

Support Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization is interested in learning more about sponsorship, please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

How the DNS Validation Method Works

The DNS-01 validation method works like this: to prove that you control www.example.com, you create a TXT record at _acme-challenge.www.example.com with a “digest value” as specified by ACME (your ACME client should take care of creating this digest value for you). When the TXT record is ready, your ACME client informs the ACME server (for instance, Let’s Encrypt) that the domain is ready for validation. The ACME server looks up the TXT record, compares it to the expected digest value, and if the result is correct, considers your account authorized to issue for www.example.com. Your new customer can set up this TXT record (or a CNAME) without interfering with normal website operations.

The Advantages of a CNAME

There’s an additional trick that I recommend for hosting providers and CDNs: Instead of giving the digest value to your new customer and telling them to make a TXT record with it, tell your customer to configure a CNAME from _acme-challenge.www.example.com to a domain name that you control and that is unique to the domain being validated. For instance, you might use www.example.com.validationserver.example.net. Then, once your software has verified that this CNAME is set up (accounting for propagation delay and anycast), your ACME client should begin the validation process for www.example.com, provisioning a TXT record at www.example.com.validationserver.example.net. Because the ACME server’s TXT lookup follows CNAMEs (as do all DNS lookups), it will see the value you provisioned, and consider your account authorized.

This approach is preferable to handing your customers a raw digest value for a few reasons. First, it gives your customers all the time they need to set up the CNAME. If you create a pending authorization up front and give your customer a digest value to deploy themselves, it has a fixed lifetime before it expires (for Let’s Encrypt this lifetime is 7 days). If your customer doesn’t complete the process in that time, you’ll have to create a new pending authorization and give your customer a new digest value. That’s annoying and time consuming for both you and your customer. The CNAME method means even if it takes your new customer a month to make the needed changes to their DNS, you can get things up and running as soon as they do.

Another reason to prefer the CNAME method over having new customers directly provision their TXT records is to support the best practice of periodically rotating your ACME account key. Because the digest value used for DNS-01 validation is computed based on your current ACME account key, it will change whenever you rotate your account key. If you asked customers to provision their TXT record manually, that means notifying potential new customers that the value you asked them to put in DNS isn’t valid anymore, and they need to use a different one. That’s pretty inconvenient! If you use the CNAME method instead, there’s only one ACME-related value you’ll ever need to have your new customers put in DNS, and it won’t change as you change your account key.

Cleaning Up Unused CNAMES

One last note: This is a good way to onboard customers, but you also need to detect when customers offboard themselves. They may simply change their A records to point at a different CDN, without telling you that their plans have changed. You should monitor for this situation and stop attempting to issue certificates. If the customer has left behind a CNAMEd _acme-challenge subdomain that points at you, you should contact that and remind them to delete it. The CNAMEd subdomain represents a delegated authorization to issue certificates, and cleaning up that delegation improves both the customer’s security posture and your own. Similarly, if a customer sets up the CNAME and you issue a certificate on their behalf, but they never point their A records at your servers, you should not reissue new certificates indefinitely without further intervention from the customer.

Update: Feb. 5 2020

The Let’s Encrypt CT logs are now included in approved log lists and are usable by all publicly-trusted certificate authorities.

Today we are announcing a new Certificate Transparency log called Oak. The Oak log will be operated by Let’s Encrypt and all publicly trusted certificate authorities will be welcome to submit certificates.

Sectigo generously provided funding to cover a significant portion of our costs to run our CT log. “Sectigo is proud to sponsor the Let’s Encrypt CT Log. We believe this initiative will provide much-needed reinforcement of the CT ecosystem,” said Ed Giaquinto, Sectigo’s CIO. We thank them for their collaboration to improve Internet security.

Certificate Transparency (CT) is a system for logging and monitoring certificate issuance. It greatly enhances everyone’s ability to monitor and study certificate issuance, and these capabilities have led to numerous improvements to the CA ecosystem and Web security. As a result, it is rapidly becoming critical Internet infrastructure. Let’s Encrypt accelerated the adoption of CT by logging every certificate since we started issuing in 2015 - approximately half a billion certificates at this point.

We decided to create and operate a CT log for a few reasons. First, operating a log is consistent with our mission to create a more secure and privacy-respecting Web. We believe transparency increases security and empowers people to make well-informed decisions. Second, operating a log helps us take control of our destiny. Google Chrome requires all new certificates to be submitted to two separate logs, so multiple log options are imperative to our operation. Finally, Let’s Encrypt often issues more than 1M certificates each day, so we wanted to design a CT log that is optimized for high volume. We’ve designed our log to be able to handle submissions from all other publicly trusted Certificate Authorities so they can use Oak to fulfill their logging requirements as well.

Our log uses Google’s Trillian software running on AWS infrastructure. We use Kubernetes for container orchestration and job scheduling and AWS RDS for database management.

We are submitting our log for inclusion in the approved log lists for Google Chrome and Apple Safari. Following 90 days of successful monitoring, we anticipate our log will be added to these trusted lists and that change will propagate to people’s browsers with subsequent browser version releases.

Continuing the forest theme, we are also announcing the launch of our open source CT monitoring tool, CT Woodpecker. We use it to monitor and ensure compliance for our log and we’ve made it open source so others in the CT ecosystem can use it as well.

We’d like to thank Google, Sectigo, Cloudflare, and DigiCert for also running open logs, and we look forward to contributing to better transparency in Web security!

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Update, September 17, 2020

Due to concerns about insufficient ISRG root propagation on Android devices we have decided to move the date on which we will start serving a chain to our own root to January 11, 2021. We had originally delayed this change until September 29, 2020.

Update, June 11, 2020

In an effort to provide more time for our community to prepare for this transition, we have moved back the date on which we will start serving a chain to our own root to September 29, 2020.

On January 11, 2021, we will change the default intermediate certificate we provide via ACME. Most subscribers don’t need to do anything. Subscribers who support very old TLS/SSL clients may want to manually configure the older intermediate to increase backwards compatibility.

Since Let’s Encrypt launched, our certificates have been trusted by browsers via a cross-signature from another Certificate Authority (CA) named IdenTrust. A cross-signature from IdenTrust was necessary because our own root was not yet widely trusted. It takes time for a new CA to demonstrate that it is trustworthy, then it takes more time for trusted status to propagate via software updates.

Now that our own root, ISRG Root X1, is widely trusted by browsers we’d like to transition our subscribers to using our root directly, without a cross-sign.

On January 11, 2021, Let’s Encrypt will start serving a certificate chain via the ACME protocol which leads directly to our root, with no cross-signature. Most subscribers don’t need to take any action because their ACME client will handle everything automatically. Subscribers who need to support very old TLS/SSL clients may wish to manually configure their servers to continue using the cross-signature from IdenTrust. You can test whether a given client will work with the newer intermediate by accessing our test site.

Our current cross-signature from IdenTrust expires on March 17, 2021. The IdenTrust root that we are cross-signed from expires on September 30, 2021. Within the next year we will obtain a new cross-signature that is valid until September 29, 2021. This means that our subscribers will have the option to manually configure a certificate chain that uses IdenTrust until September 29, 2021.

We’d like to thank IdenTrust for providing a cross-signature while we worked to get our own root trusted. They have been wonderful partners. IdenTrust believed in our mission to encrypt the entire Web when it seemed like a long-term dream. Together, in less than five years, we have helped to raise the percentage of encrypted page loads on the Web from 39% to 78%.

Let’s Encrypt is currently providing certificates for more than 160 million websites. We look forward to being able to serve even more websites as efforts like this make deploying HTTPS with Let’s Encrypt even easier. If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

Having a standardized protocol for certificate issuance and management is important for two reasons. First, it improves the quality of the software ecosystem because developers can focus on developing great software for a single protocol, instead of having many pieces of less well maintained software for bespoke APIs. Second, a standardized protocol makes switching from one CA to another easier by minimizing technical dependency lock-in.

We consider the standardized version of the ACME protocol to be the second major version of ACME, so we refer to it as ACMEv2. The first version, which we call ACMEv1, is the version of ACME that Let’s Encrypt has used since our launch in 2015. Now that ACMEv2 is standardized, we are announcing an end-of-life plan for our ACMEv1 support.

Let’s Encrypt is currently providing certificates for more than 150 million websites. We look forward to being able to serve even more websites as efforts like this make deploying HTTPS with Let’s Encrypt even easier. If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

We’re excited that Facebook is supporting our work through a three-year Platinum sponsorship! We asked them to share their thoughts on HTTPS adoption here. Please join us in thanking Facebook for their support of Let’s Encrypt and our mission to encrypt the Web! - Josh Aas, Executive Director, ISRG / Let’s Encrypt

If the web is more secure, everybody wins. A key technology for making this happen is HTTPS, which enables encrypted connections between people and the websites that they visit. Among its many benefits, HTTPS helps to prevent sensitive data from leaking over the network, and from connections being censored or otherwise maliciously manipulated. The more widely it is deployed, the more secure and private the web becomes for everyone.

We have long worked to protect Facebook users from spammy or malicious content when navigating away from our platform, and last year we extended this protection to upgrading outbound HTTP links to HTTPS where possible. In this way we can help improve people’s security and privacy as they leave our platform. While we take these steps to improve the security and safety of Facebook users, ultimately we hope to see more websites allowing HTTPS connections.

Enabling HTTPS was historically a non-trivial task for any site. It required investment in buying and installing a TLS certificate, which verifies control over the website so that HTTPS can work. The technical difficulty and cost used to serve as barriers to expanding the use of HTTPS across the web. However, things have recently started to change, largely thanks to Let’s Encrypt, a non-profit certificate authority, launched in 2015.

Let’s Encrypt provides free TLS certificates, which are often installed using a tool maintained by the Electronic Frontier Foundation, to massively simplify enabling HTTPS. With that, Let’s Encrypt is effectively upgrading the security and privacy of the web, at no cost to over 150 million websites, including those frequented by Facebook users.

We’re excited to see the continuous increase in HTTPS adoption across the internet. More websites are choosing to enable secure connections which provide the security and privacy benefits and enable a better browsing experience. For example, navigating from Facebook to another site can be faster over encrypted connections than HTTP, and an increasing number of browser features will only work when sites use HTTPS.

We have sponsored Let’s Encrypt from the start, and are proud to share that we are increasing that support as a platinum sponsor. We believe that Let’s Encrypt has played a significant and important role in bringing encryption into the mainstream and raising the number of secure sites across the internet.

As we automatically crawl web content on Facebook (for example, to generate link previews), about 38% of HTTPS domains we observe use Let’s Encrypt, making it the top certificate authority. Over 19% of outbound clicks from Facebook to HTTPS-enabled websites go to sites that use certificates from Let’s Encrypt. Overall, more than 72% of outbound clicks from Facebook are now destined for HTTPS-enabled websites, including the links that we upgrade to HTTPS in real time.

We’re proud to continue to collaborate with Let’s Encrypt on helping to improve web security. To any website owners who haven’t yet enabled encryption, we strongly encourage you to use Let’s Encrypt to protect your users and allow HTTPS connections.

Most importantly though, the Web went from 67% encrypted page loads to 77% in 2018, according to statistics from Mozilla. This is an incredible rate of change!

We’d like to thank all of the people and organizations who worked hard to create a more secure and privacy-respecting Web.

This year we created a new website for the legal entity behind Let’s Encrypt, Internet Security Research Group (ISRG), because we believe there will be other instances beyond Let’s Encrypt in which ISRG might be able to help to build, or improve access to, a better Internet.

While we’re proud of what we accomplished in 2018, we spend most of our time looking forward rather than back. As we wrap up our own planning process for 2019, we’d like to share some of our plans with you, including both the things we’re excited about and the challenges we’ll face. We’ll cover service growth, new features, infrastructure, and finances.

Service Growth

Let’s Encrypt helps to drive HTTPS adoption by offering a free, easy to use, and globally available option for obtaining the certificates required to enable HTTPS. HTTPS adoption on the Web took off at an unprecedented rate from the day Let’s Encrypt launched to the public.

The number of certificates and unique domains we support continues to grow rapidly:

We expect strong growth again in 2019, likely up to 120M active certificates and 215M fully qualified domains. You can view our recently revamped stats page for more information.

One of the reasons Let’s Encrypt is so easy to use is that our community has done great work making client software that works well for a wide variety of platforms. We’d like to thank everyone involved in the development of more than 85 client software options for Let’s Encrypt. Support for our protocol, ACME, is built in to Apache and we’re hoping 2019 will be the year that it comes to Nginx.

Other organizations and communities are also doing great work to promote HTTPS adoption, and thus stimulate demand for our services. For example, browsers are starting to make their users more aware of the risks associated with unencrypted HTTP (e.g. Firefox, Chrome). Many hosting providers and CDNs are making it easier than ever for all of their customers to use HTTPS. Government agencies are waking up to the need for stronger security to protect constituents. The media community is working to Secure the News.

New Features

In 2018 we introduced several new features, including ACMEv2 support and wildcard certificates. We’ve got some exciting features planned for 2019.

The feature we’re most excited about is multi-perspective validation. Currently, when a subscriber requests a certificate, we validate domain control from a single network perspective. This is standard practice for CAs. If an attacker along the network path for the validation check can interfere with traffic they can potentially cause certificates to be issued that should not be issued. We’re most concerned about this happening via BGP hijacking, and since BGP is not going to be secured any time soon, we needed to find another mitigation. The solution we intend to deploy in 2019 is multi-perspective validation, in which we will check from multiple network perspectives (distinct Autonomous Systems). This means that potential BGP hijackers would need to hijack multiple routes at the same time in order to pull off a successful attack, which is significantly more difficult than hijacking a single route. We are working with a talented research team at Princeton to design the most effective multi-perspective validation system we can, and have already turned parts of this feature on in our staging environment.

We are also planning to introduce a Certificate Transparency (CT) log in 2019. All certificate authorities like Let’s Encrypt are required to submit certificates to CT logs but there are not enough stable logs in the ecosystem. As such, we are moving forward with plans to run a log which all CAs will be able to submit to.

We had planned to add ECDSA root and intermediate certificates in 2018 but other priorities ultimately took precedence. We hope to do this in 2019. ECDSA is generally considered to be the future of digital signature algorithms on the Web due to the fact that it is more efficient than RSA. Let’s Encrypt will currently sign ECDSA keys from subscribers, but we sign with the RSA key from one of our intermediate certificates. Once we have an ECDSA root and intermediates, our subscribers will be able to deploy certificate chains which are entirely ECDSA.

Infrastructure

Our CA infrastructure is capable of issuing millions of certificates per day with redundancy for stability and a wide variety of security safeguards, both physical and logical. Our infrastructure also generates and signs around 40 million OCSP responses daily, and serves those responses approximately 5.5 billion times per day. We expect these numbers to grow approximately 40% in 2019.

Our physical CA infrastructure currently occupies approximately 55 units of rack space, split between two datacenters, consisting primarily of compute servers, storage, HSMs, switches, and firewalls. When we issue more certificates it puts the most stress on storage for our databases. We regularly invest in more and faster storage for our database servers, and that will continue in 2019.

All of our infrastructure is managed by our Site Reliability Engineering (SRE) team, which is comprised of six people. SRE staff are responsible for building and maintaining all physical and logical CA infrastructure. These staff are largely responsible for our high standards for security and compliance. The team also manages a 24/7/365 on-call schedule and they are primary participants in both security and compliance audits.

Finances

We pride ourselves on being an efficient organization. In 2019 Let’s Encrypt will secure a massive portion of the Web with a budget of only $3.6M. We believe this represents an incredible value and that contributing to Let’s Encrypt is one of the most effective ways to help create a more secure and privacy-respecting Web.

Our 2019 fundraising efforts are off to a strong start with Platinum sponsorships from Cisco, OVH, Mozilla, Google Chrome, Electronic Frontier Foundation, and Internet Society, as well as many other Gold and Silver sponsors. The Ford Foundation has renewed their grant to Let’s Encrypt as well. We are seeking additional sponsorship and grant assistance to meet our full needs for 2019.

Support Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

We’re grateful for the industry and community support that we receive, and we look forward to continuing to create a more secure and privacy-respecting Web!

Today’s announcement that we’re trusted by all major root programs represents a major milestone for us, but it’s not the conclusion of our journey towards being directly trusted everywhere.

Certificates from Let’s Encrypt have been widely trusted since our first issuance because of a cross-signature from another CA called IdenTrust. Browsers and operating systems have not, by default, directly trusted Let’s Encrypt certificates, but they trust IdenTrust, and IdenTrust trusts us, so we are trusted indirectly. IdenTrust is a critical partner in our effort to secure the Web, as they have allowed us to provide widely trusted certificates from day one.

While Let’s Encrypt is now directly trusted by almost all newer versions of operating systems, browsers, and devices, there are still many older versions in the world that do not directly trust Let’s Encrypt. Some of those older systems will eventually be updated to trust Let’s Encrypt directly. Some will not, and we’ll need to wait for the vast majority of those to cycle out of the Web ecosystem. We expect this will take at least five more years, so we plan to use a cross signature until then.

As a subscriber of Let’s Encrypt, today’s milestone does not require any action on your part. Just continue to use best practices, including making sure that your ACME client (e.g. Certbot or an alternative) is regularly receiving software updates.

Let’s Encrypt is currently providing certificates for more than 115 million websites. We look forward to being able to serve even more websites as efforts like this make deploying HTTPS with Let’s Encrypt even easier. If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

Certificate Transparency offers three ways to deliver SCTs to a browser: In a TLS extension, in stapled OCSP, or embedded in a certificate. We chose to implement the embedding method because it would just work for Let’s Encrypt subscribers without additional work. In the SCT embedding method, we submit a “precertificate” with a poison extension to a set of CT logs, and get back SCTs. We then issue a real certificate based on the precertificate, with two changes: The poison extension is removed, and the SCTs obtained earlier are added in another extension.

Given a certificate, let’s first look for the SCT list extension. According to CT (RFC 6962 section 3.3), the extension OID for a list of SCTs is 1.3.6.1.4.1.11129.2.4.2. An OID (object ID) is a series of integers, hierarchically assigned and globally unique. They are used extensively in X.509, for instance to uniquely identify extensions.

We can download an example certificate, and view it using OpenSSL (if your OpenSSL is old, it may not display the detailed information):

$ openssl x509 -noout -text -inform der -in Downloads/031f2484307c9bc511b3123cb236a480d451
...
CT Precertificate SCTs:
    Signed Certificate Timestamp:
        Version   : v1(0)
        Log ID    : DB:74:AF:EE:CB:29:EC:B1:FE:CA:3E:71:6D:2C:E5:B9:
                    AA:BB:36:F7:84:71:83:C7:5D:9D:4F:37:B6:1F:BF:64
        Timestamp : Mar 29 18:45:07.993 2018 GMT
        Extensions: none
        Signature : ecdsa-with-SHA256
                    30:44:02:20:7E:1F:CD:1E:9A:2B:D2:A5:0A:0C:81:E7:
                    13:03:3A:07:62:34:0D:A8:F9:1E:F2:7A:48:B3:81:76:
                    40:15:9C:D3:02:20:65:9F:E9:F1:D8:80:E2:E8:F6:B3:
                    25:BE:9F:18:95:6D:17:C6:CA:8A:6F:2B:12:CB:0F:55:
                    FB:70:F7:59:A4:19
    Signed Certificate Timestamp:
        Version   : v1(0)
        Log ID    : 29:3C:51:96:54:C8:39:65:BA:AA:50:FC:58:07:D4:B7:
                    6F:BF:58:7A:29:72:DC:A4:C3:0C:F4:E5:45:47:F4:78
        Timestamp : Mar 29 18:45:08.010 2018 GMT
        Extensions: none
        Signature : ecdsa-with-SHA256
                    30:46:02:21:00:AB:72:F1:E4:D6:22:3E:F8:7F:C6:84:
                    91:C2:08:D2:9D:4D:57:EB:F4:75:88:BB:75:44:D3:2F:
                    95:37:E2:CE:C1:02:21:00:8A:FF:C4:0C:C6:C4:E3:B2:
                    45:78:DA:DE:4F:81:5E:CB:CE:2D:57:A5:79:34:21:19:
                    A1:E6:5B:C7:E5:E6:9C:E2

Now let’s go a little deeper. How is that extension represented in the certificate? Certificates are expressed in ASN.1, which generally refers to both a language for expressing data structures and a set of formats for encoding them. The most common format, DER, is a tag-length-value format. That is, to encode an object, first you write down a tag representing its type (usually one byte), then you write down a number expressing how long the object is, then you write down the object contents. This is recursive: An object can contain multiple objects within it, each of which has its own tag, length, and value.

One of the cool things about DER and other tag-length-value formats is that you can decode them to some degree without knowing what they mean. For instance, I can tell you that 0x30 means the data type “SEQUENCE” (a struct, in ASN.1 terms), and 0x02 means “INTEGER”, then give you this hex byte sequence to decode:

30 06 02 01 03 02 01 0A

You could tell me right away that decodes to:

SEQUENCE
  INTEGER 3
  INTEGER 10

Try it yourself with this great JavaScript ASN.1 decoder. However, you wouldn’t know what those integers represent without the corresponding ASN.1 schema (or “module”). For instance, if you knew that this was a piece of DogData, and the schema was:

DogData ::= SEQUENCE {
    legs           INTEGER,
    cutenessLevel  INTEGER
}

You’d know this referred to a three-legged dog with a cuteness level of 10.

We can take some of this knowledge and apply it to our certificates. As a first step, convert the above certificate to hex with xxd -ps < Downloads/031f2484307c9bc511b3123cb236a480d451. You can then copy and paste the result into lapo.it/asn1js (or use this handy link). You can also run openssl asn1parse -i -inform der -in Downloads/031f2484307c9bc511b3123cb236a480d451 to use OpenSSL’s parser, which is less easy to use in some ways, but easier to copy and paste.

In the decoded data, we can find the OID 1.3.6.1.4.1.11129.2.4.2, indicating the SCT list extension. Per RFC 5280, section 4.1, an extension is defined:

Extension  ::=  SEQUENCE  {
      extnID      OBJECT IDENTIFIER,
      critical    BOOLEAN DEFAULT FALSE,
      extnValue   OCTET STRING
                  -- contains the DER encoding of an ASN.1 value
                  -- corresponding to the extension type identified
                  -- by extnID
      }

We’ve found the extnID. The “critical” field is omitted because it has the default value (false). Next up is the extnValue. This has the type OCTET STRING, which has the tag “0x04”. OCTET STRING means “here’s a bunch of bytes!” In this case, as described by the spec, those bytes happen to contain more DER. This is a fairly common pattern in X.509 to deal with parameterized data. For instance, this allows defining a structure for extensions without knowing ahead of time all the structures that a future extension might want to carry in its value. If you’re a C programmer, think of it as a void* for data structures. If you prefer Go, think of it as an interface{}.

Here’s that extnValue:

04 81 F5 0481F200F0007500DB74AFEECB29ECB1FECA3E716D2CE5B9AABB36F7847183C75D9D4F37B61FBF64000001627313EB19000004030046304402207E1FCD1E9A2BD2A50A0C81E713033A0762340DA8F91EF27A48B3817640159CD30220659FE9F1D880E2E8F6B325BE9F18956D17C6CA8A6F2B12CB0F55FB70F759A419007700293C519654C83965BAAA50FC5807D4B76FBF587A2972DCA4C30CF4E54547F478000001627313EB2A0000040300483046022100AB72F1E4D6223EF87FC68491C208D29D4D57EBF47588BB7544D32F9537E2CEC10221008AFFC40CC6C4E3B24578DADE4F815ECBCE2D57A579342119A1E65BC7E5E69CE2

That’s tag “0x04”, meaning OCTET STRING, followed by “0x81 0xF5”, meaning “this string is 245 bytes long” (the 0x81 prefix is part of variable length number encoding).

According to RFC 6962, section 3.3, “obtained SCTs can be directly embedded in the final certificate, by encoding the SignedCertificateTimestampList structure as an ASN.1 OCTET STRING and inserting the resulting data in the TBSCertificate as an X.509v3 certificate extension”

So, we have an OCTET STRING, all’s good, right? Except if you remove the tag and length from extnValue to get its value, you’re left with:

04 81 F2 00F0007500DB74AFEEC...

There’s that “0x04” tag again, but with a shorter length. Why do we nest one OCTET STRING inside another? It’s because the contents of extnValue are required by RFC 5280 to be valid DER, but a SignedCertificateTimestampList is not encoded using DER (more on that in a minute). So, by RFC 6962, a SignedCertificateTimestampList is wrapped in an OCTET STRING, which is wrapped in another OCTET STRING (the extnValue).

Once we decode that second OCTET STRING, we’re left with the contents:

00F0007500DB74AFEEC...

“0x00” isn’t a valid tag in DER. What is this? It’s TLS encoding. This is defined in RFC 5246, section 4 (the TLS 1.2 RFC). TLS encoding, like ASN.1, has both a way to define data structures and a way to encode those structures. TLS encoding differs from DER in that there are no tags, and lengths are only encoded when necessary for variable-length arrays. Within an encoded structure, the type of a field is determined by its position, rather than by a tag. This means that TLS-encoded structures are more compact than DER structures, but also that they can’t be processed without knowing the corresponding schema. For instance, here’s the top-level schema from RFC 6962, section 3.3:

   The contents of the ASN.1 OCTET STRING embedded in an OCSP extension
   or X509v3 certificate extension are as follows:

        opaque SerializedSCT<1..2^16-1>;

        struct {
            SerializedSCT sct_list <1..2^16-1>;
        } SignedCertificateTimestampList;

   Here, "SerializedSCT" is an opaque byte string that contains the
   serialized TLS structure.

Right away, we’ve found one of those variable-length arrays. The length of such an array (in bytes) is always represented by a length field just big enough to hold the max array size. The max size of an sct_list is 65535 bytes, so the length field is two bytes wide. Sure enough, those first two bytes are “0x00 0xF0”, or 240 in decimal. In other words, this sct_list will have 240 bytes. We don’t yet know how many SCTs will be in it. That will become clear only by continuing to parse the encoded data and seeing where each struct ends (spoiler alert: there are two SCTs!).

Now we know the first SerializedSCT starts with 0075.... SerializedSCT is itself a variable-length field, this time containing opaque bytes (much like OCTET STRING back in the ASN.1 world). Like SignedCertificateTimestampList, it has a max size of 65535 bytes, so we pull off the first two bytes and discover that the first SerializedSCT is 0x0075 (117 decimal) bytes long. Here’s the whole thing, in hex:

00DB74AFEECB29ECB1FECA3E716D2CE5B9AABB36F7847183C75D9D4F37B61FBF64000001627313EB19000004030046304402207E1FCD1E9A2BD2A50A0C81E713033A0762340DA8F91EF27A48B3817640159CD30220659FE9F1D880E2E8F6B325BE9F18956D17C6CA8A6F2B12CB0F55FB70F759A419

This can be decoded using the TLS encoding struct defined in RFC 6962, section 3.2:

enum { v1(0), (255) }
 Version;

struct {
   opaque key_id[32];
} LogID;

opaque CtExtensions<0..2^16-1>;
...

struct {
   Version sct_version;
   LogID id;
   uint64 timestamp;
   CtExtensions extensions;
   digitally-signed struct {
       Version sct_version;
       SignatureType signature_type = certificate_timestamp;
       uint64 timestamp;
       LogEntryType entry_type;
       select(entry_type) {
           case x509_entry: ASN.1Cert;
           case precert_entry: PreCert;
       } signed_entry;
      CtExtensions extensions;
   };
} SignedCertificateTimestamp;

Breaking that down:

# Version sct_version v1(0)
00
# LogID id (aka opaque key_id[32])
DB74AFEECB29ECB1FECA3E716D2CE5B9AABB36F7847183C75D9D4F37B61FBF64
# uint64 timestamp (milliseconds since the epoch)
000001627313EB19
# CtExtensions extensions (zero-length array)
0000
# digitally-signed struct
04030046304402207E1FCD1E9A2BD2A50A0C81E713033A0762340DA8F91EF27A48B3817640159CD30220659FE9F1D880E2E8F6B325BE9F18956D17C6CA8A6F2B12CB0F55FB70F759A419

To understand the “digitally-signed struct,” we need to turn back to RFC 5246, section 4.7. It says:

A digitally-signed element is encoded as a struct DigitallySigned:

struct {
   SignatureAndHashAlgorithm algorithm;
   opaque signature<0..2^16-1>;
} DigitallySigned;

And in section 7.4.1.4.1:

enum {
    none(0), md5(1), sha1(2), sha224(3), sha256(4), sha384(5),
    sha512(6), (255)
} HashAlgorithm;

enum { anonymous(0), rsa(1), dsa(2), ecdsa(3), (255) }
  SignatureAlgorithm;

struct {
      HashAlgorithm hash;
      SignatureAlgorithm signature;
} SignatureAndHashAlgorithm;

We have “0x0403”, which corresponds to sha256(4) and ecdsa(3). The next two bytes, “0x0046”, tell us the length of the “opaque signature” field, 70 bytes in decimal. To decode the signature, we reference RFC 4492 section 5.4, which says:

The digitally-signed element is encoded as an opaque vector <0..2^16-1>, the
contents of which are the DER encoding corresponding to the
following ASN.1 notation.

Ecdsa-Sig-Value ::= SEQUENCE {
   r       INTEGER,
   s       INTEGER
}

Having dived through two layers of TLS encoding, we are now back in ASN.1 land! We decode the remaining bytes into a SEQUENCE containing two INTEGERS. And we’re done! Here’s the whole extension decoded:

# Extension SEQUENCE - RFC 5280
30
# length 0x0104 bytes (260 decimal)
820104
  # OBJECT IDENTIFIER
  06
  # length 0x0A bytes (10 decimal)
  0A
    # value (1.3.6.1.4.1.11129.2.4.2)
    2B06010401D679020402
  # OCTET STRING
  04
  # length 0xF5 bytes (245 decimal)
  81F5
    # OCTET STRING (embedded) - RFC 6962
    04
    # length 0xF2 bytes (242 decimal)
    81F2
    # Beginning of TLS encoded SignedCertificateTimestampList - RFC 5246 / 6962
    # length 0xF0 bytes
    00F0
      # opaque SerializedSCT<1..2^16-1>
      # length 0x75 bytes
      0075
      # Version sct_version v1(0)
      00
      # LogID id (aka opaque key_id[32])
      DB74AFEECB29ECB1FECA3E716D2CE5B9AABB36F7847183C75D9D4F37B61FBF64
      # uint64 timestamp (milliseconds since the epoch)
      000001627313EB19
      # CtExtensions extensions (zero-length array)
      0000
      # digitally-signed struct - RFC 5426
      # SignatureAndHashAlgorithm (ecdsa-sha256)
      0403
      # opaque signature<0..2^16-1>;
      # length 0x0046
      0046
        # DER-encoded Ecdsa-Sig-Value - RFC 4492
        30 # SEQUENCE
        44 # length 0x44 bytes
          02 # r INTEGER
          20 # length 0x20 bytes
            # value
            7E1FCD1E9A2BD2A50A0C81E713033A0762340DA8F91EF27A48B3817640159CD3
          02 # s INTEGER
          20 # length 0x20 bytes
            # value
            659FE9F1D880E2E8F6B325BE9F18956D17C6CA8A6F2B12CB0F55FB70F759A419
      # opaque SerializedSCT<1..2^16-1>
      # length 0x77 bytes
      0077
      # Version sct_version v1(0)
      00
      # LogID id (aka opaque key_id[32])
      293C519654C83965BAAA50FC5807D4B76FBF587A2972DCA4C30CF4E54547F478
      # uint64 timestamp (milliseconds since the epoch)
      000001627313EB2A
      # CtExtensions extensions (zero-length array)
      0000
      # digitally-signed struct - RFC 5426
      # SignatureAndHashAlgorithm (ecdsa-sha256)
      0403
      # opaque signature<0..2^16-1>;
      # length 0x0048
      0048
        # DER-encoded Ecdsa-Sig-Value - RFC 4492
        30 # SEQUENCE
        46 # length 0x46 bytes
          02 # r INTEGER
          21 # length 0x21 bytes
            # value
            00AB72F1E4D6223EF87FC68491C208D29D4D57EBF47588BB7544D32F9537E2CEC1
          02 # s INTEGER
          21 # length 0x21 bytes
            # value
            008AFFC40CC6C4E3B24578DADE4F815ECBCE2D57A579342119A1E65BC7E5E69CE2

One surprising thing you might notice: In the first SCT, r and s are 32 (0x20) bytes long. In the second SCT, they are both 33 (0x21) bytes long, and have a leading zero. Integers in DER are two’s complement, so if the leftmost bit is set, they are interpreted as negative. Since r and s are positive, if the leftmost bit would be a 1, an extra byte has to be added so that the leftmost bit can be 0.

This is a little taste of what goes into encoding a certificate. I hope it was informative! If you’d like to learn more, I recommend “A Layman’s Guide to a Subset of ASN.1, BER, and DER.”

Footnote 1: A “poison extension” is defined by RFC 6962 section 3.1:

The Precertificate is constructed from the certificate to be issued by adding a special
critical poison extension (OID `1.3.6.1.4.1.11129.2.4.3`, whose
extnValue OCTET STRING contains ASN.1 NULL data (0x05 0x00))

In other words, it’s an empty extension whose only purpose is to ensure that certificate processors will not accept precertificates as valid certificates. The specification ensures this by setting the “critical” bit on the extension, which ensures that code that doesn’t recognize the extension will reject the whole certificate. Code that does recognize the extension specifically as poison will also reject the certificate.

Footnote 2: Lengths from 0-127 are represented by a single byte (short form). To express longer lengths, more bytes are used (long form). The high bit (0x80) on the first byte is set to distinguish long form from short form. The remaining bits are used to express how many more bytes to read for the length. For instance, 0x81F5 means “this is long form because the length is greater than 127, but there’s still only one byte of length (0xF5) to decode.”

While we’re proud of what we accomplished in 2017, we are spending most of the final quarter of the year looking forward rather than back. As we wrap up our own planning process for 2018, I’d like to share some of our plans with you, including both the things we’re excited about and the challenges we’ll face. We’ll cover service growth, new features, infrastructure, and finances.

Service Growth

We are planning to double the number of active certificates and unique domains we service in 2018, to 90 million and 120 million, respectively. This anticipated growth is due to continuing high expectations for HTTPS growth in general in 2018.

Let’s Encrypt helps to drive HTTPS adoption by offering a free, easy to use, and globally available option for obtaining the certificates required to enable HTTPS. HTTPS adoption on the Web took off at an unprecedented rate from the day Let’s Encrypt launched to the public.

One of the reasons Let’s Encrypt is so easy to use is that our community has done great work making client software that works well for a wide variety of platforms. We’d like to thank everyone involved in the development of over 60 client software options for Let’s Encrypt. We’re particularly excited that support for the ACME protocol and Let’s Encrypt is being added to the Apache httpd server.

Other organizations and communities are also doing great work to promote HTTPS adoption, and thus stimulate demand for our services. For example, browsers are starting to make their users more aware of the risks associated with unencrypted HTTP (e.g. Firefox, Chrome). Many hosting providers and CDNs are making it easier than ever for all of their customers to use HTTPS. Government agencies are waking up to the need for stronger security to protect constituents. The media community is working to Secure the News.

New Features

We’ve got some exciting features planned for 2018.

First, we’re planning to introduce an ACME v2 protocol API endpoint and support for wildcard certificates along with it. Wildcard certificates will be free and available globally just like our other certificates. We are planning to have a public test API endpoint up by January 4, and we’ve set a date for the full launch: Tuesday, February 27 Update: these features are live!

Later in 2018 we plan to introduce ECDSA root and intermediate certificates. ECDSA is generally considered to be the future of digital signature algorithms on the Web due to the fact that it is more efficient than RSA. Let’s Encrypt will currently sign ECDSA keys from subscribers, but we sign with the RSA key from one of our intermediate certificates. Once we have an ECDSA root and intermediates, our subscribers will be able to deploy certificate chains which are entirely ECDSA.

Infrastructure

Our CA infrastructure is capable of issuing millions of certificates per day with multiple redundancy for stability and a wide variety of security safeguards, both physical and logical. Our infrastructure also generates and signs nearly 20 million OCSP responses daily, and serves those responses nearly 2 billion times per day. We expect issuance and OCSP numbers to double in 2018.

Our physical CA infrastructure currently occupies approximately 70 units of rack space, split between two datacenters, consisting primarily of compute servers, storage, HSMs, switches, and firewalls.

When we issue more certificates it puts the most stress on storage for our databases. We regularly invest in more and faster storage for our database servers, and that will continue in 2018.

We’ll need to add a few additional compute servers in 2018, and we’ll also start aging out hardware in 2018 for the first time since we launched. We’ll age out about ten 2u compute servers and replace them with new 1u servers, which will save space and be more energy efficient while providing better reliability and performance.

We’ll also add another infrastructure operations staff member, bringing that team to a total of six people. This is necessary in order to make sure we can keep up with demand while maintaining a high standard for security and compliance. Infrastructure operations staff are systems administrators responsible for building and maintaining all physical and logical CA infrastructure. The team also manages a 24/7/365 on-call schedule and they are primary participants in both security and compliance audits.

Finances

We pride ourselves on being an efficient organization. In 2018 Let’s Encrypt will secure a large portion of the Web with a budget of only $3.0M. For an overall increase in our budget of only 13%, we will be able to issue and service twice as many certificates as we did in 2017. We believe this represents an incredible value and that contributing to Let’s Encrypt is one of the most effective ways to help create a more secure and privacy-respecting Web.

Our 2018 fundraising efforts are off to a strong start with Platinum sponsorships from Mozilla, Akamai, OVH, Cisco, Google Chrome and the Electronic Frontier Foundation. The Ford Foundation has renewed their grant to Let’s Encrypt as well. We are seeking additional sponsorship and grant assistance to meet our full needs for 2018.

We had originally budgeted $2.91M for 2017 but we’ll likely come in under budget for the year at around $2.65M. The difference between our 2017 expenses of $2.65M and the 2018 budget of $3.0M consists primarily of the additional infrastructure operations costs previously mentioned.

Support Let’s Encrypt

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

We’re grateful for the industry and community support that we receive, and we look forward to continuing to create a more secure and privacy-respecting Web!

We created Let’s Encrypt in order to make getting and managing TLS certificates as simple as possible. For Let’s Encrypt subscribers, this usually means obtaining an ACME client and executing some simple commands. Ultimately though, we’d like for most Let’s Encrypt subscribers to have ACME clients built in to their server software so that obtaining an additional piece of software is not necessary. The less work people have to do to deploy HTTPS the better!

ACME support being built in to one of the world’s most popular Web servers, Apache httpd, is great because it means that deploying HTTPS will be even easier for millions of websites. It’s a huge step towards delivering the ideal certificate issuance and management experience to as many people as possible.

The Apache httpd ACME module is called mod_md. It’s currently in the development version of httpd and a plan is being formulated to backport it to an httpd 2.4.x stable release. The mod_md code is also available on GitHub.

It’s also worth mentioning that the development version of Apache httpd now includes support for an SSLPolicy directive. Properly configuring TLS has traditionally involved making a large number of complex choices. With the SSLPolicy directive, admins simply select a modern, intermediate, or old TLS configuration, and sensible choices will be made for them.

Development of mod_md and the SSLPolicy directive has been funded by Mozilla and carried out primarily by Stefan Eissing of greenbytes. Thank you Mozilla and Stefan!

Let’s Encrypt is currently providing certificates for more than 55 million websites. We look forward to being able to serve even more websites as efforts like this make deploying HTTPS with Let’s Encrypt even easier. If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

Update, March 13, 2018

Wildcard certificate support is live.

Let’s Encrypt will begin issuing wildcard certificates in January of 2018. Wildcard certificates are a commonly requested feature and we understand that there are some use cases where they make HTTPS deployment easier. Our hope is that offering wildcards will help to accelerate the Web’s progress towards 100% HTTPS.

Let’s Encrypt is currently securing 47 million domains via our fully automated DV certificate issuance and management API. This has contributed heavily to the Web going from 40% to 58% encrypted page loads since Let’s Encrypt’s service became available in December 2015. If you’re excited about wildcard availability and our mission to get to a 100% encrypted Web, we ask that you contribute to our summer fundraising campaign.

A wildcard certificate can secure any number of subdomains of a base domain (e.g. *.example.com). This allows administrators to use a single certificate and key pair for a domain and all of its subdomains, which can make HTTPS deployment significantly easier.

Wildcard certificates will be offered free of charge via our upcoming ACME v2 API endpoint. We will initially only support base domain validation via DNS for wildcard certificates, but may explore additional validation options over time. We encourage people to ask any questions they might have about wildcard certificate support on our community forums.

We decided to announce this exciting development during our summer fundraising campaign because we are a nonprofit that exists thanks to the generous support of the community that uses our services. If you’d like to support a more secure and privacy-respecting Web, donate today!

We’d like to thank our community and our sponsors for making everything we’ve done possible. If your company or organization is able to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

First, it illustrates the strong demand for our services. We’d like to thank all of the sysadmins, web developers, and everyone else managing servers for prioritizing protecting your visitors with HTTPS.

Second, it illustrates our ability to scale. I’m incredibly proud of the work our engineering teams have done to make this volume of issuance possible. I’m also very grateful to our operational partners, including IdenTrust, Akamai, and Sumo Logic.

Third, it illustrates the power of automated certificate management. If getting and managing certificates from Let’s Encrypt always required manual steps there is simply no way we’d be able to serve as many sites as we do. We’d like to thank our community for creating a wide range of clients for automating certificate issuance and management.

The total number of certificates we’ve issued is an interesting number, but it doesn’t reflect much about tangible progress towards our primary goal: a 100% HTTPS Web. To understand that progress we need to look at this graph:

When Let’s Encrypt’s service first became available, less than 40% of page loads on the Web used HTTPS. It took the Web 20 years to get to that point. In the 19 months since we launched, encrypted page loads have gone up by 18%, to nearly 58%. That’s an incredible rate of change for the Web. Contributing to this trend is what we’re most proud of.

If you’re as excited about the potential for a 100% HTTPS Web as we are, please consider getting involved, making a donation, or sponsoring Let’s Encrypt.

Here’s to the next 100,000,000 certificates, and a more secure and privacy-respecting Web for everyone!

Update, April 27, 2018

ACME v2 and wildcard support are fully available since March 13, 2018.

Update, January 4, 2018

We introduced a public test API endpoint for the ACME v2 protocol and wildcard support on January 4, 2018. ACME v2 and wildcard support will be fully available on February 27, 2018.

Let’s Encrypt will add support for the IETF-standardized ACME v2 protocol in January of 2018. We will be adding a new ACME v2 API endpoint alongside our existing ACME v1 protocol API endpoint. We are not setting an end-of-life date for our ACME v1 API at this time, though we recommend that people move to the ACME v2 endpoint as soon as possible once it’s available. For most subscribers, this will happen automatically via a hosting provider or normal ACME client software update.

The ACME protocol, initially developed by the team behind Let’s Encrypt, is at the very heart of the CA service we provide. It’s the primary way in which we interact with our subscribers so that they can get and manage certificates. The ACME v1 protocol we use today was designed to ensure that our validation, issuance, and management methods are fully automated, consistent, compliant, and secure. In these respects, the current ACME v1 protocol has served us well.

There are three primary reasons why we’re starting a transition to ACME v2.

First, ACME v2 will be an IETF standard, and it’s important to us that we support true standards. While ACME v1 is a well-documented public specification, developed in a relatively open manner by individuals from a number of different organizations (including Mozilla, the Electronic Frontier Foundation, and the University of Michigan), it did not benefit from having been developed within a standards body with a greater diversity of inputs and procedures based on years of experience. It was always our intent for ACME v1 to form the basis for an IETF standardization process.

Second, ACME v2 was designed with additional input from other CAs besides Let’s Encrypt, so it should be easier for other CAs to use. We want a standardized ACME to work for many CAs, and ACME v1, while usable by other CAs, was designed with Let’s Encrypt in particular in mind. ACME v2 should meet more needs.

Third, ACME v2 brings some technical improvements that will allow us to better serve our subscribers going forward.

We are not setting an end-of-life date for the ACME v1 protocol because we don’t yet have enough data to determine when would be an appropriate date. Once we’re confident that we can predict an appropriate end-of-life date for our ACME v1 API endpoint we’ll announce one.

ACME v2 is the result of great work by the ACME IETF working group. In particular, we were happy to see the ACME working group take into account the needs of other organizations that may use ACME in the future. Certificate issuance and management protocols are a critical component of the Web’s trust model, and the Web will be better off if CAs can use a standardized public protocol that has been thoroughly vetted.

We’d like to thank our community, including our sponsors, for making everything we did this past year possible. Please consider getting involved or making a donation. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

OVH initially got in touch with Let’s Encrypt to become a Platinum sponsor shortly after our public launch in December of 2015. It was clear that they understood the need for Let’s Encrypt and our potential impact on the Web.

“Over a year ago, when Let’s Encrypt came out of beta, it was an obvious choice for OVH to support this new certificate authority, and become a Platinum sponsor,” said Octave Klaba, Founder, CTO and Chairman. “We provided free Let’s Encrypt certificates to all our Web customers. At OVH today, over 2.2 million websites can be reached over a secure connection, and a total of 3.6 million certificates were created for our customers during the first year.”

In the past year, Let’s Encrypt has grown to provide 28 million certificates to more than 31 million websites. The Web went from around 40% HTTPS page loads at the end of 2015 to 50% HTTPS page loads at the start of 2017. This is phenomenal growth for the Web, and Let’s Encrypt is proud to have been a driving force behind it.

Of course, it wouldn’t have been possible without major hosting providers like OVH making it easier for their customers to enable HTTPS with Let’s Encrypt. OVH was one of the first major hosting providers to make HTTPS available to a large number of their customers, and they are continuing to expand the scope of services that are secure by default.

“We then wanted to go one step further,” continues Octave Klaba. “We decided to launch SSL Gateway, powered by Let’s Encrypt. It’s an all-in-one front-end for your infrastructure with HTTPS encryption and anti-DDOS capability. It makes the Web even more secure and reliable. This service is now available to everyone, for free.”

Financial and product commitments like these from OVH are moving the Web toward our goal of 100% encryption. We depend on support from organizations like OVH to continue operating. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

At the start of 2016, Let’s Encrypt certificates had been available to the public for less than a month and we were supporting approximately 240,000 active (unexpired) certificates. That seemed like a lot at the time! Now we’re frequently issuing that many new certificates in a single day while supporting more than 20,000,000 active certificates in total. We’ve issued more than a million certificates in a single day a few times recently. We’re currently serving an average of 6,700 OCSP responses per second. We’ve done a lot of optimization work, we’ve had to add some hardware, and there have been some long nights for our staff, but we’ve been able to keep up and we’re ready for another year of strong growth.

We added a number of new features during the past year, including support for the ACME DNS challenge, ECDSA signing, IPv6, and Internationalized Domain Names.

When 2016 started, our root certificate had not been accepted into any major root programs. Today we’ve been accepted into the Mozilla, Apple, and Google root programs. We’re close to announcing acceptance into another major root program. These are major steps towards being able to operate as an independent CA. You can read more about why here.

The ACME protocol for issuing and managing certificates is at the heart of how Let’s Encrypt works. Having a well-defined and heavily audited specification developed in public on a standards track has been a major contributor to our growth and the growth of our client ecosystem. Great progress was made in 2016 towards standardizing ACME in the IETF ACME working group. We’re hoping for a final document around the end of Q2 2017, and we’ll announce plans for implementation of the updated protocol around that time as well.

Supporting the kind of growth we saw in 2016 meant adding staff, and during the past year Internet Security Research Group (ISRG), the non-profit entity behind Let’s Encrypt, went from four full-time employees to nine. We’re still a pretty small crew given that we’re now one of the largest CAs in the world (if not the largest), but it works because of our intense focus on automation, the fact that we’ve been able to hire great people, and because of the incredible support we receive from the Let’s Encrypt community.

Let’s Encrypt exists in order to help create a 100% encrypted Web. Our own metrics can be interesting, but they’re only really meaningful in terms of the impact they have on progress towards a more secure and privacy-respecting Web. The metric we use to track progress towards that goal is the percentage of page loads using HTTPS, as seen by browsers. According to Firefox Telemetry, the Web has gone from approximately 39% of page loads using HTTPS each day to just about 49% during the past year. We’re incredibly close to a Web that is more encrypted than not. We’re proud to have been a big part of that, but we can’t take credit for all of it. Many people and organizations around the globe have come to realize that we need to invest in a more secure and privacy-respecting Web, and have taken steps to secure their own sites as well as their customers’. Thank you to everyone that has advocated for HTTPS this year, or helped to make it easier for people to make the switch.

We learned some lessons this year. When we had service interruptions they were usually related to managing the rapidly growing database backing our CA. Also, while most of our code had proper tests, some small pieces didn’t and that led to incidents that shouldn’t have happened. That said, I’m proud of the way we handle incidents promptly, including quick and transparent public disclosure.

We also learned a lot about our client ecosystem. At the beginning of 2016, ISRG / Let’s Encrypt provided client software called letsencrypt. We’ve always known that we would never be able produce software that would work for every Web server/stack, but we felt that we needed to offer a client that would work well for a large number of people and that could act as a reference client. By March of 2016, earlier than we had foreseen, it had become clear that our community was up to the task of creating a wide range of quality clients, and that our energy would be better spent fostering that community than producing our own client. That’s when we made the decision to hand off development of our client to the Electronic Frontier Foundation (EFF). EFF renamed the client to Certbot and has been doing an excellent job maintaining and improving it as one of many client options.

As exciting as 2016 was for Let’s Encrypt and encryption on the Web, 2017 seems set to be an even more incredible year. Much of the infrastructure and many of the plans necessary for a 100% encrypted Web came into being or solidified in 2016. More and more hosting providers and CDNs are supporting HTTPS with one click or by default, often without additional fees. It has never been easier for people and organizations running their own sites to find the tools, services, and information they need to move to HTTPS. Browsers are planning to update their user interfaces to better reflect the risks associated with non-secure connections.

We’d like to thank our community, including our sponsors, for making everything we did this past year possible. Please consider getting involved or making a donation, and if your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

We decided to run a crowdfunding campaign for a couple of reasons. First, there is a gap between the funds we’ve raised and what we need for next year. Second, we believe individual supporters from our community can come to represent a significant diversification of our annual revenue sources, in addition to corporate sponsorship and grants.

We will provide updates on our progress throughout the campaign via Twitter (@letsencrypt).

Thank you for your support!

The Ford Foundation is a major philanthropic entity both in the US and globally. One of its programmatic areas, Internet Freedom, is focused on creating a more open and inclusive Internet experience for all people. Our relationship with Ford was born out of this mutual desire.

According to Michael Brennan, Ford Foundation Internet Freedom Program Officer, “We are thrilled to be able to support the growth of a Web that meets the needs of all its users through Let’s Encrypt.”

This grant will support various software development staff and activities, including the work we recently did to add support for Internationalized Domain Name (IDN) certificates.

If your company or organization would like to sponsor Let’s Encrypt, please email us at sponsor@letsencrypt.org.

We’re excited that Squarespace has decided to protect the millions of sites they host with HTTPS! While talking with their team we learned they were deploying OCSP Stapling from the get-go, and we were impressed. We asked them to share their experience with our readers in our first guest blog post (hopefully more to come).

- Josh Aas, Executive Director, ISRG / Let’s Encrypt

OCSP stapling is an alternative approach to the Online Certificate Status Protocol (OCSP) for checking the revocation status of certificates. It allows the presenter of a certificate to bear the resource cost involved in providing OCSP responses by appending (“stapling”) a time-stamped OCSP response signed by the CA to the initial TLS handshake, eliminating the need for clients to contact the CA. The certificate holder queries the OCSP responder at regular intervals and caches the responses.

Traditional OCSP requires the CA to provide responses to each client that requests certificate revocation information. When a certificate is issued for a popular website, a large amount of queries start hitting the CA’s OCSP responder server. This poses a privacy risk because information must pass through a third party and the third party is able to determine who browsed which site at what time. It can also create performance problems, since most browsers will contact the OCSP responder before loading anything on the web page. OCSP stapling is efficient because the user doesn’t have to make a separate connection to the CA, and it’s safe because the OCSP response is digitally signed so it cannot be modified without detection.

OCSP Stapling @ Squarespace

As we were planning our roll out of SSL for all custom domains on the Squarespace platform, we decided that we wanted to support OCSP stapling at time of launch. A reverse proxy built by our Edge Infrastructure team is responsible for terminating all SSL traffic, it’s written in Java and is powered by Netty. Unfortunately, the Java JDK 8 only has preliminary, client-only, OCSP stapling support. JDK 9 introduces OCSP stapling with JEP 249, but it is not available yet.

Our reverse proxy does not use the JDK’s SSL implementation. Instead, we use OpenSSL via netty-tcnative. At this time, neither the original tcnative nor Netty’s fork have OCSP stapling support. However, the tcnative library exposes the inner workings of OpenSSL, including the address pointers for the SSL context and engine. We were able to use JNI to extend the netty-tcnative library and add OCSP stapling support using the tlsext_status OpenSSL C functions. Our extension is a standalone library but we could equally well fold it into the netty-tcnative library itself. If there is interest, we can contribute it upstream as part of Netty’s next API-breaking development cycle.

One of the goals of our initial OCSP stapling implementation was to take the biggest edge off of the OCSP responder’s operator, in this case Let’s Encrypt. Due to the nature of the website traffic on our platform, we have a very long tail. At least to start, we don’t pre-fetch and cache all OCSP responses. We decided to fetch OCSP responses asynchronously and we try to do it only if more than one client is going to use it in the foreseeable future. Bloom filters are utilized to identify “one-hit wonders” that are not worthy of being cached.

Squarespace invests in the security of our customers’ websites and their visitors. We will continue to make refinements to our OCSP stapling implementation to eventually have OCSP staples on all requests. For a more in depth discussion about the security challenges of traditional OCSP, we recommend this blog post.

We’re excited about this feature because our goal is to serve the entire Web, including those who want to use domains with language-specific characters. This feature was also commonly requested by our community.

There are more details on how to request a certificate containing IDNs at our community forum. Visit our Getting Started page for information on how to request certificates in general.

Let’s Encrypt depends on industry and community support. Please consider getting involved, and if your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

Download Legal Transparency Report, January 2016 - June 2016

Let’s Encrypt will require about $2.9M USD to operate in 2017. We believe this is an incredible value for a secure and reliable service that is capable of issuing certificates globally, to every server on the Web free of charge.

We’re currently working to raise the money we need to operate through the next year. Please consider donating or becoming a sponsor if you’re able to do so! In the event that we end up being able to raise more money than we need to just keep Let’s Encrypt running we can look into adding other services to improve access to a more secure and privacy-respecting Web.

Here’s how our 2017 budget breaks down:

Staffing is our dominant cost. We currently have eight full time employees, plus two full time staff that are employed by other entities (Mozilla and EFF). This includes five operations/sysadmin staff, three software developers, one communications and fundraising person, and an executive director. Our 2017 budget covers salary and benefits for ten employees.

Our systems administration staff are at the heart of our day to day operations. They are responsible for building and improving our server, networking, and deployed software infrastructure, as well as monitoring the systems every hour of every day. It’s the critical 24/7 nature of the work that makes this our biggest team. Any issues need to be dealt with immediately, ideally with multiple people on hand.

Our software developers work primarily on boulder, our open source CA software. We needed to write our own software in order to create a secure, reliable, and fully-automated CA that is capable of issuing and managing enough certificates to serve the entire Web. Our software development staff also allow us to support new features much more quickly than we could if we relied on third party software for implementation.

The majority of our administrative support (e.g. HR, payroll, accounting) is provided by the Linux Foundation, so we don’t hire for those roles and related expenses come in under the “Legal/Administrative” category.

Hardware expenses include compute, storage, networking, and HSM hardware, as well as the associated support contracts. There is quite a bit of duplication for redundancy. Software expenses are low since the majority of the software we use is freely available open source software.

Hosting costs include space in two different highly secure geographically separated rooms inside secure data centers, as well as internet connections and power. The hardware and physical infrastructure we have in place is capable of issuing hundreds of millions of certificates - enough for every server on the Web. We need to maintain strong physical control over all hardware and infrastructure related to certificate issuance and management for security and auditing reasons.

Auditing costs include the required annual WebTrust audits as well as third party expert security review and testing. The third party security audits include code review, infrastructure review, penetration testing, and ACME protocol analysis. We are not required to do third party auditing beyond the WebTrust audits, but it would be irresponsible of us not to.

Legal costs go towards attorney time, primarily in the areas of corporate governance, contract development and review, and trademarks. Administrative costs include HR, payroll and benefits management, accounting and tax services, as well as travel and other miscellaneous operating costs.

Our 2016 budget is very similar to our 2017 budget, the major difference being that we will only spend approximately $2.0M USD due to a number of our staff starting after the beginning of the year. We will pay full staffing costs next year because all of the staff that joined us in 2016 will be on our payroll for the entirety of 2017.

Currently, the majority of our funding comes from corporate sponsorships. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We’re working to make grants and individual contributions more significant sources of income over the next year.

We’re grateful for the industry and community support that we receive, and we look forward to continuing to create a more secure and privacy-respecting Web!

Public CAs need their certificates to be trusted by browsers and devices. CAs that want to issue independently under their own root accomplish this by either buying an existing trusted root, or by creating a new root and working to get it trusted. Let’s Encrypt chose to go the second route.

Getting a new root trusted and propagated broadly can take 3-6 years. In order to start issuing widely trusted certificates as soon as possible, we partnered with another CA, IdenTrust, which has a number of existing trusted roots. As part of that partnership, an IdenTrust root “vouches for” the certificates that we issue, thus making our certificates trusted. We’re incredibly grateful to IdenTrust for helping us to start carrying out our mission as soon as possible.

Chain of Trust Between Firefox and Let's Encrypt Certificates

However, our plan has always been to operate as an independently trusted CA. Having our root trusted directly by the Mozilla root program represents significant progress towards that independence.

We have also applied to the Microsoft, Apple, Google, Oracle and Blackberry root programs. We look forward to acceptance into these programs as well.

Let’s Encrypt depends on industry and community support. Please consider getting involved, and if your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

As IPv4 address space is exhausted, more and more people are deploying services that are only reachable via IPv6. Adding full support for IPv6 allows us to serve more people and organizations, which is important if we’re going to encrypt the entire Web.

IPv6 is an exciting step forward which will allow the Internet to grow and reach more people. You can learn more about it by watching this video from Google’s Chief Internet Evangelist, Vint Cerf. We’re looking forward to the day when both TLS and IPv6 are ubiquitous.

Let’s Encrypt depends on industry and community support. Please consider getting involved, and if your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

Update, June 24 2016

We have confirmed that Comodo submitted Requests for Express Abandonment for all three trademark registration applications in question. We’re happy to see this positive step towards resolution, and will continue to monitor the requests as they make their way through the system.

We’d like to thank our community for their support.

Some months ago, it came to our attention that Comodo Group, Inc., is attempting to register at least three trademarks for the term “Let’s Encrypt,” for a variety of CA-related services [1][2][3]. These trademark applications were filed long after the Internet Security Research Group (ISRG) started using the name Let’s Encrypt publicly in November of 2014, and despite the fact Comodo’s “intent to use” trademark filings acknowledge that it has never used “Let’s Encrypt” as a brand.

We’ve forged relationships with millions of websites and users under the name Let’s Encrypt, furthering our mission to make encryption free, easy, and accessible to everyone. We’ve also worked hard to build our unique identity within the community and to make that identity a reliable indicator of quality. We take it very seriously when we see the potential for our users to be confused, or worse, the potential for a third party to damage the trust our users have placed in us by intentionally creating such confusion. By attempting to register trademarks for our name, Comodo is actively attempting to do just that.

Since March of 2016 we have repeatedly asked Comodo to abandon their “Let’s Encrypt” applications, directly and through our attorneys, but they have refused to do so. We are clearly the first and senior user of “Let’s Encrypt” in relation to Internet security, including SSL/TLS certificates – both in terms of length of use and in terms of the widespread public association of that brand with our organization.

If necessary, we will vigorously defend the Let’s Encrypt brand we’ve worked so hard to build. That said, our organization has limited resources and a protracted dispute with Comodo regarding its improper registration of our trademarks would significantly and unnecessarily distract both organizations from the core mission they should share: creating a more secure and privacy-respecting Web. We urge Comodo to do the right thing and abandon its “Let’s Encrypt” trademark applications so we can focus all of our energy on improving the Web.

[1] “Let’s Encrypt” Trademark Registration Application

[2] “Let’s Encrypt With Comodo” Trademark Registration Application

[3] “Comodo Let’s Encrypt” Trademark Registration Application

Let’s Encrypt has issued more than 5 million certificates in total since we launched to the general public on December 3, 2015. Approximately 3.8 million of those are active, meaning unexpired and unrevoked. Our active certificates cover more than 7 million unique domains.

A couple of different factors have contributed heavily to this growth. The first is large-scale deployments from companies such as OVH, WordPress.com, Akamai, Shopify, Dreamhost, and Bitly. The second is our ability to serve individual sites globally with a focus on ease-of-use. If we’re going to get to 100% HTTPS we need to reach the “long tail” of the Web, which is in many ways more challenging due to the number of parties involved and widely varying degrees of technical competency.

Our progress is accelerating the growth of HTTPS on the Web in general. When we launched in December of 2015, 39.5% of page loads on the Web used HTTPS (as measured by Firefox Telemetry). By mid-April 2016 that number was up to 42% and today it stands at 45%.

This is an incredible rate of change for the Web. We’re really excited about our early progress. Getting to 50% HTTPS page loads in 2016 used to seem like an overly ambitious goal but now it seems within reach. Let’s get there!

Let’s Encrypt depends on industry and community support. Please consider getting involved, and if your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

Since our beta began in September 2015 we’ve issued more than 1.7 million certificates for more than 3.8 million websites. We’ve gained tremendous operational experience and confidence in our systems. The beta label is simply not necessary any more.

We set out to encrypt 100% of the Web. We’re excited to be off to a strong start, and with so much support across the industry.

“From the very beginning, Akamai has been committed to supporting Let’s Encrypt’s vision of enabling greater use of SSL/TLS across the internet,” says Stephen Ludin, Chief Architect at Akamai. “This milestone is confirmation of Let’s Encrypt’s ability to execute on that vision and have a tremendous impact to the Internet ecosystem.”

“Cisco is committed to improving the security of the Internet, not only for our customers and partners, but for everyone else as well,” says David Ward, CTO of Engineering and Chief Architect at Cisco. “Let’s Encrypt has been doing impressive work toward that goal. Our support of this community towards real-time, on-demand certificates will make the Internet more secure.”

“We’re very proud to be a Gold Sponsor for Let’s Encrypt which leverages our industry-leading hardware security modules to protect their certificate authority system,” says Todd Moore, Vice President of Encryption Product Management at Gemalto. “Encryption by default is critical to privacy and security, and by working with Let’s Encrypt Gemalto is helping to deliver trust for the digital services that billions of people use every day.”

Let’s Encrypt depends on industry and community support. Please consider getting involved, and if your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

Download Legal Transparency Report, July 2015 - December 2015

Over the next few months the Let’s Encrypt client software will transition to a new name, soon to be announced, and a new home at the Electronic Frontier Foundation (EFF).

This change does not affect the Let’s Encrypt certificate authority (CA) or associated services, which will retain the Let’s Encrypt name and continue to be hosted by the Internet Security Research Group.

The goal of Let’s Encrypt is to make turning on HTTPS as easy as possible. To accomplish that, it’s not enough to fully automate certificate issuance on the certificate authority (CA) side - we have to fully automate on the client side as well. The Let’s Encrypt client is now being used by hundreds of thousands of websites and we expect it to continue to be a popular choice for sites that are run from a single server or VPS.

That said, the web server ecosystem is complex, and it would be impossible for any particular client to serve everyone well. As a result, the Let’s Encrypt community has created dozens of clients to meet many diverse needs. Moving forward, we feel it would be best for Let’s Encrypt to focus on promoting a generally healthy client and protocol ecosystem and for our client to move to the EFF. This will also allow us to focus our engineering efforts on running a reliable and rapidly growing CA server infrastructure.

The Let’s Encrypt client goes further than most other clients in terms of end-to-end automation and extensibility, both getting certificates and in many cases installing them. This is an important strategy since major servers don’t yet have built-in support, and we want to make sure it’s given a proper chance to thrive. The EFF has led development of the Let’s Encrypt client from the beginning, and they are well-qualified to continue pursuing this strategy.

The rename is happening for reasons that go beyond the move to the EFF. One additional reason for the rename is that we want the client to be distributable and customizable without having to create a complex process for deciding whether customized variants are appropriate for use with Let’s Encrypt trademarks. Another reason is that we want it to be clear that the client can work with any ACME-enabled CA in the future, not just Let’s Encrypt.

We expect the client to do well at the EFF and continue to be used by many people to get certificates from Let’s Encrypt.

We want to see HTTPS become the default on the Web, and today’s occasion gives us confidence that we’re going to get there - much faster than even we predicted. We’re growing at a current rate of more than 100,000 certificates per week and don’t see this slowing down anytime soon. This is dramatic and very rapid change for the Web.

Our rapid growth is due to strong demand for an easy-to-use, low-cost, widely trusted, and truly global solution for certificate issuance and management. We also received a considerable boost from industry endorsement, with major hosting companies like OVH, Wordpress.com, Gandi, Dreamhost, and Digital Ocean helping many sites move to HTTPS with Let’s Encrypt.

HTTPS has been around for a long time but according to Firefox telemetry only ~40% of websites and ~65% of transactions used HTTPS at the end of 2015. Those numbers should both be 100% if the Web is to provide the level of privacy and security that people expect, and Let’s Encrypt is going to lead the way.

Let’s Encrypt depends on support from a wide variety of individuals and organizations. Please consider getting involved, and if your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

According to OVH CTO and Founder Octave Klaba, “OVH is delighted to become a Platinum sponsor. With Let’s Encrypt, OVH will be able to set a new standard for security by offering end-to-end encrypted communications by default to all its communities.”

The Web is an increasingly integral part of our daily lives, and encryption by default is critical in order to provide the degree of security and privacy that people expect. Let’s Encrypt’s mission is to encrypt the Web and our sponsors make pursuing that mission possible.

OVH’s sponsorship will help us to pay for staff and other operation costs in 2016.

If your company or organization would like to sponsor Let’s Encrypt, please email us at sponsor@letsencrypt.org.

It’s time for the Web to take a big step forward in terms of security and privacy. We want to see HTTPS become the default. Let’s Encrypt was built to enable that by making it as easy as possible to get and manage certificates.

We’d like to thank everyone who participated in the Limited Beta. Let’s Encrypt issued over 26,000 certificates during the Limited Beta period. This allowed us to gain valuable insight into how our systems perform, and to be confident about moving to Public Beta.

We’d also like to thank all of our sponsors for their support. We’re happy to have announced earlier today that Facebook is our newest Gold sponsor.

We have more work to do before we’re comfortable dropping the beta label entirely, particularly on the client experience. Automation is a cornerstone of our strategy, and we need to make sure that the client works smoothly and reliably on a wide range of platforms. We’ll be monitoring feedback from users closely, and making improvements as quickly as possible.

Instructions for getting a certificate with the Let’s Encrypt client can be found here.

Update: Feb. 10 2016, instructions for getting a certificate with Certbot (Let’s Encrypt recommended client) can be found here.

Let’s Encrypt Community Support is an invaluable resource for our community, we strongly recommend making use of the site if you have any questions about Let’s Encrypt.

Let’s Encrypt depends on support from a wide variety of individuals and organizations. Please consider getting involved, and if your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

According to Alex Stamos, Chief Security Officer at Facebook, “Making it easier for websites to deploy HTTPS encryption is an important step in improving the security of the whole internet, and Facebook is proud to support this effort.”

Facebook’s sponsorship will help us produce a greater impact as we open up our public beta today and usher in more participants over the coming months.

If your company or organization would like to sponsor Let’s Encrypt, please email us at sponsor@letsencrypt.org.

Our Limited Beta started on September 12, 2015. We’ve issued over 11,000 certificates since then, and this operational experience has given us confidence that our systems are ready for an open Public Beta.

It’s time for the Web to take a big step forward in terms of security and privacy. We want to see HTTPS become the default. Let’s Encrypt was built to enable that by making it as easy as possible to get and manage certificates.

We have more work to do before we’re comfortable dropping the beta label entirely, particularly on the client experience. Automation is a cornerstone of our strategy, and we need to make sure that the client works smoothly and reliably on a wide range of platforms. We’ll be monitoring feedback from users closely, and making improvements as quickly as possible.

Let’s Encrypt depends on support from a wide variety of individuals and organizations. Please consider getting involved, and if your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org.

Ninety days is nothing new on the Web. According to Firefox Telemetry, 29% of TLS transactions use ninety-day certificates. That’s more than any other lifetime. From our perspective, there are two primary advantages to such short certificate lifetimes:

1. They limit damage from key compromise and mis-issuance. Stolen keys and mis-issued certificates are valid for a shorter period of time.
2. They encourage automation, which is absolutely essential for ease-of-use. If we’re going to move the entire Web to HTTPS, we can’t continue to expect system administrators to manually handle renewals. Once issuance and renewal are automated, shorter lifetimes won’t be any less convenient than longer ones.

For these reasons, we do not offer certificates with lifetimes longer than ninety days. We realize that our service is young, and that automation is new to many subscribers, so we chose a lifetime that allows plenty of time for manual renewal if necessary. We recommend that subscribers renew every sixty days. Once automated renewal tools are widely deployed and working well, we may consider even shorter lifetimes.

Deciding what to do here has been tough. On the one hand, we don’t like these sites any more than anyone else does, and our mission is to help build a safer and more secure Web. On the other hand, we’re not sure that certificate issuance (at least for Domain Validation) is the right level on which to be policing phishing and malware sites in 2015. This post explains our thinking in order to encourage a conversation about the CA ecosystem’s role in fighting these malicious sites.

CAs Make Poor Content Watchdogs

Let’s Encrypt is going to be issuing Domain Validation (DV) certificates. On a technical level, a DV certificate asserts that a public key belongs to a domain – it says nothing else about a site’s content or who runs it. DV certificates do not include any information about a website’s reputation, real-world identity, or safety. However, many people believe the mere presence of a DV certificate ought to connote at least some of these things.

Treating a DV certificate as a kind of “seal of approval” for a site’s content is problematic for several reasons.

First, CAs are not well positioned to operate anti­-phishing and anti-malware operations – or to police content more generally. They simply do not have sufficient ongoing visibility into sites’ content. The best CAs can do is check with organizations that have much greater content awareness, such as Microsoft and Google. Google and Microsoft consume vast quantities of data about the Web from massive crawling and reporting infrastructures. This data allows them to use complex machine learning algorithms (developed and operated by dozens of staff) to identify malicious sites and content.

Even if a CA checks for phishing and malware status with a good API, the CA’s ability to accurately express information regarding phishing and malware is extremely limited. Site content can change much faster than certificate issuance and revocation cycles, phishing and malware status can be page-specific, and certificates and their related browser UIs contain little, if any, information about phishing or malware status. When a CA doesn’t issue a certificate for a site with phishing or malware content, users simply don’t see a lock icon. Users are much better informed and protected when browsers include anti-phishing and anti-malware features, which typically do not suffer from any of these limitations.

Another issue with treating DV certificates as a “seal of approval” for site content is that there is no standard for CA anti­-phishing and anti-malware measures beyond a simple blacklist of high-­value domains, so enforcement is inconsistent across the thousands of CAs trusted by major browsers. Even if one CA takes extraordinary measures to weed out bad sites, attackers can simply shop around to different CAs. The bad guys will almost always be able to get a certificate and hold onto it long enough to exploit people. It doesn’t matter how sophisticated the best CA anti­-phishing and anti-malware programs are, it only matters how good the worst are. It’s a “find the weakest link” scenario, and weak links aren’t hard to find.

Browser makers have realized all of this. That’s why they are pushing phishing and malware protection features, and evolving their UIs to more accurately reflect the assertions that certificates actually make.

TLS No Longer Optional

When they were first developed in the 1990s, HTTPS and SSL/TLS were considered “special” protections that were only necessary or useful for particular kinds of websites, like online banks and shopping sites accepting credit cards. We’ve since come to realize that HTTPS is important for almost all websites. It’s important for any website that allows people to log in with a password, any website that tracks its users in any way, any website that doesn’t want its content altered, and for any site that offers content people might not want others to know they are consuming. We’ve also learned that any site not secured by HTTPS can be used to attack other sites.

TLS is no longer the exception, nor should it be. That’s why we built Let’s Encrypt. We want TLS to be the default method for communication on the Web. It should just be a fundamental part of the fabric, like TCP or HTTP. When this happens, having a certificate will become an existential issue, rather than a value add, and content policing mistakes will be particularly costly. On a technical level, mistakes will lead to significant down time due to a slow issuance and revocation cycle, and features like HSTS. On a philosophical and moral level, mistakes (innocent or otherwise) will mean censorship, since CAs would be gatekeepers for online speech and presence. This is probably not a good role for CAs.

Our Plan

At least for the time being, Let’s Encrypt is going to check with the Google Safe Browsing API before issuing certificates, and refuse to issue to sites that are flagged as phishing or malware sites. Google’s API is the best source of phishing and malware status information that we have access to, and attempting to do more than query this API before issuance would almost certainly be wasteful and ineffective. (Update: As of January 10, 2019, we no longer check domains against the Safe Browsing API.)

We’re going to implement this phishing and malware status check because many people are not comfortable with CAs entirely abandoning anti-phishing and anti-malware efforts just yet, even for DV certificates. We’d like to continue the conversation for a bit longer before we abandon what many people perceive to be an important CA behavior, even though we disagree.

Conclusion

The fight against phishing and malware content is an important one, but it does not make sense for CAs to be on the front lines, at least when it comes to DV certificates. That said, we’re going to implement checks against the Google Safe Browsing API while we continue the conversation.

We look forward to hearing what you think. Please let us know.

Both Let’s Encrypt intermediate certificates, Let’s Encrypt Authority X1 and Let’s Encrypt Authority X2, received cross-signatures. Web servers will need to be configured to serve the appropriate cross-signature certificate as part of the trust chain. The Let’s Encrypt client will handle this automatically.

You can see an example of a server using a Let’s Encrypt certificate under a new cross-signed intermediate here.

Vital personal and business information is flowing over the Internet more frequently than ever, and it’s time to encrypt all of it. That’s why we created Let’s Encrypt, and we’re excited to be one big step closer to bringing secure connections to every corner of the Web.

According to Olaf Kolkman, “The Internet Society sees encryption as a new norm to enhance the security of, and thereby trust in, the Internet.” We couldn’t agree more.

Internet Society’s sponsorship comes at a key time, and will help us to pay for staff and other operation costs as we move from our beta program into general availability.

If your company or organization would like to sponsor Let’s Encrypt, please email us at sponsor@letsencrypt.org.

Our cross signature is not yet in place, however this certificate is fully functional for clients with the ISRG root in their trust store. When we are cross signed, approximately a month from now, our certificates will work just about anywhere while our root propagates. We submitted initial applications to the root programs for Mozilla, Google, Microsoft, and Apple today.

We’re thrilled to finally be a live CA. We’ll be working towards general availability over the next couple of months by issuing certificates to domains participating in our beta program. You can request that your domain be included in our beta program by clicking here.

If you want to get involved with Let’s Encrypt, please visit this page.

We’re a small organization hoping to help a large portion of the Web move to HTTPS. Many people are going to have questions during the transition, and we need to be able to provide answers. Hiring an army of support staff is out of the question for us, but we’re not convinced that would be the way to go anyway, even if we could afford it.

Clay Shirky once said that you’ll make more accurate predictions about the quality and longevity of services in a Web-driven world “if you ask yourself not what’s the business model, but rather do the people who like it take care of each other?”

No amount of staffing could provide the level of support that a strong community of fellow users is capable of providing. We sincerely hope that our users will join us in taking care of each other.

We’d like to thank Discourse for providing the Let’s Encrypt Community Support platform and hosting.

We’ve decided to push our launch schedule back a bit to give us time to further improve our systems. Our new schedule is:

• First certificate: Week of September 7, 2015
• General availability: Week of November 16, 2015

In the ten weeks between these two dates we’ll gradually issue more and more certificates. We’ll start by issuing a small number of certs to whitelisted domains and expand our issuance as we gain confidence in our systems (stay tuned for instructions on getting your domains added to our early-access whitelist). When it’s time for general availability we will open up our systems to all requests.

A cross-signature will be in place before general availability. This will allow certificates from Let’s Encrypt to validate automatically for the vast majority of consumers. Prior to cross-signing, browsers will not accept our certificates as valid unless a user has installed our root as trusted.

I want to thank our staff, partners, sponsors, and contributors. We’re making great progress, despite this delay, and it’s all possible due to their hard work.

Download Legal Transparency Report, January 2015 - June 2015

• First certificate: Week of July 27, 2015
• General availability: Week of September 14, 2015

We will issue the first end entity certificates under our root under tightly controlled circumstances. No cross-signature will be in place yet, so the certificates will not validate unless our root is installed in client software. As we approach general availability we will issue more and more certificates, but only for a pre-approved set of domains. This limited issuance period will give us time to further ensure that our systems are secure, compliant, and scalable.

When it’s time for general availability, we will open up our systems to certificate requests for any domain. A cross-signature from IdenTrust will be in place for general availability, so that our certificates will validate automatically for the vast majority of consumers.

Engineering and policy development for Let’s Encrypt began in earnest in mid-October of 2014. If we stay true to the schedule outlined above we will have built an innovative CA, capable of operating at Internet scale and without cutting corners, in just eleven months. That’s quite a feat, given all that’s involved, and a testament to the skill and dedication of our staff, partners, sponsors, and contributors.

• Key pair and self-signed cert for the ISRG root
• Key pair and certificate for the ISRG root’s OCSP
• Key pairs and certificates for two Let’s Encrypt intermediate CAs
• CRL under the ISRG root showing that the Let’s Encrypt intermediates have not been revoked

The certificates over the public keys, of course, can be made public:

• ISRG Root X1 Certificate
• Let’s Encrypt Intermediate X1 CA Certificate
• Let’s Encrypt Intermediate X2 CA Certificate

Let’s Encrypt will issue certificates to subscribers from its intermediate CAs, allowing us to keep our root CA safely offline. IdenTrust will cross-sign our intermediates. This will allow our end certificates to be accepted by all major browsers while we propagate our own root.

Under normal circumstances, certificates issued by Let’s Encrypt will come from “Let’s Encrypt Intermediate X1”. The other intermediate, “Let’s Encrypt Intermediate X2”, is associated with our disaster recovery site and will only be used should we lose the ability to issue with “Let’s Encrypt Intermediate X1”.

The private keys for the ISRG root CA and the Let’s Encrypt intermediate CAs are stored on hardware security modules (HSMs), which provide a high degree of protection against the keys being stolen.

All ISRG keys are currently RSA keys. We are planning to generate ECDSA keys later this year.

The generation of these keys and certificates is an important step in getting Let’s Encrypt ready to issue certificates. In the next few weeks, we’ll be saying some more about our plans for going live. In the meantime, we would love for you to get involved.

Today we’re publishing the first public draft of the Let’s Encrypt Subscriber Agreement. We’ve attempted to create terms that are sensible for both our organization and our subscribers, and we want to give people a chance to review them before we start issuing certificates. Please let us know if you have any questions or comments.

Today we’re publishing an updated draft of our Certificate Policy (CP) and the first public draft of our Certification Practice Statement (CPS). These are important documents for any Certificate Authority (CA), and we want to give people a chance to review ours before we start issuing certificates. Please let us know if you have any questions or comments.

The NCC Group audit will take place prior to the general availability of Let’s Encrypt’s service, and is intended to provide additional assurance that our systems are secure. The NCC Group audit does not replace the WebTrust audit, which will happen after general availability.

“I’m very much looking forward to the general availability of Let’s Encrypt - lowering the bar both technically and financially for people to deploy TLS will result in a more encrypted Internet that helps increase security and preserve people’s privacy,” said Tom Ritter, Practice Director, NCC Group.

Let’s Encrypt’s certificate authority software is called boulder. It’s largely written in the go language and makes use of CloudFlare’s CFSSL tools, which are also written in go. Our boulder software contains modules including a web front-end and registration, validation, certificate, and storage authorities.

ACME, short for Automated Certificate Management Environment, is the protocol that Let’s Encrypt will use for automatic certificate issuance and management. We hope to standardize ACME in the IETF, starting with the formation of a working group later this year.

The Linux Foundation is a non-profit organization dedicated to advancing Linux and collaborative development. Founded in 2000, The Linux Foundation sponsors the work of Linux creator Linus Torvalds and is supported by leading Linux and open source companies and developers from around the world.

While ISRG and Let’s Encrypt are not specifically oriented towards Linux, we share an enthusiasm for solving problems with open and collaborative initiatives. The Linux Foundation hosts a variety of projects that embrace open source and collaborative development practices and principles, ranging from Yocto Project (embedded Linux) to AllSeen Alliance (Internet of Things) and Cloud Foundry (Platform-as-a-Service). It also hosts the Core Infrastructure Initiative, which identifies and funds critical open source projects that support the world’s infrastructure. This was established in response to Heartbleed a year ago and is today funding important work to ensure a secure Internet.

ISRG’s collaboration with The Linux Foundation will allow our staff and management to focus on carrying out our mission. While we do so, we’ll be supported by a team at The Linux Foundation with a wealth of experience helping similar organizations. Furthermore, sharing support infrastructure with other projects allows us to put more of our donors’ money directly towards carrying out our mission.

Today we’re publishing a draft of our Certificate Policy (CP). This is an important document for any Certificate Authority (CA), and we want to give people a chance to review it before we start issuing certificates. Please let us know if you have any questions or comments.

We’re also working on our Certification Practice Statement (CPS), which describes how we’re going to issue certificates under the policies outlined in our CP, but it isn’t quite ready for public review yet. We’re hoping to publish a draft within the next month.

The challenge is server certificates. The anchor for any TLS-protected communication is a public-key certificate which demonstrates that the server you’re actually talking to is the server you intended to talk to. For many server operators, getting even a basic server certificate is just too much of a hassle. The application process can be confusing. It usually costs money. It’s tricky to install correctly. It’s a pain to update.

Let’s Encrypt is a new free certificate authority, built on a foundation of cooperation and openness, that lets everyone be up and running with basic server certificates for their domains through a simple one-click process.

Mozilla Corporation, Cisco Systems, Inc., Akamai Technologies, Electronic Frontier Foundation, IdenTrust, Inc., and researchers at the University of Michigan are working through the Internet Security Research Group (“ISRG”), a California public benefit corporation, to deliver this much-needed infrastructure in Q2 2015. The ISRG welcomes other organizations dedicated to the same ideal of ubiquitous, open Internet security.

The key principles behind Let’s Encrypt are:

• Free: Anyone who owns a domain can get a certificate validated for that domain at zero cost.
• Automatic: The entire enrollment process for certificates occurs painlessly during the server’s native installation or configuration process, while renewal occurs automatically in the background.
• Secure: Let’s Encrypt will serve as a platform for implementing modern security techniques and best practices.
• Transparent: All records of certificate issuance and revocation will be available to anyone who wishes to inspect them.
• Open: The automated issuance and renewal protocol will be an open standard and as much of the software as possible will be open source.
• Cooperative: Much like the underlying Internet protocols themselves, Let’s Encrypt is a joint effort to benefit the entire community, beyond the control of any one organization.

If you want to help these organizations in making TLS Everywhere a reality, here’s how you can get involved:

• Sponsor ISRG
• Help Us Build Let’s Encrypt

To learn more about the ISRG and our partners, check out our About page.