The story of how WoSign gave me an SSL certificate for GitHub.com

It was rather surreal when I realized I had actual valid SSL/TLS certificates for the primary GitHub domains. Https is supposed to prevent eavesdropping, yet with these keys, I could become a man-in-the-middle with relative ease.

Let me take you back to the beginning and walk you through how I acquired these certificates from the certificate authority WoSign.

StartSSL

As the owner of this website and domain schrauger.com, I wanted to ensure the pages were encrypted with https. I don't have anything particularly interesting on the site, but it wouldn't hurt, and my server could handle the minute increase in loads. So about 3 1/2 years ago, I went to StartSSL. At the time, they were the only trusted certificate authority that gave away free SSL certificates.

Certificate Authority (CA)

Anyone can generate a certificate for any domain. The catch is that your browser will not see these self-signed certificates as valid. It only trusts a few select operators, called certificate authorities. Only these entities are able to sign your certificate, which basically tells the world that they have vetted you and confirmed that you are the proper owner and operator of a specific domain.

As I learned and experimented, I started adding subdomains for specific services. StartSSL gives free certificates, but each cert only works for one domain. So each subdomain I created required a new certificate. This became a hassle to manage, as they all expired a year from creation, and I was constantly logging into my StartSSL control panel to generate a replacement for various subdomains.

WoSign

Enter WoSign. This Chinese CA popped on my radar in the spring of 2015. They had announced that they would be giving away free SSL certificates, and not only were the certificates valid for 3 years instead of 1, they would also allow up to 100 domains and subdomains on a single cert.

Since WoSign is another trusted CA, nothing would change to end users of my site, but I would have a single certificate to manage.

University of Central Florida

As my day job, I am a linux system admin and web developer for the University of Central Florida, specifically working at the College of Medicine. I manage the site med.ucf.edu along with a few various other subsites. I decided to create a test certificate from WoSign for our website to try them out.

Domain Validation

In order to acquire a certificate for any domain, the trusted CAs require that you prove ownership in one of several ways. One of WoSign's methods (also used by many other CAs) is to give you a text file filled with unique data. You must put the file in a specific location on your website such that WoSign can browse to that location and download the file. They compare their download to the file they originally gave you, and if it all matches up, it proves that you are in control. Your request is approved, and your certificate is signed and given.

Also note that by proving control of a primary domain, you are assumed to control any subdomains. This is because the DNS operator of a given domain has full control over where any subdomains go to. Most CAs will assume this, and it is a reasonable assumption that the owner of a website at the primary domain is also in control of the websites of any subdomains.

Exploit Discovered!

I was, of course, approved for med.ucf.edu. As usual, I decided to have the subdomain 'www' added on.

However, instead of applying for 'www.med.ucf.edu', I accidentally put in 'www.ucf.edu'. I realized my error and was going to remove the second request when I realized that WoSign showed it thought I was authorized for all domains I had listed.

I am not the admin for the University's primary site. I certainly do not control ucf.edu or www.ucf.edu. Yet WoSign was happy to sign my request and give me a certificate for this base domain.

Verification of Exploit - GitHub

I had to test further. Would this work on other sites? Was it just a fluke?

GitHub is used to store programming projects. It is a code repository. But it also gives each GitHub user their own subdomain to have a main webpage for their code. The users can do whatever they want with the page.

Thus, I naturally had full control of the subdomain schrauger.github.com and (schrauger.github.io). After verifying I owned these domains to WoSign by uploading their text file, I was allowed to generate a certificate for these two subdomains. Perfectly fine.

Then I requested that WoSign also sign the same certificate for the base domains github.com and github.io. WoSign's domain validation did not kick in. They had a bug in their validation process that assumed I controlled the base domain, despite only proving access to a specific subdomain.

I added www.github.io for good measure, and I didn't add www.github.com because I stupidly forgot to. Oops. By this point, my heart was beating and adrenaline was pumping, because I was breaking The Internet's trust in SSL at this very moment. Cut me a little slack, will ya?

WoSign signed my certificate, and lo and behold, I had a certificate that was valid for github.com, github.io, www.github.io, schrauger.github.com, and schrauger.github.io. After overriding a DNS entry (via /etc/hosts), I set up a test website on my local machine that responded to GitHub's domains. I loaded the site, saw that the location was https://github.com, and the browser said my connection was encrypted by a valid certificate signed by WoSign.

I also opened a second GitHub account and got a second certificate, but it was only valid for github.io, since the redirect from user.github.com to user.github.io was only available to users active in the system before GitHub decided to use that second domain for user pages.

Request for Help

At this point, I was in over my head. I had no idea how to report this major vulnerability. I couldn't just post to a public forum explaining all the details, because that would let everyone know how to follow the exploit before it was patched.

I also couldn't go directly to WoSign. Primarily because I don't speak Chinese, and only a few pages on their site are in English. But also secondarily because I didn't want them to brush me off, or to fix the vulnerability and keep it quiet.

I was concerned they would keep it quiet. After all, just four years prior, the certificate authority DigiNotar went bankrupt after it was shown they mis-issued a bunch of certificates. The major browsers (Chrome, Firefox, Internet Explorer) had revoked DigiNotar as a trusted CA, and all of DigiNotar's customers were forced to go to a different company to get their certificates.

Since I, quite figuratively, stumbled across the exploit, I had to assume that others had come across it as well. (Spoiler alert: I was right.)

Stack Exchange

In the end, I created a pseudonym account on Stack Exchange. I posted a question asking how to proceed. I kept the details generic, not mentioning the CA WoSign, the details of the exploit, or anything else that could be used to figure out how to recreate the vulnerability. The community responded by posting a few answers and voting for the best answer.

I contacted Dan Kaminsky, a well-known person in the field of web encryption and security. We emailed back and forth and then talked on the phone. I gave him the public key for my GitHub certificate, and he was able to contact WoSign on my behalf.

Edit 2016-09-08: Apparently, the emails were never forwarded to WoSign, so the CA never received word of the vulnerability. However, they had a secondary system in place to flag certain high-level domain names, including GitHub. Thus, they caught the GitHub certificate and revoked it soon after it was created. A few months later, they discovered the validation bug on their own and fixed it as well.

WoSign fixed the domain validation vulnerability, and they revoked my GitHub certificate. All seemed well.

Still valid?

A year went by, and I was cleaning up my server when I saw the old test site for github.com. For fun, I tried to load the site once again, and Firefox blocked it with a warning saying the certificate was revoked.

For some reason, I decided to check my test GitHub site Chrome as well. I typed in github.com, made sure that my hosts file had the redirect, and hit enter. Chrome loaded my site without a hitch.

Wait, what? Why didn't Chrome tell me my site was using a revoked certificate?

Looking back in my emails, I saw the revocation email for my GitHub certificate. It was definitely revoked. But Chrome didn't seem to be checking the revocation servers. So my certificate was still valid for any Chrome users!

But then another realization hit me. There was no email about the www.ucf.edu certificate being revoked. I looked around and found the certificate I had generated, ran a couple tests, and saw that it was in fact still valid. WoSign had not revoked it.

Which meant that WoSign didn't do any retroactive searches for other certificates to revoke. Or worse, they had done their searches and didn't catch all the certificates created by this vulnerability.

GitHub Bug Program

With both of these in mind, I contacted GitHub through their bug program. I showed them how anyone could use Chrome and load my GitHub test site without seeing any certificate errors. I also let them know about the other unrevoked certificate. With this information in hand, they contacted the security teams for Google Chrome, who then went on to contact the teams of Firefox, Internet Explorer, and Safari.

Mozilla and Chrome talk to WoSign

Suddenly, I was a part of the conversation between Google and Mozilla about WoSign. WoSign had never reported the mis-issued GitHub certificate, nor the vulnerability they had patched that I discovered (Edit 2016-09-08: The vulnerability accidentally went unreported, but Wosign later independently discovered and fixed it). All certificate authorities go through yearly audits, and they are required to report any major issues found.

There were a couple other reports about WoSign with different vulnerabilities that also went unreported. One involved validating a domain using a non-standard port that could be controlled by a non-server-admin user. Another was an API bug that allowed deprecated SHA1 certificates to be generated with backdating, which kept the certificate valid in browsers that had deprecated that type of certificate after a certain date.

WoSign Responds

WoSign joined the conversation, saying they didn't realize the vulnerabilities had to be reported. The spokesperson apologized and said they would do better in the future. He later posted a list of 33 certificates that were generated using my exploit.

Strangely, he said that WoSign would only revoke those certificates if the people who created them contacted WoSign to request revocation!!

That is essentially the same as saying the police would post Wanted posters of criminals, but only if the criminals emailed them a photo of themselves and requested the police to put their photo on a Wanted poster! No one who maliciously exploited the validation bug would ever contact the CA to request their certificate be revoked.

WoSign apparently doesn't understand that all certificates generated by the exploit should be revoked no matter what. If any were actually valid, that site owner could reapply using the now-fixed validation system to get a new certificate.

Chrome Revocation Checks

There are major problems with the current system of certificate revocation. The main issue is that someone with the ability to do a man-in-the-middle attack (which is needed in order to take advantage of a mis-issued certificate) is also able to block a browser's request to check if a certificate has been revoked or not. If the revocation check fails to respond, most browsers will default to accepting the certificate.

That means that any revoked certificate can still be used to launch a man-in-the-middle attack. The current system of checking for revoked certificates doesn't work very well under actual attacks.

CRLSet

Google decided to forgo checking for certificate revocation status. Instead, it takes a select subset of all revoked certificates and includes it with their browser updates. That way, high-profile sites can be added to their Certificate Revocation List Set (CRLSet) and be protected from a MITM attack using a revoked certificate.

Unfortunately, including every revoked certificate is neither practical nor possible. It isn't practical because browser updates would become large and waste space, and it isn't possible anyway because not all certificate authorities publish a full list of revoked certificates. Instead, you must first load a specific certificate, and then look at the details of the certificate in order to get a link for checking revocation status.

CRLSets have their problems as well, so while Google uses them primarily, Mozilla uses both instant revocation checking and OneCRL (their version of a CRLSet).

Certificate Transparency

Google, and other browser teams, have recently been pushing for more CAs to publish a certificate transparency report. This would be a list of all certificates generated by that CA for every single person. Domain owners could review these lists and keep an eye out for certificates generated for their domain that were unapproved.

WoSign is now joining the list of CAs who publish a full transparency report, partly due to this vulnerability coming to light.

Conclusion

Domain validation is hard. It isn't as simple as one may think, and WoSign isn't the first to have a problem. They are still a trusted CA for now, and hopefully they will get their act together quickly.

Having competition is good; only just recently did Lets Encrypt join as a free SSL provider alongside StartSSL and WoSign. If WoSign doesn't stay, we'll be back to two providers.

Even still, some people have decided to manually revoke WoSign as a trusted CA in their browsers. Which means they'll be unable to view my website, as it still uses a WoSign certificate (that may be changing very soon).

The story is still ongoing. You can follow the discussion at the Mozilla Security Policy group.