Turktrust, the Turkish certificate authority (CA) responsible for issuing an intermediate CA certificate that was later used to generate an unauthorized certificate for google.com, claims that the bad Google certificate was not used for dishonest purposes.
On Thursday, Google, Mozilla and Microsoft announced that they are blacklisting two intermediate CA certificates that were mistakenly issued by Turktrust in August 2011 for two of its customers who should have received normal end-entity certificates for their respective domain names.
An intermediate or subordinate CA (sub-CA) certificate inherits the trust given by browsers to the issuing CA and allows its owner to sign and create trusted SSL certificates for virtually any domain name on the Internet.
Turktrust's error was discovered after one of the mis-issued sub-CA certificates was used to sign a wildcard certificate for *.google.com without authorization from Google, the domain name owner. The rogue certificate was used on Dec. 24, 2012, in an attempt to inspect the encrypted communications of a Google Chrome user using a man-in-the-middle technique that consisted of re-encrypting the user's traffic en-route using the fraudulent certificate.
Because the Google Chrome browser contains a hardcoded list of legitimate Google certificates -- a feature called certificate pinning -- it is able to detect the use of unauthorized certificates and report the incidents back to Google.
In a statement published on its website, Turktrust explained that the mis-issuing of the two intermediate CA certificates was the result of an undetected certificate profile migration error between the company's testing and production systems. As a result, the production system used for certificate issuing ended up with two certificate profiles that contained CA extensions and were intended to be used only for testing purposes.
"The wrong profiles were only used on August 8, 2011 to issue those two faulty certificates which were certainly not intended to be sub-CA certificates," the company said.
The presence of the bad certificate profiles on the production system was detected a couple of days later and the error was corrected. However, the company failed to notice that two certificates with sub-CA status had been issued in the meantime.
According to Turktrust, the production system went through a successful audit by professional services company KPMG in November 2011 and was found compliant with the policy requirements for certification authorities that issue public key certificates, ETSI technical specification 102 042.
The company also claims to have implemented two additional verification procedures to prevent similar situations from happening in the future, one for checking various certificate characteristics during the certificate issuing process and one for checking already issued certificates before they are sent to customers.
One of the mis-issued sub-CA certificates had been revoked at the customer's request before being used, Turktrust said. The other was used on a Microsoft IIS (Internet Information Services) server that operated as a Web mail server for over a year, it said.
However, on Dec. 6 someone installed the Web mail sub-CA certificate and its corresponding private key in a firewall appliance manufactured by Check Point that was configured to run as a man-in-the-middle proxy, Turktrust said. That same day, the firewall used it to generate a fraudulent certificate for *.google.com.
"It appears that the firewall automatically generates MITM certificates once a CA cert is installed," Turktrust said.
The CA did not name the customers that received the two intermediate CA certificates, but according to a Microsoft security advisory published Thursday, they were issued for e-islem.kktcmerkezbankasi.org, a domain that belongs to the Central Bank of the Turkish Republic of Northern Cyprus and *.EGO.GOV.TR, the domain of the EGO General Directorate, an agency of the Municipality of Ankara that provides public services related to electricity, gas and transportation.
The unauthorized *.google.com certificate appears to have been issued using the *.EGO.GOV.TR sub-CA certificate.
According to documentation published on Check Point's website, some of its gateway security products do have HTTPS inspection capabilities. By default this feature uses a self-signed CA certificate that needs to be deployed on the network computers before it can be used to inspect HTTPS traffic without triggering certificate warnings in browsers.
However, the feature also allows customers to import their own CA certificate, which is what happened with the *.EGO.GOV.TR sub-CA certificate.
"The available data strongly suggests that the *.google.com cert was not issued for dishonest purposes or has not been used for such a purpose," Turktrust said. The company also stressed that there is no evidence of a security breach on its systems.
"I don't have enough experience with Checkpoint firewalls, but after looking at the details, this seems like a plausible scenario," Robert Graham, the CEO of security firm Errata Security, said Thursday in a blog post. "It's quite possible that the MitM was essentially accidental."
Other people are not that convinced that this was an accident. "Why would a certificate intended for end-entity SSL use (albeit actually enabled for CA use) be installed on a 'firewall'? What was the system administrator's intent?," Stephen Schultze, the associate director of the Center for Information Technology Policy at Princeton University, asked late Thursday on a Mozilla mailing list where the incident is being discussed.
"On what network was this Checkpoint device installed, and what set of users were being MITM'ed? Specific IP [Internet Protocol] blocks would be helpful," Schultze said in response to a message posted on the list by Mert Ozarar, project manager at Turktrust.
"We are certainly not in position to answer these questions to full extent," Turktrust said in a response posted Friday on the same mailing list. "However, it is almost apparent that the agency has wanted to configure the firewall as a MITM proxy for their internal users. Our thorough OCSP [Online Certificate Status Protocol] analysis has also supported this in the sense that almost 96% of OCSP requests stemmed from the EGO domain."
The situation unfolded when EGO changed their firewall vendor and device model, Turktrust said. "Odds are too low that there is a malicious intent on their side. There is also no evidence at all of any other malicious use of the faulty cert."
In response to questions sent via email on Friday, a Turktrust spokesman advised this reporter to follow the discussion on the Mozilla mailing list and ask any questions there so they can be seen by the community.
Some people have called for browser vendors to revoke their trust in Turktrust's root CA certificates, a harsh punishment that was taken before in the case of former Dutch certificate authority DigiNotar that filed for bankruptcy after its own root CA certificates were removed from browsers as a result of a serious security breach on its systems.
Mozilla removed a newer Turktrust root CA certificate that had been added in the beta version of Firefox 18 and Google also said that it will update Chrome in January to no longer display Extended Validation (EV) status for certificates issued by Turktrust. However, the main Turktrust root CA certificates, including the one that was used to sign the bad sub-CA certificates, have not been removed from either browser. Both companies said that they might decide to take additional actions after further discussion.
Microsoft did not express any intention to completely remove the Turktrust root CA certificates from the Windows trusted certificate store either.
Security experts have long warned that having hundreds of certificate authorities trusted by default in browsers poses a big security risk, because if any of them is compromised it can impact the whole ecosystem. This happened when the CA systems of DigiNotar and a Comodo reseller were compromised and the attackers managed to issue hundreds of rogue certificates for popular domain names. In the case of DigiNotar, the certificates were even used to spy on a large number of users in Iran.
The existence of subordinate CAs poses even bigger risks, because they are usually not held up to the same standards as the CAs they inherit their trust from. Some CAs don't even disclose the sub-CA certificates they issue.
"I have spent the past couple of years trying to convince Mozilla to strengthen their requirements so that CAs must disclose all subordinate certificates that they have issued, so that researchers can better map the risk landscape and so that users can make more informed trust decisions and detect unexpected subordinates," Schultze said in a blog post. "They're getting closer, but it is taking an awfully long time."
In February 2012, certificate authority Trustwave revealed that it had issued a sub-CA certificate that enabled an unnamed private company to spy on SSL-protected connections within its corporate network. Trustwave defended itself by saying that this is a common practice in the industry.
"So once again we go through the process of revoking these [sub-CA] certificates and deciding how much future trust to put in Turktrust," Chester Wisniewski, a senior security advisor at antivirus vendor Sophos, said Friday in a blog post. "It is really time we move on from this 20-year-old, poorly implemented system. Whether it is the Public Key Pinning Extension for HTTP, Convergence, Trusted Assertions for Certificate Keys (TACK) or DNSSEC-TLS [technologies proposed to fix or replace the CA-based model] we've got to pick something and start implementing it."