Not even that. An attacker with local root can just extract the wireguard keys from process memory, or use the TPM to decrypt the state file like Tailscale would.
The only scenario where it helps is a local attacker who can read the state file on disk, but is not full root. Kinda unlikely on Linux, but could happen on Windows.
Windows uses TPM for Bitlocker. A very common scenario where TPMs get reset is BIOS updates (when a TPM is implemented in firmware).
AFAIK, Windows cheats here because it also manages BIOS updates. When an update happens, it takes extra steps to preserve the Bitlocker encryption key in plaintext, and re-seals it to the TPM after the update completes.
Apart from Windows, there are many setups that fail in fun ways: Kubernetes pods that migrate from one VM with a TPM to another one, hypervisors that mount a virtual TPM to VMs, containers or VM images that do Tailscale registration on one machine and then get replicated to others, etc.
Tailscale already did some attempts at cleverness when deciding whether to enable features using a TPM (e.g. probing for TPM health/version on startup, disabling node state encryption on Kubernetes pods), but there was still a long tail of edge cases.
Actually, this is not the case. BitLocker wraps the key, meaning even if the TPM were compromised, one would still have to brute-force the PIN for the actual key. It’s cryptsetup on Linux that stores the key on the TPM in plaintext. This vulnerability has been known for quite a while and nothing has been done about it so far.
Slightly off-topic: it also cheats in how TPM works for Bitlocker when you do TPM + PIN. One would assume PIN becomes part of the encryption key, but in reality, it's just used as the auth for TPM to release the key. So while it sounds like a two-factor solution, in reality it's just single factor.
So the Bitlocker without TPM is actually a better idea and Windows makes it very painful to do if TPM is on.
I don’t know much about the TPM but if it’s anything like Apple’s Secure Enclave, it should require exponentially longer time after each incorrect PIN past the first one, making it so you can’t reasonably brute force it without getting lucky.
I’m not sure how the typical “two factor” best practices would interpret one of the factors basically self destructing after 10 guesses, but IMO it’s a pretty decent system if done right.
That's not the issue. The TPM isn't blinded in the above description meaning that if someone cracks the TPM they can get your key. Ideally both factors are always required to access the secret.
If you're wondering, yes this is a security issue in practice. There have been TPM vulnerabilities in the past that enabled exfiltration of secrets.
Aren't PINs usually short, and might even be really be made out of just digits in the first place? So would there be real security benefits in adding that to the key?
You can make PINs as complex as you want, there's only a maximum length limitation of 20 characters. There's no difference between passwords and PINs in Windows except that Windows calls it a PIN if it's used with TPM. And yes, it does nudge you in the direction of making it simple because "TPM guarantees security", but you don't have to.
> Windows cheats here because it also manages BIOS updates
Is this (relatively) new?
I don't use TPM and I rarely update BIOS unless I really need to, but I thought there was an option on my BIOS/UEFI to use USB drive to update it. How would Windows know about it?
Window can get BIOS updates through windows update, if the OEM participates and packages them. I haven't seen BIOS updates through windows update on my systems where I built it from components, I've only seen it on integrated systems from major builders (HP, Lenovo, etc).
The BIOS update instructions for my retail packaged motherboard indicate to turn off BitLocker before doing upgrades to prevent loss of TPM turning into a loss of access, but it'd be easier if it were automated.
You can update with a USB drive, but if you have bitlocker enabled and don't temporarily disable it before the BIOS update, you'll need to reformat and reinstall Windows.
I believe you can also get it from your online Microsoft account if that's what you logged in with once. I ran into this a while ago and had to do it that way. I didn't even know I'd set up Bitlocker.
I'm one of the Tailscale engineers who built node state encryption initially (@awly on Github), and who made the call to turn it off by default in 1.92.5.
TPMs are a great tool for organizations that have good control of their devices. But the very heterogeneous fleet of devices that Tailscale users have is very difficult to support out of the box. So for now we leave it to security-conscious users and admins to enable, while avoiding unexpected breakage for the broader user base.
We should've provided more of this context in the changelog, apologies!
Those issues are a surprising read. I would expect issues with TPM on old or niche devices, but not Dell XPS laptops, or a variety of VMs. But I guess I'm not entirely sure how my vms handle TPM state, or if they even can.
I'm running nearly all of my personal tailscale instances in containers and VMs. Looking now at the dashboard, it appears this feature really only encrypted things on my primary linux and windows pc, my iphone, and my main linux server's host. None of the VMs+containers i use were able to take advantage of this, nor was my laptop. Although my laptop might be too old.
Stuff breaks all the time, you just need a bigger sample size.
Overseeing IT admins for corp fleets is part of my gig, and from my experience, we get malfunctioning TPMs on anything consumer - Lenovo, Dell, HP, whatever. I think the incidence is some fraction of a percent, but get a few thousand devices and the chance of eventually experiencing it is high, very high. I can't imagine a vTPM being perfect either, since there isn't a hypervisor out there someone hasn't screwed up a VM on.
Many, many more devices here... And good/typical enterprise level hardware... And failing TPMs are just something that happen. It's pretty expected these days. And on Windows when it causes a loss of certificates, it's actually a good bit more of a pain than just a dying disk or display or something, because it's not immediately obvious what's wrong, it just doesn't talk to the network properly anymore, or so.
I'm not surprised by Tailscale's change here. It's a good move.
The issue could be a bug in the host OS not in the VM. I had a Windows update that broke VMs when the guest OS was Windows running in real-time mode. This was the only issue and if I didn't run real-time VMs I would have never known. The only resolution was to reinstall Windows.
Just had a system board replaced on a device in my org, Dell laptop.
As part of setting up a device in our org we enroll our device in Intune (Microsoft's cloud-based device management tool aka UEM / RMM / MDM / etc). To enroll your device you take a "hardware hash" which's basically TPM attestation and some additional spices and upload it to their admin portal.
After the system board replacement we got errors that the device is in another orgs tenant. This is not unusual (you open a ticket with MS and they typically fix it for you), and really isn't to blame on Dell per se. Why ewaste equipment you can refurbish?
Just adding 5c to the anecdata out there re: TPM as an imperfect solution.
When I replaced a motherboard (rest of the hw was OK) Microsoft was of the opinion I had a 'new computer' and would need to buy a new Windows 10 license (of IIRC 150 EUR → scoundrels). I went to G2A and bought one for 20 EUR. Then it hit me. This occurred before when my previous motherboard/CPU was broken, and back then I actually called Microsoft where they insisted on selling me a new license. I did exactly the same back then.
I've handled technical+legal concerns for licensing for a very small org in a different lifetime, and yes, that's exactly how Microsoft used to think of licenses. I don't know how it works these days, it's someone else's problem.
We had to archive invoices+servicing documentation for warrantied mobos from the supplier to keep a legal licensing chain.
I remember the path my license had: it was a free upgrade to Windows 10, from Windows 7 (right before they removed said free upgrade; I tend to be slow with adapter Windows versions). The original Windows 7 license was a pirated one, but that didn't matter (we know why: before GDPR, Microsoft could spy on Windows 10 users, and the pirated Windows 7 was already a lost sale).
Apparently the free upgrade was OEM, bound to the hardware. I did not know. Either way, I'm from Europe (EU), and here a software license cannot be exhausted via second hand market, so it stands to reason I can buy one second hand. That this isn't what Microsoft support is told to discuss, suuure (even when I explicitly asked for it, they insisted I had to buy it via them).
I've had quite the opposite experience with Microsoft.
One time their support just give me a licence for a newer version of Windows - I've replaced the HDD/SSD, cloned/copied it and it was not activated. I contacted their chat support from that laptop and when they asked me for licence on the sticker I mentioned I'll have to come back in 5 minutes since I'll have to turn off laptop, and take out battery to see the MS sticker/hologram.
Support said "No worries, here's a new activation key".
Can't recall if it was from XP to Win 7, or Win 7 to 10.
--
And after buying 2 or 3 licences from another website just like G2A (Win 10 was ~€10 on Instant-Gaming) - a bunch of new computers (even brand new assembled desktops) were automatically activated.
My eyes have opened up to the pitfalls of TPM recently while upgrading CPUs and BIOS/UEFI versions on various hardware in my home.
VMs typically do not use TPMs, so it is not surprising that the feature was not being used there. One common exception is VMware, which can provide the host's TPM to the VM for a better Windows 11 experience. One caveat is this doesn't work on most Ryzen systems because they implement a CPU-based fTPM that VMware does not accept.
It is in fact surprising that TPMs can be wiped so easily. It makes them almost useless compared to dedicated solutions like physical FIDO keys or smartcards, and does not bode well for hardware-backed Passkeys that would also be inherently reliant on TPM storage.
Not all TPM. I've yet to manage it on my MBP M1 Pro or my Pixel. Of course, M1-M3 have broken secure enclave which cannot be fixed by the user.
On AMD with fTPM I get a fat warning if I want to reset the fTPM keys. I think earlier implementations failed here.
> and does not bode well for hardware-backed Passkeys that would also be inherently reliant on TPM storage.
So you revoke the key and auth in another way (or you use a backup). One passkey is never meant to be the one sole way of auth.
I actually like the concept. Consider a situation where you would log into your webmail while in a café or bus. If the password is tied to your hardware, nobody can watch over your shoulder to use it on theirs.
I don't use them much (I've been forced to) because I already use a self-hosted password manager where I never see the password myself. But for the average person, passkeys are better.
Now, if you compare with FIDO2, those are supposed to be with you all the time (something you have). So they can be used on multiple platforms, while a TPM is tied to hardware.
You can DoS many physical FIDO tokens by using the wrong PIN on purpose several times.
They're programmed to lock or reset as a security measure. If they're locked, they need a special process, software and credentials to unlock them, which you might not have immediate, or any, access to.
If they reset, it's no different than wiping a TPM.
I had a Ryzen 3900x on a gigabyte motherboard and the fTPM was just totally unreliable for a pretty mainstream combination. Not fully sure which was to blame there.
At least it was fixed in the 5900x (and _different_ gigabyte motherboard, but from the same lineup) that replaced it.
I'm not sure what makes any of this "surprising". Each ticket reads like "we replaced the computer that tailscale was on, it doesn't work anymore" pikachu face.
Yeah, that was a feature and the exact reason why we use TPMs. I guess it should have been better advertised.
VMs don't have TPMs as they are hw devices, although you can run a software TPM (potentially backed by the host TPM) and pass it to them, which you might want to do for this use case.
Hardware key attestation is a yet-unfinished feature that we're building. The idea is to generate a signing key inside of the TPM and use it to send signatures to our control plane and other nodes, proving that it's the same node still. (The difference from node state encryption is that an attacker can still steal the node credentials from memory while they are decrypted at runtime).
We started by always generating hardware attestation keys on first start or loading them from the TPM if they were already generated (which seemed safe enough to do by default). That loading part was causing startup failures in some cases.
To be honest, I didn't get to the bottom of all the reports in that github issue, but this is likely why for some users setting `--encrypt-state=false` didn't help.
It was designed mostly for mechanisms where in the event of certain changes (BIOS upgrades, certain other firmware changes, some OS changes) there is a fallback mechanism to unlock the system and reset the key. This is why Windows BitLocker is so insistent about you saving your key somewhere else - if you do a BIOS update and it can’t decrypt, it’ll require your copy of the key and then reset the TPM-encrypted copy with the new BIOS accounted for.
A TPM’s primary function works by hashing things during the boot process, and then telling the TPM to only allow a certain operation if hashes X & Z don’t change. Depending on how the OS/software uses it, a whole host of things that go into that hash can change: BIOS updates being a common one. A hostile BIOS update can compromise the boot process, so some systems will not permit automatic decryption of the boot drive (or similar things) until the user can confirm that they have the key.
Thank you for your openness here - and yes, it would be nice to see this kind of reasoning in the changelog, even if it's tucked a little out of the way! Those of us who care will read it.
Also very welcome is to separate it into a small blogpost providing details, if the situation warrants a longer, more detailed format.
Your suspicion is correct. I have an AMD AM5 motherboard, and everytime I update it's BIOS it warns me that the fTPM will be reset, and I know it does so because afterwards Bitlocker prompts me to introduce the recovery key since it can't unlock the drive anymore.
> There's also tailscaled-on-macOS, but it won't have a TPM or Keychain bindings anyway.
Do you mean that on macOS, tailscaled does not and has never leveraged equivalent hardware-attestation functionality from the SEP? (Assuming such functionality is available)
The third one is just the open-source tailscaled binary that you have to compile yourself, and it doesn't talk to the Keychain. It stores a plaintext file on disk like the Linux variant without state encryption. Unlike the GUI variants, this one is not a Swift program that can easily talk to the Keychain API.
In fact, SecurityFramework doesn’t have a real Swift/Obj-C API. The relevant functions are all direct bindings to C ABIs (just with wrappers around the CoreFoundation types).
> The third one is just the open-source tailscaled binary that you have to compile yourself, and it doesn't talk to the Keychain.
I use this one (via nix-darwin) because it has the nice property of starting as a systemwide daemon outside of any user context, which in turn means that it has no (user) keychain to access (there are some conundrums between accessing such keychains and "GUI" i.e user login being needed, irrespective of C vs Swift or whatever).
Maybe it _could_ store things in the system keychain? But I'm not entirely sure what the gain would be when the intent is to have tailscale access through fully unattended reboots.
Good to know, my understanding of the macOS system APIs is fairly limited.
I'm sure it's doable, with some elbow grease and CGO. We just haven't prioritized that variant of the client due to relatively low usage.
A BIOS update to my PC reset the TPM only this week. I did get a warning that Bitlocker keys would be wiped as a result before acting at least.
(I believe this was because it was fixing an AMD TPM exploit - presumably updating the TPM code wipes the TPM storage either deliberately or as an inevitable side effect.)
TPMs are basically storing the hashes of various pieces of software, then deterministically generating a key from those. Since the BIOS software changed, that hash changed, and the key it generates is completely new.
If someone had messed with your BIOS maliciously, that's desirable. Unfortunately you messing with your BIOS intentionally also makes the original key pretty much unrecoverable.
IIUC, it's a bit more nuanced: TPM stores hashes of various things like firmware in PCRs, and when creating keys in the TPM you can optionally bind the key to specific PCR values.
But you also don't have to (and Tailscale doesn't), in which case keys survive firmware updates for example.
Coincidentally this was a feature unknown to me until I performed a SSD migration from one server to another and Tailscale failed to connect because ("of course!" in hindsight) it failed to decrypt whatever.
So not a TPM failure but certainly a gotcha! moment; luckily I had a fallback method to connect to the machine, otherwise in the particular situation I was in I would have been very sorry.
The "whoever needs this will enable it" + support angle makes total sense.
IPsec or TLS-based overlays which use AES encryption and NIST-approved ECC curves or (gasp) RSA for key exchange and authentication. They generally suck in comparison with wireguard, which is a clean-sheet modern cryptographic protocol.
The tailscale client generates WireGuard key pairs, but only sends public keys to the control plane. The private keys remain on the device only.
With only the public keys, tailscale control plane cannot snoop on your traffic.
Want to donate to help Ukraine but there are too many options?
A group of volunteers (including myself) put together this aggregator site with structured organization info and some filtering options.
You can skim through ~100 orgs quickly and narrow down your list of options.
We also made sure to only put trustworthy orgs there, manually researching each one.
So, the "something you have" element of security is to primarily avoid remote compromise. And even make local compromise require an additional theft step.
If the hacker gets your key, and your password it's game over.
But hacking the password will take some time hopefully, and systems usually have retry limits etc. so if you discover your lost key, you hopefully have some time to revoke the lost key.
I personally would not use it for password-less login, as it is only good as a second factor.
If your threat model includes any real likelihood of people capable of stealing your keys and cracking your passwords, then 2FA is only a small part of the opsec you need.
Yubikey still acts as a second factor, so for the average person's threat model this should be fine provided you use strong, unique passwords for your accounts (i.e. a password manager). 1Password supports U2F as a second factor to access your account, but you still need to know your secret key and master password; you cannot decrypt your vault with only a hardware token.
PKCS#11 is a C API. It does not describe the wire format for talking to the actual hardware.
To use PKCS#11 for a particular device, you need a module (shared library) to translate between the C API and the actual hardware. This module is usually vendor-specific.
If I develop software with PKCS#11 support, I'm basically asking every user to find a PKCS#11 module from their device vendor and install it in the right place.
The wire format is standardized: ISO 7816. Even U2F uses ISO 7816.
This issue with existing smart card technology is not lack of standardization, it's too much standardization--too much flexibility and stacks that are too deep.
Vendors ship their own PKCS#11 drivers as a convenience. But PKCS#11 isn't the only high-level API. The other is PC/SC, which is actually simpler than PKCS#11, though it often requires more local support from the OS. But not necessarily. You can write PC/SC shims that talk directly to hardware, or even to Vault servers if you want, w/o OS support. I have my own rapid driver framework that supports all of these. For example, I have a PKCS#11 and PC/SC client driver which can use the Apple T1 chip to authenticate to a Vault server for remote signing using Transit keys--the only engine that supports ad hoc remote key operations. This permits sharing GnuPG (via PC/SC) and OpenSSH (via PKCS#11) keys between users, without actually disclosing the keys, though Vault actually makes it difficult to do this securely as you need to write ACLs to prevent transit keys from being exportable.
BTW, you don't need special drivers to use Yubikeys, either. They just provide them as a convenience because the FOSS ecosystem is confusing and... non-optimal.
I'm hoping to release a macOS product soon and as part that may release some of my framework as FOSS.
That level of standardization is a feature, not a bug. PKCS#11 lets you use any compliant hardware device with any compliant software package, as long as they both implement the spec. Compliant software packages include: ssh, Java's keytool, the GnuTLS utilities, the openssl utilities, wpa_supplicant, various web browsers, and VPN clients. Nowadays many popular Linux distributions come with p11kit configured out of the box, which lets openssl/GnuTLS autoselect the correct PKCS#11 shared library based on the matching information in the PKCS#11 URI.
While the low level API is complex and the UI often isn't ideal, PKCS#11 has been a godsend for interoperability because it abstracts out the low level hardware interfaces and other implementation details. It lets your application seamlessly access hardware-backed keys whether the keystore is sitting on USB (Yubikey), ISO7816 (smartcard), I2C (TPM), or something else. On the application side, adding PKCS#11 support only takes about a dozen lines of code, after which the app can use hardware backed keys/certs to perform TLS negotiations.
There are a few differences between an OpenSSH jump host and Teleport:
- you have to actively manage authorized_keys for every person using openssh; Teleport manages a PKI and can be backed by your existing SSO
- it hard to restrict any given user to a subset of hosts (e.g. only allow select few to access prod database); Teleport has RBAC
- hosts with Teleport also get SSH certificates, so you don't need to trust-on-first-use (which everyone has been conditioned to ignore)
Yes, but it's rarely how companies use OpenSSH because it takes a ton of work to set up.
Teleport gives you all this functionality by default out of the box. So it's not inventing anything new, just provides better UX.
Maintaining a CA (and dealing with cert rotation) is some work.
Other things are indeed just a flag or config option (like jumphosts). But it takes work for a sysadmin/devops to educate all engineers in the company and make sure everyone uses the correct setup and doesn't end up dropping authorized_keys around random servers.
It's not that difficult technically as it is socially.
Yeah, you are right about this. Just thought about a personal setup, but getting it all up and running in enterprise scape will be more work than just a few flags
The only scenario where it helps is a local attacker who can read the state file on disk, but is not full root. Kinda unlikely on Linux, but could happen on Windows.