SQL Server 2005 introduced native database encryption with the use of encryption functions. This allowed users to secure data within the database so that even admnistrators may be prevented from viewing the data. While that is a great addition, it was not a “flip the switch” implementation. Applications had to be modified to enable this feature (although it was pretty easy to do) and you had to deal with potential performance issues. The latter is the bigger challenge as you cannot index an encrypted column so you effectively end up scanning the table with an encrypted column.

This isn’t very appealing to users who truly need the encryption capability as the column they need to encrypt is often the column which they search by. A common example is a credit card company where the credit card number is the column they want to encrypt but that is also the  column they would search when customers access their accounts or when employees need to lookup a customer. When you have hundreds of millions of accounts and need to do thousands of lookups every minute, it’s a bit of a problem.

Katmai takes this a (big) step further with Transparent Data Encryption. The concept isn’t new and at least one other major database has it. Basically, think of having a layer between the storage  engine and your disks that will encrypt data before it is written to disk and decrypt it before being read into memory. Turn it on is a lot simpler than Yukon’s encryption too. You just need to create a database encryption key and set the database encryption option on. Note that your database encryption key needs to be protected by a certificate that is protected by the database master key. This ensures it can be accessed after a restart or recovery.

Unlike Yukon’s table encryption, you don’t have the same challenges with indexes like table level encryption in Yukon. The following is a simple example for encrypting your the pubs database and a query whose plan will show an Index Seek. Not very exciting in itself but considering the past encryption technique where indexes don’t even come into play, this is pretty cool.

USE master;
GO
CREATE MASTER KEY ENCRYPTION BY PASSWORD = ‘p@55w0rd’;
GO

CREATE CERTIFICATE DBEncryptKeyCert  WITH SUBJECT = ‘DB_Encryption_Key_Cert’;
GO

USE pubs
GO
– Note that not all operating systems support AES so choose an algorithm that is supported by your OS
– SQL Server supports most common algorithm like AES, DES, 3DES, RC4, etc… See Books Online for details.

CREATE DATABASE ENCRYPTION KEY
WITH ALGORITHM = AES_256
ENCRYPTION BY SERVER CERTIFICATE DBEncryptKeyCert;
GO

ALTER DATABASE pubs
SET ENCRYPTION ON;
GO

USE pubs
GO
SELECT  fname, lname, emp_id
FROM  employee
WHERE  emp_id=’A-C71970F’;
GO

Just pretend emp_id is credit card number. If you had encrypted the column emp_id (i.e. credit card number) and created the encrypt/decrypt function along with the view, triggers, etc… you’d be scanning the table just to retrieve that one row. That’s what you had to work with in 2005 (unless you had some 3rd party solution). Sure, there are several ways to reduce/avoid that but solutions typically involve some variant of storing a substring of the column (hashed or as-is) and indexing that column. Not a very viable solution for credit cards since the first 6 digits make up the credit card issuer’s identification number. A bunch of other concerns pop up and though not insurmountable technically, they do present challenges and in some cases, regulations may disallow this approach. No such concern with TDE in 2008.

So what about EFS? Well, just consider it as another option to be used as appropriate. EFS encrypts the entire file system while TDE encrypts just the database. Picking the right option depends on your transportability requirements and concerns. For example, if your company uses a centralized repository for storing ALL backups and you don’t trust those pesky Windows admins that own that repository, TDE ensures nobody can copy the backup file and restore on another SQL Server instance. It is also possible to use both but be aware of the performance and administration implications especially during recovery (EFS access then database access).

Upgrade Effort & Impact
What’s the effort to benefit from this? Well, from an execution perspective, just the few steps in the sample scripts above or you can use Management Studio. If you are upgrading a SQL Server 2005 database that is already using column level encryption, suggest you revert to plain text before the upgrade. Though they are different features, there really isn’t a need for using both. Same with those who are using 3rd party encryption products. That said, there are a few challenges in the upgrade process. For instance, you might not be allowed to store data in clear text to facilitate a simple upgrade. As such, you will need to upgrade first then reverse the process which you employed to encrypt a column using the function based approach. That means you’ll turn on TDE on the database then move the data from the encrypted column to a non-encrypted column so you can do away with the UDFs (and views if you had those). Bottom line is, this may mean a longer upgrade/migrate time window plus a longer rollback time also.

There are also other considerations to your production server. This is a database wide option and will impact other components such as the transaction log and tempdb. To ensure data is not stored unencrypted once you turn it on, the current virtual log file will be “zeroed out” and new transactions will begin on the new VLF in encrypted form. Your tempDB will also be encrypted. This means you will have performance overhead on the system initially (both CPU and disk) and alters user behaviour later (e.g. can’t snoop around in tempdb easily anymore). Another important consideration is backup. With TDE, your backups are also encrypted so make sure you backup your keys/certificates also. Else, in a disaster recovery situation, you will not be able to restore your database. These and other considerations are well documented in BOL so do review in detail.

From a planning perspective, think real hard about why you need to encrypt your database. If the only reason you can come up with is “to make it more secure”, talk to your friendly neighbourhood Microsoft engineer or trusted partner before turning this on. You might eventually still employ TDE or you may not even need it (but you’ll likely discover other security concerns). Remember, security is not just about technology. If you don’t educate your staff and users or fail to practice secure processes diligently, no amount of encryption will save your data. Flipping over keyboards and turning photo frames around on users desks STILL reveal passwords (last verified Sept 2007) and backup tapes can still be found on DBAs/operators’ desks or at the reception awaiting picking. TDE is just another layer of defense in your arsenal for specific intents and attack vectors; don’t forget the others.
 

joe yong.