Microsoft SQL Server 2008 continues the trend of improving its BI capabilities. Every major BI component (including the relational engine) has been enhanced in terms of both performance (scalability) and capabilities (features). Two other items I want to mention quickly (neither of which are in the corresponding article) is that of Change Data Capture (CDC) and Resource Governor (RG). CDC is a great feature for enabling your SQL Server 2008 based OLTP systems (yes, you would have to upgrade your OLTP servers) to expose their changed data asynchronously for ETL consuming purposes! In addition, RG can be used to prioritize OTLP workloads including that of ETL processes, this makes it easier to ensure that BI requirements do not adversely affect the more important OLTP-based applications (business transactional systems). So the big question still remains…is SQL Server 2008 BI worth the cost (and effort) of an upgrade? Unfortunately, like most difficult questions in life the answer is that it depends upon your unique business intelligence environment and requirements!

Cheers

Derek Comingore

This fairly recent Microsoft purchase of the Dundas source code for integration into SSRS 2008 is a great move as the Dundas suite of SSRS add-ons have become the premier choice for such advanced visualization needs. By including the Dundas technologies into SSRS 2008, Reporting Services customers will not only gain access to a much improved Report Server Architecture (without requiring IIS) but also an enhanced Visualization platform. There are literally about three times as many chart types in SSRS 2008 as compared to SSRS 2005. Some of the brand new chart types include the Funnel, Range, Pyramid, and Polar. In addition to the added chart types customers will also gain access to the Dundas Gauge capabilities via a new Gauge Data Region. As of the latest SQL Server 2008 CTP there are two main gauge types: Radial and Linear. Finally, we get a few other ‘goodies’ with the inclusion of the Dundas suite including:

• Secondary Axes
• Runtime Calculated Series
• WYSIWYG Chart Editor (design-time)

Below are some sample screenshots of the new SSRS 2008 visualization capabilities…

Cheers

Derek Comingore

Managed to find some time for an extended lunch so I figured I’d come back and share some other learnings - an easy one this time: backup compression. This is not the same as data compression so don’t confuse the two. I’ll blog how to on the latter later when I find some time. That’s pretty cool too.

I’ve worked with a lot of customers who use 3rd party backup software primarily for the compression capabilities followed by improved speed, especially during restore. With one of the more popular 3rd party backup solutions, I’ve restored a 1.2 TB database in under 30 minutes on a beefy HP Integrity rx8640 server with very a large EMC Dx3 SAN so there was no IO or CPU bottlenecks at any point. No official hard data on Katmai’s backup compression or performance right now but I know the folks in building 35 will be working hard to at least match the popular 3rd party solutions.

From some of the tests we’ve done both internally and with a couple of customers with tens and hundreds of GB databases, both compression ratio and performance seem to map closely to the popular 3rd party software we used. At least for default compressions on the 3rd party software. You can change the level of compression with some of these 3rd parties but SQL Server 2008 only has 1 level so I didn’t bother testing against different settings. I’d imagine the delta would not be significant percentage wise but if you have tens of Terabytes to deal with, even 5% can be a noteworthy difference.

How it’s done

Taking advantage of backup compression once you’ve upgraded to Katmai is just a matter of adding an extra option (in bold below) in your backup scripts 

– My Backup Job with compression 

BACKUP DATABASE MyBigDatabase
TO  DISK = ‘N:\Tier1Backups\MyBigDatabase.bak’
WITH
NOFORMAT,
NOINIT,
NAME = ‘Full DB Backup - MyBigDatabase’,
SKIP,
NOREWIND,
NOUNLOAD,
COMPRESSION;
GO

or selecting the compression option if you are using Management Studio GUI.  Note that you can change the global setting at the instance level with

SP_CONFIGURE ‘backup compression default’, 1;

so that all backup commands will always compress by default. This can be changed at the command level also when you execute your backup so you can backup with compression for specific databases only and not for everything in the instance. This is too easy.

Note that the default setting is ‘no compression’ so your DBA team does not get alarmed by the sudden reduction in the backup file sizes; just in case not everyone is keeping close tabs Katmai enhancements. Last thing you want is the graveyard shift operator re-running backup all night and finally paging you at 3am because “the backup file looks too small so it can’t be good…”. Don’t laugh (unless it’s someone else’s turn to be on-call).

Sample results

Simple tests with the AdventureWorks OLTP database yielded the following:

Path	Backup time	Restore time	File size
Original database	n/a	n/a	174,080 MB
NO COMPRESSION backup	23s	23s	171,093 MB
WITH COMPRESSION backup	11s	14s	40,097 MB

This was done on a VPC with 1.5GB of dedicated RAM and its own dedicated 7200rpm EIDE disk (8MB cache) on a USB 2 interface. The host machine is a notebook with an Intel Core 2 Duo 2.2GHz CPU (T7500), 4GB RAM and a 7200rpm SATA disk. The times are based on averages from 3 runs - all were with a couple of seconds. Other tests with real customer databases ranging from 5GB to 800GB in size yielded similar levels of compression and percentage elapsed time reduction. Sorry, can’t publish any of the customer data but those tests were run on x64 workstations with local SATA drives and IA64 servers with enterprise class SANs.

Not bad at all. Note the benefits from a performance perspective since you have much less to send to disk or retrieve when restoring. There was some impact on CPU utilization but it’s hard to say what the real impact would be when doing hot backups in a production environment since I don’t have other workloads running. You can expect some CPU load but the level will really depend on your database. Looks like it’ll be low to moderate. So long as you’re not already facing CPU pressure in your current system, I expect this to work fine but don’t take my word for it; test (no really, test). This is important because while few environments do full DB backups during peak hours, many companies have other system jobs or reports run during maintenance windows so make sure you don’t saturate the CPU. All that aside, the numbers are very decent especially since it’s still in beta.

Of course, with any compression technology, database or file system, your mileage can vary significantly. Character data types typically compress very well while binary types not as well (or very poorly in some cases). However, most traditional OLTP systems (e.g. accounting, sales, HR, etc…) will probably benefit from this since the bulk of the data is of character types.I expect this will be a widely used feature both to improve backup/restore performance and to save disk space (enterprise storage is NOT cheap).

One thing to remember is that smaller backup files also benefit some disaster recovery scenarios (basically those that don’t have good DR plans). Companies that do not have a SAN replication solution or some kind of Log Shipping/Database Mirroring solution in place, often transfer backup files to their warm DR site. Smaller files means faster transfer time which means better chance of recovery (you have a smaller window of exposure).

Password lock is available as with regular backups but there is no encryption capability with this release as far as I know. Now if you’re thinking, “hey, I’ve got Transparent Data Encryption, I can mix and match!!“. Well, don’t. Backup compression and TDE, at least based on the current release, do not compliment each other. You get next to nothing in space savings with backup compression on a TDE enabled database but you will burn the extra CPU cycles trying. As far as I can tell, this is true regardless of whether you use SQL Server’s backup compression or some 3rd party backup software with compression. However, some 3rd party solutions do offer backup compression with encryption but that’s a whole different deal. It’s a compressed backup that’s stored encrypted on disk with a non-TDE database.

Chalk this up as one of the “little to no effort” benefits when you move to SQL Server 2008. Just be aware that while it does great at the intended task, it doesn’t have all the bells & whistles that popular common 3rd party solutions do. For example, having different levels of compression and encrypting backups of non-TDE database are the most common I hear from our customers. Got something you want the SQL Server product group to add something for the next release? Go to SQL Server Connect

joe.

Cheers

Derek Comingore

Since Microsoft’s 1996 acquisition of the Israeli-based Panorama Software, Microsoft’s Business Intelligence platform has been continually refined and is spearheaded by the later renamed Analysis Services analytical platform! Analysis Services continues to rank as the number one OLAP product according to such institutes as Gartner, IDC, and the OLAP Report. With each iteration that passes Analysis Services continues to ‘up the ante’ in the OLAP market.

Analytical platforms in general are growing and are realizing more mainstream use. Every potential business and industry can benefit from analytics and thus the larger field of Business Intelligence! Competition is fiercer today in business than ever before and the ability to reap more value from your existing corporate data then your competitors is one of the many ‘winning factors’ in business today!

Every major area of the SQL Server product has been enhanced in the 2008 release including Analysis Services. The new enhancements as described later touch both multidimensional and data mining objects. Analysis Services 2008 improvements are substantial and thus warrant an evaluation of Analysis Services 2008 and the potential benefits it may bring to your organization!

Why Upgrade from Analysis Services 2000?

The same reasons for upgrading to Analysis Services 2005 from 2000 apply to why you should consider upgrade to Analysis Services 2008. In addition to those original improvements found in the 2005 release, 2008 contains additional improvements which are listed below in the topic of Why Upgrade from 2005. Beyond the analytical platform improvements there is the end of mainstream support for SQL Server 2000 matter. Microsoft mainstream support for SQL Server 2000 ends on 4/8/2008, more information can be found on this at http://support.microsoft.com/lifecycle/?LN=en-us&p1=2852&x=17&y=10 .

Why Upgrade from Analysis Services 2005?

Like the DTS to SSIS 2008 migration, if you are already running on the 2005 platform you will realize less benefits when compared to those migrating from Analysis Services 2000. That being said, there are still a good amount of new features and improvements found in Analysis Services 2008 including but not limited to:

• MultiDimensional Data
  • Enhanced Cube Design
    • Simplified & Enhanced Cube Wizard ‘IntelliCube’
    • AMO & UI Best Practice Alerts
    • Aggregations Designer
    • Simplified & Enhanced Aggregations Designer & Usage-Based Optimizations Wizards
  • Enhanced Scalability
    • MOLAP-enabled write back
    • Scale-out databases
    • Backup Compression
  • Enhanced Dimension Design
    • Attribute Relationships Designer
    • Enhanced Dimension Wizard
    • Key Columns Dialog
  • Enhanced MDX
    • Dynamic Named Sets
    • Subspace Computations
• Data Mining
  • ARIMA Time Series Algorithm
  • Creation of Holdout Test Sets
  • Filtering on Model Cases
  • Cross-validation of multiple Mining Models
  • Drill though to Structure Cases & Columns
  • Aliasing Mining Model Columns

As you can see there are still several added benefits by migrating to Analysis Services 2008! Microsoft continues to devote a large amount of its resources to continual refining the coveted MSFT BI platform. So should you upgrade to Analysis Services 2008? Ultimately, this decision should be based on your organization’s analytical applications and use of Analysis Services and the degree you can benefit from the new capabilities of the 2008 platform.   

We are still running on OLAP Services (SQL 7.0), what do we do?

Things get a bit more difficult if your organization is still running on the original SQL 7.0 OLAP Services release! First, you will need to upgrade to one of the two supported SQL Server 2008 upgrade paths:

1.       SQL Server 2000 w/sp4

2.       SQL Server 2005 w/sp2

Once you have performed this preliminary upgrade of your SQL Server (and thus OLAP Services) server you can start planning your SQL Server 2008 upgrade if you desire to get the latest advances in the analytics platform! Note that mainstream support for SQL Server 7.0 concluded on 12/31/2005! For more information on Microsoft’s Support Lifecycle for SQL Server 7.0 please see http://support.microsoft.com/lifecycle/?p1=2862 .

Effort Estimation Phase

Yet another parallel with the DTS to SSIS 2008 migration, you should consider performing a migration effort inquiry phase. There are fairly large differences between the 2000 version of Analysis Services and those versions that followed thereafter. Some of the items you may want to consider when performing your Analysis Services migration effort estimation include but not limited to:

·         Cube & Mining Structure/Models Complexities

·         SQL Server Upgrade Advisor (SSUA) 2008’s alerts on your existing Analysis Services Installations

·         64-bit Environments

Summary & Part2

In this post we have reviewed why an organization should consider upgrading their existing Analysis Services (and OLAP Services) installations to Analysis Services 2008. Much like the DTS to SSIS 2008 migration, Analysis Services migrations (particularly from SQL Server 2000) are a somewhat difficult yet high value proposition. A second parallel is having a solid, well thought out migration plan and effort estimation phase completed. Part 2 of our Analysis Services 2008 migration topic will go into the actual migration itself, rather how we perform the migration once we have decided to go forward!

Cheers

Derek Comingore

Well, we’re back and it’s time to perform the actual DTS to SSIS migration. Before we get started let’s review a few key points of interest. First, the migration process from DTS to SSIS is the same regardless of which version of SSIS (2005 or 2008) you are migrating too. In the context of these blog posts we are discussing SSIS 2008, just keep in mind that these are the same processes to migrate to SSIS 2005 as well. Second, we need to review the available DTS to SSIS migration strategies as mentioned in my first post, The Key SQL Server 2008 BI Migration: DTS to SSIS 2008, Part 1 of 2:

·         Proactive (resolving DTS to SSIS issues prior to upgrade)

·         Reactive w/Package Migration Wizard

·         Reactive w/o Package Migration Wizard

There are various business and technical requirements that will assist you in this decision such as:

·         Can you still work in/edit the DTS Packages?

·         Can the majority of the DTS to SSIS issues be resolved in DTS or must you wait until the package is in SSIS format? (some DTS items do not have direct SSIS equivalents)

·         What is the average effectiveness of the Package Migration Wizard for your packages?

The DTS to SSIS Migration Process

At this point you have been given the ‘green light’ to perform the actual migration and you have selected a migration strategy. So now what? Well, first off don’t do anything on production! If you have not yet performed a DTS Package scan with SQL Server Upgrade Advisor (SSUA) for the effort estimation, that will be the first task to perform. Again, we want to minimize the load on the production server(s) as they still have to do their daily ‘jobs’. To perform a SSUA scan of your DTS Packages perform the following steps:

1.       Export the DTS Packages to the Structured Storage File Format.

a.       In SQL Server 2000 Enterprise Manager, open each package and do a Save As command to the Structured Storage File Format.

*If the production server has already been upgraded to SQL Server 2005 and is still running the DTS Packages you will need the SQL Server 2000 DTS Designer Components to save the package to a storage file.

*You can save each package to a new file or consolidate all of them into a single file. The SSUA facilitates scanning all files in a supplied directory so either method will work with the SSUA.

2.        Copy or Move the files to a SQL Server 2008 development server.

3.        Scan the DTS Packages w/SSUA 2008

*The SSUA requires the detection of the DTS runtime which gets installed as part of the SQL Server 2005|2008 backwards compatibility files prior to allowing you to scan individual .dts (structured storage) files.

4.        Note the SSUA scan results

With your SSUA scan results ‘in hand’ let’s move on. You should already have the DTS packages on a development server. At this point you should either use your current development server or another server to create a true test environment. By test environment I mean a SQL Server 2008 environment that mimics production. So items like files, directories, databases (with a sampling of data) etc are on the development server, this will facilitate the actual testing of your SSIS 2008 packages after being converted!

At this point our development environment has been built using SQL Server 2008. It is now time to migrate the packages in the development environment by leveraging your chosen migration strategy detailed above. Once you have your SSIS (.dtsx) equivalents of the DTS packages you need to perform testing of each individual SSIS package. Make sure they execute as expected producing the correct results. Once you are confident that the new SSIS packages are working as expected you should:

·         Enable SSIS Package Configurations for all external resources

·         Create a Package Deployment Utility

You are now ready to deploy, test, and ensure that the new SSIS packages run as expected in your production environment! Definitely coordinate the following steps with the server’s DBA. Depending upon the overall SQL Server 2008 upgrade strategy the DBA choose (in-place or side-by-side) you may or may not have both instances on the current production machine. If the production servers have not yet been upgraded then you must wait until they are upgraded to SQL Server 2008. Once the production server(s) are upgraded, deploy the packages with the previously created Package Deployment utility to the intended servers. Update any package configuration settings needed and test each individual package during periods of low activity on the production server (if possible). Once you have confirmed the new SSIS Packages work as expected your DTS to SSIS 2008 migration is complete!

Migrating from SSIS 2005 to 2008

If you are already running on SQL Server 2005’s Integration Services platform and you wish to get the new benefits of Integration Services 2008 you will still need to perform a migration (or upgrade), however it’s a relatively simple one. As of the time of this writing (SQL Server 2008 CTP5), SSIS 2008 does support being installed side-by-side with SSIS 2005. SSIS 2008 is for the most part a pure enhancement of the platform. I say for the most part because there are a few ‘gotchas’ (breaking changes) between the two builds:

·         SSIS 2008 uses Visual Studio 2005 Tools for Applications (VSTA) for its scripting environment

·         SSIS 2008 uses a newer build of the SQL Server Native Client

·         SSIS 2008 contains an updated OLEDB Provider for Analysis Services (10.0)

·         SSIS 2005 custom developed components will need to be altered to work in SSIS 2008

The script environment differences are the biggest change between the two builds of SSIS. When you open/add a SSIS 2005 package which contains a Script Task in SQL Server 2008 BIDS it will automatically attempt to convert the script for you. You can also attempt to execute the older SSIS 2005 package with the 2008 dtexec utility, this operation will also convert the script; however the script conversion is only temporary for the currently running instance of the package.

Removing the old DTS Packages & SQL Server Agent Jobs from Production

This is the step that is the easiest to forget about! You have now successfully migrated your older DTS packages to the SSIS 2008 format, we are all done right? No, that’s not right, we still have the original DTS packages running on our production server. To remove the older packages from a production environment perform the following steps (again make sure the DBA is aware of your actions):

·         Stop all corresponding SQL Agent jobs

·         Open each package and do a Save As command to export the production version to the structured storage file format (yes, I know we already did this, but you never know if any of them changed or not, good practice)

·         Backup, check in, etc the storage files just in case we ever need to refer to them again!

*Assuming the same schedules apply to the new packages you may want to just update each SQL Server Agent Job with the corresponding SSIS Package step replacing the older DTSRun steps. If you need to recreate the jobs too on a different server consider scripting out all of the jobs and then editing/recreating them in bulk with the updated job steps in the scripts.

Summary

In this post we have explored the various DTS to SSIS migration solutions and best practices processes. Once the migration is approved based upon the business and technical factors (see my prior post The Key SQL Server 2008 BI Migration: DTS to SSIS 2008, Part 1 of 2) you need to perform the actual migration. The DTS to SSIS migration summarized consists of:

1.       Choosing a DTS to SSIS Migration Strategy (Reactive/Proactive)

2.       Capturing SSUA DTS Package Alerts (all categories of notifications)

3.       Building a dev/test environment

4.       Migrating the packages using the selected DTS to SSIS Migration Strategy

5.       Testing/Correcting the resulting SSIS 2008 Packages in the dev/test environment

6.       Deploying and reconfirming the resulting SSIS 2008 Packages work in production as expected

7.       Removing the old DTS Packages from production w/optional SQL Server Agent Jobs

By using the prescriptive guidance found in both of my posts you will have a great head start on the DTS to SSIS migration process we all face as SQL Server customers!

Cheers

Derek Comingore

Fire up those Zune’s and enjoy!

-Derick

SQL Server Integration Services or SSIS, saying those words around IT personnel who have a plethora of DTS packages will shiver! Naturally, they tremble because of two key reasons. First, SSIS is a completely different and more powerful tool, with power does come a certain degree of complexity. DTS has been used for a range of tasks beyond simple Extract, Transform, Load (ETL) purposes. Recently I was at a client’s site that entailed over 400 DTS packages, few of which were used for actual decision support purposes. Analyst, Information Workers, and others that are common non-IT personnel users of DTS are going to feel awkward (at least at first) using SSIS!

The second reason that folks get nervous about any potential DTS to SSIS migrations is because they are to some degree aware of the migration ‘cost’ or effort involved. DTS to SSIS is not a trivial migration, especially for those clients who have the 100s or even 1000s of production packages. There is a package migration wizard I will discuss in part 2 of this post that ships with SQL Server 2005 and 2008.  The package migration wizard attempts to migrate your DTS packages, however this wizard roughly migrates only around 40% of the various possible package configurations!

So Why Migrate Given the High Effort Required?

This is a natural question and the answer is also simple: power and speed. First, SSIS is a much more powerful data integration platform. SSIS contains several features that the older DTS platform did not have. Here are some of the features of SSIS that DTS did not have (or at least to the degree that SSIS contains):        

• Separation of control flow and data flow        
• Some connections have been optimized        
• Several additional various transformations        
• Wider range of data sources        
• Security enhancements        
• Toolset unification via BIDS with the other BI projects, source control integration
• Checkpoints & in place restarts        
• Dynamic runtime configurations        
• Superior logging capabilities 

The second reason to migrate is speed or rather performance. SSIS entails two different yet cooperating engines, the Run-Time Engine and the Data Flow Engine. SSIS’s Data Flow Engine leverages a buffer-oriented architecture to load and transform data in memory. As a result, SSIS is fairly fast compared to DTS, especially in terms of transformations. If you are performing simple data loads without many transformations involved then there will be a small delta in performance leaning towards SSIS (SSIS’s SQL Server destination component is faster than the older DTS connections). That said, as you increase package transformation complexity, SSIS performance will continue to increase over DTS. In fact, on average SSIS is believed to be roughly 7 times as fast as DTS. Your organization may experience a lesser increase in performance or even higher but this is the average gain seen.

A third reason though it is not a prevalent one yet (which is why I did not mention it before) is because the DTS product is an official deprecated feature of SQL Server. What this means is that Microsoft is delivering a message to its customer that they need to migrate to the SSIS platform sooner than later. Some would argue with me that this should be the main reason why organizations should migrate but let’s be honest about the matter. SQL Server 2008 is the last version to support DTS (as of now) and thus as a customer you will have a long time until you are absolutely forced into this migration but note that the day will come.

We Are Already Running on SSIS 2005, Do We Need SSIS 2008?

This is where things get a bit more difficult. Do you really need to perform an upgrade to SQL Server 2008 just for the SSIS enhancements? Like most things in life, that will depend on your organization’s usage of SSIS and the larger SQL Server product. Product enhancements beyond SSIS are out of scope for this blog post but suffice it to say there are several BI related enhancements in the 2008 product beyond those found in SSIS. That said, below are the enhancements from SSIS 2005 to 2008:        

• New Data Profiling Task & Viewer        
• Enhanced Performance and Caching for the Lookup Transformation        
• Enhanced Data Flow Engine Tuned for Parallelism       
• Various smaller enhancements for supporting new data types and TSQL capabilities

OK, I Want SSIS…How Do We Migrate?

There are three core approaches to the DTS to SSIS 2005|2008 migration. I will be listing these solutions in order of ease. Also, only the last two are possible solutions if you must migrate immediately for some reason.·        

• Proactive

SolutionBased on the SQL Server Upgrade Advisor’s results, take corrective action on the existing DTS packages then re-scan them w/Upgrade Advisor. Once the Upgrade Advisor reports no outstanding issues leverage the 2005 Package Migration Wizard to automate the package migration process. Any packages that cannot have their upgrade issues resolved will need to be rebuilt from scratch in SSIS. 

• Reactive Approach w/Package Migration Wizard

Disregard the Upgrade Advisor’s alerts and use the Package Migration Wizard to perform the migrations. Review each new resulting SSIS package to ensure it is built correctly. For those packages which did not migrate correctly either modify the resulting SSIS package or create it from scratch in the SSIS designer. 

• Reactive Approach w/o Package Migration Wizard

Disregard the Upgrade Advisor’s alerts and rebuild all of the packages from scratch in the SSIS designer. 

Scalability Experts DTS to SSIS Effort Estimation Framework & Considerations

The first step in performing a DTS to SSIS migration project is the effort estimation itself. This is not a trivial migration process and thus warrants a short yet focused study on the effort required. At Scalability Experts we have developed a DTS to SSIS Effort Estimation Framework for such projects. The framework assists us in both the time and accuracy of our DTS to SSIS effort estimations. When you are performing a DTS to SSIS effort estimation some of the items you want to consider include:

• Package Complexities
• Package Upgrade Advisor Alerts
• Parent/Child Package Environments
• Package External Dependencies
• 64-Bit Environments

Summary & Part 2

In conclusion, DTS to SSIS is a somewhat difficult yet highly valuable migration for those organizations that have come to rely on the Microsoft data integration platform(s). With the arrival of SSIS 2008 there are even more reasons to perform the migration. Having a solid migration planned out and effort estimation phase completed will be key steps to ensuring a smooth migration. In part 2 of this blog post we will discuss the actual migration process itself (How to perform the upgrade). This is the next logical step in the process once management has given the thumbs up for the migration effort to continue. Cheers
Derek Comingore

So what is this ‘heavy lifting’ I am referring to? The ‘heavy lifting’ represents the major infrastructure changes that occurred between SQL Server 2000 and 2005 in the Business Intelligence stack. The APIs that power all of the User Interfaces (UIs) in SQL Server (and facilitate your own custom administration applications) were changed and complimented. SQL Server Distributed Management Objects (DMO) was complimented with a new relational admin object model called Server Management Objects (SMO). SQL Server SSAS’ (SSAS) Decision Support Objects (DSO) object model was complimented with a new analytical admin object model called Analysis Management Objects (AMO). Additionally, for programmatic replication admin Replication Management Objects (RMO) was added to the 2005 product. If you are curious I use the term ‘complimented’ because the older object models still shipped with the 2005 & 2008 products. The newer object models are managed (CLR/.Net based) and expose the new features, while the older object models continue to support the older feature sets and are COM-based.

*In SQL Server 2008 Books Online CTP5 there are several instances mentioning SQL Server 2008 will be the last version of the product to entail the older COM-based object models. Plan to port your scripts and admin applications that leverage these older object models to their newer managed equivalents!

We saw the release of SQL Server Integration Services (SSIS) with the 2005 release which required either a manual migration or a partial-manual migration using the Package Migration Wizard. SSIS was a completely different data integration platform when compared to the SQL Server 7.0 originating Data Transformation Services (DTS). DTS contained (like the other 2000-based and older APIs) a COM-based object model implemented in three DLL files: dtspkg.dll, dtspump.dll, and custtask.dll. With SSIS we received a managed object-model like the other 2005-based services.

*In SQL Server 2008 Books Online CTP5 there are several instances mentioning SQL Server 2008 will be the last version of the product to entail the DTS platform. If you have not already done so, plan/migrate your DTS packages to SSIS!

SSAS beyond the API changes was drastically changed. XMLA, Actions, KPIs, perspectives, the concept of measure groups, there were huge changes in the SSAS environment. Like SSIS, SSAS provided a migration wizard for the 2000 database formats to the 2005. In addition, for the first time SSAS supported a project mode in addition to the online mode of SSAS databases. The SSAS project mode is part of a larger difference, which was the introduction of the 2005-based Business Intelligence Development Studio (BIDS). Prior to SQL Server 2005 we had to use a combination of Enterprise Manager, Analysis Manager, Query Analyzer, and the MDX Sample Application for developing SQL Server BI solutions. BIDS consolidated all of the BI development needs into one tool based on the larger Visual Studio shell.

SQL Server Reporting Services (SSRS) was a bit different from the rest of the SQL Server BI components for a good reason; it was originally designed for the SQL Server 2005 infrastructure, it was ‘down ported’ the SQL Server 2000 product in 2004 due to customer demand. SSRS is the outlier because of the previous statement and that it is undergoing the most changes from the SQL Server 2005 to the 2008 product. The Report Server in SSRS 2008 has been completely re-architected. The Report Server in SQL Server 2008 has been consolidated into a single Windows-based service that provides the report manager, web services, and background processing cores. This new Report Server architecture is enabled by leveraging the SQL Server’s networking stack and the HTTP.SYS which is part of the Windows networking subsystem and eliminates the requirement of Internet Information Services (ISS) in the process. SQL Server gurus foresaw this occurring because of the relational engine’s addition of native HTTP endpoints in the 2005 product that leveraged the HTTP.SYS module. Because of this re-architecture effort in SSRS 2008 we do see a few IIS-centric issues in the Upgrade Advisor 2008 release for SSRS.

In conclusion, you will have to incur the ‘heavy lifting’ of the significant differences in the APIs and BI components (SSIS/SSRS/SSAS) regardless of whether you upgrade to the 2005 or the 2008 product. If you were to compare the SQL Server Upgrade Advisor (SSUA) accompanying help you will find a large number of the BI component’s upgrade issues are the same from the 2005 SSUA tool, this is because of the ‘heavy lifting’ I have described in this post that exists between the 2000 & later versions of Microsoft SQL Server. Mainstream support for SQL Server 2000 is coming to an end and so it behooves you to start considering an upgrade to either the 2005 or 2008 version. I’d like to conclude this post with a brief summary table per BI component of the severity of differences between the product versions (with accompanying notes) and thus the more SQL server  upgrade issues you will face.

Figure 1: SQL Server 2000 to 2005/2008 BI Upgrade Effort Summary Table

Derek Comingore

In Analysis Services 2000 you configure the drillthough settings via the Cube Editor dialog and then going Tools->drillthrough options. You specify the tables, columns and an optional drillthrough filter. In Analysis Services 2005 you configure drillthrough settings by creating a drillthough action in the cube editor’s Actions tab. Like its predecessor, in Analysis Services 2005 you can optionally provide a filtering condition. Because the drillthrough settings are significantly different in Analysis Services 2000 compared to Analysis Services 2005/2008, this issue only applies to 2000 to 2008 in-place upgrades. I have personally tested Analysis Services 2005 to 2008 in-place upgrades with OLAP cubes that contain drillthough settings and they do in fact get migrated.

In my Analysis Services 2000 test case I am using the out-of-the-box FoodMart 2000 AS database. It contains two objects that have drillthough settings defined, the HR & Sales cubes. Upgrade Advisor will detect and alert you when it scans an Analysis Services 2000 instance which contains drillthrough settings. The Upgrade Advisor corresponding help topic simply states that ‘While drillthrough exists in Microsoft SQL Server 2005 Analysis Services, drillthrough settings are not migrated from previous versions of Analysis Services’. Since Analysis Services 2005 is not subjected to this issue the Upgrade Advisor does not report anything back when it scans an AS 2005 instance.   

Prior to performing the actual SQL Server 2008 upgrade from a SQL Server 2000 instance, make note of the existing drillthough settings. Perform the actual upgrade (we always recommend using a test machine configured to mimic your production environments to test upgrade first) and then recreate your drillthough settings in your SQL Server 2008 Analysis Services instances.

Derek Comingore

One of the better 3rd party write-ups on the subject is from Steve Jones who recommends users to skip 2005 and move straight to 2008. I’ve known Steve for a while though we typically only catch up at conferences (TechEd, PASS, etc…). Still, he’s been working with SQL Server a long time and knows what he’s talking about so it’s worth taking a peek at his article on SQL Server Central.

Of course, Microsoft too has a couple of papers on the subject that are worth reviewing. While they may sound like a gentle nudge towards upgrade to 2005 first, some of the reasons are good ones. There’s the original “Why upgrade to SQL Server 2005” paper which is ok if you really are brand new to 2005. While quite dated, the info is still relevant. The newer “SQL Server 2005 Upgrade FAQ” has the latest and greatest information including a short section on the benefits of moving to 2005 first then 2008 later. It’s a good start but IMHO, doesn’t provide enough to help the average user decide.

I really don’t believe there is a hard rule for this; it really depends on each individual deployment (yes, deployment not company). Here are a few important points to consider when trying to decide whether to first upgrade SQL to 2005 then on to 2008 or do both in 1 step. I’ve laid them in in pros/cons format to make it easier to get most folks started but the right approach really depends on your specific deployment and what’s important to you.

Note the the lists is not ordered or weighted in anyway. They are just intended to get folks thinking about some of the issues involved. For example, testing efforts for a 1-step SQL Server upgrade might be significantly more involved than a 2-step approach. This is because there are (a small number of) deprecated features that are fully de-supported/removed from SQL Server 2008. If you had taken a 2-step approach, you would have some time to setup alerts (use of unsupported features/command will fire an event create an entry in the SQL Server log) and phase them out gradually. With the 1-step upgrade, you’d have to hunt down every line of code including dynamic SQL to be analyzed by Upgrade Advisor or run through the Application Compatibility Testing tool.

Not purely an IT decision.
Sure, the DBAs and application admins are intimately involved since they need to handle the testing and deployment. However, there’s a fair bit of learning involved especially if you want to take advantage of Policy Based Management (aka DMF) so there’s the first involvement from the business side - training budgets. Still, the most important points, IMHO, from a business perspective are the downtime window and regulatory compliance. I’ve worked with some customers where extended downtime is schedule up to 2 years in advanced. This doesn’t leave much room for doing a lot of upgrade operations so you might have to just bite the bullet and make one big jump. On the other hand, if there are specific regulations that require you to run only products that have mainstream support, you’ll likely have to get to 2005 asap (2008 will not be available till summer).

The SP1 rule
One important point I’d like to make is about SP1. I know of a lot of companies that have this rule - no RTM deployments, wait for SP1. I used to subscribe to this rule also, a very long time ago when I helped manage some Oracle DBs and in my early SQL Server days. As of 7.0, this SP1 rule really made less sense with every release. My rationale is simple. Beta versions of SQL Server have been running in production environments for SAP customers and a bunch of Microsoft’s own internal applications (its own SAP and other business critical applications) since 7.0 and the practice continues today. If you know anything about SAP, you’d know running it in production on a beta database is no small feat regardless of how many modules you run. I don’t know of any other enterprise database that does this. So if multi-billion dollar companies can run their very complex business critical applications on beta versions of the product, I’m pretty comfortable running the released version.

If you still can’t decide after all this, do a POC. Download a copy of the AppCompat Tool, aka SSUA for SQL Server 2008 (currently slated for March release) and do your own application compatibility testing. That should give you a good idea of the effort it’ll take to upgrade your database/application which hopefully, puts you in a place to make the right decision on which path to take.

In short, there are no easy answers and your decision should not be based on features alone. Though the most common response to this question is “do you need the new features”, it is hardly the most important one. Most features can be worked around with some creative code or 3rd party applications. Downtime windows and regulatory compliance requirements are generally very inflexible. Of course, your environment might be such that neither of these apply so all you really care about are the new features. Like I said from the start, it depends on each individual deployment.

joe.

ODBC or Open Database Connectivity is a platform neutral, universal approach for providing an Application Programming Interface (API) to access relational database management systems. ODBC was an early modular approach to enable database access. The ODBC library itself is an independent Operating System (OS) component which acts like a bridge passing requests from the consuming application to the corresponding ODBC drivers. The various data store vendors develop their product’s ODBC drivers and then either ship those with the product and or make them available over the Internet. Microsoft designed Object Linking and Embedding Database (OLEDB) as a replacement for ODBC mainly due to the desire to support a much larger variety of data stores beyond those that are pure relational databases. A nice alternative point to this post is that we take so much for granted in today’s technological landscape. At one point in time, performing database access was not a trivial task.

In today’s world, particularly the SQL Server BI world we have a plethora of drivers and providers for accessing databases, email systems, spreadsheets you name it. Like SQL Server native extended store procedures (XPs), Microsoft is slowly (but surely) phasing out the old ODBC API. Currently, Microsoft OSs still support the ODBC platform/technology, however the higher-level components are withdrawing support. SQL Server Analysis Services (SSAS) is one such component, since SQL Server 2005, SSAS has not supported the ODBC specification. The following is a quote describing the non-support of ODBC by SSAS 2005, the entire technical article can be located here http://technet.microsoft.com/en-us/library/aa964120.aspx :

‘Microsoft SQL Server Analysis Services 2000 supported a range of OLE DB providers. In particular, it supported OLE DB for ODBC. You could configure an ODBC data source to access your database.Analysis Services 2005 provides greater functionality by providing data-source views. Because it supports different syntaxes, Analysis Services 2005 imposes stricter requirements on client libraries than does Analysis Services 2000. Analysis Services 2005 does not support Microsoft OLE DB Provider for ODBC. To access your database, you must find an appropriate version of the OLE DB provider for Analysis Services.

Besides OLE DB in Analysis Services 2005, you can use managed providers to access your relational database.’

Detecting SSAS 2000 ODBC Data Sources with the Katmai Upgrade Advisor

When you analyze a SQL Server 2000sp4 SSAS instance that contains ODBC data sources with the SQL Server 2008 Upgrade Advisor (UA) utility you will receive an advisory stating that ‘ODBC Data Sources are not supported’. As described in my previous post, this is one of the more common issues that the UA will find when analyzing Analysis Services instances. A lot of folks out there are still running on the SQL Server 2000 platform including Analysis Services.

Post Katmai Installation & Resolutions

Once you perform the SQL Server 2008 upgrade you will need to update or recreate your Analysis Services data source to use a corresponding OLEDB provider. This issue applies only to SQL Server 2000sp4 upgrades to SQL Server 2008. That being said, because of the sheer number of Analysis Services 2000 installations running in production that do in fact use the ODBC drivers this is a fairly common Katmai upgrade issue.

Derek Comingore

*The UA utility requires the .Net Framework 2.0 redistributable

In our testing of this utility based on the November 2007 CTP build of Katmai (CTP5) we found that the UA will allow the analysis of both local and remote instances of the BI components excluding Reporting Services. Reporting Services instances will require the UA to be installed on the report server itself. Another discovery we found from our tests is that in the CTP5 build the Reporting Services analysis does not seem to be working as designed. Integration Services is not yet supported in UA, however the tool does note that in a future CTP/build of Katmai UA will support analyzing SSIS. In addition, SQL Server 2008 will continue to support the Data Transformation Services (DTS) runtime and thus the UA allows you to analyze DTS Packages in both a SQL Server 2000 and 2005 environment. A nice bonus is the ability to analyze DTS packages stored in the file system as well. Also note that when you analyze a SSAS 2000 instance the UA utility will require the Decision Support Objects (DSO) object model.

*DTS Packages are an official deprecated feature in SQL Server 2008. You should migrate your organization’s DTS packages as soon as possible to help support future upgrades of your SQL Server environments.

The UA utility provides you with both analysis and reports. UA reports provide you with the alerts/warnings that you should consider or take action on prior to upgrading your SQL Server environments to the 2008 platform. A nice feature of the UA utility is its ability to provide the user with links directly to its alert documentation. Some of the more important UA alerts are as follows:

SQL Server Data Transformation Services UA Key Alerts

· SQL Server DTS Packages are now a deprecated feature

· To continue running and managing your existing DTS Packages, the SQL Server 2008 Backwards Compatibility Files are required (these files were required for the same reason in SQL Server 2005 platform)

· The DTS Designer Components are a required add-on if you wish to edit DTS packages in Management Studio (these files were also required for the same reason in SQL Server 2005 platform)

SQL Server Analysis Services UA Key Alerts

· VBA functions handle NULL values and empty values differently from SQL Server Analysis Services 2005

· Custom Aggregations are not migrated

· ODBC Connections are not migrated

· MDX Changes

SQL Server Reporting Services UA Key Alerts

· Custom Reporting Services extensions (DLLs) will not be moved

· Client Certificates were detected (Reporting Services 2008 does not support them)

· Reporting Services 2000 Report Server database is on a remote 2000 instance (blocks a 2000 to 2008 SSRS upgrade)

Be aware that UA cannot analyze every aspect of your SQL Server environments and thus it cannot detect every issue that could affect an upgrade. Some examples of items not analyzed by the UA are middle-tier or front-tier code containing TSQL, encrypted stored procedures, and extended stored procedures. Regardless of the UA limitations, it is a very good tool for the BI professional to leverage for prepping their environments for SQL Server 2008.

Happy Upgrading!

Derek Comingore

This isn’t very appealing to users who truly need the encryption capability as the column they need to encrypt is often the column which they search by. A common example is a credit card company where the credit card number is the column they want to encrypt but that is also the  column they would search when customers access their accounts or when employees need to lookup a customer. When you have hundreds of millions of accounts and need to do thousands of lookups every minute, it’s a bit of a problem.

Katmai takes this a (big) step further with Transparent Data Encryption. The concept isn’t new and at least one other major database has it. Basically, think of having a layer between the storage  engine and your disks that will encrypt data before it is written to disk and decrypt it before being read into memory. Turn it on is a lot simpler than Yukon’s encryption too. You just need to create a database encryption key and set the database encryption option on. Note that your database encryption key needs to be protected by a certificate that is protected by the database master key. This ensures it can be accessed after a restart or recovery.

Unlike Yukon’s table encryption, you don’t have the same challenges with indexes like table level encryption in Yukon. The following is a simple example for encrypting your the pubs database and a query whose plan will show an Index Seek. Not very exciting in itself but considering the past encryption technique where indexes don’t even come into play, this is pretty cool.

USE master;
GO
CREATE MASTER KEY ENCRYPTION BY PASSWORD = ‘p@55w0rd’;
GO

CREATE CERTIFICATE DBEncryptKeyCert  WITH SUBJECT = ‘DB_Encryption_Key_Cert’;
GO

USE pubs
GO
– Note that not all operating systems support AES so choose an algorithm that is supported by your OS
– SQL Server supports most common algorithm like AES, DES, 3DES, RC4, etc… See Books Online for details.

CREATE DATABASE ENCRYPTION KEY
WITH ALGORITHM = AES_256
ENCRYPTION BY SERVER CERTIFICATE DBEncryptKeyCert;
GO

ALTER DATABASE pubs
SET ENCRYPTION ON;
GO

USE pubs
GO
SELECT  fname, lname, emp_id
FROM  employee
WHERE  emp_id=’A-C71970F’;
GO

Just pretend emp_id is credit card number. If you had encrypted the column emp_id (i.e. credit card number) and created the encrypt/decrypt function along with the view, triggers, etc… you’d be scanning the table just to retrieve that one row. That’s what you had to work with in 2005 (unless you had some 3rd party solution). Sure, there are several ways to reduce/avoid that but solutions typically involve some variant of storing a substring of the column (hashed or as-is) and indexing that column. Not a very viable solution for credit cards since the first 6 digits make up the credit card issuer’s identification number. A bunch of other concerns pop up and though not insurmountable technically, they do present challenges and in some cases, regulations may disallow this approach. No such concern with TDE in 2008.

So what about EFS? Well, just consider it as another option to be used as appropriate. EFS encrypts the entire file system while TDE encrypts just the database. Picking the right option depends on your transportability requirements and concerns. For example, if your company uses a centralized repository for storing ALL backups and you don’t trust those pesky Windows admins that own that repository, TDE ensures nobody can copy the backup file and restore on another SQL Server instance. It is also possible to use both but be aware of the performance and administration implications especially during recovery (EFS access then database access).

Upgrade Effort & Impact
What’s the effort to benefit from this? Well, from an execution perspective, just the few steps in the sample scripts above or you can use Management Studio. If you are upgrading a SQL Server 2005 database that is already using column level encryption, suggest you revert to plain text before the upgrade. Though they are different features, there really isn’t a need for using both. Same with those who are using 3rd party encryption products. That said, there are a few challenges in the upgrade process. For instance, you might not be allowed to store data in clear text to facilitate a simple upgrade. As such, you will need to upgrade first then reverse the process which you employed to encrypt a column using the function based approach. That means you’ll turn on TDE on the database then move the data from the encrypted column to a non-encrypted column so you can do away with the UDFs (and views if you had those). Bottom line is, this may mean a longer upgrade/migrate time window plus a longer rollback time also.

There are also other considerations to your production server. This is a database wide option and will impact other components such as the transaction log and tempdb. To ensure data is not stored unencrypted once you turn it on, the current virtual log file will be “zeroed out” and new transactions will begin on the new VLF in encrypted form. Your tempDB will also be encrypted. This means you will have performance overhead on the system initially (both CPU and disk) and alters user behaviour later (e.g. can’t snoop around in tempdb easily anymore). Another important consideration is backup. With TDE, your backups are also encrypted so make sure you backup your keys/certificates also. Else, in a disaster recovery situation, you will not be able to restore your database. These and other considerations are well documented in BOL so do review in detail.

From a planning perspective, think real hard about why you need to encrypt your database. If the only reason you can come up with is “to make it more secure”, talk to your friendly neighbourhood Microsoft engineer or trusted partner before turning this on. You might eventually still employ TDE or you may not even need it (but you’ll likely discover other security concerns). Remember, security is not just about technology. If you don’t educate your staff and users or fail to practice secure processes diligently, no amount of encryption will save your data. Flipping over keyboards and turning photo frames around on users desks STILL reveal passwords (last verified Sept 2007) and backup tapes can still be found on DBAs/operators’ desks or at the reception awaiting picking. TDE is just another layer of defense in your arsenal for specific intents and attack vectors; don’t forget the others.
 

joe yong.