A quick glimpse at your Log Shipping Status

Almost every shops I’ve worked in, Log Shipping is an important part of their disaster recovery plan (2012 has changed the game with availability groups). When you configure log shipping natively, it also creates a couple of alerts – Log shipping Primary Server Alert and Log shipping Secondary Server Alert (hope you have a SMTP Server and have configured database mail correctly). These alerts are invoked by a job which keeps calling the following stored proc:

exec sys.sp_check_log_shipping_monitor_alert

It does a fair job of notifying team members (DBAs/System Engineers, etc.) when the log shipping falls behind. Most of these are configurable parameters, and you can also modify these thresholds by updating metadata/system tables (be careful while doing this however)!

I have been in situations where the alert system (SMTP) connectivity stops working (briefly, until restarted), or there are some network connectivity issues. Sometimes with ad-hoc index rebuilds (online rebuilds), I try to keep close eyes on how far behind the log shipping is, without relying heavily on log shipping monitors/alerts. The following combination of T-SQL and PowerShell comes in in the process. With PowerShell, I do not have to run the same query on multiple machines anymore.

What gets called

USE [DBARepo]
GO
 
 
SET ANSI_NULLS ON
GO
 
SET QUOTED_IDENTIFIER ON
GO
 
create view [dbo].[DBA_restore_history_checker]
as
(
 
SELECT
rsh.destination_database_name AS [Database],
rsh.user_name AS [Restored By],
CASE WHEN rsh.restore_type = 'D' THEN 'Database'
  WHEN rsh.restore_type = 'F' THEN 'File'
  WHEN rsh.restore_type = 'G' THEN 'Filegroup'
  WHEN rsh.restore_type = 'I' THEN 'Differential'
  WHEN rsh.restore_type = 'L' THEN 'Log'
  WHEN rsh.restore_type = 'V' THEN 'Verifyonly'
  WHEN rsh.restore_type = 'R' THEN 'Revert'
  ELSE rsh.restore_type 
 END AS [Restore Type],
rsh.restore_date AS [Restore Started],
bmf.physical_device_name AS [Restored From], 
 rf.destination_phys_name AS [Restored To]
FROM msdb.dbo.restorehistory rsh
INNER JOIN msdb.dbo.backupset bs ON rsh.backup_set_id = bs.backup_set_id
INNER JOIN msdb.dbo.restorefile rf ON rsh.restore_history_id = rf.restore_history_id
INNER JOIN msdb.dbo.backupmediafamily bmf ON bmf.media_set_id = bs.media_set_id
WHERE rsh.restore_date >= DATEADD(dd, ISNULL(-1, -30), GETDATE())
AND destination_database_name = ISNULL('DBName', destination_database_name)
 
 
)

Caller:

foreach ($ServerInstance in get-content "C:\serverlist.txt")
{
    ##$ServerInstance = "My Hardcoded Server was here. This is commented out."
    $Database = "DBARepo"
    $ConnectionTimeout = 30
     
    $Query = "select top 1 * from dbo.DBA_restore_history_checker order by [restore started] desc"

    $QueryTimeout = 120

    $conn=new-object System.Data.SqlClient.SQLConnection
    $ConnectionString = "Server={0};Database={1};User Id=username;Password=password;;Connect Timeout={2}" -f $ServerInstance,$Database,$ConnectionTimeout
    $conn.ConnectionString=$ConnectionString
    $conn.Open()
    $cmd=new-object system.Data.SqlClient.SqlCommand($Query,$conn)
    $cmd.CommandTimeout=$QueryTimeout
    $ds=New-Object system.Data.DataSet
    $da=New-Object system.Data.SqlClient.SqlDataAdapter($cmd)
    [void]$da.fill($ds)
    $conn.Close()
    $ds.Tables
}

As you might have noticed, the above PowerShell script can be used with almost everything, and the best part is that it allows you to iterate through the list of servers you provide.

Happy PowerShelling!

How I do my backups

Several years ago, I got my hands on one of the database servers that I did not know existed within my organization (ex-employer). It was managed by a completely different department, on a different VLAN (or else SQLCMD -L would have most likely found it), and they needed some help shrinking it – the log had grown tremendously (it was a onetime thing). After shrinking the log, I talked to the department so that the DB could be backed up (no backups in the backup history, anyone?). Backed it up, restored it to a test, and ran DBCC CHECKDB. Corruption was found. Tried various methods of repair, ultimately it came down to REPAIR_ALLOW_DATA_LOSS.

Risk was – data loss. Plus no backup. The first thing to do was to do a full backup, and restore it on a different server just in case it causes more harm (be careful of this – if you restore 2000 to 2005, or 2005 to 2008, you cannot backup from there and restore that to earlier version). After talking to developers on what the purpose of the database was, and if any quick stopping of the services were allowed, the answer was usual (24/7/365) operation. I resorted to the last possible way – create an empty shell with all the tables (transfer the schema), switch the database (so that all the inserts are not lost), do a quick repair, check against the restored copy of the database and transfer the lost data, switch it back, and then transfer all the data from the shell.

It was a painful process, and needed careful planning plus management approval. The key takeaway from this lesson was to design an automated process which not only backup up the database, but does automated restores (based on your preferences – weights/values of databases, on the given date/time), automated CHECKDB, and build reports so that you know what was happening. Additionally, send an email out if there was a problem with the restore or if the CHECKDB failed. It got more interesting in the process, and I started keeping tabs on how long the backups were taking, growth sizes, and all kind of interesting things.

The experience did help re-enforce the conviction in my mind – have a good backup plan or you are going to express highway towards unemployment. The same process that I built over several years ago has been refined as time goes by, and I occasionally dwell into network transfer speeds, etc. A basic structure of what has worked for me over several years is outlined below.

test-backup

If you are reading this, and for some reason focusing on this has not been your primary concern, here are a few things that might help

  • RichCopy – you can script out from the GUI and create a batch file. Make sure you have a log file too.
  • Instant File Initialization – makes the restore go fast. No zeroing out of database file.
  • Databases cannot be downgraded. Do not expect the database you restored in newer version server can be restored to lower version
  • More to add..

A little bit of code to help you get started

RichCopy Portion – What gets called

My CopyToNetworkLocation.bat looks like this:

"Drive:\LocationOfRichCopy\RichCopy 4.0\RichCopy64.exe"  "Drive:\BackupLocation\"  "\\NetworkLocation\Backups\ServerName\" /FSD /TSD /TSU /CCD /CLW /NE /P /V /QA /QP "Drive:\LocationOfRichCopy\RichCopyScripts\ServerName.log" /UE /US /UD /UC /UPC /UFC /UCS /USC /USS /UPR

SQL Job Configuration – The caller

EXEC xp_cmdshell 'Drive:\LocationOfRichCopy\RichCopyScripts\CopyToNetworkLocation.bat'

Remember, you have to enable xp_cmdshell to run this command. You can briefly enable, and disable it. Or you can use Windows Scheduled Task to schedule this job.

Adieu 2012!

As we are getting closer to the end of 2012, I wanted to thank all of you who have been with me through all thick-and-thins. I also wanted to thank those who have parted ways; I learned a lot from you (without any hard feelings).

2012 has been great so far; both personally and professionally. By the time the year started, I was adjusted to being in college again, working under deadlines (we did have kinda-sorta deadline at work); getting my name acknowledged in a book, and still able to enjoy little things life has to offer.

What’s up with the types?

This rant deserved a post!

Early this morning, I was jet set to write a script that would duplicate a (package) and give it a new name. So basically, idea was to get the identity value of the old package, and insert it as a new with same attributes but different package names. Based on the insert, it would do the same for all corresponding child tables. To help understand better, something like this:

INSERT INTO dbo.PackageTable(PackageDesc, attribute1, attribute2)
SELECT
@newPackageDesc AS PackageDesc
, attribute1
, attribute2
FROM dbo.PackageTable
WHERE PackageID = @oldPackageID

I got a bunch of error messages such as:

There was an error! Error converting data type varchar to numeric.

After dwelling around for almost 45 minutes, (when I was planning to go back to take Database 101 after all these years) guess what I discover? One of the columns is using user defined type such as:

[PackageDesc] [dbo].[Desc_Short] NOT NULL

and the Desc_Short was defined as:

CREATE TYPE [dbo].[Desc_Short] FROM [varchar](30) NOT NULL
GO

Firstly, I see no benefit of doing that, and I wanted to find out some use cases where you could benefit from defining a UDF. Came across this blog post:

http://blogs.msdn.com/b/bartd/archive/2010/08/25/t-sql-udts-what-are-they-good-for.aspx

I will quote a couple of paragraphs from this post just to make this blog post a bit more convincing:

“The attempt to make the app developer’s life easier can backfire and actually made it a little harder.”

Now, I was not going to do this, but I wanted to check how much effort is really needed to get away from UDTs. Turns out:

1) There is no easy way to drop types without first modifying all the dependencies.
2) Even if you change all the object definitions (for columns), you still have to find all the parameters/variables that is being used. (SQL Search will come handy here).
3) What happens when your type has to be modified? Lets say, Desc_Short in the above example has to accomodate varchar(40) instead of varchar(30)? There is no alter types, so you will have to repeat all the steps above (I am assuming that if you have to do this, you will not go back to types again).

Bottom line: We should not add more dependencies than there already is. So, please, do not use user defined types unless you can really justify it.

Trending SQL Server Job Duration by Day

As I am getting used to the new environment, I wanted to keep tabs on SQL Server Agent job and see how the database growth is impacting SQL Server Agent jobs in terms of duration (daily). I had stumbled upon a query to analyze it by hour and that would work in data warehouse situations where ETL is happening every hour. My objective was to analyze it by days, and have it dynamic (had to monkey around with PIVOT function and dynamic columns). Finally, bingo!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
 
DECLARE
@columns VARCHAR(2000)
, @query VARCHAR(8000)
 
 
SELECT @columns = COALESCE
(
@columns + ',[' + CONVERT(VARCHAR, run_date) + ']', + '[' + CONVERT(VARCHAR, run_date) + ']'
)
from msdb.dbo.sysjobhistory
group by run_date
order by run_date
SET @query = '
;WITH JobHistory AS
(
SELECT
b.name as JobName
, a.run_date as [RunDate]
,(a.run_duration / 10000 * 60 * 60 -- Hours
+ a.run_duration % 10000 / 100 * 60 -- Minutes
+ a.run_duration % 100 -- Seconds
) / 60.0 AS [DurationMinutes]
FROM
msdb.dbo.sysjobhistory a WITH(NOLOCK)
INNER JOIN msdb.dbo.sysjobs b WITH(NOLOCK)
ON
a.[job_id] = b.[job_id]
-- AND b.[name] = @JobName
AND step_id = 0
AND run_status = 1
)
SELECT *
FROM
JobHistory
PIVOT
(
SUM(DurationMinutes)
FOR [RunDate]
in
(' +
@columns
+
')
) as pvt
 
order by JobName'
exec(@query)

The original idea about trending by hour was acquired from:

http://timlaqua.com/2009/12/trending-sql-server-agent-job-duration-by-hour/

Drive Space Alert

I am not sure whether there is already a mechanism in SQL Server (without any third party product) to send out alerts when your drive space is running low (below a certain threshold). Not much familiar with writing WMI, so I decided to put together a script that does this automatically – the idea is to send out an alert every hour in my email so that I know the alerting mechanism is working (have seen this break before), and to send out an alert if the drive space is running below certain threshold. In my experience, drive spaces run low when:
1) When too many autogrows happen.
2) When you have too many backups that you have not archived.
3) When the Windows Paging file grows (unless you have set up max size for this).

Please feel free to copy and modify the script to your needs and any comment/feedback will be most welcome. Please be sure to create a group email that sends alerts to every DBA or at least those who are on call.

Script as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
 
USE [DBATasks]
GO
 
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[dba_DiskFreeSpaceAlert]
 
	@DriveCBenchMark int = 13000,
	@DriveKBenchMark int = 38044,
	@DriveLBenchMark int = 35780,
	@DriveWBenchMark int = 85811
 
 
AS
/**********************************************************************************
Author: Subhash Pant
 
**********************************************************************************/
SET NOCOUNT ON;
 
DECLARE @DiskFreeSpaceC INT
DECLARE @DiskFreeSpaceK  int
DECLARE @DiskFreeSpaceL INT
DECLARE @DiskFreeSpaceW INT
DECLARE @DriveLetter CHAR(1)
DECLARE @AlertMessage VARCHAR(500)
DECLARE @MailSubject VARCHAR(100)
 
IF EXISTS (SELECT * FROM tempdb..sysobjects
WHERE id = object_id(N'[tempdb]..[#disk_free_space]'))
DROP TABLE #disk_free_space
CREATE TABLE #disk_free_space (
	DriveLetter CHAR(1) NOT NULL,
	FreeMB INTEGER NOT NULL)
 
/* Populate #disk_free_space with data */
INSERT INTO #disk_free_space
	EXEC master..xp_fixeddrives
--SELECT * FROM #disk_free_space
 
SET @DiskFreeSpaceC = (SELECT FreeMB FROM #disk_free_space where DriveLetter = 'C')
SET @DiskFreeSpaceK = (SELECT FreeMB FROM #disk_free_space where DriveLetter = 'K')
SET @DiskFreeSpaceL = (SELECT FreeMB FROM #disk_free_space where DriveLetter = 'L')
SET @DiskFreeSpaceW = (SELECT FreeMB FROM #disk_free_space where DriveLetter = 'W')
 
--SELECT @DiskFreeSpaceC as DiskFreeSpaceC, @DiskFreeSpaceK as DiskFreeSpaceK, @DiskFreeSpaceL as DiskFreeSpaceL, @DiskFreeSpaceW as DiskFreeSpaceW
 
IF (@DiskFreeSpaceC < @DriveCBenchMark)
BEGIN
	SET @MailSubject = 'Drive C free space is low on ' + @@SERVERNAME
	SET @AlertMessage = 'Drive C on ' + @@SERVERNAME + ' has only ' +  CAST(@DiskFreeSpaceC/1024 AS VARCHAR) + ' GB left.'
 
	-- Send out email
	EXEC msdb.dbo.sp_send_dbmail 
		@profile_name='DBA',
	@recipients='dbagroup@companyname.com',
		@subject= @MailSubject,
		@body= @AlertMessage
END
 
--SELECT @DiskFreeSpaceK = FreeMB FROM #disk_free_space where DriveLetter = 'K'
 
IF (@DiskFreeSpaceK < @DriveKBenchMark)
BEGIN
	SET @MailSubject = 'Drive K free space is low on ' + @@SERVERNAME
	SET @AlertMessage = 'Drive K on ' + @@SERVERNAME + ' has only ' +  CAST(@DiskFreeSpaceK /1024 AS VARCHAR) + ' GB left.'
	-- Send out email
	EXEC msdb.dbo.sp_send_dbmail 
		@profile_name='DBA',
	@recipients='dbagroup@companyname.com',
		@subject= @MailSubject,
		@body= @AlertMessage
END
 
--SELECT @DiskFreeSpaceL = FreeMB FROM #disk_free_space where DriveLetter = 'L'
 
IF (@DiskFreeSpaceL < @DriveLBenchMark)
BEGIN
	SET @MailSubject = 'Drive L free space is low on ' + @@SERVERNAME
	SET @AlertMessage = 'Drive L on ' + @@SERVERNAME + ' has only ' +  CAST(@DiskFreeSpaceL /1024 AS VARCHAR) + ' GB left.'
	-- Send out email
	EXEC msdb.dbo.sp_send_dbmail 
		@profile_name='DBA',
	@recipients='dbagroup@companyname.com',
		@subject= @MailSubject,
		@body= @AlertMessage
END
 
--SELECT @DiskFreeSpaceW = FreeMB FROM #disk_free_space where DriveLetter = 'W'
 
IF (@DiskFreeSpaceW < @DriveWBenchMark)
BEGIN
	SET @MailSubject = 'Drive W free space is low on ' + @@SERVERNAME
	SET @AlertMessage = 'Drive W on ' + @@SERVERNAME + ' has only ' +  CAST(@DiskFreeSpaceW /1024 AS VARCHAR) + ' GB left.'
	-- Send out email
	EXEC msdb.dbo.sp_send_dbmail 
		@profile_name='DBA',
	@recipients='dbagroup@companyname.com',
		@subject= @MailSubject,
		@body= @AlertMessage
END
 
DROP TABLE #disk_free_space
 
 
EXEC msdb.dbo.sp_send_dbmail 
	@profile_name='DBA',
	@recipients='dbagroup@companyname.com',
	@subject= 'Drive space alert on [Database Name]',
	@body= 'I successfully ran. I alert you if the drive space is running low on C, K, L, W. If you did not receive other emails, you are probably fine. This email ensures that alerts are working.'

Executing SSIS Packages from SQL without SSIS

I wanted to automate collecting logs from some production servers that I manage and the best way to get remote data is using SSIS. However, we typically do not install SQL Server Integration Services (SSIS) on our boxes – and I do not have access to any server that has this feature installed (its a new job, things are going to change!). So package store is not there, nor can you schedule the SSIS Package easily. DTEXEC came to rescue:

You can simply execute package via the SQL Server agent job by doing the following. Please note that you will have to have xp_cmdshell enabled if you want to use T-SQL.

 
EXEC xp_cmdshell 'dtexec /f "C:\SSISProjects\DBA\DBA\Test.dtsx"'

SSRS 2008 R2 Permissions

I spent a couple of hours this morning trying to figure out the permission with one of our SSRS 2008 R2 servers. Here is how the problem started, and the steps I took to fix the issue. When I RDP to the server, I can open the Reporting Services “Site Settings” and view the reports that I have published. However, when I am on my local computer, there was a permission issue which said the following:

“does not have required permissions. Verify that sufficient permissions have been granted and Windows User Account Control (UAC) restrictions have been addressed”

I was wondering what the problem was because I was already added to the “System Administrator” role in the site setting. That did not resolve the problem. After looking at the permission in the ReportServer Database, I added myself to RSExecRole. That still did not fix the problem. Finally it came down to Adding myself to the root level Folder Permission.

SSRS 2008 Permission