MSDB too large

C drive was growing very fast and then saw that the msdb  data file and log file was huge.

I shrank the msdb log file first.
I ran the following query to see the top higly populated table

SELECT object_name(i.object_id) as objectName,
i.[name] as indexName,
sum(a.total_pages) as totalPages,
sum(a.used_pages) as usedPages,
sum(a.data_pages) as dataPages,
(sum(a.total_pages) * 8) / 1024 as totalSpaceMB,
(sum(a.used_pages) * 8) / 1024 as usedSpaceMB,
(sum(a.data_pages) * 8) / 1024 as dataSpaceMB
FROM sys.indexes i
INNER JOIN sys.partitions p
ON i.object_id = p.object_id
AND i.index_id = p.index_id
INNER JOIN sys.allocation_units a
ON p.partition_id = a.container_id
GROUP BY i.object_id, i.index_id, i.[name]
ORDER BY sum(a.total_pages) DESC, object_name(i.object_id)
GO

Once running the query on MSDB , I found the follwing large tables

I ran the follwing to clear the history

use msdb
GO
exec sp_maintplan_delete_log @oldest_time = '05/12/2017'


This expands the log file so make sure you shrink file


DBCC SHRINKFILE(MSDBLog, 50) --50MB

GO


Then the following maintenance plan was scheduled to run every week.

Backup SSISDB

I came across a very good instructions on MSDN for this .

https://blogs.msdn.microsoft.com/mattm/2012/03/23/ssis-catalog-backup-and-restore/

SSIS Catalog – Backup and Restore

 

March 23, 2012By Kaarthik Sivashanmugam

 

 

 

 

Integration Services catalog (SSISDB) uses the encryption mechanism available in SQL Server to protect its sensitive data. So backup and restore of SSIS catalog across machines need some extra steps in addition to the usual backup and restore steps. This post describes all the steps involved in backup and restore of SSIS catalog.

Backup
1. Backup the master key used for encryption in SSISDB database and protect the backup file using a password. This is done using BACKUP MASTER KEY statement.

 USE SSISDB
 BACKUP MASTER KEY TO FILE = 'c:\DMK\SSISDB\key'
 ENCRYPTION BY PASSWORD = 'SS1SC@talogMKBKUP'

 This step is not necessary every time you do backup unless you have lost the file or the password or if you have changed the master key of the SSISDB database.
2. Backup SSISDB database using SQL Server Management Studio or BACKUP DATABASE statement in TSQL.
3. If SSISDB database will be restored in a SQL Server instance that never had SSIS catalog, follow steps 3.1. & 3.2. listed below in additional backup steps section.

Restore
4. If SSISDB database will be restored in a SQL Server instance that never had any SSIS catalog, follow preparatory steps 4.1. & 4.2. listed below in additional restore steps section.
5. Restore SSISDB database using SQL Server Management Studio or RESTORE DATABASE statement in TSQL.
6. If SSISDB is restored in a SQL Server instance that never had any SSIS catalog, follow steps 6.1. thru 6.3. listed below in additional restore steps section.
7. Restore backup of the master key from the source server using the backup file created in step 1 in Backup section above.

 USE SSISDB
 RESTORE MASTER KEY FROM FILE = 'D:\MasterKeyBackup\SSIS-Server1234\key'
 DECRYPTION BY PASSWORD = 'SS1SC@talogMKBKUP'
 ENCRYPTION BY PASSWORD = 'NewC@talogPassw0rd'
 FORCE
 

“SS1SC@talogMKBKUP” is the password used to protect the file containing backup of the master key and “NewC@talogPassw0rd” is the new password to encrypt database master key.
The warning reported when carrying out this step “The current master key cannot be decrypted. The error was ignored because the FORCE option was specified.” can be ignored.
Decrypting all keys using the currently active master key and re-encrypting using the restored master key is a resource-intensive operation should be scheduled when server usage is low.
SSIS catalog should be fully functional in the restored server now.

Additional Backup Steps
Following steps are necessary during backup if you will be restoring SSISDB database to a SQL Server instance where the catalog was never created.
3.1. Generate create script for sp_ssis_startup stored procedure which is created when SSIS catalog is created in a SQL Server. The script is generated using SQL Server Management Studio. Right-click on the stored procedure in master database in SSMS Object Explorer, select “Script Stored Procedure as” -> “CREATE TO” -> “File” options to save the script to a file (for example, sp_ssis_startup.sql).
3.2. Generate create script for Agent job [SSIS Server Maintenance Job] which is created in SQL Server Agent automatically when SSIS catalog is created in a SQL Server. The script is generated using SQL Server Management Studio. Right-click on the Agent job in SSMS Object Explorer, select “Script Job as” -> “CREATE TO” -> “File” options to save the job to a file (for example, ssis_server_maintenance_job.sql).

Additional Restore Steps
Following steps are necessary during restore if you are restoring SSISDB database to a SQL Server instance where the catalog was never created.
4.1. Enable CLR functionality in SQL Server that SSISDB database depends on.

 USE MASTER
 EXEC sp_configure 'clr enabled', 1
 RECONFIGURE

4.2. Create asymmetric key and unsafe assembly loading principal that SSISDB database depends on. The login is used only for granting permission and hence does not have to be mapped to a database user.

 USE MASTER
 CREATE ASYMMETRIC KEY MS_SQLEnableSystemAssemblyLoadingKey FROM EXECUTABLE FILE = 'C:\Program Files\Microsoft SQL Server\110\DTS\Binn\ISServerExec.exe'
 CREATE LOGIN MS_SQLEnableSystemAssemblyLoadingUser FROM ASYMMETRIC KEY MS_SQLEnableSystemAssemblyLoadingKey
 GRANT UNSAFE ASSEMBLY TO MS_SQLEnableSystemAssemblyLoadingUser

You may need to change the file system path to ISServerExec.exe depending on your install location.

6.1. Create login [##MS_SSISServerCleanupJobLogin##] using CREATE LOGIN TSQL statement. This login is for internal use in SSISDB maintenance.
6.2. Map SSISDB user ##MS_SSISServerCleanupJobUser## to server login ##MS_SSISServerCleanupJobLogin##

 USE SSISDB
 ALTER USER [##MS_SSISServerCleanupJobUser##] with LOGIN = [##MS_SSISServerCleanupJobLogin##]

6.3. Create startup stored procedure and agent job by executing scripts (sp_ssis_startup.sql  & ssis_server_maintenance_job.sql) created in steps 3.1 & 3.2 under additional backup steps section above in the master database in the SQL Server instance in which SSISDB database is restored. You need to update @servername parameter for sp_add_jobserver step in ssis_server_maintenance_job.sql with the name of the new server replacing old server name.

Work Adeventure: Backup database to network

I had a situation where I had to perform a backup of the database on a network. I performed the following

Mapped the  nwMachine\L share to Y drive.
Thus I had (\\nwMachine\L) Y:

If I do a backup at this point the SQL server will not see the mapped drive (Y:) yet .
I had to run few scripts to make SQL server see the mapped Y drive.

EXEC sp_configure 'advanced', 1

RECONFIGURE WITH override

GO

--to enable the xp_cmdshell
--once the job is done make sure this is turned off

EXEC sp_configure 'xp_cmdshell',1

RECONFIGURE WITH override

GO


now make sql server see the mapped drive
EXEC xp_cmdshell 'net use Y: \\nwMachine\L
GO



--EXEC xp_cmdshell 'net use <local mapped drive> (space)  <shared path>
 
--Once we get result as "command ran successfully" we are successful.

--Now in the backup wizard the Y drive should appear along with the other local drives.

--now turn off the xp_cmdshell

EXEC sp_configure 'xp_cmdshell',0

RECONFIGURE WITH override

GO

Create and restore database using backup files from the network

I had a project where I had to create and restore the database from the backup files in a network.
I achieved it by following these steps.

• I moved all the backup files from the network to the local server in a backup  folder.
• Created a table BackupFileList to store the current database name and the backup file names using the command shell script
• Executed a dynamic SQL script to create all the databases.
• Executed the dynamic script to restore all the databases I made use of the created table to build a dynamic query to find the database name and the backup file.

( Note I had to use the MOVE command in the restore script since the physical path of the data file and the log file were different in the local server than the other server from where the backup was done.)


First of all need to make sure that the 'xp_cmdshell' is enabled.


EXEC sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO

EXEC sp_configure 'xp_cmdshell',1
GO
RECONFIGURE
GO

I ran  the following query to see the  physical path of the back up sets that was moved over from the network.

restore filelistonly from disk V:\SQL\Backup\Daily\DB1_backup_2015_06_22_194001_8960340.bak'
with file = 1
go

[In SQL Server 2008 and later, you need to have CREATE DATABASE permission to get this information

 

 
If there are multiple backup sets on the backup device, specify the backup set number using the “WITH FILE = n” option.

 To specify the second backup set on the device run this code:

restore filelistonly from disk V:\SQL\Backup\Daily\DB1_backup_2015_06_22_194001_8960340.bak'
with file = 2
go ]


Script 1 :
Important!!!!
BEFORE RUNNING MAKE SURE THAT THE SERVICE ACCOUNT HAS PERMISSION IN THE PATH DIR.

--To create a backupfiles list and populate a table in the work db

--delete from HCFS_WORKDB..DBBackUpFilesList

CREATE TABLE #DirectoryList (
  line  VARCHAR(512))
DECLARE @f  table (Fname varchar(100))
DECLARE @Path varchar(256) = 'dir V:\SQL\Backup\Daily\'
DECLARE @Command varchar(1024) =  @Path+  '/b /a-d  /s *.bak ' --' /s /b /o:gn' --/a:d /s /b | sort'
PRINT @Command
INSERT #DirectoryList
EXEC MASTER.dbo.xp_cmdshell @Command
DELETE #DirectoryList WHERE  Line IS NULL

delete from #DirectoryList where ( line not like ('%2015_06_24%')) --keep the recent backup file

delete from #DirectoryList where line like '%system%'
delete from #DirectoryList where line like '%\Diffferntial%'
delete from #DirectoryList where line not like '%\client_%'
--select line from #DirectoryList
insert into  @f
select SUBSTRING(line,charindex('\client_',line,1)+1,512 ) from #DirectoryList
--capture the database name and the filename in the table
insert into DB_WORKDB..DBBackUpFilesList (DBname,BackupfileName)
select rtrim(left(SUBSTRING(fname ,charindex('\client_',fname,1)+1,100 ),charindex('_backup',SUBSTRING(fname ,charindex('\client_',fname,1)+2,100 ),1))),SUBSTRING(fname ,charindex('\client_',fname,1)+1,100 ) from @f
--select substring(SUBSTRING(line,charindex('\client_',line,1)+1,512 ),charindex('\',line,1)+1,512)  from #DirectoryList
DROP TABLE #DirectoryList
GO

 


-------------------------------------------------------------------------------

Script 2 : Create multiple databases using a cursor


declare @curClientCode as cursor ,@db varchar(100),
@sqlCreateDb nvarchar(max),
@dbName varchar(100)
SET @curClientCode = CURSOR FOR
--crete db for arm clients too
select name from sys.databases
 where name like 'client_%'

OPEN   @curClientCode
FETCH NEXT
FROM @curClientCode into @db
WHILE @@FETCH_STATUS = 0
BEGIN
SET @dbName = cast(@db as varchar)
SET @sqlCreateDb = 'USE  master' + CHAR(13) +
--'DROP DATABASE  ' +  @dbName + CHAR(13) +
' CREATE DATABASE ' + @dbName +  CHAR(13) +
' ON (NAME = ' + @dbName + '_data, FILENAME = ''E:\SQL\Data\' + @dbName +'.mdf'', Size = 100MB,MAXSIZE=unlimited, FILEGROWTH = 10)' + char(13) +
' LOG ON (NAME = ' + @dbName +  '_log, FILENAME = ''L:\SQL\Data\' + @dbName +'.ldf'', Size = 5MB,MAXSIZE=500MB, FILEGROWTH = 10); '
   

print @sqlCreateDb
exec sp_executesql @sqlCreateDb;
FETCH NEXT
FROM @curClientCode into @db
END
CLOSE @curClientCode
DEALLOCATE @curClientCode


---------------------------------------------------------
Script 3 : Restore the database with MOVE command from the file
The backup files name were found from the table in the DB_workdb that was used earlier to capture the database name and the backup files


 use master
Go
declare @curClientCode as cursor ,@db varchar(10),
@sqlRestoreDb nvarchar(max),
@dbName varchar(20),
@backupFile varchar(100)
SET @curClientCode = CURSOR FOR
select name from sys.databases
 where name like 'client_%'
OPEN   @curClientCode
FETCH NEXT
FROM @curClientCode into @db
WHILE @@FETCH_STATUS = 0
BEGIN
SET @dbName = @db
SELECT  @backupFile = BackupFileName from HCFS_WORKDB..dbBackUpFilesList where dbname = @dbName
SET @sqlRestoreDb = 'ALTER DATABASE ' +  @dbName + char(13) +
' SET SINGLE_USER WITH  ROLLBACK IMMEDIATE ' + char(13)  + char(13)
SET @sqlRestoreDb  = @sqlRestoreDb + ' RESTORE DATABASE ' +  @dbName + char(13) +
' FROM DISK = ''V:\SQL\Backup\Daily\' + @dbName + '\'+  + @backupFile  + ''''+ char(13)+
 ' WITH  MOVE N''' + @dbName + ''' TO  N''E:\SQL\Data\' + @dbName + '.mdf'','  + char(13)+
 ' MOVE N''' + @dbName + '_log'' TO  N''L:\SQL\Data\' + @dbName + '.ldf'','  + char(13)+
 ' REPLACE, STATS = 10 ' +  char(13)
 
SET @sqlRestoreDb  = @sqlRestoreDb  + 'ALTER DATABASE ' +  @dbName + char(13) +
' SET MULTI_USER WITH  ROLLBACK IMMEDIATE ' + char(13)
 

print @sqlRestoreDb
exec sp_executesql @sqlRestoreDb;
FETCH NEXT
FROM @curClientCode into @db
END
CLOSE @curClientCode
DEALLOCATE @curClientCode

GO
      

restore msdb database

USE master
GO
--just to verify if the below command succeeds the restore should succeed too
RESTORE VERIFYONLY
FROM DISK = N'V:\SANSQL01Backups\Daily\msdb\msdb_backup_2014_10_05_194001_4668459.bak'
--restore the database
RESTORE DATABASE [msdb]
FROM DISK = N'V:\SANSQL01Backups\Daily\msdb\msdb_backup_2014_10_05_194001_4668459.bak'
WITH REPLACE
GO
--msdb database was curripted and when to suspect mode. So fixed it by restoring a copy of a backup

Recovery database from suspect mode

The database was in suspect mode, inaccessible Ran the following query Then restored from the latest full back up

EXEC sp_resetstatus client_367
ALTER DATABASE client_367 SET EMERGENCY
use client_367
DBCC checkdb
ALTER DATABASE client_367 SET SINGLE_USER WITH ROLLBACK IMMEDIATE
DBCC CheckDB ('client_367', REPAIR_ALLOW_DATA_LOSS)
ALTER DATABASE client_367 SET MULTI_USER

Check the backup process

Script to see how the backup is progressing.

SELECT percent_complete , (estimated_completion_time/1000)/60 Estimated_completion_time_Mins , (total_elapsed_time/1000)/60 Total_Elapsed_Time_Mins ,DB_NAME(Database_id) DBName ,* FROM
sys.dm_exec_requests WHERE session_id = (the spid for running the back up - you can find by running sp_who2 active)