Troubleshooting SQL Server AlwaysOn - vijayrod



Troubleshooting SQL Server AlwaysOnApplies to: SQL Server 2012, SQL Server 2014.ContentsSQL Server AlwaysOn errors (errors in SQL Server)SQL Server AlwaysOn (errors in SQL Server Configuration Manager)SQL Server AlwaysOn (New Availability Group wizard errors)SQL Server AlwaysOn (SSMS errors)SQL Server AlwaysOn (performance issues)SQL Server AlwaysOn errors (network)SQL Server AlwaysOn errors (Application issues)SQL Server AlwaysOn errors (cluster log)SQL Server AlwaysOn errors (event log messages)SQL Server AlwaysOn errors (Windows messages obtained through “net helpmsg”)SQL Server AlwaysOn errors (WMI errors)SQL Server AlwaysOn (patching/updates in SQL 2012)Tips to search this document: Try searching on error number, or on part of error message, or on performance issue like “hang”, wait type like “HADR_SYNC_COMMIT”, or on database state like “RECOVERY_PENDING” or “RESOLVING” (without quotes).Ideally this information is available through below logs:Executing TSQL etc. commands through SQL Management Studio.SQL ERRORLOG filesCluster logs, cluster validation report.SQL default traces, system_health_* files (or Output of “EXEC sp_server_diagnostics;”), AlwaysOn files in LOG folderEvent logs.SQL Server AlwaysOn errors (errors in SQL Server)errorseveritydescription95717Database '%.*ls' is enabled for database mirroring or has joined an availability group. The name of the database cannot be changed.97614The target database, '%.*ls', is participating in an availability group and is currently not accessible for queries. Either data movement is suspended or the availability replica is not enabled for read access. To allow read-only access to this and other If dm_hadr_availability_replica_states.role_desc=3(INVALID), then check service broker. Because Availability group is using Service Broker to communicate between SQL Server then need to check Service Broker first. Open SSMS>“Server Objects” > “Endpoints”, there are two “Service Broker” folder>Expand both, and if the state is Stopped or Disabled, then can change the state to Started.Similar steps as in error 35250.97814The target database ('%.*ls') is in an availability group and is currently accessible for connections when the application intent is set to read only. For more information about application intent, see SQL Server Books Online.97914The target database ('%.*ls') is in an availability group and currently does not allow read only connections. For more information about application intent, see SQL Server Books Online.98214Unable to access the '%.*ls' database because no online secondary replicas are enabled for read-only access. Check the availability group configuration to verify that at least one secondary replica is configured for read-only access. Wait for an enabled r98314Unable to access database '%.*ls' because its replica role is RESOLVING which does not allow connections. Try the operation again later.Check SQL ERRORLOGs, event logs, etc. for network, storage related messages etc.Check for issues as in error 19406. Search for “RESOLVING_NORMAL” in this document.140816The remote copy of database "%.*ls" is not recovered far enough to enable database mirroring or to join it to the availability group. You need to apply missing log records to the remote database by restoring the current log backups from the principal/prim140916Database "%.*ls" requires database logs to be restored either on the future mirror database before you can enable database mirroring or on a secondary availability database before you can join it to the availability group. Restore current log backups from146516Database "%.*ls" database is not in full recovery mode on each of the server instances. The full recovery model is required for a database to participate in database mirroring or in an availability group.146616Database "%.*ls" is read-only on one of the server instances which is incompatible with participating in database mirroring or in an availability group. Set the database to read-write mode, and retry the operation.146716Database "%.*ls" database is in emergency or suspect mode on one of the partners which is incompatible with participating in database mirroring or in an availability group.146816The operation cannot be performed on database "%.*ls" because it is involved in a database mirroring session or an availability group. Some operations are not allowed on a database that is participating in a database mirroring session or in an availabilit146916Database "%.*ls" is an auto-close database on one of the partnerswhich is incompatible with participating in database mirroring or in an availability group.148010The %S_MSG database "%.*ls" is changing roles from "%ls" to "%ls" because the mirroring session or availability group failed over due to %S_MSG. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.148816Database "%.*ls" database is in single user mode which is incompatible with participating in database mirroring or in an availability group. Set database to multi-user mode, and retry the operation.183316File '%ls' cannot be reused until after the next BACKUP LOG operation. If the database is participating in an availability group, a dropped file can be reused only after the truncation LSN of the primary availability replica has passed the drop LSN of the310416RESTORE cannot operate on database '%ls' because it is configured for database mirroring or has joined an availability group. If you intend to restore the database, use ALTER DATABASE to remove mirroring or to remove the database from its availability gro340210The database '%ls' is marked %ls and is in a state that does not allow recovery to be run.If the second %ls is RESTORING, then one option is below.1) Asked the customer to stop the SQL Server Service.2) Had the customer rename the actual files on disk with a "_old" suffix so that the sql server would not find them and have the db come up suspect.3) Asked the customer to start the SQL Server Service.Once the server came back up we were able to delete the database.4) Had the customer create a new database with the same name and location as the original database. Used the output from sysaltfiles to verify this.5) Had the customer stop the SQL Server Service.6) Had the customer rename these new database files by appending a "_new" on the end of the file.7) Had the customer take the old database files with the "_old" suffix and changed them back to the original name.8) Had the customer start the SQL Server Service.363316The operating system returned the error '%ls' while attempting '%ls' on '%ls' at '%hs'(%d).Verify error in %ls at DOS prompt with “net helpmsg”. Then proceed depending on this error.375216The database '%.*ls' is currently joined to an availability group. Before you can drop the database, you need to remove it from the availatility group.Before you can drop the database, you need to remove it from the availability group. If AG exists, try ALTER AVAILABILITY GROUP [AGname] REMOVE DATABASE [DbName]. If AG does not exist, try removing with ALTER DATABASE [DbName] SET HADR OFF.If replica_id and group_database_id in sys.databases show as null, then this may be due to earlier meta-data when database was part of an AG so stop SQL Server, rename the database files and start SQL Server. Then you should be able to remove the database.552916Failed to remove a FILESTREAM file. The database is a primary database in an availability group. Wait for the FILESTREAM data files to be hardened on every secondary availability replica. Then retry the drop file operation.1078616The ALTER AVAILABILITY GROUP command failed because it contained multiple MODIFY REPLICA options: %ls. Enter a separate ALTER AVAILABILITY GROUP ... MODIFY REPLICA command for each replica option that you want to modify.1325110availability group1940116The READ_ONLY_ROUTING_URL '%.*ls' specified for availability replica '%.*ls' is not valid. It does not follow the required format of 'TCP://system-address:port'. For information about the correct routing URL format, see the CREATE AVAILABILITY GROUP docum1940316The availability replica '%.*ls' specified in the READ_ONLY_ROUTING_LIST for availability replica '%.*ls' does not exist. Only availability replicas that belong to the specified availability group '%.*ls' can be added to this list. To get the names of ava1940516Failed to create, join or add replica to availability group '%.*ls', because node '%.*ls' is a possible owner for both replica '%.*ls' and '%.*ls'. If one replica is failover cluster instance, remove the overlapped node from its possible owners and try agMay occur if AG involves two or more SQL failover instances. Check possible owners of SQL resource on both/more so as to verify no node name overlap. There should not be node-overlap.TSQL: SELECT * FROM sys.dm_os_cluster_nodes;Powershell: get-clusterownernode -resource "XXXBSQL"Powershell: set -clusterownernode -resource "XXXASQL" -owners NODE1,NODE21940610The state of the local availability replica in availability group '%.*ls' has changed from '%ls' to '%ls'. The replica state changed because of either a startup, a failover, a communication issue, or a cluster error. For more information, see the availabiIf the “changed to” state is PRIMARY_PENDING, then check sys.dm_hadr_database_replica_states. If database_state_desc = RECOVERY_PENDING (synchronization_health_desc will be NOT_HEALTHY), then try “ALTER DATABASE db SET HADR RESUME;”. Else if this is the only replica (no secondary replica), then (consider first taking a database snapshot as a backup if required) try “ALTER DATABASE db SET HADR OFF;” to remove AlwaysOn so as to then manually recover database with SQL service or database restart (ALTER DATABASE ONLINE).TSQL: ALTER DATABASE DbName SET HADR RESUME;TSQL: ALTER DATABASE DbName SET ONLINE;TSQL: RESTORE DATABASE DbName WITH RECOVERY;If the “changed to” state is RESOLVING_NORMAL, check for additional messages. Also check for issues as in (Troubleshooting automatic failover problems in SQL Server 2012 AlwaysOn environments).If “change to” state is PRIMARY_NORMAL / SECONDARY_NORMAL, then this may this indicates successful failover. Check for additional messages if failover was not expected.1940716The lease between availability group '%.*ls' and the Windows Server Failover Cluster has expired. A connectivity issue occurred between the instance of SQL Server and the Windows Server Failover Cluster. To determine whether the availability group is fail1945216The availability group listener (network name) with Windows Server Failover Clustering resource ID '%s', DNS name '%s', port %hu failed to start with a permanent error: %u. Verify port numbers, DNS names and other related network configuration, then retry1945316The availability group listener (network name) with Windows Server Failover Clustering resource ID '%s', DNS name '%s', port %hu failed to start with this error: %u. Verify network and cluster configuration and logs.1945416The availability group listener (network name) with Windows Server Failover Clustering resource ID '%s', DNS name '%s', port %hu failed to stop with this error: %u. Verify network and cluster configuration and logs.1945516The WSFC cluster does not have a public cluster network with an IPv4 subnet. This is a requirement to create an availability group DHCP listener. Configure a public network for the cluster with an IPv4 subnet, and try to create the listener.1945616None of the IP addresses configured for the availability group listener can be hosted by the server '%.*ls'. Either configure a public cluster network on which one of the specified IP addresses can be hosted, or add another listener IP address which can bAdded a new IP address (with different subnet) manually to the existing Listener [to be tested].1945816The WSFC nodes that host the primary and secondary replicas belong to different subnets. DHCP across multiple subnets is not supported for availability replicas. Use the static IP option to configure the availability group listener.1946016The availability group listener with DNS name '%.*ls' is configured to use DHCP. For listeners with this configuration, IP addresses cannot be added through SQL Server. To add IP addresses to the listener, drop the DHCP listener and create it again config1946816The listener with DNS name '%.*ls' for the availability group '%.*ls' is already listening on the TCP port %u. Please choose a different TCP port for the listener. If there is a problem with the listener, try restarting the listener to correct the problem1946916The specified listener with DNS name, '%.*ls', does not exist for the Availability Group '%.*ls'. Use an existing listener, or create a new listener.1947116The WSFC cluster could not bring the Network Name resource with DNS name '%ls' online. The DNS name may have been taken or have a conflict with existing name services, or the WSFC cluster service may not be running or may be inaccessible. Use a differentVerify cluster name has "Create Computer objects" permission in AD. Steps in error 1193.Check for associated errors like error 19476.[Below to be tested].1. Temporarily move the CNO account into the Computers container [optional].2. Log into one of the cluster nodes with a domain account that had the Reset Password right in the domain 3. Take the cluster Network Name offline. If this does not work to solve the problem, simulate failures for the cluster Network Name resource until it is in a permanent failed state4. Once the resource was in an offline state, right-click on the resource, choose More Actions and then click Repair5. The previous action caused the password for the CNO to be reset in the domain6. Restart the cluster service on all nodes.1947616The attempt to create the network name and IP address for the listener failed. The WSFC service may not be running or may be inaccessible in its current state, or the values provided for the network name and IP address may be incorrect. Check the state of the WFSC cluster and validate the network name and IP address with the network administrator.Check for associated errors like error 41009.1947716The availability group '%.*ls' already has a listener with DNS name '%.*ls'. Availability groups can have only one listener. Use the existing listener, or drop the existing listener and create a new one.1948616The configuration changes to the availability group listener were completed, but the TCP provider of the instance of SQL Server failed to listen on the specified port [%.*ls:%d]. This TCP port is already in use. Reconfigure the availability group listener2187216The availability group associated with Virtual Network Name '%s' has no replicas. 2188016The virtual network name '%s' has been used to identify the redirected publisher for original publisher '%s' and database '%s'. The availability group associated with this virtual network name, however, does not include the publisher database. 2188216The database '%s' at the redirected publisher '%s' for original publisher '%s' and database '%s' belongs to a HADRon availability group and must be redirected to its associated HADRon Virtual Network Name. 2188316The query at the redirected publisher '%s' to determine whether the publisher database '%s' belonged to an availability group failed with error '%d', error message '%s'. 2188416The query at the redirected publisher '%s' to determine the health of the availability group associated with publisher database '%s' failed with error '%d', error message '%s'. 2188716The query at the redirected publisher '%s' to determine whether the publisher database '%s' belonged to an availability group failed with error '%d', error message '%s'. 2189216Unable to query sys.availability_replicas at the availability group primary associated with virtual network name '%s' for the server names of the member replicas: error = %d, error message = %s.',3344516The database '%.*s' is a readable secondary database in an availability group and cannot be enabled for FILESTREAM non-transacted access.3344616The FILESTREAM database configuration cannot be changed for database '%.*s'. The database is either a mirror database in Database Mirroring, or is in a secondary replica of an AlwaysOn availability group. Connect to the server instance that hosts the pr3344910FILESTREAM File I/O access is enabled, but no listener for the availability group is created. A FILESTREAM PathName will be unable to refer to a virtual network name (VNN) and, instead, will need to refer to a physical Windows Server Failover Clustering (3345010FILESTREAM File I/O access is enabled. One or more availability groups ('%ls') currently do not have a listener. A FILESTREAM PathName will be unable to refer to a virtual network name (VNN) and, instead, will need to refer to a physical Windows Server F3520210A connection for availability group '%ls' from availability replica '%ls' with id [%ls] to '%ls' with id [%ls] has been successfully established. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.3520516Could not start the AlwaysOn Availability Groups transport manager. This failure probably occurred because a low memory condition existed when the message dispatcher started up. If so, other internal tasks might also have experienced errors. Check the SQL3520610A connection timeout has occurred on a previously established connection to availability replica '%ls' with id [%ls]. Either a networking or a firewall issue exists or the availability replica has transitioned to the resolving role.If there is a login failed for domain\node$ account around same time, then it may be because NT AUTHORITY\SYSTEM account lacked the required permissions to failover the availability group. To failover the group to the other node, NT AUTHORITY\SYSTEM must have permission to connect to SQL (CONNECT SQL), failover the availability group (ALTER ANY AVAILABILITY GROUP) and execute sp_server_diagnostics (VIEW SERVER STATE). The NT AUTHORITY\SYSTEM account is used to automatically execute sp_server_diagnostics. Another option is to add the account to sysadmin role in SQL Server.CREATE LOGIN [NT AUTHORITY\SYSTEM] FROM WINDOWS WITH DEFAULT_DATABASE=[master], DEFAULT_LANGUAGE=[us_english];ALTER SERVER ROLE [sysadmin] ADD MEMBER [NT AUTHORITY\system];3520716Connection attempt on availability group id '%ls' from replica id '%ls' to replica id '%ls' failed because of error %d, severity %d, state %d.3521016Failed to modify options for availability replica '%.*ls' in availability group '%.*ls'. The specified availability group does not contain an availability replica with specified name. Verify that availability group name and availability replica name are c3521216The %ls operation is not allowed by the current availability-group configuration. This operation would exceed the maximum number of %d synchronous-commit availability replicas in availability group '%.*ls'. Change one of the existing synchronous-commit 3521316The %ls operation is not allowed by the current availability-group configuration. This operation would exceed the maximum number of %d automatic failover targets in availability group '%.*ls'. Change one of the existing synchronous-commit replicas to th3521716 The thread pool for AlwaysOn Availability Groups was unable to start a new worker thread because there are not enough available worker threads. This may degrade AlwaysOn Availability Groups performance. Use the "max worker threads" configuration option3522016Could not process the operation. AlwaysOn Availability Groups replica manager is waiting for the host computer to start a Windows Server Failover Clustering (WSFC) cluster and join it. Either the local computer is not a cluster node, or the local cluster 3522116Could not process the operation. AlwaysOn Availability Groups replica manager is disabled on this instance of SQL Server. Enable AlwaysOn Availability Groups, by using the SQL Server Configuration Manager. Then, restart the SQL Server service, and retry t3522216Could not process the operation. AlwaysOn Availability Groups does not have permissions to access the Windows Server Failover Clustering (WSFC) cluster. Disable and re-enable AlwaysOn Availability Groups by using the SQL Server Configuration Manager. The3522316Cannot add %d availability replica(s) to availability group '%.*ls'. The availability group already contains %d replica(s), and the maximum number of replicas supported in an availability group is %d.3522416Could not process the operation. AlwaysOn Availability Groups failed to load the required Windows Server Failover Clustering (WSFC) library. Verify that the computer is a node in a WSFC cluster. You will need to restart the SQL Server instance to reload3522516Could not process the operation. The instance of SQL Server is running under WOW64 (Windows 32-bit on Windows 64-bit), which does not support AlwaysOn Availability Groups. Reinstall SQL Server in the native 64-bit edition, and re-enable AlwaysOn Availab3522616Could not process the operation. AlwaysOn Availability Groups has not started because the instance of SQL Server is not running as a service. Restart the server instance as a service, and retry the operation.3522816The attempt to set the failure condition level for availability group '%.*ls' failed. The specified level value is out of the valid range [%u, %u]. Reenter the command specifying a valid failure condition level value.3522916The attempt to set the health check timeout value for availability group '%.*ls' failed. The specified timeout value is less than %u milliseconds. Reenter the command specifying a valid health check timeout value.3523316Cannot create an availability group containing %d availability replica(s).3523716None of the specified replicas for availability group %.*ls maps to the instance of SQL Server to which you are connected. Reenter the command, specifying this server instance to host one of the replicas. This replica will be the initial primary replica. 3523816Database '%.*ls' cannot be added to availability group '%.*ls'. The database does not exist on this SQL Server instance. Verify that the database name is correct, then retry the operation.3523916The ALTER DATABASE <database-name> SET HADR SUSPEND (or SET HADR RESUME) statement failed on database '%.*ls' of availability group '%.*ls''. Either the availability group does not contain the specified database, or the database has not joined the availab3524016Database '%.*ls' cannot be joined to or unjoined from availability group '%.*ls'. This operation is not supported on the primary replica of the availability group. TSQL to unjoin is “ALTER DATABASE db SET HADR OFF”. Note that failing over thus making it a secondary, and then retrying ALTER DATABASE HADR is not an option, because the alter database will then give error 921 that database has not been recovered yet.If this replica can be made a secondary, then fail over thus making it a secondary, and then drop replica when in this secondary role. TSQL is “ALTER AVAILABILITY GROUP AgName REMOVE REPLICA ON ‘ReplicaName’;”.If no secondary replica for failover and getting this error on primary, then “DROP AVAILABILITY GROUP AgName”. This will automatically drop listener. If do not want listener dropped, then create a temporary AG adding same name, then try the “DROP AVAILABILITY GROUP”.3524216Cannot complete this ALTER DATABASE <database-name> SET HADR operation on database '%.*ls'. The database is not joined to an availability group. After the database has joined the availability group, retry the command.3524316Failed to set resource property '%.*ls' for availability group '%.*ls'. The operation encountered SQL Server error %d. When the cause of the error has been resolved, retry the ALTER AVAILABILITY GROUP command later.3524416Database '%.*ls' cannot be added to availability group '%.*ls'. The database is currently joined to another availability group. Verify that the database name is correct and that the database is not joined to an availability group, then retry the operati3524616Failed to create the availability group. A SQL Server instance name could not be validated because the dynamic link library (DLL) file '%ls' could not be located (Windows System Error %d). Verify that the specified server instance exists. If it exists, th3524716Failed to create the availability group. A SQL Server instance name could not be validated because the dynamic link library (DLL) file '%ls' could not be loaded (Windows System Error %d).3524916An attempt to add or join a system database, '%.*ls', to an availability group failed. Specify only user databases for this operation.3525016The connection to the primary replica is not active. The command cannot be processed.Check SQL ERRORLOG to verify SQL is listening on the ports. If SQL is listening on port, check if SQL is listening on all IPs (e.g. not “Server is listening on [ 192.192.192.192 <ipv4> 5022]”) (should ideally be “Server is listening on [ 'any' <ipv4> 1433]”). If not listening on any/all IPs for port, then two options. One option is to recreate hadr_endpoint to listen on ALL ip. Second option is to enter the replication adapter IP in DNS for the configured endpoint).Check ping, telnet, nslookup to replica(s). If firewall exception is required, then create new inbound rule for TCP port 5022.3525416An error occurred while accessing the availability group metadata. Remove this database or replica from the availability group, and reconfigure the availability group to add the database or replica again. For more information, see the ALTER AVAILABILITY G3525716 AlwaysOn Availability Groups Send Error (Error code 0x%X, "NOT OK") was returned when sending a message for database ID %d. If the partner is running and visible over the network, retry the command using correctly configured partner-connection parameters3525916 Database '%.*ls' is already participating in a different availability group. 3526016 During an attempted database recovery, an availability database manager was not found for database id %d with availability group ID %d and group database ID %ls. Recovery was terminated. The most likely cause of this error is that the availability group3526116 Attempt to perform an AlwaysOn Availability Groups operation on a system database, '%ls', failed. System databases are not supported by AlwaysOn Availability Groups.3526217Skipping the default startup of database '%.*ls' because the database belongs to an availability group (Group ID: %d). The database will be started by the availability group. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.3526410AlwaysOn Availability Groups data movement for database '%.*ls' has been suspended for the following reason: "%S_MSG" (Source ID %d; Source string: '%.*ls'). To resume data movement on the database, you will need to resume the database manually. For inforLink redirects to and the mention steps should be followed to resume data movement.SSMS: AlwaysOn High Availability node>Availability Groups node>expand availability group>Availability Databases node>right-click database>Resume Data Movement>OK.TSQL on secondary: ALTER DATABASE database_name SET HADR RESUME;3526510AlwaysOn Availability Groups data movement for database '%.*ls' has been resumed. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.3526610AlwaysOn Availability Groups connection with %S_MSG database established for %S_MSG database '%.*ls' on the availability replica with Replica ID: {%.8x-%.4x-%.4x-%.2x%.2x-%.2x%.2x%.2x%.2x%.2x%.2x}. This is an informational message only. No user action is This is an informational message only. If there is a problem, then look for additional messages.3526710AlwaysOn Availability Groups connection with %S_MSG database terminated for %S_MSG database '%.*ls' on the availability replica with Replica ID: {%.8x-%.4x-%.4x-%.2x%.2x-%.2x%.2x%.2x%.2x%.2x%.2x}. This is an informational message only. No user action is rThis is an informational message only. If there is a problem, then look for additional messages.3527516A previous RESTORE WITH CONTINUE_AFTER_ERROR operation or being removed while in the SUSPECT state from an availability group left the '%.*ls' database in a potentially damaged state. The database cannot be joined while in this state. Restore the database3527617Failed to allocate and schedule an AlwaysOn Availability Groups task for database '%ls'. Manual intervention may be required to resume synchronization of the database. If the problem persists, you might need to restart the local instance of SQL Server.3527916The attempt to join database '%.*ls' to the availability group was rejected by the primary database with error '%d'. For more information, see the SQL Server error log for the primary replica.3528016Database '%.*ls' cannot be added to availability group '%.*ls'. The database is already joined to the specified availability group. Verify that the database name is correct and that the database is not joined to an availability group, then retry the ope3528116Database '%.*ls' cannot be removed from availability group '%.*ls'. The database is not joined to the specified availability group. Verify that the database name and the availability group name are correct, then retry the operation.3528216Availability replica '%.*ls' cannot be added to availability group '%.*ls'. The availability group already contains an availability replica with the specified name. Verify that the availability replica name and the availability group name are correct, t3528316Availability replica '%.*ls' cannot be removed from availability group '%.*ls'. The availability group does not contain an availability replica with the specified name. Verify that the availability replica name is correct, then retry the operation.3528416Availability replica '%.*ls' cannot be removed from availability group '%.*ls', because this replica is on the local instance of SQL Server. If the local availability replica is a secondary replica, connect to the server instance that is currently hostin3528716AlwaysOn Availability Groups transport for availability database "%.*ls" failed to decompress the log block whose LSN is %S_LSN. This error can be caused by a corrupt network packet or a compression version mismatch. The database replica has been put in3528816AlwaysOn Availability Groups log apply for availability database "%.*ls" has received an out-of-order log block. The expected LSN was %S_LSN. The received LSN was %S_LSN. The database replica has been put into the SUSPENDED state. Resume the availabil3529910Nonqualified transactions are being rolled back in database %.*ls for an AlwaysOn Availability Groups state change. Estimated rollback completion: %d%%. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4100916The Windows Server Failover Clustering (WSFC) resource control API returned error code %d. The WSFC service may not be running or may not be accessible in its current state, or the specified arguments are invalid. For information about this error code,Net helpmsg on the %d value, and proceed accordingly. May also be accompanied by Microsoft-Windows-FailoverClustering messages in event log. Check cluster log.If there error is in cluster in Windows Azure, then it may be because Windows Azure currently does not support more than one IP address per VM, which is required by the availability group listener to resolve to the availability group owner node. This is documented in (High Availability and Disaster Recovery for SQL Server in Windows Azure Virtual Machines).4101516Failed to obtain the Windows Server Failover Clustering (WSFC) node handle (Error code %d). The WSFC service may not be running or may not be accessible in its current state, or the specified cluster node name is invalid. For information about this erroCheck if Windows cluster service is helpmsg on the %d value, and proceed accordingly.4103916An availability group replica already exists on the Windows Server Failover Clustering (WSFC) node '%.*ls'. Each WSFC node can contain only one replica of an availability group. Please choose another WSFC node to host the new replica.4104016Failed to remove the availability group replica '%.*ls' from availability group '%.*ls'. The availability group does not contain a replica with the specified name. Verify the availability group and replica names and then retry the operation.4104216The availability group '%.*ls' already exists. This error could be caused by a previous failed CREATE AVAILABILITY GROUP or DROP AVAILABILITY GROUP operation. If the availability group name you specified is correct, try dropping the availability group aIf AG still exists, DROP AVAILABILITY GROUP.Open regedit, backup the entry from the old Availability Group that starts with HADR_Ag_???? key, then delete this key [to be tested].Try with new name for Availability Group.4104316For availability group '%.*ls', the value of the name-to-ID map entry is invalid. The binary value should contain a Windows Server Failover Clustering (WSFC) resource ID, a WSFC group ID, and their corresponding lengths in characters. The availability g4104416Availability group name to ID map entry for availability group '%.*ls' cannot be found in the Windows Server Failover Clustering (WSFC) store. The availability group name may be incorrect, or the availability group may not exist in this Windows Server Fa4104516Cannot add database '%.*ls' to the availability group '%.*ls', because there is already a database with the same name in the availability group. Please verify that the database and availability group names specified are correct.4104616Cannot add replica '%.*ls' to the availability group '%.*ls', because there is already a replica with the same name in the availability group. Please verify the replica and availability group names specified are correct.4104810AlwaysOn Availability Groups: Local Windows Server Failover Clustering service has become unavailable. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4104910AlwaysOn Availability Groups: Local Windows Server Failover Clustering node is no longer online. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4105010AlwaysOn Availability Groups: Waiting for local Windows Server Failover Clustering service to start. This is an informational message only. No user action is required.Notice this error says cluster “service” not node.Check services.msc if cluster service is started on that node. If not started, then start the service.This is an informational message only. If there is a problem, then look for additional messages.4105110AlwaysOn Availability Groups: Local Windows Server Failover Clustering service started. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4105210AlwaysOn Availability Groups: Waiting for local Windows Server Failover Clustering node to start. This is an informational message only. No user action is required.Notice this error says cluster “node” not service.If node is paused in Failover Cluster Manager, resume it.This is an informational message only. If there is a problem, then look for additional messages.4105310AlwaysOn Availability Groups: Local Windows Server Failover Clustering node started. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4105410AlwaysOn Availability Groups: Waiting for local Windows Server Failover Clustering node to come online. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4105510AlwaysOn Availability Groups: Local Windows Server Failover Clustering node is online. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4105616Availability replica '%.*ls' of availability group '%.*ls' cannot be brought online on this SQL Server instance. Another replica of the same availability group is already online on the local Windows Server Failover Clustering (WSFC) node. Each WSFC node4105810AlwaysOn: The local replica of availability group '%.*ls' is starting. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4105910AlwaysOn: Availability group '%.*ls' was removed while the availability replica on this instance of SQL Server was offline. The local replica will be removed now. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4106016The Cyclic Redundancy Check (CRC) value generated for the retrieved availability group configuration data from the Windows Server Failover Clustering (WSFC) store does not match that stored with the data for the availability group with ID '%.*ls'. The av4106110AlwaysOn: The local replica of availability group '%.*ls' is stopping. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4106216The ID of availability group '%.*ls' in local data store is inconsistent with that in the Windows Server Failover Clustering (WSFC) data store. The availability group may have been dropped and recreated while the SQL Server instance was offline, or while4106316Windows Server Failover Clustering (WSFC) detected that the availability group resource with ID '%.*ls' was online when the availability group was not actually online. The attempt to synchronize the WSFC resource state with the availability group state f4106616Cannot bring the Windows Server Failover Clustering (WSFC) resource (ID '%.*ls') online (Error code %d). The WSFC service may not be running or may not be accessible in its current state, or the WSFC resource may not be in a state that could accept the rIn DOS prompt, run “Net helpmsg” on the %d value, and proceed accordingly.4106916The existence of availability group data for the availability group '%.*ls' in the Windows Server Failover Clustering (WSFC) store could not be determined. The local WSFC node may be down, or a previous CREATE AVAILABILITY GROUP or DROP AVAILABILITY GROU4107016Configuration data for the availability group with Windows Server Failover Clustering (WSFC) resource ID '%.*ls' is not found in the WSFC data store. The availability group may have been dropped, or a previous CREATE AVAILABILITY GROUP or DROP AVAILABILI4107116Cannot read the persisted configuration of AlwaysOn availability group with corresponding Windows Server Failover Clustering (WSFC) resource ID '%.*ls'. The persisted configuration is written by a higher-version SQL Server that hosts the primary availabi4107216The ID of availability group '%.*ls' in local data store does not exist in the Windows Server Failover Clustering (WSFC) data store. The availability group may have been dropped but the current WSFC node was not notified. To resolve this error, try to r4107316The database '%.*ls' cannot be removed from availability group '%.*ls'. This database does not belong to the availability group.4107410AlwaysOn: The local replica of availability group '%.*ls' is preparing to transition to the primary role in response to a request from the Windows Server Failover Clustering (WSFC) cluster. This is an informational message only. No user action is requiredThis is an informational message only. If there is a problem, then look for additional messages.4107510AlwaysOn: The local replica of availability group '%.*ls' is preparing to transition to the resolving role in response to a request from the Windows Server Failover Clustering (WSFC) cluster. This is an informational message only. No user action is requirThis is an informational message only. If there is a problem, then look for additional messages.4107610AlwaysOn: Availability group '%.*ls' is going offline because it is being removed. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4108016Failed to delete SQL Server instance name to Windows Server Failover Clustering node name map entry for the local availability replica of availability group '%.*ls'. The operation encountered SQL Server error %d and has been terminated. Refer to the SQL4108116Failed to destroy the Windows Server Failover Clustering group corresponding to availability group '%.*ls'. The operation encountered SQL Server error %d and has been terminated. Refer to the SQL Server error log for details about this SQL Server error 4108910AlwaysOn Availability Groups startup has been cancelled, because SQL Server is shutting down. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4109110AlwaysOn: The local replica of availability group '%.*ls' is going offline because either the lease expired or lease renewal failed. This is an informational message only. No user action is required.Lease expired event from the cluster. Lease cannot be renewed maybe because the SQL Server was busy. Will need to checks default traces etc. General suggestions include updating statistics, index rebuild, set max memory if not set.Possible causes include loss of lease, possible network issues and sp_server_diagnostic query timeout.This is an informational message only. If there is a problem, then look for additional messages.4109210AlwaysOn: The availability replica manager is going offline because %ls. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4109310AlwaysOn: The local replica of availability group '%.*ls' is going offline because the corresponding resource in the Windows Server Failover Clustering (WSFC) cluster is no longer online. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4109410AlwaysOn: The local replica of availability group '%.*ls' is restarting because the existing primary replica restarted or the availability group failed over to a new primary replica. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4109510AlwaysOn: Explicitly transitioning the state of the Windows Server Failover Clustering (WSFC) resource that corresponds to availability group '%.*ls' to Failed. The resource state is not consistent with the availability group state in the instance of SQL Verify pre-requisite (for AlwaysOn) Windows fixes are installed.4109610AlwaysOn: The local replica of availability group '%.*ls' is being removed. The instance of SQL Server failed to validate the integrity of the availability group configuration in the Windows Server Failover Clustering (WSFC) store. This is expected if th4109710AlwaysOn: The local replica of availability group '%.*ls' is going offline. This replica failed to read the persisted configuration because of a version mismatch. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4109810AlwaysOn: The local replica of availability group '%.*ls' is restarting, because it failed to read the persisted configuration. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4109910AlwaysOn: The local replica of availability group '%.*ls' is going offline. This replica failed to read the persisted configuration, and it has exhausted the maximum for restart attempts. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4110016The availability group '%.*ls' and/or its local availability replica does not exist. Verify that the specified availability group name is correct, and that the local availability replica has joined the availability group, then retry the operation.4110116The availability group with Windows Server Failover Clustering resource ID '%.*ls' and/or its local availability replica does not exist. Verify that the specified availability resource ID is correct, and that the local availability replica has joined the4110210Failed to persist configuration data of availability group '%.*ls' in the Windows Server Failover Clustering (WSFC) cluster. The local availability replica either is not the primary replica or is shutting down.4110310Startup of the AlwaysOn Availability Replica Manager has been terminated, because the 'FixQuorum' property of Windows Server Failover Clustering (WSFC) is not present. The prerequisite QFE hotfix, KB 2494036, might not yet be installed on your Windows Ser4110416Failover of the availability group '%.*ls' to the local replica failed because the availability group resource did not come online due to a previous error. To identify that error, check the SQL Server error log and the Windows Server Failover Cluster logs4110616Cannot create an availability replica for availability group '%.*ls'. An availability replica of the specified availability group already exists on this instance of SQL Server. Verify that the specified availability group name is correct and unique, the4110716Availability group '%.*ls' failed to create necessary events for the WSFC Lease mechanism. Windows returned error code (%d) when obtaining handles for Lease events. Resolve the windows error and retry the availability group operation.4110816An error occurred while removing availability group '%.*ls'. The DROP AVAILABILITY GROUP command removed the availability group configuration from the local metadata. However, the attempt to remove this configuration from the Windows Server Failover Clust4110917Could not enqueue a task (SQL OS error: 0x%x) for process actions for the availability group '%.*ls'. Most likely, the instance of SQL Server is low on resources. Check the SQL Server error log to determine the cause of the failure. Retry the operation l4111010AlwaysOn: The availability replica manager is starting. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4111110AlwaysOn: The availability replica manager is waiting for the instance of SQL Server to allow client connections. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4111216A Windows Server Failover Clustering (WSFC) API required by availability groups has not been loaded. AlwaysOn Availability Groups replica manager is not enabled on the local instance SQL Server. If the server instance is running an edition of SQL Server t4111316Cannot failover availability group '%.*ls' to this instance of SQL Server because a failover command is already pending on the local replica of this availability group. Wait for the pending failover command to complete before issuing another command on t4111416Cannot create an availability group named '%.*ls' because it already exists in a system table.4111516Cannot create the availability group named '%.*ls' because its availability group ID (ID: '%.*ls') already exists in a system table.4111616Cannot create an availability group named '%.*ls' with replica ID '%.*ls' because this ID already exists in a system table.4111716Cannot map local database ID %d to the availability database ID '%.*ls' within availability group '%.*ls'. This database is already mapped to an availability group.4111816Cannot map database ID %d to the availability database ID '%.*ls' within availability group '%.*ls'. Another local database, (ID %d). is already mapped to this availability database.4111916Could not find the availability group ID %d in the system table.4112110The local availability replica of availability group '%.*ls' cannot accept signal '%s' in its current replica role, '%s', and state (configuration is %s in Windows Server Failover Clustering store, local availability replica has %s joined). The availabil4112216Cannot failover availability group '%.*ls' to this instance of SQL Server. The local availability replica is already the primary replica of the availability group. To failover this availability group to another instance of SQL Server, run the failover c4112616Operation on the local availability replica of availability group '%.*ls' failed. The local copy of the availability group configuration does not exist or has not been initialized. Verify that the availability group exists and that the local copy of the4112716Attempt to set database mapping state where the local database id %d is not mapped to any availability group.4112816Failed to perform database operation '%s' on database '%.*ls' (ID %d) in availability group '%.*ls'. The database might be in an incorrect state for the operation. If the problem persists, you may need to restart the SQL Server instance.4112916Failed to schedule or execute database operation '%s' on database '%.*ls' (Database ID: %d) in availability group '%.*ls' (SQL OS error: %d). The instance of SQL Server may have insufficient resources to carry out the database operation. If the problem p4113016Operation '%s' on a database '%.*ls' (Database ID: %d) in availability group '%.*ls' failed with SQL Server error %d (Error details: "%.*ls"). The operation has been rolled back. See previous error messages in the SQL Server error log for more details. 4113110Failed to bring availability group '%.*ls' online. The operation timed out. Verify that the local Windows Server Failover Clustering (WSFC) node is online. Then verify that the availability group resource exists in the WSFC cluster. If the problem persisCheck logs for additional errors.4113216Cannot join database '%.*ls' to availability group '%.*ls'. The specified database does not belong to the availability group. Verify the names of the database and the availability group, and retry the command specifying the correct names.4113310Cannot remove database '%.*ls' from availability group '%.*ls'. Either the database does not belong to the availability group, or the database has not joined the group. Verify the database and availability group names, and retry the command.4113416Cannot bring the availability group '%.*ls' online. The local instance was not the previous primary replica when the availability group went offline, not all databases are synchronized, and no force failover command was issued on the local availability r4113510Startup of AlwaysOn Availability Groups replica manager failed due to SQL Server error %d. To determine the cause of this error, check the SQL Server error log for the preceding error.4113616Failed to join the availability replica to availability group '%.*ls' because the group is not online. Either bring the availability group online, or drop and recreate it. Then retry the join operation.4113710Abandoning a database operation '%ls' on availability database '%.*ls' of availability group '%.*ls'. The sequence number of local availability replica has changed (Previous sequence number: %u, current sequence number: %u). This is an informational mesThis is an informational message only. If there is a problem, then look for additional messages.4113817Cannot accept AlwaysOn Availability Groups operation operation on database '%.*ls' of availability group '%.*ls'. The database is currently processing another operation that might change the database state. Retry the operation later. If the condition per4113910Failed to set database information for availability group %.*ls. The local availability replica is not the primary replica, or it is shutting down. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4114016Availability group '%.*ls' cannot process the ALTER AVAILABILITY GROUP command, because the local availability replica is not the primary replica. Connect to the server instance that is currently hosting the primary replica of this availability group, an4114116Failed to set availability group database information for availability group %.*ls. The local availability replica is not the primary, or is shutting down. This is an informational message only. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4114216The availability replica for availability group '%.*ls' on this instance of SQL Server cannot become the primary replica. One or more databases are not synchronized or have not joined the availability group, or the WSFC cluster was started in Force QuorumSelect availability group databases and select option to resume data movement. This will work if cluster was not started in forced_quorum mode.Check if *_FailoverCluster_health_XeLogs confirm that the cluster was started in forced_quorum mode (forced_quorum True in availability_replica_automatic_failover_validation event). When a cluster is started in forced_quorum mode (net.exe start clussvc /forcequorum, OR Failover Cluster Manager>left pane>right-click Windows cluster>choose Force Cluster Start>Confirm by clicking Yes.) on primary replica, the availability group is required to be started by issuing command ALTER AVAILABILITY GROUP <AGNAME> FORCE_FAILOVER_ALLOW_DATA_LOSS;. Because this is the original primary, despite issuing force allow data loss, NO data loss occursWhen a cluster is started in forced_quorum mode on secondary replica, first check sys.dm_hadr_database_replica_cluster_states.is_failover_ready on secondary. If this value is 1, and be assured no data loss will occur. If this value is not 1 (for example because it’s asynchronous commit), then can check sys.dm_hadr_database_replica_states.last_commit_time, so as to know estimate start time of data loss. The sys.dm_hadr_database_replica_states.last_hardened_lsn value may be used to consider other secondaries for failover. Then can initiate failover with ALTER AVAILABILITY GROUP <AGNAME> FORCE_FAILOVER_ALLOW_DATA_LOSS; on secondary.TSQL: SELECT database_name, is_failover_ready FROM sys.dm_hadr_database_replica_cluster_states WHERE replica_id = (SELECT replica_id FROM sys.availability_replicas WHERE replica_server_name = '<Replica server name>') Otherwise, once all local secondary databases are joined /synchronized, a planned manual failover can be performed to this secondary replica (without data loss).4114316Cannot process the operation. The local replica of availability Group '%.*ls' is in a failed state. A previous operation to read or update persisted configuration data for the availability group has failed. To recover from this failure, either restart 4114416The local availability replica of availability group '%.*ls' is in a failed state. The replica failed to read or update the persisted configuration data (SQL Server error: %d). To recover from this failure, either restart the local Windows Server Failov4114510Cannot join database '%.*ls' to availability group '%.*ls'. The database has already joined the availability group. This is an informational message. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4114616Failed to bring Availability Group '%.*ls' online. The Windows Server Failover Clustering (WSFC) service may not be running, or it may not be accessible in its current state. Please verify the local WSFC node is up and then retry the operation.4114710AlwaysOn Availability Groups was not started because %ls. This is an informational message. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4114816Cannot add or join database '%.*ls' to availability group '%.*ls'. The database does not exist on this instance of SQL Server. Verify the database name and that the database exists on the server instance. Then retry the operation, specifying the correc4114916Operation on the availability group '%.*ls' has been cancelled or terminated, either because of a connection timeout or cancellation by user. This is an informational message. No user action is required.This is an informational message only. If there is a problem, then look for additional messages.4115016Failed to take availability group '%.*ls' offline. The Windows Server Failover Clustering (WSFC) service may not be running, or it may not be accessible in its current state. Verify the local WSFC node is up and then retry the operation.4115116Error accessing the Availability Groups manager. The local Availability Groups manager has not been initialized. Wait until the Availability Groups manager is in a state that allows access, and retry the operation.4115216Failed to create availability group '%.*ls'. The operation encountered SQL Server error %d and has been rolled back. Check the SQL Server error log for more details. When the cause of the error has been resolved, retry CREATE AVAILABILITY GROUP commandCheck this document for error number mentioned in %d.4115316Failed to create availability group '%.*ls'. The operation encountered SQL Server error %d. An attempt to roll back the operation failed. Check the SQL Server error log for more details. Then execute the DROP AVAILABILITY GROUP command to clean up any4115416Cannot failover availability group '%.*ls' to this SQL Server instance. The availability group is still being created. Verify that the specified availability group name is correct. Wait for CREATE AVAILABILITY GROUP command to finish, then retry the op4115516Cannot failover availability group '%.*ls' to this instance of SQL Server. The availability group is being dropped. Verify that the specified availability group name is correct. The availability group may need to be recreated if the drop operation was 4115616Cannot drop availability group '%.*ls' from this instance of SQL Server. The availability group is either being dropped, or the local availability replica is being removed from the availability group. Verify that the specified availability group name is4115716Cannot remove the local availability replica from availability group '%.*ls' from this instance of SQL Server. The availability group is either being dropped, or the local availability replica is being disjoined. Verify that the specified availability g4115816Failed to join local availability replica to availability group '%.*ls'. The operation encountered SQL Server error %d and has been rolled back. Check the SQL Server error log for more details. When the cause of the error has been resolved, retry the ACheck for SQL/Windows messages around same time.4115916Failed to join local availability replica to availability group '%.*ls'. The operation encountered SQL Server error %d. An attempt to rollback the operation failed. Check SQL Server error log for more details. Run DROP AVAILABILITY GROUP command to cl4116016Failed to designate the local availability replica of availability group '%.*ls' as the primary replica. The operation encountered SQL Server error %d and has been terminated. Check the preceding error and the SQL Server error log for more details aboutChecks logs for additional messages.4116116Failed to validate the Cyclic Redundancy Check (CRC) of the configuration of availability group '%.*ls'. The operation encountered SQL Server error %d, and the availability group has been taken offline to protect its configuration and the consistency of 4116216Failed to validate sequence number of the configuration of availability group '%.*ls'. The in-memory sequence number does not match the persisted sequence number. The availability group and/or the local availability replica will be restarted automatical4116316An error occurred while waiting for the local availability replica of availability group '%.*ls' to transition to the primary role. The operation encountered SQL OS error %d and has been terminated. Verify that the Windows Server Failover Clustering (WS4116416An error occurred while waiting for the local availability replica of availability group '%.*ls' to transition to the resolving role. The operation encountered SQL OS error %d and has been terminated. Verify that the Windows Server Failover Clustering (4116516A timeout error occurred while waiting to access the local availability replica of availability group '%.*ls'. The availability replica is currently being accessed by another operation. Wait for the in-progress operation to complete, and then retry the 4116616An error occurred while waiting to access the local availability replica of availability group '%.*ls'. The operation encountered SQL OS error %d, and has been terminated. Verify that the local availability replica is in the correct state, and then retr4116716An error occurred while attempting to access availability replica '%.*ls' in availability group '%.*ls'. The availability replica is not found in the availability group configuration. Verify that the availability group and availability replica names are4116816An error occurred while attempting to access availability replica with ID '%.*ls' in availability group '%.*ls'. The availability replica is not found in the availability group configuration. Verify that the availability group name and availability repl4116916An error occurred while attempting to access the availability group database with ID '%.*ls' in availability group '%.*ls'. The availability database is not found in the availability group configuration. Verify that the availability group name and avail4117010Post-online processing for availability group '%.*ls' has been terminated. Either post-online processing has already completed, the local availability replica is no longer the primary replica, or the availability group is being dropped. This is an infor4117116Failed to create availability group '%.*ls', because a Windows Server Failover Cluster (WSFC) group with the specified name already exists. The operation has been rolled back successfully. To retry creating an availability group, either remove or rename4117216An error occurred while dropping availability group '%.*ls' from Windows Server Failover Clustering (WSFC) cluster and from the local metadata. The operation encountered SQL OS error %d, and has been terminated. Verify that the specified availability grIn DOS prompt, run “Net helpmsg” on the %d value, and proceed accordingly.4117316An error occurred while removing the local availability replica from availability group '%.*ls'. The operation encountered SQL OS error %d, and has been terminated. Verify that the specified availability group name is correct, and then retry the command4117610Failed to acquire exclusive access to local availability group configuration data (SQL OS error: %d). If the problem persists, you might need to restart the instance of SQL Server.4117716The availability replica of the specified availability group '%.*ls' is being dropped. Wait for the completion of the drop command and retry the operation later.4117816Cannot drop availability group '%.*ls' from this SQL Server instance. The availability group is currently being created. Verify that the specified availability group name is correct. Wait for the current operation to complete, then retry the command if4117916Cannot remove the local availability replica from availability group '%.*ls' from this instance of SQL Server. The availability group is currently being created. Verify that the specified availability group name is correct. Wait for the current operati4118016Attempt to access non-existent or uninitialized availability group with ID '%.*ls'. This is usually an internal condition, such as the availability group is being dropped or the local WSFC node has lost quorum. In such cases, and no user action is requir4118116The local availability replica of availability group '%.*ls' did not become primary. A concurrent operation may have changed the state of the availability group in Windows Server Failover Cluster. Verify that the availability group state in Windows Serv4118216Failed to set the local availability replica of availability group '%.*ls' as joined in Windows Server Failover Clustering (WSFC) database. Either the local availability replica is no longer the primary, or the WSFC service is not accessible. Verify tha4118316Failed to modify availability replica options for availability group '%.*ls'. Before the availability group configuration could be updated, the operation encountered SQL Server error %d. The operation has been rolled back. Refer to the SQL Server error4118416Failed to modify availability replica options for availability group '%.*ls'. The availability group configuration has been updated. However, the operation encountered SQL Server error %d while applying the new configuration to the local availability re4118510Replica option specified in ALTER AVAILABILITY GROUP '%.*ls' MODIFY DDL is same is cached availability group configuration.4118616Availability group '%.*ls' cannot process an ALTER AVAILABILITY GROUP command at this time. The availability group is still being created. Verify that the specified availability group name is correct. Wait for CREATE AVAILABILITY GROUP command to finis4118716Availability group '%.*ls' cannot process an ALTER AVAILABILITY GROUP command at this time. The availability group is being dropped. Verify that the specified availability group name is correct. The availability group may need to be recreated if it was4118816Availability group '%.*ls' failed to process %s-%s command. The operation encountered SQL Server error %d before the availability group configuration could be updated, and has been rolled back. Refer to the SQL Server error log for details. Verify that4118916Availability group '%.*ls' failed to process the %s-%s command. The availability group configuration has been updated. However, the operation encountered SQL Server error %d while applying the new configuration to the local availability replica, and has4119016Availability group '%.*ls' failed to process %s-%s command. The local availability replica is not in a state that could process the command. Verify that the availability group is online and that the local availability replica is the primary replica, the4119116The local availability replica of availability group '%.*ls' cannot become the primary replica. The last-known primary availability replica is of a higher version than the local availability replica. Upgrade the local instance of SQL Server to the same 4119217Creating and scheduling a worker task for AlwaysOn Availability Groups failed due to lack of resources (SQL OS error %d). Processing of new actions might be delayed or stalled until the resource limits are resolved. Reduce the memory or thread count on 4119310Cannot join database '%.*ls' to availability group '%.*ls'. The database is in the process of being removed from the availability group. When the remove-database operation completes, the database will no longer be joined to the availability group. Then4119416An error occurred while waiting for the local availability replica for availability group '%.*ls' to complet post-online work. The operation encountered SQL OS error %d and has been terminated. Verify that the Windows Server Failover Clustering (WSFC) c4119516Availability group '%.*ls' failed to process the WSFC lease-renewal command. The local availability replica lease is no longer valid to process the lease renewal command. Availability replica lease expired. This is an informational message only. No user aThis is an informational message only. If there is a problem, then look for additional messages.4119616Failed to create availability group '%.*ls', because a Windows Server Failover Cluster (WSFC) group with the specified name already exists. An attempt to rollback the operation failed. Check the SQL Server error log for more details. To manually clean 4119916The specified command is invalid because the AlwaysOn Availability Groups feature is not supported by this edition of SQL Server. For information about features supported by the editions of SQL Server, see SQL Server Books Online.4140216The WSFC cluster is offline, and this availability group is not available. This issue can be caused by a cluster service issue or by the loss of quorum in the cluster.4140316Availability group is offline.4140416The availability group is offline, and is unavailable. This issue can be caused by a failure in the server instance that hosts the primary replica or by the WSFC availability group resource going offline.4140516Availability group is not ready for automatic failover.4140616The availability group is not ready for automatic failover. The primary replica and a secondary replica are configured for automatic failover, however, the secondary replica is not ready for an automatic failover. Possibly the secondary replica is unavailCheck for additional messages on secondary replica(s).4140816In this availability group, at least one secondary replica has a NOT SYNCHRONIZING synchronization state and is not receiving data from the primary replica.4141016In this availability group, at least one synchronous replica is not currently synchronized. The replica synchronization state could be either SYNCHONIZING or NOT SYNCHRONIZING.4141216In this availability group, at least one availability replica does not currently have the primary or secondary role.4141416In this availability group, at least one secondary replica is not connected to the primary replica. The connected state is DISCONNECTED.Check connectivity as in error 35250.4142416This secondary database is not joined to the availability group. The configuration of this secondary database is incomplete. For information about how to join a secondary database to an availability group, see SQL Server Books Online.4142816This secondary replica is not joined to the availability group. For an availability replica to be successfully joined to the availability group, the join state must be Joined Standalone Instance (1) or Joined Failover Cluster (2). For information about hoSQL Server AlwaysOn (errors in SQL Server Configuration Manager)An error occurred while loading the AlwaysOn High Availability properties [return code: 0x80070005].May get this error if don't have the appropriate permission. Try right click ‘SQL Server Configuration Manager’ and select ‘Run as Administrator’.Unable to save the AlwaysOn High Availability settings [return code: 0x80041033].Proceed depending on error code. This is generally a WMI code e.g. 0x80041033.The AlwaysOn Availability Groups feature requires the x86(non-WOW) or x64 Enterprise Edition of SQL Server 2012 (or later version) running on Windows Server 2008 (or later version) with WSFC hotfix KB 2494036 installed. This SQL Server edition and/or Windows Server System does not meet one or more of the requirementsVerify OS is Windows 2008 or later version.If OS is Windows 2008 or Windows 2008 R2, install Windows hotfix KB 2494036, if not already installed.Unable to retrieve cluster name [return code: (0x800706d9)].Verify cluster service is started. One option may be to EVICT and ADD again that node. Steps: Right click node in failover cluster manager > More Actions... > Evict.SQL Server AlwaysOn (New Availability Group wizard errors)The current WSFC cluster quorum vote configuration is not recommended for this availability group. For more information, see the following topic in SQL Server Books Online: validating WSFC quorum vote configuration, the AlwaysOn Availability Group Wizard shows a warning if any of the following conditions are true:The cluster node that hosts the primary replica does not have a vote. If this vote is not required for that environment, then this message may be ignored. Powershell command to check vote is “Get-ClusterNode | f1 Name, Nodeweight”.A secondary replica is configured for automatic failover and its cluster node does not have a vote. If this vote is not required for that environment, then this message may be ignored.KB2494036 is not installed on all cluster nodes that host availability replicas. This patch is required to add or remove votes for cluster nodes in multi-site deployments. However, in single-site deployments, it is usually not required and you may safely ignore the warning.While setting up the availability group in the wizard, you configured a replica to asynchronous availability mode. This warning is only reported in SQL 2012 RTM, not in SP1 of SQL 2012 RTM.While setting up the availability group in the wizard, you configured a replica for manual failover mode. This warning is only reported in SQL 2012 RTM, not in SP1 of SQL 2012 RTM.Check the link. Redirects to for the database files on the secondary replica resulted in an error. (Microsoft.SqlServer.Management.HadrTasks)------------------------------Program Location: at Microsoft.SqlServer.Management.Hadr.TestDatabaseFileExisting.DoWork() at Microsoft.SqlServer.Management.TaskForms.SimpleWorkItem.Run()===================================The following database files already exist on the server instance that hosts secondary replica HA-VM3\TEST1: \\path\file.mdf\\path\file_log.ldf (Microsoft.SqlServer.Management.HadrTasks)------------------------------Program Location: at Microsoft.SqlServer.Management.Hadr.TestDatabaseFileExisting.DoWork()Check if database-file paths are identical on source and on destination servers. If different, manual (rather than using AG wizard) restore may be required with WITH MOVE syntax.An error was encountered while modifying the quorum settings.Your cluster quorum settings have not been changed.There was an error configuring the file share witness '\\XXX\ABC'.Unable to save property changes for 'File Share Witness'.File share associated with file share witness resource cannot be hosted by this cluster or any of its nodesRun cluster validation report and ensure there are no errors.The Endpoints tab lists at least one endpoint that uses only Windows Authentication. However, the server instance might be running under a nondomain account. To use the listed endpoint, change the corresponding SQL Server service account to a domain account. To continue using the nondomain account, alter the endpoint to use a certificate. Do you want to use the listed endpoints?Check SQL Server Configuration Manager to see if SQL startup account is domain user. ------------------------------TITLE: Microsoft SQL Server Management Studio------------------------------The primary server 'XXX' cannot write to '\\XXX\e$\mssql\data'. (Microsoft.SqlServer.Management.HadrTasks)------------------------------ADDITIONAL INFORMATION:Backup failed for Server 'XXX'. (Microsoft.SqlServer.SmoExtended)For help, click: ((SQL11_PCU_Main).121019-1325+)&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Backup+Server&LinkId=20476------------------------------An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)------------------------------Cannot open backup device '\\XXX\e$\mssql\data\HADR Seeding Test 237a7aa8-9d03-4f62-bb8b-de1322338467.bak'. Operating system error 5(Access is denied.).BACKUP DATABASE is terminating abnormally. (Microsoft SQL Server, Error: 3201)For help, click: SQL startup account to see if it’s domain account. SQL Server AlwaysOn (SSMS errors)SSMS shows Availability Group state as Resolving.Check that cluster service is started on that node. Also check for issues as in error 19406.Back up database task backs up the database availability groups contained in the set, you receive a warning message like the one below.This backup type is not supported in the secondary this task will fail if executed on the secondary.SQL Server Management Studio, in Object Explorer, expand the availability of high-availability – availability – AlwaysOn group – group – propertiesBackup preferences page screen captureConsider using the maintenance plan Wizard to create a backup job since it checks the sys.fn_hadr_backup_is_preferred_replica function call and automatically includes the following script logic. This logic is if the default backup the current replica replica (the return value to 1) grants the COPY_ONLY option in your backup and not backup default backup replicas. The maintenance plan Wizard when you create a database backup using the scripts are automatically added.IF (NOT sys.fn_hadr_backup_is_preferred_replica(@DBNAME))BEGINSelect 'This is not the preferred replica, exiting with success';RETURN 0 – This is a normal, expected condition, so the script returns successENDBACKUP DATABASE @DBNAME TO DISK=<disk>WITH COPY_ONLY;AlwaysOn. 'ApplcationIntent=ReadOnly’ doesn't work for registered serverThis is a known issue. Consider specifying “ApplicationIntent=ReadOnly” everytime you make connection to SQL server 2012.The local node is not part of quorum and is therefore unable to process this operation. This may be due to one of the following reasons:The local node is not able to communicate with the WSFC cluster.No quorum set across the WSFC cluster.For more information on recovering from quorum loss, refer to SQL Server Books Online.(Microsoft.SqlServer.Management.HadrTasks)Start cluster service. If it’s not started due to quorum, last option may be start without a quorum. After cluster is running, check cluster nodes/votes are appropriate.SQL Server AlwaysOn (performance issues)Disks are not detected if there is only one node at the secondary site.This is a limitation of Windows 2008 R2 cluster. This issue does not occur in Windows 2012. PowerShell can be used in Windows 2008 R2 to add the disk and it will work. Will need to modify the Possible Owners for the resource, as by default it will have all nodes checked. Powershell 1: add-clusterresource -Group "Available Storage" - Cluster "myclustername" -Name "diskname" -ResourceType "Physical Disk"Powershell 2: get-clusterresource "diskname" -Cluster "myclustername" | set-clusterparameter DiskPath "F:" #F: is the drive letter assigned in disk management for the disk.Slow synchronization. Waittime for HADR_SYNC_COMMIT grows anywhere from 500ms to 900ms (compared to less than 15-20 ms).If KB2723814 not applied, then try the KB workaround of suspend secondary replica and then resume, so that AlwaysOn knows that availability mode has changed back to synchronous commit.SQL Server Agent Jobs do not automatically failover, when participating in AlwaysOnThis is by design. Suggestion is to create the job at both Primary and Secondary and enable them. Include logic in job step that checks the role_desc in sys.dm_hadr_availability_replica_states of the database. If role_desc is primary then execute the job, and if role_desc is secondary then exit out of the job.TSQL: select role_desc from sys.dm_hadr_availability_replica_states where is_local=1 and role=1;Reason why the secondary replica becomes unavailable when SQL Server service is stopped on primary node (node2).ORAlwaysOn failback is not working.As expected. Increment "the maximum number of failures during this period" count. Its default value on n node cluster is n-1.The secondary connects to the Primary and not the other way around. If secondary is trying to connect to the primary, and primary is down, the state will be RESOLVING. For example, if the SAN was taken offline that hosted the AlwaysOn database on the primary, the secondary was no longer able to connect to that database, so was not synchronized and could not come online. This is an expected behavior (By Design). Primary replica database becoming unresponsiveWhile checking root cause, ensure latest SQL/Windows fixes and can set the following on availability groups so they are not adversely effected by any non-yielding events temporarily:Set the availability group FAILURE_CONDITION_LEVEL to 1 which will reduce the SQL Server symptoms that can result in health detection failure alert.To specifically address the lease timeout, increase the availability group HEALTH_CHECK_TIMEOUT setting to a higher setting (e.g. 90 seconds) than the default 30 seconds. The signal interval for the lease is 1/3 the HEALTH_CHECK_TIMEOUT so a 10 sec gap can result in lease expiration.Questions on secondary replicas. The function sys.fn_hadr_backup_is_preferred_replica returns 0 no matter what the backup preference is set to.Check if @@SERVERNAME returns correct value on that server.TSQL: SELECT @@SERVERNAME;Linked server configuration with AlwaysOn listenerTSQL: EXEC master.dbo.sp_addlinkedserver @server = N'MYLISTENER', @srvproduct=N'SQL2012', @provider=N'SQLNCLI11', @datasrc=N'MYLISTENER', @provstr=N'Provider=SQLNCLI11.1; Data Source=myListener;ApplicationIntent=READONLY', @catalog=N'DB1'As in the previous version in order to use the SQL Server Native Client (11.x) in SQL Server Feature Pack 2012 2012 must be installed. Microsoft? SQL Server? 2012 Feature Pack .SQL Native Client (SNAC) can also be installed on application server by running SQL Setup and installing ‘Client Tools Connectivity.’ Client Tools includes components for communication between clients and servers, including network libraries for DB-Library, OLEDB for OLAP, ODBC, ADODB, and ADOMD+.SQL Native Client DLL is available "C:\Windows\System32\sqlncli11.dll"Application encounters ODBC error after AlwaysOn group is failed over to secondary. Works fine when AlwaysOn group is on primary. [Microsoft][ODBC SQL Server Driver][SQL Server]The EXECUTE permission was denied on the object 'FN_ADJUSTED_DATE', database 'iDeliver', schema 'dbo'.. (IES 10901) (WIS 10901)SQL Native Client 11.x does support the new connection parameters. Older versions of SQL Native Client do NOT support ApplicationIntent parameter. Upgrade SQL Native client on the client application server. Please search this document for “SQL Native client” for install steps. This will upgrade ODBC etc. components on application server.***Stack Dump being sent to E:\MSSQL11.MSSQLSERVER\MSSQL\LOG\SQLDump0007.txt * BEGIN STACK DUMP:* Location: HadrAvailabilityGroupReplica.cpp:943* Expression: *pcbActualData <= cbRemainingBufferCheck if memory messages in SQL ERRORLOG which indicates possible memory pressure. If such messages exist, then ensure sp_configure ‘max server memory’ is set - point to note is that CLR is part of BP memory in 2012. If you’re using CLR, it should be accommodated within Max Server Memory. If memory messages present and if LogPool memory appears to be high, are the replicas connected through a fast network? Also, it is possible that it is so heavily transactional that the number of log records generated is high and with the amount of databases it pushes this memory above the roof. As such, you may want to consider increasing the memory/RAM on the box.FAIL_PAGE_ALLOCATION 1 in SQL ERRORLOG when using AlwaysOn Functionality.Check the max and min server memory correctly.TSQL: EXEC sp_configure 'max server memory';Slow commit performance problem for replica in synchronous commit mode.Check ‘Replica:Transaction Delay counter’, ‘Replica:Log Send Queue’ perfmon counters.High HADR_WORK_QUEUE wait.AlwaysOn Availability Groups background worker thread waiting for new work to be assigned. This is an expected wait when there are ready workers waiting for new work, which is the normal state.High HADR_LOGCAPTURE_WAIT wait.Check perfmon counters average log bytes flushed / sec, log bytes received /sec. If log bytes received /sec is much higher, then this may indicate that the log scan could be a bottleneck.After failing over from one subnet to another in AG, the ping command (to listener) from the remote client is not resolving to newly current active IP. DNS entry for the Listener virtual shows both the IP’sipconfig /flushdnsHostRecordTTL is now being set to 60, RegisterAllProvidersIP is set to 0 (default is 1, value change requires the cluster service to be cycled or the CAP resources restarted), but the DNS is still returning wrong IP on Ping cmd for over a minute.windows issue (cluster attribute?)Import-Module FailoverClustersGet-ClusterResource yourListenerName|Set-ClusterParameter RegisterAllProvidersIP 0cluster /cluster:<ClusterName> res <NetworkNameResource> /priv RegisterAllProvidersIP=0Always on Availability Group. Application is using SQL Login to access the databases. The application is not able access the database after the database is failed over to the secondary.SID may be different for the user in both instances.TSQL: SELECT name, sid FROM sys.database_principals;TSQL: CREATE LOGIN [LLL] WITH PASSWORD='dddd', DEFAULT_DATABASE=[master], DEFAULT_LANGUAGE=[us_english], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF,SID=0xABC;SQL Server AlwaysOn errors (network)TCP Provider, error: 0 - An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was Stat output may show hundreds of entries in TIME_WAIT state leading to buffer/port exhaustionAdd registry setting for MaxUserPort registry setting for TCPTIMEWAITDELAY.App/IIS restart, machine reboot are additional options.SQL Server AlwaysOn errors (Application issues)Application connection string.Connection string should use ODBC or SQL OLE DB in SNAC (application intent=readonly is optional and depends on whether read only connections are supported):To use SQL Native Client SQL OLEDB, change your connection string to this (this one is using integrated security): provider=sqlncli11;data source= tcp:AGListener,1633;database=ag;integrated security=sspi;application intent=readonly;MultiSubnetFailover=True To use .Net SQLClient: data source= tcp: AGListener,1633;database=ag;user=sa;password=Password2;applicationintent=readonly Finally, to use SQL Native Client ODBC, connection string can be like below:driver={SQL Server Native Client 11.0};server= tcp: AGListener,1633;database= CGFData;trusted_connection=yes;applicationintent=readonly;MultiSubnetFailover=TrueApplication reconnects takes 1mins and 30 seconds, even though Failover on dB takes 6 to 10 seconds.Filter driver (like anti-virus etc.) may be causing the slowness in connection. Check if anti-virus is up to date, and if SQL files are excluded as indicated in .Application/osql using AlwaysOn database gets disconnected when executing a failover of the Availability Group.This is expected. Application should have connection retry logic.Application hang after AlwaysOn group failover.If Java based application, then Java does not have command timeout (be default, it is not limited). But .net has 30 sec default command timeout, that's why .NET has no issue. Set commandtimeout in Java.Application connects to primary replica every time even when the parameter Connection Intent =Read Only is specified in the connection string.Check if Routing URL is defined for each server, and if Routing List was also not defined.If the database part is omitted (in connection string that has AG listener), readonly routing does not workSQL Server AlwaysOn errors (cluster log)[RES] Network Name <SPEPD_Tier2>: Unable to update password for computer account DCR-SPEPD on DC \\ DC-04., status 5.[RHS] Online for resource SPEPD_Tier2 failed.Status 5 is Access DeniedIn Windows AD, check for CNO (cluster network object) permissions.For example, below to grant the "create computer objects" to the cluster virtual host (virtual cluster name) for the OU in Active Directory on "Computers" and "DomainServer" OU that was created. [To be tested, steps in event 1193]Instructions [to be tested]1) Log into the Domain Controller machine.2) Select Start - Administrative Tools - Active Directory users and computer.--> Active Directory users and computer screen displays.3) Display-select advanced features.4) Select the appropriate domain - Computers.5) Verify that the cluster virtual hosts.6) Select the appropriate domain - Computers - right click - Properties.--> Computers Properties screen appears.7) Security tab-click the Advanced button.--> Advanced Security for Computers "screen appears.8) If you have access permissions tab - cluster virtual host registration registration no "Edit" button if you choose the "add" button.+ Case "add": conducted a check in the "computer", click the "object type" button, search for cluster virtual hosts.--> [Permission entries on Computers] screen is displayed.9) Target: Select this object and all child objects check the permission "create computer objects" and click the "OK" button.--> Returns to the advanced security for Computers "screen.10) Click "OK" button after clicking "the apply" button.--> Returns to the Computers Properties screen.11) And click the "OK" button and exit the Active Directory users and computer [RES] Network Name: [NNLIB] NetUserAdd object aglisten11 on DC: \\DC., result: 8557ERR [RES] Network Name: [NNLIB] Failed to create Computer Object aglisten11 in the Active Directory, error 8557net helpmsg 8557 says "Your computer could not be joined to the domain. You have exceeded the maximum number of computer accounts you are allowed to create in this domain. Contact your system administrator to have this limit reset or increased."Cluster log has "[RES] SQL Server Availability Group: [hadrag] Run 'EXEC sp_server_diagnostics 10' returns following information" "ERR [RES] SQL Server Availability Group: [hadrag] ODBC Error: [42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]The user does not have permission to perform this action. (297)"NT AUTHORITY\SYSTEM account lacked the required permissions to failover the availability group. To failover the group to the other node, NT AUTHORITY\SYSTEM must have permission to connect to SQL (CONNECT SQL), failover the availability group (ALTER ANY AVAILABILITY GROUP) and execute sp_server_diagnostics (VIEW SERVER STATE).Please view error 35206 for additional details.Lease renewal failed with timeout errorCheck this document for “lease” related errors/troubleshooting. This may be a performance issue.ERR [RES] SQL Server Availability Group <WSDBMEMAVG>: [hadrag] Availability Group is not healthy with given HealthCheckTimeout and FailureConditionLevelERR [RES] SQL Server Availability Group <WSDBMEMAVG>: [hadrag] Resource Alive result 0.ERR [RES] SQL Server Availability Group: [hadrag] Failure detected, diagnostics heartbeat is lostThis may be a performance issue.SQL Server AlwaysOn errors (event log messages)Event IDSourceMessage1193Microsoft-Windows-FailoverClusteringCluster network name resource '%1' failed to create its associated computer object in domain '%2' for the following reason: %3.The associated error code is: %5Please work with your domain administrator to ensure that:- The cluster identity '%4' can create computer objects. By default all computer objects are created in the 'Computers' container; consult the domain administrator if this location has been changed.- The quota for computer objects has not been reached.- If there is an existing computer object, verify the Cluster Identity '%4' has 'Full Control' permission to that computer object using the Active Directory Users and Computers tool.If %5 says “Access is denied”, then verify “Create Computer objects permission” for cluster network name resource (CNO).Steps: 1. On a domain controller, click Start, click Administrative Tools, and then click Active Directory Users and Computers. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue. 2. Expand the default Computers container or the folder in which the cluster name account (the computer account for the cluster) is located. Computers is located in Active Directory Users and Computers/domain node/Computers. 3. Examine the icon for the cluster name account. It must not have a downward-pointing arrow on it, that is, the account must not be disabled. If it appears to be disabled, right-click it and look for the command Enable Account. If you see the command, click it. 4. On the View menu, make sure that Advanced Features is selected. When Advanced Features is selected, you can see the Security tab in the properties of accounts (objects) in Active Directory Users and Computers. 5. Right-click the default Computers container or the folder in which the cluster name account is located. 6. Click Properties. 7. On the Security tab, click Advanced. 8. In the list of accounts with permissions, click the cluster name account, and then click Edit. Note If the cluster name account is not listed, click Add and add it to the list. 9. For the cluster name account, ensure that Allow is selected for the """Create Computer objects permission. """10. Click OK until you have returned to the Active Directory Users and Computers snap-in.1194Microsoft-Windows-FailoverClusteringCluster network name resource '%1' failed to create its associated computer object in domain '%2' for the following reason: %3.The text for the associated error code is: %4Please work with your domain administrator to ensure that:- The cluster identity '%5' can create computer objects. By default all computer objects are created in the 'Computers' container; consult the domain administrator if this location has been changed.- The quota for computer objects has not been reached.- If there is an existing computer object, verify the Cluster Identity '%5' has 'Full Control' permission to that computer object using the Active Directory Users and Computers tool.Like in event 1193, verify “Create Computer objects permission” for cluster network name resource (CNO).Try creating Client Access Point (CAP) with same name and IP to try to reproduce the issue outside SQL.Try pre-staging the name in AD (also add node name $ to the ACL with full control). Now re-try creating CAP. More details in (WS.10).aspx#BKMK_steps_precreating .Cluster validation.1206Microsoft-Windows-FailoverClusteringCluster network name resource '%1' cannot be brought online. The computer object associated with the resource could not be updated in domain '%2'. The error code was '%3'. The cluster identity '%4' may lack permissions required to update the object. Please work with your domain administrator to ensure that the cluster identity can update computer objects in the domain.1207Microsoft-Windows-FailoverClusteringCluster network name resource '%1' cannot be brought online. The computer object associated with the resource could not be updated in domain '%2' for the following reason:%3.The text for the associated error code is: %4The cluster identity '%5' may lack permissions required to update the object. Please work with your domain administrator to ensure that the cluster identity can update computer objects in the domain.SQL Server AlwaysOn errors (Windows messages obtained through “net helpmsg”)665The requested operation could not be completed due to a file system limitationIf database has FILESTREAM/FileTable, then check if 8dot3 name creation is enabled for applicable drive(s). If it is enabled, disable it.DOS command1: fsutil 8dot3name query C:DOS command2: fsutil 8dot3name set C: 15057The cluster IP address is already in use.Checked if IP is already in use. May require a new/unused IP depending on what is being done.Check permission for cluster network object CNO.5942The resource failed to come online due to the failure of one or more provider resources.Check for additional messages in logs.Depending on the resource type, this generally requires involvement of Windows team.For multi-subnet cluster, check the HostRecordTTL property for network name. If it’s set to 1200 (20 minutes), suggested value is 300 (5 minutes). Try to remove SQL from this issue. For example, if this is a network name or an IP address resource, then try creating a Client Access Point with same name and IP if possible, and see it comes online. If it does not come online, then this requires investigation by Windows team.8557Your computer could not be joined to the domain. You have exceeded the maximum number of computer accounts you are allowed to create in this domain. Contact your system administrator to have this limit reset or helpmsg 8557 says "Your computer could not be joined to the domain. You have exceeded the maximum number of computer accounts you are allowed to create in this domain. Contact your system administrator to have this limit reset or increased."For example, when a listener is created, a Computer Object is automatically created in Active Directory. Even after Listener is dropped, that Computer Object may remain in Active Directory. The other thing to know is that by default, authenticated (domain) users are limited to create 10 Computer Objects in the domain. Once they hit the limit, attempting to create the 11th will fail with this error. User needs permission to create more than 10 Computer Objects or, more likely, user needs to engage the Windows network team to clean up these Computer Objects.SQL Server AlwaysOn errors (WMI errors)0x80070005May get this error if don't have the appropriate permission. Try right click ‘SQL Server Configuration Manager’ and select ‘Run as Administrator’.Verify wbemtest queries are working against root/default, cimv2 and SQL namespaces.0x80041033WBEM_E_SHUTTING_DOWN 2147749939 (0x80041033) --> User has requested an operation while WMI is in the process of shutting down. At the same time of error, check if app event log has message "Windows Management Instrumentation has stopped WMIPRVSE.EXE because a quota reached a warning value. " (Microsoft-Windows-WMI, event id 5612). If this message is present, then this issue may be because WMIPrvse process is not able to create the required number of handles, it may be shutting down before completing the entire process. Increase the maximum number of handles per host in WMI.Steps to increase max number of handles per host: ?Go to Start--> Run and type wbemtest.exe. ?Click Connect. ?In the namespace text box type "root" (without quotes). ?Click Connect. ?Click Enum Instances… ?In the Class Info dialog box enter Superclass Name as "__ProviderHostQuotaConfiguration" (without quotes) and press OK. Note: the Superclass name includes a double underscore at the front. ?In the Query Result window, double-click "__ProviderHostQuotaConfiguration=@" ?In the Object Editor window, double-click HandlesPerHost. ?In the Value dialog, type in 8192 ?Click Save Property. ?Click Save Object. ?Close Wbemtest. ?Restart the Windows Management Instrumentation service.Note: There should not be any impact as such, since this is a 64 bit server and we have ample resources on the server. This change means: the WMIPrvse which was eligible to create 4096 handles at the max will be able to make 8196 handles at the max only if required. Even if something makes the WMIprvse host to leak handles it will not cross 8192 handles. Handles consumed are proportional to amount of resources consumed; especially Pool resources. But I don’t see that being a problem unless, we have a large number of WMIPrvse leaking handles all at the same time and the system does not have much RAM/Processor. However, I would not recommend changing the default values unless there is a similar demand or any other reason to increase the default value.SQL Server AlwaysOn (patching/updates in SQL 2012)AG on Standalone InstancePatching steps:Patch the secondary replica (B)Bring the secondary replica online (will be new version)Log the original synchronization configurations for each replica. Change the secondary replica and primary replica to “Synchronous Commit” mode, waiting for the secondary replica (B) to be “synchronized.”This will ensure there is no data loss during failover.You can check dashboard or dmv: dm_hadr_database_replica_states for the statusIssue a failover through SSMS or T-SQL to failover the AG to the secondary replica. So now the new primary is B, the new secondary is APatch the original primary replica (A)Bring the original primary replica (A) online (will be new version )Wait for A to become “Synchronized” Failover the AG back to AChange each replica’s synchronization mode to the original configurations you logged in step3.Caveat list:Before you patch, you can still keep automatic/manual failover setting no change. Just a reminder: if during patching time, primary is down, automatic failover may fail if the secondary hasn’t completed the patching.If primary and secondary replicas are in multisubnet, your client may experience a little bit longer time of disconnection or timeout during failover.Please do remember to switch back to your original synchronization mode.AG on FCIs (Failover Cluster Instances)FCI on AG deployment ExamplePrimary replica(A): FCI1 Node1 – activeNode2 - passiveSecondary Replica(B): FCI2 Node3 – activeNode4 - passive Choose between one of the following two options:Longer downtime, less steps(FCI rolling patching).Patching steps (basic):Patch Node4Move FCI2 from Node3 to Node4 Patch Node3Move FCI2 from Node4 to Node3Patch Node2Move FCI1 from Node1 to Node2Patch Node1Move FCI1 from Node2 to Node1Optimized downtime, more steps(Leverage AG failover patching). Patching steps (optimized):Patch Node4Move FCI2 from Node3 to Node4Patch Node3Move FCI2 from Node4 to Node3Log the original synchronization configurations for each replica. Change the secondary replica and primary replica to “Synchronous Commit” mode, waiting for the secondary replica (B) to be “synchronized”.Manually failover AG from FCI1 to FCI2 (now the new primary is on FCI2, the new secondary is on FCI1)Patch Node2Move FCI1 from Node1 to Node2Patch Node1Move FCI1 from Node2 to Node1Manually failover AG from FCI2 back to FCI1Change each replica’s synchronization mode to the original configurations you logged in step #5.Caveat list:** Please use basic patching steps if primary and secondary are on different data centers/subnets **FCI rolling patch guarantees zero data lostUse synchronous commit secondary on a high latency network would impact OLTP performanceAG failover cross subnets might cause up to 20 seconds delay on client first connections. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download