SQL Server Failover Cluster Takes Disk Offline

This is very interesting case. One day we saw that a database is not accessible in a failover cluster due to an offline disk. And we examine the event log about the situation. We saw below errors:

Disk has crossed a capacity utilization threshold and used bytes. When the threshold was crossed,the pool had 0 bytes of remaining capacity.

Disk has reached a logical block provisioning permanent resource exhaustion condition.

When we examine the errors, we see that the disk cannot grow because it exceeds a defined threshold, even though there is free space on the disk. After the necessary investigations, we found that the error was due to the allocation of disk space for the snapshot on the storage. Failover Cluster takes the disk offline with the following errors because there is free space on the disk but it can not grow.

Ownership of cluster disk has been unexpectedly lost by this node. Run the Validate a Configuration wizard to check your storage configuration.

Since we cannot access the status of the disks from Failover Cluster Manager, we used the following powershell scripts to bring the disk online.

Get Failover Cluster Resource Status with Powershell

Take Online the Disk with Powershell

After taking the disk online we fix the problem on storage. If you do not fix the problem on the storage you will encounter the same problem again on other disk of the same storage.