Content not updating on all servers in a web farm

Overview

When page content updates are not synced between the web servers in the web farm, this may mean that web farm cache synchronization has been failing.

For the purpose of troubleshooting, the important steps of the cache synchronization process are as follows: 

• Evoq uses a provider called WebRequestCachingProvider to synchronize the cache between the enabled web servers automatically. 
• When there is more than one server in the server list, and the web farm flag is enabled then the SynchronizeCache.aspx request is then sent to the servers on the list. The request reaches the uniquely defined server URLs in the list to synchronize the cache between the servers.
• Every time the request returns a 200 message, this means that the SynchronizeCache request was successful and the cache has been synced between the webheads.

Web Farm cache synchronization issues occur when any of the points above fail to complete.

You can also confirm you are having cache synchronization issues by going into the SiteRoot/Portals/_default/Logs folder and seeing the following error:

Synchronization Error in Request:
[Thread:55][ERROR] DotNetNuke.Services.Exceptions.Exceptions - System.Exception: Synchronization Error in Request: http://www.website.com/SynchronizeCache.aspx?message=1yDD6Y0tSOM%3d&data=bYPl_hRFaUFi3RPdZCZLRy608cAwf-HVD2nEq5COLZ8%3d ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 0.0.0.0.0.
at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at DotNetNuke.Professional.Providers.CachingProviders.WebRequestCachingProvider.WebRequestCachingProvider.OnSynchronizeResponseCallback(IAsyncResult asynchronousResult)

StackTrace:
InnerMessage: The remote server returned an error: (404) Not Found.

Another error you may encounter is:

Message: Synchronization Error in Request

InnerMessage: The underlying connection was closed: An unexpected error occurred on a send.
InnerStackTrace: at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at DotNetNuke.Professional.Providers.CachingProviders.WebRequestCachingProvider.WebRequestCachingProvider.OnSynchronizeResponseCallback(IAsyncResult asynchronousResult)

Solution

1. Confirm the web farm is configured correctly. Follow this guide: How to Setup a Web Farm
   1. Pay special attention to the instance specific URIs on each webserver. Without these the requests to SynchronizeCache.aspx might not go to the correct location. For example, if you are using the top level domain as the URL for the webserver, the request will go out to the DNS, then back in through the load balancer, and get randomly assigned to one of the nodes.
   2. The second thing to pay attention to is to set the cache provider to WebRequestCachingProvider.
      1. From Persona Bar, navigate to Settings > Servers > Server Settings > Web Servers.
      2. From the Server Web Request Adapter drop-down, select DotNetNuke.Entities.Host.ServerWebRequestAdapter.
      3. Click Save.

If it still does not work, you could be encountering an edge case. Please review and test the following suggestions:

• There are old/unused servers in the server list.
  1. You need to delete the old servers or run the TRUNCATE TABLE webservers query through Settings > SQL Console.
  2. Recycle the application pool. This will clear out the webservers list. By recycling the application pool, the application will regenerate the Web Server entries. Alternatively, you can enable the webserver monitor scheduled tasks to perform the same process.
• The servers' unique URLs are not present in the Portal Alias table.
  
  • You will need to add the server's unique URLs to the list of Portal Aliases.
• The servers' unique URLs are incorrect in the WebServers table.
  • Access Settings > Servers and review the URL setting for each server to confirm it is correct.
  • To review these settings in bulk run this SQL query: SELECT * FROM WebServers
• The unique URLs defined for the webservers are not accessible from all other webservers.
  1. This can be verified by accessing each web server remotely (use the server's unique URL).
  2. From the remote server, make a request to other servers (use the server's unique URL, not the URL of the load balancer). You can simply open a web browser and navigate to the SynchronizeCache.aspx page on the target machine.
  3. If any errors are returned, you will need to check the connection between the webheads. You can also verify this by checking the IIS logs for the corresponding webhead (find the SynchronizeCache request and make sure it returns a 200 message). Note: The SynchronizeCache.aspx cannot be accessed directly via a browser.
  4. Make sure you set the webserver URLs to their unique FQDNs/IP addresses.

Testing

To test that the cache synchronization has been resolved:

1. Make a content update on any modules and publish the page on one web server.
2. Go to Settings > Servers > Clear Cache.
3. Go to another web server in the web farm by using its unique URL.

You should see that the content is consistent between the servers. Make sure you are logged in and Output Cache is not enabled on the page/module.