Scaling sitecore

One of the sites I look after is www.mcfc.co.uk . I am not sure I am allowed to release the stats, but  on a match day we can get a lot of page views in 24 hours. With the announcement of a high profile transfer signing, it can be many more. What follows is a brief overview of the setup and some of the things we do to help the site scale.

The site currently runs on sitecore 6.2, although we have plans to move to 6.4. In production we currently have 2 cms servers ( one is a redundant backup ) and 4 front end servers sitting behind a load balancer just serving content.

To manage the state across the servers we use memcached to store session data. We have two memcached servers to provide redundancy.

Cache clearing across the front end servers is done using the staging module ( it does have its quirks admittedly).

The database server is a huge clustered beast to ensure we have some redundancy. Each web server has it's own web database, so that we can safely update one server at a time.

We also employ a CDN provider called limelight to handle the serving of media such as images, video, CSS & JS. This takes a huge strain off of the servers as sitecore usually stores the images in the database.

To help reduce the load on the database server, we store all of the news, videos ect in a lucene index. This saves us having to do lots of really slow GetItems() queries.

We have also set up some maintenance tasks to ensure the database indexes do not get fragmented. At one point, our indexes were 98% fragmented and our database server was taking a pounding. A quick rebuild on the main tables and it made a huge difference.

Below is a query to help you find out how fragmented a table is:

 DECLARE @db_id SMALLINT;  
 DECLARE @object_id INT;  
 SET @db_id = DB_ID(N'ManCitySitecore_Web_Phase3');  
 SET @object_id = OBJECT_ID(N'Items');  
 IF @db_id IS NULL  
 BEGIN;  
   PRINT N'Invalid database';  
 END;  
 ELSE IF @object_id IS NULL  
 BEGIN;  
   PRINT N'Invalid object';  
 END;  
 ELSE  
 BEGIN;  
   SELECT * FROM sys.dm_db_index_physical_stats(@db_id, @object_id, NULL, NULL , 'LIMITED');  
 END;  
 GO  

Model View Presenter (MVP) and Sitecore

There are various ways to try and abstract our interaction with sitecore. The two most common that we have used are the Model View Presenter (MVP)  and MVC ( Model View Controller) patterns. In this post I will talk about how to use MVP with sitecore.

FIrst of: Don't use XSLT. It is untestable.

We start of with a base user control that all our controls will extend. All we are doing here is injecting the Presenter using structure map ( abstracted away inside DependencyInjector) and then calling a common method that will exist on all presenters, Init(). It also provides some common properties that we will be using later on.

 public class MVPUserControl<TPresenter> : BaseUserControl  
     where TPresenter : MVPPresenter  
   {  
     protected override void OnInit(EventArgs e)  
     {  
       var presenter = DependencyInjector.Instance.GetDefaultInstance<TPresenter>();  
       presenter.View = this;  
       IntegrateWithPresenter(presenter);  
       presenter.Init();  
       base.OnInit(e);  
     }  
     public bool IsFirstViewing  
     {  
       get { return !IsPostBack; }  
     }  
     public Guid CurrentItemId  
     {  
       get { return Sitecore.Context.Item.ID.ToGuid(); }  
     }  
     public string CurrentItemName  
     {  
       get { return Sitecore.Context.Item.Name; }  
     }  
     public string LanguageCode { get { return "UK"; } }  
     public string RegionCode { get { return "2"; } }  
     public virtual void IntegrateWithPresenter(TPresenter presenter) { }  
   }  

In the user control ( the view) we keep it very simple and it should just be used to pass information from and to. This way we can make it conform to an interface and as a result we can mock it using different frameworks.

The interface:

 public interface IView : IMVPView  
   {  
     string SectionTitle { set; }  
     IList<FaqItem> Questions { set; }  
   }  

The Implementation:

 public partial class FaqSection : MVPUserControl<Presenter>, IView  
   {  
     public string SectionTitle { protected get; set; }  
     public IList<FaqItem> Questions  
     {  
       set  
       {  
         rptFaqs.DataSource = value;  
         rptFaqs.DataBind();  
       }  
     }  
   }  

As you probably saw from our base user control, our presenter get magically wired up and it calls the Init() method when loaded.

The Presenter base class:

 public class MVPPresenter<TView> : MVPPresenter where TView : IMVPView  
   {  
     public new TView View  
     {  
       get { return (TView)base.View; }  
       set { base.View = value; }  
     }  
     protected internal override void CheckViewType(object view)  
     {  
       Type viewType = view.GetType();  
       Type allowedType = typeof(TView);  
       if (viewType != allowedType && allowedType.IsAssignableFrom(viewType) == false)  
         throw new InvalidOperationException(Resources.Constants.ObjectType + " " + viewType.Name + Resources.Constants.NotAllowed + " " + GetType().Name);  
     }  
   }  
   public abstract class MVPPresenter  
   {  
     private object m_view;  
     public object View  
     {  
       get { return m_view; }  
       set  
       {  
         CheckViewType(value);  
         m_view = value;  
       }  
     }  
     public virtual void Init() { }  
     protected internal virtual void CheckViewType(object view) { }  
   }  

The Presenter:

  public class Presenter :MVPPresenter<IView>  
   {  
     private readonly IFaqRepository m_repository;  
     public Presenter(IFaqRepository repository)  
     {  
       m_repository = repository;  
     }  
     public override void Init()  
     {  
       base.Init();  
       var section = m_repository.GetFaqSection(View.CurrentItemId);  
       View.Questions = section.Questions;  
       View.SectionTitle = section.SectionTitle;  
     }  
   }  

As you can see, all the asp.net and sitecore nastiness (Sitecore.Context.Item) is hidden in the view. The presenter just deals with initiating the repositories and doing some simple logic and passing it down to the view.

I can test this code. Happy days!

Sitecore cache clear

Sometime you just need to clear the caches on a sitecore server. Use the script below to save having the sitecore folder on your front end web server.

 <script language="CS" runat="server">  
  protected override void OnLoad(EventArgs e)  
  {  
  if (Request.QueryString["ClearCache"] != null)  
  {  
   Sitecore.Context.Database = this.WebContext.Database;  
   Sitecore.Context.Database.Engines.TemplateEngine.Reset();  
   Sitecore.Context.ClientData.RemoveAll();  
   Sitecore.Caching.CacheManager.ClearAllCaches();  
   Sitecore.Context.Database = this.ShellContext.Database;  
   Sitecore.Context.Database.Engines.TemplateEngine.Reset();  
   Sitecore.Caching.CacheManager.ClearAllCaches();  
   Sitecore.Context.ClientData.RemoveAll();  
   InfoLiteral.Text = "All sitecore cache has been cleared. DateTime:" + DateTime.Now.ToString();  
  }  
  else  
  {  
   InfoLiteral.Text = "You haven't entered correct query parameter. DateTime:" + DateTime.Now.ToString();  
  }  
  base.OnLoad(e);  
  }  
  private Sitecore.Sites.SiteContext WebContext  
  {  
  get  
  {  
   return Sitecore.Configuration.Factory.GetSite(Sitecore.Configuration.Settings.GetSetting("stagingSiteContext", "website"));  
  }  
  }  
  private Sitecore.Sites.SiteContext ShellContext  
  {  
  get  
  {  
   return Sitecore.Configuration.Factory.GetSite("shell");  
  }  
  }  
 </script>  

Enjoy!

Sitecore rebuild

If you run a sitecore setup with multiple front end servers, you will find that you occasionally need to manually rebuild the lucene indexes or the link database.

Place the script below in the root of your site. Very Handy.

  <script runat="server">  
     public string Database  
     {  
       get  
       {  
         return string.IsNullOrEmpty(Request["dbname"]) ? "web" : Request["dbname"];  
       }  
     }  
     protected void Page_Load(object sender, EventArgs e)  
     {  
       var database = Sitecore.Configuration.Factory.GetDatabase(Database);  
       if (Request.QueryString["index"] == "true")  
         RebuildIndexes(database);  
       if (Request.QueryString["link"] == "true")  
         RebuildLinksDatabase(database);  
     }  
     protected void RebuildIndexes(Sitecore.Data.Database database)  
     {  
       if (database != null)  
       {  
         DateTime start = DateTime.Now;  
         for (int i = 0; i < database.Indexes.Count; i++)  
         {  
           WriteToResponse(String.Format("Rebuilding {0} index", database.Indexes[i].Name));  
    try {  
           database.Indexes[i].Rebuild(database);  
    }  
    catch(Exception ex)  
    {  
    WriteToResponse("Error while rebuilding index: " + ex.Message);  
    }  
           WriteToResponse(String.Format("Index {0} rebuilt", database.Indexes[i].Name));  
         }  
         TimeSpan elapsed = DateTime.Now - start;  
         WriteToResponse("Indexes rebuilt in " + elapsed.TotalMilliseconds + " ms");  
       }  
     }  
     protected void RebuildLinksDatabase(Sitecore.Data.Database database)  
     {  
       WriteToResponse("Rebuilding Links database");  
       DateTime start = DateTime.Now;  
       Sitecore.Globals.LinkDatabase.Rebuild(database);  
       TimeSpan elapsed = DateTime.Now - start;  
       WriteToResponse("Links database rebuilt in " + elapsed.TotalMilliseconds + " ms");  
     }  
     protected void WriteToResponse(string str)  
     {  
       Response.Write(str + "<br/>");  
       Response.Flush();  
     }      
   </script>