Re-ghost items present in the master page gallery

Introduction

I write this post because I recently had to clean up for a customer a MOSS 2007 application where most of the master pages, layout pages and css were customized or were deployed with an upload of a file in the master page gallery. As I wanted to be able to deploy this application changes with a solution (.wsp) and wanted to perform my development work with Visual Studio, I had to re-ghost all the pages.

What I noticed first was it is not so easy to know what files were customized or were deployed by an upload, and for the customized files, what was the feature that has deployed them. I had to develop a custom tool to perform an audit, and had to think about specific practices in order to re-ghost the pages doing the less changes as possible to the original applicative architecture.
This work gave me the idea of this post that will present these practices, and the custom tool you may need to use if you are one day in the same case. By the way, I think it will be usefull for some, because deployment with SharePoint stays one of its dificult tasks, and because it is not the first time I see this scenario in a company:
Some development teams forget that with SharePoint, the deployment tasks can take long time, is risky, and don't think so much about preparing and testing it.
Then, delivery time comes and is difficult, and finaly, to respect the delivery dead line, the project is often deployed without respecting the best practices. Features and files are deployed by x-copy, or by a straight upload to the gallery, last bugs fixing are made directly on Production Environment with SharePoint Designer.
Then one day, the company needs to perform changes to the application and ask for support from a SharePoint consultant. And one of the first task of the consultant is to clean up the consequences of the first "quick" deployment.
To enumerate the different cases that lead to unghosted state, the way to identify them with the custom tool, the different practices to re-ghost a page, what is best than a tutorial?

I - Examining the different cases of unghosting and the way to obtain them

To start the tutorial, I will prepare a feature involving 3 master pages, will deploy it in Litware Portal, and perform actions to unghost the pages. I will examine the differences. I will also compare with a master page deployed by a straight upload to the gallery.

1 - the feature

I prepare a feature with 3 minimal master pages :
minimal1.master,
minimal2.master,
minimal3.master.

here is the feature xml file code :

<?xml version="1.0" encoding="utf-8" ?>
<Feature xmlns="https://schemas.microsoft.com/sharepoint/"
 Id="ABC820CB-B2BB-4779-868D-33F1EBCE3202"
 Title="e-marguerite : test re-ghosting"
 Description="Add three master pages to the master page gallery in order to perform unghosting and re-ghosting tests"
 Scope="Web"
 Hidden="TRUE"
 Version="1.0.0.0"
  AlwaysForceInstall="TRUE">
<ActivationDependencies></ActivationDependencies>
<ElementManifests>
 <ElementManifest Location="Elements.xml"/>
</ElementManifests>
<Properties>
</Properties>
</Feature>

And this is the Elements.xml code:

<?xml version="1.0" encoding="utf-8" ?>
<Elements xmlns="https://schemas.microsoft.com/sharepoint/">
 <Module Name="PlaceHolderMaster" List="116" Url="_catalogs/masterpage">
      <File Url="minimal1.master" Type="GhostableInLibrary" IgnoreIfAlreadyExists = "TRUE"></File>
      <File Url="minimal2.master" Type="GhostableInLibrary" IgnoreIfAlreadyExists = "TRUE"></File>
      <File Url="minimal3.master" Type="GhostableInLibrary" IgnoreIfAlreadyExists = "TRUE"></File>
 </Module>
</Elements>

2 - A look to Master page gallery

So when I have installed and activated my feature I can see my master pages in the Master Page Gallery.

3 - the 3 ways to obtain unghosted files

Now, we are going to unghost the master pages minimal2.master and minimal3.master.
We will also upload a new master page minimal4 that will be also unghosted.

• With SharePoint designer
  I am now opening my site in SharePoint Designer. I open minimal2.master with check-out, I perform a change and save.
  SharePoint Designer warn me I am going to customized the file.
  I accept, then, I publish the page, and as a result, a "i" icon appears beside the file meaning it's customized.
  
  
  
  
• Replacing an existing page by an upload in the master page gallery.
  In the master page gallery I click Upload and choose upload a document.
  
  
  
  I upload a file with the name : "minimal3.master". Then I go back to SharePoint Designer, and hit F5. Nothing has change for minimal3.master item except the modified date.
  However, the file is actually customized.
  
  
  
  


• Straight Upload of a file in the gallery
  Now, using the Upload functionnality we used before, we upload a new master page in the gallery: minimal4.master
  It's appearing with the other deployed by feature.
  
  
  
  
  When we check in SharePoint Designer, the page is present but no indication about its deployment mode.
  
  
  
  

So, SharePoint Designer indicate just one case of the three:
When the page has been unghosted with SharePoint Designer.
So how to know for the other page if you haven't unghosted them yourself?

II - A custom tool to audit files in order to know if they are ghosted.

I wrote an Application Page that check the files of the lists of the site root web of a site collection.
Let's see how does it work with the previous files.
I call my page from any sub site of my site collection by typing the site url and my page name (in my case "_layouts/_checkcustomizedstatus.aspx).



I click the first link (Get root web list name) to have the name of all the lists of my Site Collection site root .
I can see my master page gallery.



I copy the name of the list which I want to check files and paste the name in the text box.
Then I click the second link.



So, let's examine the result:

1. We have a table that give for each item:
• the value of the file property vti_setuppath.
  This indicates the path to the physical file in the server File System that was used to deployed the item in the list.
  here is the MSDN entry.
  
• The value of the file property: hasdefaultcontent
  If it set to true, that means that the file is templated, so linked to a template present in the server File System.
  here is the MSDN entry.
• One of the three possible value regarding customization:
  customized, uncustomized, none.
  


2. Notice the specific values for the files we have worked with:
   
   
   
   


3. The Application Page has written a physical file in the server File System for each "customized" or "none" files in a "C:\Unghosted Files" directory.
   
   
   
   Notice that we have our unghosted files, and that the other files are always unghosted in an out of the box MOSS 2007 portal.
   
   

Conclusion :

• For a file deployed with a feature, then, customized with SharePoint designer or replaced by an upload, we have:
  - hasdefaultcontent set to empty.
  - set up path non empty.
• For a file deployed by a straight upload, we have:
  - hasdefaultcontent set to empty.
  - set up path set to empty.
  

here is the html and c# code of the Application Page:

<%@ Page Language="C#" AutoEventWireup="true" %>
<%@ Register TagPrefix="SharePoint" Namespace="Microsoft.SharePoint.WebControls" Assembly="Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Register TagPrefix="Utilities" Namespace="Microsoft.SharePoint.Utilities" Assembly="Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Import Namespace="Microsoft.SharePoint" %>
<%@ Import Namespace="Microsoft.SharePoint.ApplicationPages" %>
<%@ Import Namespace="System.IO" %>
<html xmlns="https://www.w3.org/1999/xhtml">
<head>
    <title></title>
</head>
<body>
    <script runat="server">
        void getList(object sender, EventArgs e)
        {
            string myResult = "";
            string myListName = txt1.Text;
            //SPList myList = SPContext.Current.Web.Lists[myListName];
            myResult += "<table style='width:80%;border:solid 1px gray;border-collapse:collapse'>";
            myResult += "<tr>" + "<td style='border:solid 1px gray;font-weight:bold' >" + "Display Name" + "</td>" + "<td style='border:solid 1px gray;font-weight:bold' >" + "Item Count" + "</td>" + "<td style='border:solid 1px gray;font-weight:bold' >" + "Base Type" + "</td>" + "</tr>";
            foreach (SPList aList in SPContext.Current.Site.RootWeb.Lists)
            {
                myResult += "\n<tr>" + "<td style='border:solid 1px gray;' >" + aList.Title + "</td>" + "<td style='border:solid 1px gray;' >" + aList.ItemCount + "</td>" + "<td style='border:solid 1px gray;' >" + aList.BaseType + "</td>" + "</tr>";
            }
            myResult += "</table>";
            result.Text = myResult;
        }
        void WriteCustomizedAndNone(object sender, EventArgs e)
        {
            string myResult = "";
            string myListName = txt1.Text;
            SPList myList = SPContext.Current.Web.Site.RootWeb.Lists[myListName];
            if (!Directory.Exists("C:\\Unghosted Files\\"))
            {
                Directory.CreateDirectory("C:\\Unghosted Files\\");
            }
            myResult += "Status of items<br><br>";
            myResult += "<table style='width:80%;border:solid 1px gray;border-collapse:collapse'>";
            myResult += "<tr>" + "<td style='border:solid 1px gray;font-weight:bold' >" + "Name" + "</td>" + "<td style='border:solid 1px gray;font-weight:bold' >" + "Setup Path" + "</td>" + "<td style='border:solid 1px gray;font-weight:bold' >" + "has default content" + "</td>" + "<td style='border:solid 1px gray;font-weight:bold' >" + "is customized" + "</td>" + "</tr>";
            foreach (SPListItem anItem in myList.Items)
            {
                myResult += "<tr>" + "<td style='border:solid 1px gray;' >" + anItem.Name + "</td>" + "<td style='border:solid 1px gray;' >" + anItem.File.Properties["vti_setuppath"] + "</td>" + "<td style='border:solid 1px gray;' >" + anItem.File.Properties["vti_hasdefaultcontent"] + "</td>" + "<td style='border:solid 1px gray;' >" + anItem.File.CustomizedPageStatus + "</td>" + "</tr>";
                if (anItem.File.CustomizedPageStatus.ToString() != "Uncustomized")
                {
                    byte[] myBytes = anItem.File.OpenBinary();
                    System.IO.File.WriteAllBytes("c:\\Unghosted Files\\" + anItem.File.Name, myBytes);
                }
            }
            myResult += "</table>";
            result.Text = myResult;
        }
    </script>
    <form runat="server">
        <asp:LinkButton runat="server" ID="btn1" Text="Get root web lists name" OnClick="getList" />
        <br />
        <br />
        List Name
        <asp:TextBox runat="server" ID="txt1"></asp:TextBox>
        <br />
        <br />
        <asp:LinkButton runat="server" ID="btnItemSample" Text="Get files customized status and write customized and none on disk"
            OnClick="WriteCustomizedAndNone" />
        <br />
        <br />
        <br />
        <br />
        <asp:Label runat="server" ID="result"></asp:Label>
        
    </form>
</body>
</html>

III - Re-ghosting the pages

We are now going to re-ghost the pages in order to:

• make their changes performed with Visual Studio be visible in the portal,
• make them upgradable by a solution (.wsp) deployment.

We will see that the policy choosen to re-ghost the page will depend on the way it has been un-ghosted.

1 - reset to Site Definition with Sharepoint designer

We are going to re-ghost minimal2.master.
As it was customized with SharePoint designer, SharePoint designer offers us the possibility to roll back the actions we have done on this page, aka:

• Made changes.
• un-ghosted it.

So if you make just a roll back you will loose all the changes performed on the page. If you want to re-ghost the page while saving the changes do the following :

• Go to "c:\unghosted files\" and copy the file minimal2.master.
• check the feature path on the _checkcustomizedstatus.aspx page.
• overwrite the minimal2.master in the feature directory by the one you have just copied from "c:\unghosted files\".

Now, we have saved the changes, we can re-ghost the page.
Right click the page in SharePoint Designer and select Reset to Site Definition. In the warning pop-up click OK. SharePoint Designer make a copy of the customized file anyway, thus you cannot loose your changes.



Now refresh the _checkcustomizedstatus.aspx page. You see that your file has been re-ghosted. You see also that the changes you had performed to the page are still there.

2 - reset to Site Definition with SharePoint Administration

We are now going to re-ghost minimal3.master, and we cannot do it with SharePoint Designer since this page has not been un-ghosted with SharePoint Designer. By the way if you right click the page in SharePoint designer, you will not see the menu entry : "Reset to Site Definition".
In order to reset minimal3.master to Site Definition we have to do the following :

• First of all do not forget to replace the file minimal3.master present in the feature directory by the one that is in "c:\unghosted files\" in order to not loose your modifications. (Except if you mis-overwrite the page when you performed the upload and want a real roll back).
• We need the page url to make a Reset to Site definition in SharePoint so, go to master page gallery, right click the page and choose "View Porperties".
• On the page click the page name Link.
  
  
  
  
• In the case of a master page, you will obtain an error message, but you will also have the Url. Copy it.
  
  
  
  
• Go tohe Site Settings page and in the column Look And Feel, click to Reset to site definition.
  
  
  
  
• Paste the Url in the text box, and click reset.
• On the warning pop up clik yes.

We can now check on the _checkcustomizedstatus.aspx page. You see that your file has been re-ghosted. You see also that the changes you had performed to the page are still there.

3 - Ghosting a file that has never been ghosted.

To finsih, we have to ghost the file that was just uploaded in the Gallery: minimal4.master. I don't write re-ghost since it has never been ghosted.
This is the most delicate operation.
We can imagine to go to the "C:\unghosted files\" directory, copy the file in the feature, change the Elements.xml file do add minimal4.master entry, then deactivate the feature and reactivate it in order to overwrite minimal4.master in the master page gallery.
We cannot do that because all we can do with a provisionning feature is to add new files but neither overwrite nor remove any files.
Then, the only way to ghost minimal4.master is to add the same master page with a new name, then change from minimal4.master to this new file for all the publishing pages that are using it.

• We go to "C:\unghosted files\" directory and copy the file (don't forget that in real life, you will certainly have not the original minimal4.master file, but only its name in the master page gallery). Paste it in the feature directory with a new name. For instance:
  minimal4_Reghost.master
• Then, add a new entry in Elements.xml for this new page.
• Deactivate the feature in order to be allowed to re-activate it.
• Activate the feature.
• Now chek _checkcustomizedstatus.aspx page. You can see your new master with the status uncustomized.
  
  
  
  
• Change the reference to minimal4.master to minimal4_Reghost.master for all the pages of your site collection that use it.
• Once it is done, delete minimal4.master. (you will not be allowed to delete it by SharePoint untill there is no more reference to this master for any page of your Site Collection).
• By the way, you can also delete the copy of minimal2.master made by SharePoint Designer.

When it's done clean up job is finished you may obtain that in _checkcustomizedstatus.aspx page.