Feeds:
Posts
Comments

We resolved an issue today where a SharePoint Admin mistakenly deleted an out-of-the-box SharePoint Timer Job. Many of the “resolutions” we found while searching for a fix to the “uh oh!” involved the recreation or restoration of several–if not all–of the timer jobs which come with SharePoint. Honestly, that made me nervous because I’d rather not be touching something that is working just fine.

Knowing that custom timer jobs get attached through a few lines of code, I figured the same could probably be done to fix our single missing timer job. And, sure enough, it could! Here’s the PowerShell we used to restore the mistakenly deleted Information management policy job:

[System.Reflection.Assembly]::Load("Microsoft.Office.Policy, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c");
$wa = get-spwebapplication http://webappurl;
[Microsoft.SharePoint.SPSchedule] $schedule = [Microsoft.SharePoint.SPSchedule]::FromString("weekly at fri 23:00:00");
[Microsoft.Office.RecordsManagement.Internal.PolicyUpdatesJobDefinition] $policyJob = New-Object "Microsoft.Office.RecordsManagement.Internal.PolicyUpdatesJobDefinition" ($wa);
$policyJob.Schedule = $schedule;
$policyJob.Update($true);

To restore your mistakenly deleted job, all you need to do is find the assembly which contains the job (“Microsoft.Office.Policy” above) and the class used to instantiate the SPTimerJob (“Microsoft.Office.RecordsManagement.Internal.PolicyUpdatesJobDefinition” above). Substitute your assembly and class as needed above. Of course, you’ll want to use your web application URL instead of “webappurl.”

Mark Arend has a wonderful post on MSDN Blogs about the strings for SPSchedule.FromString.

We have several SharePoint farms in our control spanning the range of SP2010 to SP2013 to SharePoint Online (SPO). When it comes to providing a unified view of all of the site collections usage data to their owners, it can be a slight challenge. To meet those needs, I created some SharePoint timer jobs last year that iterate through all of the site collections and–using CSOM–store their reporting information (such as URL, title, owner, hits, unique users, and other classification information) in one list. From that list, we generate reports for the site owners and anyone else in need of the information.

When it came time to add SharePoint Online into the mix, I ran into a slight problem: How to get usage hits and unique users? The data is in there. You can find it in an Excel workbook under Site Settings > Site Collection Administration - Popularity and Search Reports > Usage. However, none of the APIs would return me anything but big fat zeroes!

I’ve asked around on several of the Microsoft social channels (such as TechNet–“Client UsageInfo ‘implementation specific period’“–and no less than three Yammer groups), but have yet to get any answers (guess I’m not very socially noticeable on this topic–we’ll see how this post does). As a last resort, I turned to teaching my timer job how to behave like a person on a browser. Let me show you what I mean…

First, let’s start with the timer job itself. This will be quick. I’m not going to cover much here because it is well documented in the OfficeDev PnP Solution: Core.TimerJobs.Samples. If you need help creating your timer job, look at the samples.

Now, let’s get to the meat. As I stated in my TechNet post, I’ve resorted to using a POST WebRequest to pull down the Excel document that contains the hits and unique users information (Site Settings > Popularity and Search Reports > Usage) and DocumentFormat.OpenXml to read it. I’m not very fond of this solution, though, because the ASPX pages don’t support OAuth and, thus, I can’t use app permissions to do it. I’d much rather be using an API than pretending to be a user with a browser but, as I said, the APIs always return zero (0) and I’ve not found any other alternative.

“So, how do I programmatically retrieve the usage data from SharePoint Online?” you ask? Below is a walkthrough to show how I did it in Visual Studio 2013. In this walkthrough, I’m going to “put the cart before the horse.” In other words, I’m going to build up the supporting methods before I walk you through building the method that uses them.

  1. Add a reference to .NET Framework 4.0’s WindowsBase assembly.
  2. Add the following NuGet packages:
    1. App for SharePoint Web Toolkit (for SharePoint Online)
    2. OfficeDevPnPCore16 (this will also install Microsoft.AspNet.WebApi.Client, Microsoft.AspNet.WebApi.Core, Microsoft.SharePointOnline.CSOM, and Newtonsoft.Json)
    3. DocumentFormat.OpenXml
  3. Add the following using statements:
    1. using System.Net
    2. using System.Security
    3. using Microsoft.SharePoint.Client
    4. using DocumentFormat.OpenXml.Packaging
    5. using DocumentFormat.OpenXml.Spreadsheet
  4. Since OAuth is not supported, we will need to get SharepointOnlineCredentials. This method takes a password in the form of a SecureString. In this walkthrough, I use a simple getter method called MyPassword with my password hard-coded as a string. You, of course, will want to replace or modify this method with something a bit more secure.
    unsafe private SecureString MyPassword
    {
      get
      {
        char[] chars = "myPassword".ToCharArray();
        SecureString myPassword;
        fixed (char* pChars = chars)
        {
          myPassword = new SecureString(pChars, chars.Length);
        }
        return myPassword;
      }
    }

    Note that you will have to check Allow unsafe code in the project’s build properties in order to compile with this method.

  5. The next two methods, GetCellValue and GetCellValues<T> are used to read the Excel spreadsheet we get back from SharePoint. The bulk of this code came from a Microsoft sample. I’ve modified it to reduce the overhead involved in opening the Stream object so we don’t have to do it for every cell we want to read.
    /// &lt;summary&gt;
    /// Retrieve the value of a cell, given a file stream, sheet name, and address name.
    /// &lt;/summary&gt;
    private string GetCellValue(System.IO.Stream stream, string sheetName, string addressName)
    {
      string value = null;
    
      // Open the spreadsheet document for read-only access.
      using (SpreadsheetDocument document = SpreadsheetDocument.Open(stream, false))
      {
        // Retrieve a reference to the workbook part.
        WorkbookPart wbPart = document.WorkbookPart;
    
        // Find the sheet with the supplied name, and then use that 
        // Sheet object to retrieve a reference to the first worksheet.
        Sheet theSheet = wbPart.Workbook.Descendants&lt;Sheet&gt;().Where(s =&gt; s.Name == sheetName).FirstOrDefault();
    
        // Throw an exception if there is no sheet.
        if (theSheet == null)
        {
          throw new ArgumentException("sheetName");
        }
    
        // Retrieve a reference to the worksheet part.
        WorksheetPart wsPart = (WorksheetPart)(wbPart.GetPartById(theSheet.Id));
    
        // Use its Worksheet property to get a reference to the cell 
        // whose address matches the address you supplied.
        Cell theCell = wsPart.Worksheet.Descendants&lt;Cell&gt;().Where(c =&gt; c.CellReference == addressName).FirstOrDefault();
    
        // If the cell does not exist, return an empty string.
        if (theCell != null)
        {
          value = theCell.InnerText;
    
          // If the cell represents an integer number, you are done. 
          // For dates, this code returns the serialized value that 
          // represents the date. The code handles strings and 
          // Booleans individually. For shared strings, the code 
          // looks up the corresponding value in the shared string 
          // table. For Booleans, the code converts the value into 
          // the words TRUE or FALSE.
          if (theCell.DataType != null)
          {
            switch (theCell.DataType.Value)
            {
              case CellValues.SharedString:
    
                // For shared strings, look up the value in the
                // shared strings table.
                var stringTable = wbPart.GetPartsOfType&lt;SharedStringTablePart&gt;().FirstOrDefault();
    
                // If the shared string table is missing, something 
                // is wrong. Return the index that is in
                // the cell. Otherwise, look up the correct text in 
                // the table.
                if (stringTable != null)
                {
                  value = stringTable.SharedStringTable.ElementAt(int.Parse(value)).InnerText;
                }
                break;
    
              case CellValues.Boolean:
                switch (value)
                {
                  case "0":
                    value = "FALSE";
                    break;
                  default:
                    value = "TRUE";
                    break;
                }
                break;
            }
          }
        }
      }
      return value;
    }
    
    private List&lt;T&gt; GetCellValues&lt;T&gt;(System.IO.Stream stream, string sheetName, string[] addressNames)
    {
      var vals = new List&lt;T&gt;();
    
      // Open the spreadsheet document for read-only access.
      using (SpreadsheetDocument document = SpreadsheetDocument.Open(stream, false))
      {
        // Retrieve a reference to the workbook part.
        WorkbookPart wbPart = document.WorkbookPart;
    
        // Find the sheet with the supplied name, and then use that 
        // Sheet object to retrieve a reference to the first worksheet.
        Sheet theSheet = wbPart.Workbook.Descendants&lt;Sheet&gt;().Where(s =&gt; s.Name == sheetName).FirstOrDefault();
    
        // Throw an exception if there is no sheet.
        if (theSheet == null)
        {
          throw new ArgumentException("sheetName");
        }
    
        // Retrieve a reference to the worksheet part.
        WorksheetPart wsPart = (WorksheetPart)(wbPart.GetPartById(theSheet.Id));
    
        foreach (string addressName in addressNames)
        {
          // Use its Worksheet property to get a reference to the cell 
          // whose address matches the address you supplied.
          Cell theCell = wsPart.Worksheet.Descendants&lt;Cell&gt;().Where(c =&gt; c.CellReference == addressName).FirstOrDefault();
    
          // If the cell does not exist, return an empty string.
          if (theCell != null)
          {
            var value = (T)Convert.ChangeType(theCell.InnerText, typeof(T));
            vals.Add(value);
          }
        }
      }
      return vals;
    }
    
  6. Now comes the method which drives it all: GetSPOUsageStats. This method takes a site collection URL and two references which it will populate: one for hits and one for unique users.I give specific cell references and do some math in lines 202 through 212. These cells refer to the last three full months of data. In my case, I want the sum of the last three full months of hits and an average daily usage for the last three full months (you may note that an individual unique user is probably being counted up to three times since there is no way to filter this individual as one visit over the sum of the three months). I never use the current month’s data because the time period it represents varies in length. You will likely want to modify lines 202 through 212 to suite your needs.
    private void GetSPOUsageStats(string siteUrl, ref long hitsCnt, ref double userAvg)
    {
      Uri siteUri = new Uri(siteUrl);
      ClientContext clientContext = new ClientContext(siteUri);
    
      var creds = new SharePointOnlineCredentials("me@mytenant.onmicrosoft.com", MyPassword);
      var authCookie = creds.GetAuthenticationCookie(siteUri);
    
      string postData = "__EVENTTARGET=__Page&__EVENTARGUMENT=ReportId%3D6bbf6e1c-d79a-45da-9ba0-d0c3332bf6e2";
      byte[] byteArray = Encoding.UTF8.GetBytes(postData);
    
      HttpWebRequest webRequest = HttpWebRequest.CreateHttp(siteUrl + "/_layouts/15/Reporting.aspx?Category=AnalyticsSiteCollection");
      webRequest.Method = "POST";
      webRequest.Accept = "text/html, application/xhtml+xml, */*";
      webRequest.ContentType = "application/x-www-form-urlencoded";
      webRequest.ContentLength = byteArray.Length;
      webRequest.CookieContainer = new CookieContainer();
      webRequest.CookieContainer.Add(new Cookie("SPOIDCRL",
            authCookie.TrimStart("SPOIDCRL=".ToCharArray()), //This is to remove the prefix from the cookie's value
            String.Empty,
            siteUri.Authority));
    
      using (System.IO.Stream dataStream = webRequest.GetRequestStream())
      {
        // Write the data to the request stream.
        dataStream.Write(byteArray, 0, byteArray.Length);
        // Close the Stream object.
        dataStream.Close();
        WebResponse webResponse = null;
        try
        {
          webResponse = webRequest.GetResponse();
        }
        catch (Exception ex)
        {
          Console.WriteLine("----------------------------------");
          Console.WriteLine("ERROR processing GetSPOUsageStats for {0}", siteUrl);
          Console.WriteLine();
          Console.WriteLine(ex.Message);
          Console.WriteLine();
          Console.WriteLine(ex.StackTrace);
          Console.WriteLine("----------------------------------");
        }
        if (null != webResponse)
        {
          using (webResponse)
          {
            byte[] buffer = new byte[1024];
            System.IO.MemoryStream memStream = null;
            try
            {
              memStream = new System.IO.MemoryStream();
              using (System.IO.Stream input = webResponse.GetResponseStream())
              {
                //total = input.Length;
    
                int size = input.Read(buffer, 0, buffer.Length);
                while (size &gt; 0)
                {
                  memStream.Write(buffer, 0, size);
                  //received += size;
    
                  size = input.Read(buffer, 0, buffer.Length);
                }
                input.Flush();
              }
    
              var hits = GetCellValues&lt;long&gt;(memStream, "Views", new string[] { "B54", "B55", "B56" });
              hitsCnt = hits.Sum();
    
              var uniqueUsers = GetCellValues&lt;int&gt;(memStream, "Views", new string[] { "C54", "C55", "C56" });
              var months = GetCellValues&lt;double&gt;(memStream, "Views", new string[] { "A54", "A56" });
              var dateStart = new DateTime(1900, 1, 1).AddDays(months[0]);
              dateStart = new DateTime(dateStart.Year, dateStart.Month, 1);
              var dateEnd = new DateTime(1900, 1, 1).AddDays(months[1]);
              dateEnd = new DateTime(dateEnd.Year, dateEnd.Month + 1, 1);
              var days = (dateEnd - dateStart).Days;
              userAvg = Math.Round((double)uniqueUsers.Sum() / days, 4);
            }
            catch (Exception ex)
            {
              Console.WriteLine("----------------------------------");
              Console.WriteLine("ERROR reading GetSPOUsageStats for {0}", siteUrl);
              Console.WriteLine();
              Console.WriteLine(ex.Message);
              Console.WriteLine();
              Console.WriteLine(ex.StackTrace);
              Console.WriteLine("----------------------------------");
            }
            finally
            {
              if (null != memStream)
              {
                memStream.Flush();
                memStream.Close();
                memStream.Dispose();
              }
            }
          }
        }
      }
    }
    
  7. Now, all we have to do is call GetSPOUsageStats. In a console application, this looks something like this:
    static void Main(string[] args)
    {
      Program p = new Program();
      long hitsCnt = 0;
      double userAvg = 0;
      p.GetSPOUsageStats("https://mytenant.sharepoint.com/sites/testsite", ref hitsCnt, ref userAvg);
      Console.WriteLine("Last three full months hits: {0}", hitsCnt);
      Console.WriteLine("Average unique users per day for last three full months: ", userAvg);
      Console.WriteLine("Press any key to quit.");
      Console.ReadKey();
    }
    

Download Program.cs DOCX for a source example used to generate this walkthrough. This Word document contains a copy of the Program.cs file.

Extra! Extra! Office 365 User Voice has helped shed some light on the dark side. Just a few days ago, the O365 Feedback Team told us, “We are working on many of these reports areas right now.” Check out the full thread here.

I’ve been working on Apps quite a lot this month and learning all sorts of things like: the difference between Office 365 D (Dedicated) and Office 365 MT (Multi-Tenant); how to setup Access Control Service (ACS); and when to use F5 in Visual Studio and when to package and deploy. It’s been a roller coaster of confusion and trial-and-error as I’ve attempted to wade through all of the How-To’s created over the last couple of years and merge them with all of the changes made available to the products during that same period. This week has brought a lot more clarity and understanding; in short, I think I’m finally starting to grasp the hang of it.

At the end of the day yesterday, I ran into an interesting phenomenon I thought would be worth sharing. Several of the documents I’ve read regarding app event receivers have said they are not supported from an on-premise install of SharePoint (see Where can I use app event receivers? in the MSDN App event receivers FAQ). Thus, online being my “only” option, I was banging my head trying to get event receivers to work in SharePoint Online for a provider-hosted app. No matter what I tried, my attached W3SVC wouldn’t stop on the breakpoint for the IRemoteEventService's ProcessEvent.

Having lost nearly all hope, I shelved my efforts and moved on to something else in the project. Upon installing the app to my on-premise SharePoint server, Visual Studio surprisingly hit my breakpoint in IRemoteEventService's ProcessEvent! Whoa! What’s going on here?

Well, it turns out that my host web (the IIS site hosting the service being called by the event) is on a Dev server. Naturally, it isn’t available publicly. As such, when SharePoint Online went to call the service, it couldn’t establish a connection. Of course, our on-premise SharePoint can connect to the Dev server because they’re on the same side of the firewall. So, what about all the documentation implying this can’t be done? Well, most of the documentation out there doesn’t consider that you can now run ACS from an on-premise server (many were written before ACS was a known option for on-premise).

Notice how the MSDN article doesn’t say it’s not possible; it only indicates that it’s not supported. I suppose such configuration as I have is just too new to support as of the last update of that article (December 31, 2014)–that more testing is needed. I for one am glad app event receivers for provider-hosted apps work when you have an accessible host web and other configurations (like ACS), either way!

Let me know if you encounter any issues using app event receivers with provider-hosted apps in an on-prem SharePoint app web and/or host web.

Steve Peschka put together a great blog post about Bypassing the Multi Authentication Provider Selection Page in SharePoint 2010 that was also incorporated in a Microsoft MSDN article. (If you’re not concerned with SharePoint 2010 and are here for info regarding SharePoint 2013, skip down to the “SharePoint 2013” heading.) The workaround Peschka provides works, but I had a few issues with how it was put together:

  1. It’s not a solution deployment. Manual.😐
  2. You’re replacing an out-of-the-box page. What happens to your changes if a CU updates that page?:-/
  3. The change affects all web applications ran on the WFE server you make the change on. What if you have a web application you don’t want to auto-select a provider?

My first issue was disappointing. My second issue was risky. My third issue? Just not acceptable because we do have multiple web applications and don’t want all of them to auto-select. Continue Reading »

I’ve been struggling for the past week to get some custom Access Denied, Signout, and Login pages implemented in our environment. There is a lot of agreement that Get-SPCustomLayoutsPage, Set-SPCustomLayoutsPage, and Microsoft.SharePoint.Administration.SPWebApplication.UpdateMappedPage() were great additions to SharePoint 2010, but there’s been quite the struggle getting these to work in SharePoint 2013.

The first question to ask is: Are you running the April 2013 CU or later? This is critical! There was a bug in earlier versions of SharePoint which made it so you could set a custom layouts page, but it wouldn’t use it. Be patient after you upgrade your farm, it took several days before ours finally started behaving itself after the CU (go ask the Computer Gods if you want to know why–I just thanked them when it finally worked.)

The next question to ask is: What UI is your site using? Is it using the 2010 UI or the 2013 UI? In many cases, you have upgraded your farm to SP2013, but haven’t upgraded your sites, yet.

As it turns out, the question about the UI is quite important. If your site is using the 2013 UI, you’re fine: By default Get-SPCustomLayoutsPage, Set-SPCustomLayoutsPage, and SPWebApplication.UpdateMappedPage(...) work with a CompatibilityLevel = 15. That’s right, CompatibilityLevel! There is an undocumented parameter on these cmdlets and method (presumably added with the April 2014 CU). If your site is using the 2010 UI, you’ll need to specify a CompatibilityLevel = 14 when using these cmdlets or method. Just be sure to give the correct path for the correct hive/UI: The path must start with “/_layouts/15/” for the 2013 UI (CompatibilityLevel = 15, or omitted), and must start with “/_layouts/” for the 2010 UI (CompatibilityLevel = 14).

Just be sure you deploy your custom pages to both hives. For me, I had packaged the custom pages in a farm solution. Just needed to deploy the pages to both hives by specifying CompatibilityLevel = All when executing Install-SPSolution. (Unfortunately, there’s no way to do this from the deployment in CA.)

Of course, this means you are able to have different pages for each of the mapped pages for each of the hives. Want one Access Denied page for the 2010 UI and a different one for the 2013 UI? No problem!

My thanks to Peter Holpar for his blog post on “How to Override the Default Item Save Event Using Custom Fields.” Gave me the information I needed to create a handy custom field.

During our testing, though, we found that save errors were not being handled correctly. For example, when the “Allow duplicate values” was set to “No” and you tried to set a duplicate value in the field, you got a very ugly error page. So, I went back to Peter’s blog: No solution. I searched all over: no solution. So, back to Peter’s blog. There was one little tidbit that turned into the key to solving the problem and getting SharePoint to properly handle the SPDuplicateValuesFoundException. Peter mentioned an alternative to using SPContext.Current.ListItem.Update() was to use SaveButton.SaveItem(). Close, very close! The parameterless method is actually not available to our custom field, but SaveButton.SaveItem(SPItemContext, bool, string) is. Calling this method instead of the update method took care of the problem. Now, when someone tries to enter a duplicate value in our custom field, we get the nice, red message “This value already exists in the list.”

Now a happy Monday morning!

Radu Tut did a wonderful blog about how to get Search Analytics Reports programmatically in SharePoint 2013. I am quite grateful for his guidance and it’s undoubtedly the most referenced article on the topic (even being referenced in Microsoft blogs and other documentation).

I followed Radu’s guidance and have successfully retrieved usage data from my site collections in 2013, but I’ve discovered a few things I’ve not seen anyone talking about:

  1. It is important to note that the DateTime value you use should be UTC. If not, you will, at some time during the day–depending on your offset–get a “Specified argument was out of the range of valid values” error when trying to get the furthest back data (example: -14 days for Day) if you use Local time. Internally, SharePoint uses DateTime.UtcNow.Date (see reflection of Microsoft.Office.Server.Search.Analytics.AnalyticsItemData in the Microsoft.Office.Server.Search.Applications.dll).
  2. To my surprise, I was able to retrieve usage data from a 2010 site collection in SP2013! Apparently, SharePoint is tracking and compiling this data, but the 2010 UI has no way to interface with the data. Thus, if you need usage data for a 2010 UI site collection in SharePoint 2013, turn to your Devs or PowerShell-savvy Admins: If your site is being crawled, the 2013 usage data you seek is in there!
  3. The time period for SP2013 to compile usage data is fairly lengthy (at least 15 minutes in my tiny dev farm). If you attempt to retrieve the data while SharePoint is compiling it, you’ll get back a big fat ‘0’. For expected results, make sure you’re not retrieving the data while SharePoint is compiling it.🙂 I believe this compilation is done by a timer job once a day at midnight, by default (I was up pretty late working on my project when I discovered this behavior).

Happy reporting!