This gallery contains 5 photos.
Blog Archives
Microsoft Graph API, Throttling & SharePoint Lists/Libraries – HTTP 429 Error Code
When developing against the Microsoft Graph you may find yourself experiencing HTTP 429 Error Codes now that resource throttling is being implemented in different areas of the Graph.
I came up against a strange and somewhat misleading one this week which is worth being aware of if you are using the Graph to access SharePoint lists and libraries using the /sites/ area of the Graph.
I had a service running which started reporting HTTP 429 error codes. I read through all the latest published documentation to try a figure out how the throttling has been implemented and what the limitations are to see what part of the code could be triggering the throttling. As you’ll find the documentation is very non-committal and mostly serves to justify why there are no specific limits and rather algorithms that dynamically determine the throttling based on a large number of dynamic criteria. All of this sounds really fancy and advanced but is not very helpful when trying to identify what could be causing the throttling issue, or what limit your code is hitting.
Here’s the Microsoft documentation links which are well worth the read:
Microsoft Graph throttling guidance
Updated guidance around SharePoint web service identification and throttling
Avoid getting throttled or blocked in SharePoint Online
Most of the above advice is summarised in this section I took from one of those official documents on handling throttling with the Graph API (Feb 2018)
Best practices to handle throttling
The following are best practices for handling throttling:
- Reduce the number of operations per request.
- Reduce the frequency of calls.
- Avoid immediate retries, because all requests accrue against your usage limits.
When you implement error handling, use the HTTP error code 429 to detect throttling. The failed response includes the Retry-After field in the response header. Backing off requests using the Retry-After delay is the fastest way to recover from throttling because Microsoft Graph continues to log resource usage while a client is being throttled.
-
Wait the number of seconds specified in the Retry-After field.
-
Retry the request.
-
If the request fails again with a 429 error code, you are still being throttled. Continue to use the recommended Retry-After delay and retry the request until it succeeds
This advise all makes sense so that if your code is making a lot of calls (think migrating SharePoint items or doing bulk updates) that the Graph may tell you to slow down. When I was investigating my scenario however, it just didn’t make sense that the code was generating enough traffic to worry the Graph (Office 365 service). The telemetry was telling me the code had made around 2,500 Graph calls spread over a period of 24 hours and this was also spread across more than 100 users from a number of different Office 365 tenants.
Diving deeper into the telemetry a pattern quickly emerged, the 429 errors were being returned in response to a Graph call to get a list item based on a column value. Something along these lines:
https://graph.microsoft.com/v1.0/sites/{site-id}/lists/{list-id}/items?filter=Fields/Title eq 'testitem'
This call didn’t fail all the time, if fact it only seemed to get the 429 error in less than 10% of the cases.
Having spend many hours over the past few years ‘working with’ SharePoint thresholds and query limitations on large lists and libraries, my mind started to move towards thinking that maybe the 429 error was a bit misleading and was actually failing due to the Graph API hitting SharePoint threshold limitations.
Off to prove my theory, I’ve got a library with just under 5000 items (which is the SharePoint list threshold)
Using the Graph API Explorer I can make a call that queries this SharePoint library for a specific item matching on the Title column value being equal to “upload.log” (a file which I know exists in the SharePoint library).
As expected the item is found and a Success code 200 is returned along with the JSON payload in the response body shown above. Time to prove the theory, what if I now add 2 more files to the same document library and repeat the process?
After uploading 2 more files, the library settings now indicate that we have exceeded the list view threshold.
Now executing the same query in the Graph API explorer gives us the 429 error code. Inspecting the response body we can see the additional error code of “activityLimitReached” and message of “The application or user has been throttled”
Why was this error misleading? Neither the error code or message specifically indicate the issue being related to SharePoint thresholds. The documentation and best practice articles (linked to at the start of this article) regarding this 429 response are written on the premise that the volume and frequency of calls is responsible for the error and hence the guidance to handle the error should be to incrementally back-off and keep trying until you get success. This guidance is totally misguided in the case of hitting the underlying SharePoint threshold limitation as the call will always fail and has nothing to do with the volume or frequency of calls you are making. It will fail if it’s the only call you make all day and no matter how many times you retry, it will always fail.
Sydney SharePoint User Group – The Transition to Modern Office Add-in Development
I had the pleasure this week of speaking at the Sydney SharePoint User Group on the topic of transitioning to the modern Office Add-in development model.
We discussed:
- The existing COM/VSTO Office Add-in development model
- The reasons and drivers for needing a new development model
- What the modern Office Add-in development is and how it works
- Benefits of the modern model
- What this transition means for Office developers
- A look at the typical modern add-in technology stack and discussing some of the options
- The wider Office Developer Vision (Extending Office through add-ins + accessing Office 365 data via Graph)
Thanks to those who attended and as promised here’s a link to the slide deck from the nights presentation.
Transitioning to Modern Office Add-in Development (slide deck)
I’m talking modern Office add-in development at this months Sydney SharePoint User Group
The Transition to Modern Office Add-in Development
Microsoft SharePoint is developed by the same team responsible for the Microsoft Office desktop applications. There has always been a strong connection between these products. Up until recently, developers who wanted to integrate Word, Excel, PowerPoint with SharePoint would use the COM add-in model. But now Microsoft has a new Modern Office Add-in development model.
In this month’s session, Cameron Dwyer – CTO at OnePlace Solutions – will discuss the high level architecture of this modern add-in model, the new technologies and choices involved, transitioning developer skillsets, advantages / disadvantages of the modern model. We will mix in a bit of Azure, Angular, TypeScript, Office 365 APIs, Office UI Fabric, Authentication, and maybe some Applications Insights and DevOps.
OnePlace Solutions is a local success story. Based in Sydney, they have developed products using Microsoft Office COM add-ins. They are one of the first companies globally to have launched add-ins based on the new model for the desktop, iPhone and Android platforms.
For more details on attending this free event in Sydney on 17 October
https://www.meetup.com/en-AU/Sydney-SharePoint-User-Group/events/242349531/
Microsoft Ignite 2017 Highlights for the Office & SharePoint Developer
What a massive week in Orlando for Office developers. So many sessions, so much new information, it really was a case of drinking from the fire hose!
Thankfully most sessions were recorded and are progressively being made available along with the relevant slide decks on https://myignite.microsoft.com/videos
If you only have time to watch one session my pick for the SharePoint/Office Developer would be Accelerate your digital transformation with SharePoint and OneDrive presented by Jeff Teper, Dan Holme, Omar Shahine, Naomi Moneypenny. This session was just non-stop announcements on what’s new in SharePoint, Office and OneDrive. Some of the announcements:
- Office 2019 is coming (Office clients, SharePoint, Exchange, Skype for Business servers – all servers to be available on-prem) – preview due mid 2018
- A SharePoint specific conference (backed by Microsoft) is coming back in 2018 – and yes it’s going to be in Las Vegas
- SharePoint hub sites to group related sites together, share branding and navigation and provide rollup of some information
- SharePoint site provisioning model to allow for scripted customizations during the provisioning process
- SharePoint Company Theming and an online tool to help (https://aka.ms/spthemebuilder)
- SharePoint File Preview webpart (supports over 270 file formats)
- SharePoint Migration tool to move SharePoint content from on-prem to the the cloud (Office 365)
- Improved external link sharing
- OneDrive – deeper integration into Windows 10
- Improved large list support
- New version of the SharePoint mobile app
Going one level deeper into the Office/SharePoint developer world, here’s some of the news that came out regarding:
Microsoft Graph (Build smarter apps with Office using the Microsoft Graph (Yina Arenas))
- Generally Available
- SharePoint Lists
- File Versions
- People API
- App-only support for OneDrive
- Access to Outlook shared calendars, contacts and mail folders
- In Preview
- Graph Extensions for Azure Functions
- Outlook Categories
- Outlook Rules
- Outlook Email Headers
- Outlook Rooms
Office Add-ins (What’s new and what’s coming for Outlook add-ins (Wey Love, Harshit Kumur))
- Generally Available
- Outlook Add-ins for Android
- Single sign-on API (Web clients)
- Auto-open task pane
- Preview
- Single sign-on API for Outlook
- Centralized Deployment for Outlook Add-ins
- Adaptive card support in Outlook (OWA)
SharePoint Framework and Patterns and Practices
- SharePoint Framework became available on-prem with SharePoint Server Feature pack 2 (this was released a couple of weeks before Ignite)
- SharePoint Framework Extensions – custom snippets on every page (aka delegate controls) context buttons/actions, customized field rendering
- Reusable controls and Office Fabric Core
One of the more complex areas of doing modern Office development is authentication and authorization (especially when connection back into Office 365 services). This story is becoming a lot simpler if you can go exclusively through the Graph API. Vittorio Bertocci did a very entertaining session (Office development: Authentication demystified) that really shows you an on-rails experience for how to accomplish this from many different technologies and context. It is well worth a watch and has the potential to save you a lot of time and head scratching. This area of Office development has changed drastically over the last couple of years and there is a lot of conflicting information on the web and much of it is out of date.
Yina Arenas
Vittorio Bertocci
Wey Love
Getting Started: The Basics of Using PowerShell with SharePoint
I’m a SharePoint guy, and certainly no PowerShell expert. I know enough PowerShell to get things done in SharePoint from time to time. I can often go months without using PowerShell and talking to colleagues I don’t think I’m alone in the SharePoint community.
So if you are just starting out with using PowerShell to do something with SharePoint, or it’s been a while and you need a refresher on the basics then you’ve come to the right place.
Before we get started I’m writing this article based on SharePoint 2013 running on a Windows 2012 server.
The ground rules
1) Run your PowerShell scripts from the SharePoint server itself
It is possible to run PowerShell against a remote SharePoint server rather than on the server itself, but it requires some setup on the SharePoint server side and let’s face it, it’s not basic. If you need to do it you need to be searching for “Remote PowerShell in SharePoint”.
2) Use Windows PowerShell ISE (not SharePoint Management Shell)
PowerShell needs to be executed/run at a PowerShell Console or PowerShell Window – this window looks like a standard Command Line window and I don’t find it too inviting.
Instead of the SharePoint 2013 Management Shell, I use the Windows PowerShell ISE.
Why? It’s like comparing Visual Studio with Notepad. The ISE is an environment for developing PowerShell scripts that gives you nice syntax highlighting, debug with breakpoints, intellisense and more. It’s more like a development environment than a command line. I think you’d agree it looks slightly more advanced.
3) Ensure the SharePoint PowerShell snapin is loaded
When using the “SharePoint 2013 Management Shell” (the ugly black one) it automatically loads a “snapin” which is basically a PowerShell extension that gives you a series of commands for working with SharePoint objects. When you use the Windows PowerShell ISE it has no idea of SharePoint, so you need to load the SharePoint snapin manually. The simplest way to do this is just to add the following code snippet to the start of all your scripts.
if((Get-PSSnapin "Microsoft.SharePoint.PowerShell") -eq $null)
{
Add-PSSnapin Microsoft.SharePoint.PowerShell
}
4) Save a script to file before you try to run it
Windows PowerShell ISE will have trouble running a script if you haven’t saved it to disk yet. PowerShell scripts are saved as files with a .ps1 extension.
5) Talk to SharePoint via the SharePoint PowerShell Cmdlets
Here’s a reference of all the SharePoint 2013 PowerShell Cmdlets you can use to work with SharePoint.
Your first script
Pre-flight checklist:
- Start Windows PowerShell ISE
- Add code snippet for loading the SharePoint snapin
- Save the script as a .ps1 file
You can now start cutting and pasting examples from the internet and modifying them to work with your environment. The following 2 lines get a handle on the SharePoint website at the URL http://vs-server12 and then output the ID of the website.
To run the script and see if it work, click the Run button.
Any output from running your script is shown in the output window below the script editor.
You’re on your way, just remember you can only access the SharePoint server on which you are running your scripts. Accessing remote SharePoint servers is possible but you need to do special magic stuff to make that happen 
If you encounter an error along the lines of:
Get-PSSnapin : No Windows PowerShell snap-ins matching the pattern “Microsoft.SharePoint.PowerShell” were found.
You’ve ignored ground rule 4 and the pre-flight checklist by forgetting to save your script before running it (yes, I still do this too).
SharePoint PowerShell How To: Create SharePoint Test Documents in Library Folders for Load/Performance Testing
SharePoint PowerShell How To Series – Foreword
I often find myself scratching around the local drive of my development machines for remnants of PowerShell scripts to generate list & libraries in SharePoint with all sorts of different folder hierarchies to test the performance of code I’m working on to ensure it scales well once the folder & item counts start getting up to high numbers. So rather than keep scratching I’m going to start posting my scripts as I create and use them so I can find/reuse them later on and someone else might find them useful as well. I’m sure they won’t work under all scenarios and situations (and they are not designed to). They are just quick scripts that serve a specific purpose to me at the time and you may be able to tweak to fit your needs.
Create documents in root (or any folder) of a document library
Script description: Takes a file located on c:\ and uploads it to a folder in a document library (renaming it automatically on the way up) to generate 1000 documents in the SharePoint library.
# ---- Script settings ----
$sourceDocumentPath = "C:\temp\Test.docx" # Source document to spawn new documents from for the creation
$newFilenamePrefix = "Test document "
$newFilenameExtension = ".docx"
$numberDocsToCreate = 1000
# Settings for the destination to create documents in
$webUrl = "http://vs-server38"
$docLibraryName = "Document Set Performance (6000)"
$folderPathWithinDocLibrary = "/Doc Set Load Test 1001" # Leave empty e.g. "" to create documents in root folder of library otherwise specify path relative to root folder e.g. "/Testing/Folder A"
# -------------------------
#Open web and library
$web = Get-SPWeb $webUrl
$docLibrary = $web.Lists[$docLibraryName]
$docLibraryUrl = $docLibrary.RootFolder.ServerRelativeUrl
$uploadfolder = $web.getfolder($docLibraryUrl + $folderPathWithinDocLibrary)
#Open file
$file = get-item $sourceDocumentPath
$fileStream = ([System.IO.FileInfo] (Get-Item $file.FullName)).OpenRead()
# Create documents in SharePoint
write-host "Creating $i documents based on the file $sourceDocumentPath"
for($i=1; $i -le $numberDocsToCreate; $i++)
{
$newFilePath = $docLibraryUrl + $folderPathWithinDocLibrary + "/" + $newFilenamePrefix+$i+$newFilenameExtension
write-host "Creating document: $newFilePath ..."
$spFile = $uploadfolder.Files.Add($newFilePath, [System.IO.Stream]$fileStream, $true)
}
write-host "Completed"
#Close file stream
$fileStream.Close()
#Dispose web
$web.Dispose()
Related articles
- SharePoint PowerShell How To: Create SharePoint List Items (in Root and in Folders) for Load/Performance Testing
- SharePoint PowerShell How To: Create SharePoint Test Documents in Library Folders for Load/Performance Testing
- SharePoint PowerShell How To: Create SharePoint Document Sets for Load/Performance Testing
- SharePoint PowerShell How To: Create SharePoint Library Folders for Load/Performance Testing
- How to import (upload) an entire folder of files to SharePoint using PowerShell
SharePoint PowerShell How To: Create SharePoint Document Sets for Load/Performance Testing
SharePoint PowerShell How To Series – Foreword
I often find myself scratching around the local drive of my development machines for remnants of PowerShell scripts to generate list & libraries in SharePoint with all sorts of different folder hierarchies to test the performance of code I’m working on to ensure it scales well once the folder & item counts start getting up to high numbers. So rather than keep scratching I’m going to start posting my scripts as I create and use them so I can find/reuse them later on and someone else might find them useful as well. I’m sure they won’t work under all scenarios and situations (and they are not designed to). They are just quick scripts that serve a specific purpose to me at the time and you may be able to tweak to fit your needs.
Create document sets in the root of a document library
Script description: Create 6000 document sets within the “Document Set Performance Test” library of the “LocationContentTesting” site on the SharePoint Server “vs-server82”.
Note: this assumes you have made have the standard “Document Set” content type available in the library.
#Script settings
$webUrl = "http://vs-server82/locationcontenttesting"
$listName = "Document Set Performance Test"
$numberDocSetsToCreate = 6000
$docSetNamePrefix = "Doc Set Load Test "
### Get web and list
$web = Get-SPWeb $webUrl
$list = $web.Lists[$listName]
### Get Document Set Content Type from list
$cType = $list.ContentTypes["Document Set"]
### Create new Document Set
[Hashtable]$docsetProperties = @{}
# Create desired number of subfolders
for($i=1; $i -le $numberDocSetsToCreate; $i++)
{
$newDocumentSet = [Microsoft.Office.DocumentManagement.DocumentSets.DocumentSet]::Create($list.RootFolder,"$docSetNamePrefix$i",$cType.Id, $docsetProperties)
write-host "$docSetNamePrefix$i ... created"
}
$web.Dispose()
Related articles
- SharePoint PowerShell How To: Create SharePoint List Items (in Root and in Folders) for Load/Performance Testing
- SharePoint PowerShell How To: Create SharePoint Test Documents in Library Folders for Load/Performance Testing
- SharePoint PowerShell How To: Create SharePoint Document Sets for Load/Performance Testing
- SharePoint PowerShell How To: Create SharePoint Library Folders for Load/Performance Testing
- How to import (upload) an entire folder of files to SharePoint using PowerShell
SharePoint Document Sets – Saving emails & documents
This new video is a great starting point for getting the most out of OnePlaceMail if you are looking to store related Email and Documents in Documents Sets in SharePoint 2010.
The video shows SharePoint 2010 Documents Sets being used as a container to store email messages and documents for projects/tasks. The screencast shows interacting with Document Sets from within Microsoft Outlook.
Interacting with Document Sets from within Microsoft Outlook
