Cleaning up obsolete SharePoint groups

During the lifetime of a SharePoint implementation, sites come and go. When a site collection grows, you typically see the amount of SharePoint groups growing as well because you want to give people access to those sites in a sort of organized way. When sites go, those groups are left behind. Removing those obsolete SharePoint groups can be a challenging task because groups which have been created for a specific site can be used for other sites as well. So, before removing a group, you need to be sure that it’s not used on any other sub sites.

SharePoint groups live on the root web of a site collection.

If a group is created as part of the site creation process, it will have a description which clearly states for which site is has been created. This doesn’t mean it can’t be used on any other sites. If you want to get a list of all SharePoint groups which exist in a site collection, you can also use the following PowerShell snippet to get the collection.

obsolete-sharepoint-groups-1

If you want to know which groups are used on a specific subsite of the site collection, you can use the UI and check the Site Permissions section of a site. This will give you all permissions for that site. You can also use the following PowerShell snippet to get these.

obsolete-sharepoint-groups-2

See the difference? The SiteGroups has a group “MyCustomGroup” which is not part of the Groups collection of the same web. This means that the group exists in the site collection but at this point, it’s not used. When I give this group explicit permissions to my site, it will be added to this collection and it will be in use.

So, the process of cleaning up obsolete groups is to check the Groups collection on each sub site and see which site groups are used for giving people access. If you have a site group which is not part of any Groups collection of any site, it’s not used and you can remove that group from the site collection.

You can do this manually, or you can automate this process and use the following script for this task.

This script will do 2 things.
If run in Simulation mode, it will look for obsolete groups and ouput them to the console. Nothing more.
If run in Execution mode, it will look for obsolete groups and delete them from the site collection.

My advice… run it in Simulation mode before running it in Execution mode. That way, you have an idea of what groups were found and will be deleted.

There are some situations which need clarifications.

Inherited permissions

What happens when you have subsites which inherit permissions? This is no issue. Suppose you have a subsite which inherits permissions of the root web of the site collection. When a SharePoint group is given access to the root web of the site collection, it will be given access to all sub sites which inherit their permissions and as such, that group will be part of the Groups collection of those sub sites.

Audience targeting

What happens when a group is exclusively used for audience targeting? Well, this is a problem because that group is not part of the Groups collection of the site where you have used it as an audience. In my opinion, this is a situation you should avoid doing because you are going to give a collection of users access to a site. In theory, an audience is a subset of authorized users, right? You want to target specific content on a site to specific users. If they can’t reach the site, what’s the point in targeting content to them?

If you do find yourself in such a situation where you have SharePoint groups which are exclusively used for audience targeting, a good approach would be to give those groups distinctive names, clearly indicating they are audience targeting groups. For example, you could start each group name with “AUD_”. This way, you can extend the script above and include a check to skip groups which start with “_AUD”.

SharePoint issues when using a trust with Selective Authentication

If you have some experience with SharePoint, the issue where you get a credential request three times before hitting the 401 Unauthorized is probably not new to you. We all know this happens when you try to navigate to a SharePoint site from the web front-end servers. Resolving this is common knowledge for SharePoint admins… You disable the loopback check in the registry or you use the recommended BackConnectionHostNames registry key. This has been documented in KB896861.

Last week, I was at a customer doing an assessment of a SharePoint implementation and one of their developers approached me with a weird issue on their Extranet. They have a SharePoint farm in a separate extranet domain. Between the internal domain and the extranet domain is a one-way trust to allow users from the internal domain to use their accounts to log on to a site on the Extranet. He was able to do this from the web front-end servers of the Extranet farm but not from his laptop. On his laptop, he had to enter his credentials and this kept failing… seems familiar right?

I double-checked the BackConnectionHostNames on the servers and sure enough, the key and hosts were there.

I tried the same thing on my machine with my account and this worked! I was able to go to the site from my machine. When he tried to do it from my machine with his account, it failed. We tested this on several other clients with several users… ALL of them had the same issue. Nobody was able to sign-in. Only I was able to sign-in from any place.

I will spare you the checks and comparisons we did, but I will tell you that we were able to solve it!

Servers in a domain are, like user accounts, just objects in Active Directory. When you open the properties of such a computer object in AD, and you go to the Security tab, you can specify a lot of permissions which specific AD objects can have on this computer. One of those permissions is “Allowed to authenticate”. For the servers in that Extranet farm, I was explicitly granted that permission, while the “Authenticated Users” group was not…

allowed-to-authenticate

In normal circumstances, this doesn’t pose any issue. If you have 1 domain which contains your users and servers, this permission is not required. Furthermore, if you have multiple domains and a one-way trust and you keep the default trust authentication level (Forest-wide authentication), you will not have any issues with users from the trusted domain authenticating to resources in the trusting domain.

selective-authentication-02

However, when you are using “Selective Authentication”, you need to explicitly grant the “Allowed to authenticate” permission to all users on the resources they need to access. When we verified this authentication level at the customer, we got confirmation that they were using selective authentication. So, we had to give “Authenticated Users” this permission on the SharePoint servers in the AD of the Extranet to resolve this issue.

See following articles for more information on selective authentication on trusts.

Hiep Hiep Hoera!!!

Hiep Hiep Hoera!!!

Hiep Hiep Hoera… je zou denken dat vandaag niet echt een dag is om te vieren. 8 september is inderdaad een dag die nooit meer hetzelfde zal zijn. Een dag waar ik nooit nog naar zal uitkijken maar eigenlijk toch wel een beetje. 8 september is uiteindelijk de dag waarop ik eindelijk papa werd, na zoveel jaren. Jammer genoeg een papa die dit nooit zal horen van zijn zoon die toen geboren werd. Maar ik voel mij wel een papa. Een papa die een schat aan herinneringen heeft van een zeer korte tijd samen met zijn zoon. En dus is 8 september ook wel een dag met hele mooie herinneringen die ik nooit wil vergeten.

Dus ja, hiep hiep hoera!! Gelukkige verjaardag jongen!!!! Het had zoveel leuker geweest als jij dat kaarsje echt had kunnen uitblazen en en met je handjes in de slagroom had gezeten, maar we weten dat je daarboven even hard zal blazen. Er is dan wel geen taart, maar ik eet een extra appeltje vandaag, speciaal voor jou.

10 jaar geleden… het lijkt een eeuwigheid

10 jaar geleden… het lijkt een eeuwigheid

Dit jaar is het 11 jaar geleden dat we elkaar hebben ontmoet. Het lijken er meer, maar het is niet zo. We hebben elkaar voor het eerst gezien op het huwelijk van een gemeenschappelijke vriendin. Ik als gast, zij als getuige van de bruid. Dat was het begin van een vriendschap, niets meer… zij had op dat moment een relatie en van een vonk was er absoluut geen sprake toen.

IMG_0833

In het anderhalf jaar dat daarop volgde is de vriendschap gegroeid en versterkt. Tot die memorabele week in Denemarken in 2006. We beseften het zelf nog niet op dat moment maar ergens is er tijdens die week een vonk geweest. De gezellige rit met ons tweetjes naar huis, het ongemakkelijke moment bij het afscheid nemen na thuiskomst… de dagen en weken nadien waren de kriebels bij momenten absurd… net pubers. Uiteindelijk hebben we elkaar de liefde verklaard op 19 november 2006.

mixtape-136395972199803901-150204161947

Een relatie die ondertussen al 10 jaar is, en zoals alle relaties met ups and downs is verlopen. We hebben mooie momenten gekend, maar ook vreselijke moeilijke momenten. De fijne momenten die we tijdens de jaren hebben gehad, het uitkijken naar een eigen huisje, het uitzichtloze proces voor een gezinsuitbreiding, de sprankeltjes hoop en het uiteraard het grote moment waar we jaren naar hebben uitgekeken, maar ook de bijzonder grote pijn van het verlies van Kobe… Het zijn die momenten die de band enkel maar hebben versterkt.

10 jaar is dan denk ik ook een mooi moment om uiteindelijk die laatste stap te zetten waar iedereen eigenlijk al een tijdje op aan het wachten is. Een evidente stap, ook voor ons… Zoals het hoort voor een echte grillmeister, hebben we dit uiteraard beslist tijdens een BBQ, tussen een kipfilet en een kalkoen-brochette… “Misschien moeten we er ook eens werk van maken? Wat denkte?” Het antwoord was niet verrassend… maar toch had ik iets later even een momentje dat ik in mijn eigen dacht…

36fd30b5b90456d8bf399400fb7c45e9

Een datum zoeken was ook al niet moeilijk: 19/11/2016. De dag waarop we exact 10 jaar samen zijn. Een snel telefoontje naar het stadhuis met de vraag of die datum nog vrij is en voila… het was geregeld. Nu nog de praktische zaken regelen en we kunnen aftellen.

De Efteling

De Efteling

Gisteren naar de Efteling geweest in Kaatsheuvel samen met wat vrienden. Het was een last minute deal. Donderdag in de voormiddag een telefoontje met de vraag om nog eens af te spreken. Sure. Wanneer? Morgen! OK. Laat mij even de agenda checken… ah, kijk, lukt nog. Cool! Wat gaan we doen? Efteling?

Ik ben geen pretpark-liefhebber maar de laatste keer dat ik in de Efteling ben geweest is al zolang geleden dat ik mij enkel nog het sprookjesbos vaag kan herinneren. Dus ja, waarom niet. Maar tegelijkertijd ook een dubbel gevoel want iemand zal er niet bij zijn en ondanks dat het bijna een jaar geleden is, zijn dergelijke uitstapjes momenten waarop we keihard met de realiteit worden geconfronteerd. Buggy’s, koetsen, gelukkige gezinnetjes… zo had het ook voor ons moeten zijn. Ik ben iemand die graag mensen observeert en op rustige momenten kijk ik dus rond en naar de mensen. Maar ik heb mijzelf erop betrapt dat ik nu meer oog heb voor het merk buggy… is het dezelfde als deze van onze Kobe? Als dit dan dezelfde is, dan voel ik mij even een momentje slecht. Niet lang, enkele tellen maar. Weer die realiteit die komt aankloppen.

Maar de beste manier om dergelijke zaken te verwerken is de confrontatie aangaan. En kijk, we hebben ons geamuseerd. Een dagje onder vrienden, de 4 kindjes die erbij waren hebben ons ook bezig gehouden. Een gezellige, drukke bende. Net een stel Duracel konijntjes die een onuitputbare bron van energie hebben. Als mijn voeten een time-out nodig hadden, dan gingen die beentjes nog altijd even hard en snel als toen ze uit de auto stapten. Een gezellig uitstapje, even geen zorgen en gewoon even de sfeer opsnuiven.

De Efteling is niet meer zoals ik het mij kan herinneren… ik denk dat we oud worden.  🙂 Maar gelukkig zijn er nog de vaste waarden… de goudstukken-schijtende ezel, Langnek, Holle Bolle Gijs,…

Verfwerken

Na 6 jaar zijn vorige week eindelijk de laatste werken gestart aan het huis. Iets wat we altijd voor ons uit hebben geschopt… verfwerken! We hadden zelf al de slaapkamers en dergelijke geverfd maar de grootste brok, de leefruimte, keuken, hal, nachthal, plafonds en deuren moest nog gebeuren. Maar dit zagen we niet zitten om zelf te doen. Verven doe je niet elk jaar en dus wilden we ineens wél zeker zijn dat de kleuren een beetje passen bij het interieur en dat het deftig is gedaan. We hadden al gauw door dat er ook aanzienlijk wat retoucheerwerk nodig was aan de bepleistering en daarenboven moesten alle muren nog worden geschuurd. Het vooruitzicht om hier een volledige verlofperiode aan te besteden was nu niet echt bemoedigend.

Read more

Manage Office 365 using PowerShell

As someone who’s primary job is being a SharePoint administrator, the Rise of the Cloud (Office 365) was something which scared me in the beginning. Was my job doomed? Well, not exactly. You see, Office 365 is complex and it evolves constantly. Yes, a lot of stuff you do as an administrator on-premise is taken off your hands by Microsoft engineers at the datacenters but there’s plenty of stuff to “administer” in a cloud environment. You still need to manage users and their habits of losing passwords. You still need to create site collections, manage mailboxes, set up retention policies, and so on. Microsoft did a great job of providing a nice administration portal for all of these tasks but they also provide PowerShell management modules for all of the services in Office 365 to allow administrators to manage their tenant locally from their machine. This post gives an overview of the things you need to manage the different areas (services) in your Office 365 tenant.

Prerequisites

The prerequisite you always need to manage anything in Office 365, is the Microsoft Online Services Sign-In Assistant for IT Professionals. Without this, you won’t be able to do anything from your machine in Office 365. To download this package, click the link below.

Users/Licenses/Subscriptions

To do general management of Office 365, like adding/removing users, manage groups, manage your subscriptions, assign licenses,… you need to install the following package:

Once this package is installed, you can start managing Office 365 users. This package includes a PowerShell Console window which already has the necessary module preloaded. You can use this console, or you can open up a regular PowerShell console and load the module manually.

After loading the module (or opening the console which has the module preloaded), you can connect to Office 365.

Skype for Business Online

To manage Skype for Business Online, you need to install the following package:

Once the package is installed, you can open a PowerShell console and connect to Skype for Business Online.

SharePoint Online

To manage SharePoint Online, you need to install the following package:

Once the package is installed, you can open a PowerShell console and connect to SharePoint Online.

The URL you need to provide to connect to SharePoint Online, is the URL of the SharePoint admin center in the Office 365 portal. For instance: https://mytenant-admin.sharepoint.com.

Exchange Online

To manage Exchange Online, you don’t need to install additional packages. You can execute the lines of PowerShell below to get started. When these lines are executed, the Exchange Online cmdlets are imported in your session. When you change Management Roles in Exchange Online however, you need to relaunch the session because changing roles also means that the cmdlets you CAN use, will change.

Rights Management

To manage Azure RMS in your Office 365 tenant, you need to install the following package:

When this package is installed, the PowerShell module will be installed on your machine.

 

The SharePoint 2013 installer doesn’t like .NET Framework 4.6.x

One of the Prerequisites of SharePoint 2013 is the .NET Framework 4.5. I have been installing countless SharePoint 2013 environments without issues but recently, I started noticing that the SharePoint installer fails with the following error :

Setup is unable to proceed due to the following error(s):
This product requires Microsoft .NET Framework 4.5.

 

NET Framework 4.6 - pic 2

This might seem strange since the prerequisite installer installed everything. When this happens, you might want to have a look at the actual installed versions of the .NET Framework. You will probably notice that .NET Framework 4.6 or higher is installed. To check which versions have been installed, you can download the .NET Framework Setup Verification Utility from Microsoft to verify which versions of the .NET Framework have been installed. When you run this tool, you will see an overview of all installed versions.

NET Framework 4.6 - pic2

When you see the .NET Framework 4.6 or 4.6.1 listed, the SharePoint installer cannot detect the presence of version 4.5.

With the release of Service Pack 1 for SharePoint 2013, SharePoint is compatible with version 4.6 but the installer isn’t. You can have version 4.6, but only AFTER you installed SharePoint 2013.

So, in order to get SharePoint installed on that machine, you need to uninstall version 4.6 or higher before you can install SharePoint 2013. And this is where it gets a bit annoying. The tool to verify the .NET Framework also has the ability to uninstall a version. Except… this doesn’t work! You can try it but nothing happens. When you recheck, it’s still there. Even after a reboot.

This version of the .NET Framework can be uninstalled by uninstalling Windows update KB3102467. But this is just the first action… if you are installing SharePoint in an organization which pushes Windows Updates, you probably want to  block this framework from installing on that machine for the time being.

To do that, you can execute the following PowerShell script. This is going to add a key ‘BlockNetFramework461’ to the registry and set it to 1 to block the installation of that version. Once SharePoint is installed, you can remove this key if you want.

To see the official article for the blocking of the installation of the .NET Framework 4.6.1, see the following link:

https://support.microsoft.com/en-us/kb/3133990

Once that version is uninstalled, you can proceed with the installation of SharePoint 2013 on that machine.

Update (14/09/2016)

Microsoft released a fix for the installer issue. You can find this fix in KB3087184.

Cleaning up large content databases and reclaiming unused disk space

Dealing with large SharePoint content databases can be a daunting task. Microsoft has the recommendation to keep your content databases below the 200GB mark for databases which are used daily. But when you are dealing with backup/restore, migrations and general operations which involve moving around those kind of databases, even 200GB can be a huge pain in the ass and will cost you in terms of time you are spending looking at a progress bar or watching a percentage creeping slowly to 100%.

A solution for this is to split up the content database into smaller databases provided that you have multiple site collections in that database which can be moved out.
Relocating a site collection to a different content database is very easy. You can do this with the Move-SPSite cmdlet.

Once you moved out a site collection to a different database, you will notice that your huge database is not getting smaller. That’s because the site collection which has been moved, is still in that database but it’s marked for deletion.
The actual removal of that site collection is done by the Gradual Site Delete timer job. This job runs once a day. Once it has run, the site collection is completely removed from the database.
But still, if you look at the database, it will not be any smaller than before. When you look at the used space in the database, you will see that this has decreased and the unused space has grown. The unused space is not released.
To release unused space, … *ducks for cover* … you can shrink the database. There, I said it.

Generally speaking, shrinking content databases is not done. You can do this, but it has no advantages and it has a negative effect on performance because when someone adds something to a site in that database, it has to expand to be able to store anything.
So, shrinking is definitely something you should avoid at any cost… except for the case where you have such a huge content database that you’ve split up into smaller content databases. The reason for splitting up the database in the first place was to make it smaller, right? To have a size which is much more manageable. But in order to get it back to a reasonable size, you need to shrink it. There’s no way around it.

During a migration from SharePoint 2007 to SharePoint 2013, I had to migrate a content database of 220GB. All things considered, this is not huge. It’s large, but not huge. This content database contained around 20 site collections. Backup of this database was not an issue… 20 minutes to backup this database. Copying this backup to a SharePoint 2010 migration server was frustrating. It took over an hour. Yeah, It SHOULD go faster but if you pass through a 10Mbit router, you are copying at USB 2.0 speed! But this was nothing compared to the time the Mount-SPContentDatabase cmdlet needed to complete to attach this database to the web application and do the actual upgrade from SP2007 to SP2010. This attach/upgrade took almost 3 hours and then it just aborted due to lack of disk space. The migration server had a data disk of 600GB and it just filled completely with the transaction log that was created as part of the attach/upgrade process. So, I lost 3 hours, had to wait until extra disk space was added and restart the whole thing again. By the time it had attached and upgraded, the size of the database was actually increased to 330GB.

When everything was attached and upgraded, I decided that I’m not going through this again when I do the migration from SP2010 to SP2013. I needed to have databases which are easier to handle. So, I split up this database into 5 databases of which the largest was still 115GB. But ok, nothing I could do about that in the short term.

Running the Gradual Site Delete job however proved to be a pain as well… took almost 6 hours to complete! Started around 2PM. Went home at 5PM. Next day, I noticed it finished around 8PM. So, I started the shrink operation of the database… lost half a day with that. Wasn’t able to do anything with that database for the larger part of the day.

Since this was only a “test” migration, I realised that history was going to repeat itself during the final migration and that I needed a way to make use of those lost hours between the finishing of the gradual site delete and the shrink. When the gradual site delete is done, start with the shrinking to have it done in the morning.

Enter… PowerShell!

The script below is going to kick off the Gradual Site Delete timer job for a specific web application and it then will monitor this job every 5 minutes to see if has completed. Once it has completed, it will continue with the shrinking of the database. The shrinking happens in 2 stages. The first stage is done with NOTRUNCATE. This means that SQL will move all pages to the front of the file but it will not reclaim unused space. The size stays the same. The next step is a shrink operation with TRUNCATEONLY. This will just remove all unused space from the end of the file. It’s basically the same thing that happens when you do a Shrink Database from the Management Studio.

Again, don’t do this as a part of a weekly maintenance routine because the first step of the shrink will introduce index fragmentation in your database. Seriously! For me, this was a necessary cleanup I had to do as part of a migration project to reorganize the content database and minimize the migration time! The environment I was doing this in, was a intermediate SharePoint 2010 environment, not a live environment.

Also, the Shrink operation in the script allows you to specify a percentage of free space it should reserve for unused space.

I used 5%. This way, for a content database of 100GB, 5GB of free space is retained. You can change this if you want, or you can add an additional parameter which allows you to specify the amount of free space it should keep.

You can find this script in my PowerShell repository on GitHub.

An eye for details… changing the ImageUrl for migrated lists

Migrating from SharePoint 2007 to SharePoint 2013 can cause all kind of headaches but this post is not about those headaches. It’s about details, or better… having an eye for details. Ever noticed that after you migrate a site to SharePoint 2013 and you complete the actual visual upgrade to SharePoint 2013 mode, the list icons which are used, are not the fancy list icons which you get when you create a new list or library in SharePoint 2013? The icons for migrated lists and libraries are still the old icons from the early days of SharePoint 2007.

ImageUrl - 1

The icon for a list or library is stored in the ImageUrl property of a list and this property points to a gif or png in the “/_layouts/images/” folder for migrated lists. When you create a new list in SharePoint 2013, the value of the property points to “/_layouts/15/images”. Furthermore, if you compare for instance a migrated document library with a new document library, you notice that the value of the property differs, not only in the location where the icon is displayed from, but also the type of file. For instance, a simple document library.

  • Migrated document library : /_layouts/images/itdl.gif
  • New document library : /_layouts/15/images/itdl.png?rev=23

While I can imagine that a lot of people really don’t see any issue with this and don’t care how those icons look like, I don’t like loose ends. If you migrate an environment, you might as well get it done completely and replace the list icons with new versions as well and get the following result in the end.

ImageUrl - 2

Admit it, this looks much better than those old school icons. It’s a small detail, but it just makes more sense. If you have a smart user who actually cares about the environment, the question why new lists have different icons than existing lists, will eventually pop up anyway and if you tell them that this the result of the migration, the next question will be whether you can change them to resemble the new lists. Show your customers or users you have an eye for detail and do it proactively.

Changing these icons can be done very easily using PowerShell. The only thing you need is a mapping between the old and new icon.

I created a script which replaces all icons for lists and libraries. In this script, a mapping is done for the mostpart of the icons which are used. It might not be the complete list, but feel free to add missing icons. There are some scripts out there which replace icons, not based on a mapping but just replace all .gif icons with .png’s. However, there are some icons which don’t have .png counterparts. So, if you replace those, your list icon will be broken.

You can find this script in my PowerShell repository on GitHub