please follow the instruction as described on this MSDN page. It seems that you missed one step. The authorization token you got from the Authorization end point needs to be traded for the actual OAuth token, see chapter "Step Three: Trade-In for an access/refresh token" in provided MSDN page.
We also found out that Google OAuth token format is very different from the one Microsoft uses (that format you are using resembles the google format to me, so it may be also an issue here). We used Microsoft’s libraries Microsoft.IdentityModel.Clients.ActiveDirectory to obtain correct OAuth token for us. The libraries can be found in NuGet repository.