How to make friends with a hedgehog and a man: the experience of using PowerShell in web applications

This article does not pretend to be a full PowerShell programming guide or step-by-step instructions for developing high-loaded .NET services. But it contains useful techniques and an explanation of some features of PowerShell integration with .NET, which are still difficult or even impossible to find on the Web.

It should be noted right away: PowerShell versions 1.0 and 2.0 is not a .NET programming language. The ETS PowerShell version 3.0 type system is already based on .NET, i.e. PSObject is a dynamic DLR object. Since these and other innovations of PowerShell 3 are not fundamental for the main topic of the article, I will consider PowerShell version 2.0. When not specified explicitly, under PowerShell is meant exactly version 2.0.

Terms:

• Cmdlet - PowerShell Team
• Runspace - .NET class object representing the PowerShell runtime of objects
• Snap-In - build .NET with a set of cmdlets, extending the PowerShell with new functionality

Our product PA (Parallels Automation) has a web application for activating and managing various services from cloud service providers such as Microsoft Exchange, IIS, SharePoint and others. To manage most of its services, for example, the Exchange server, Microsoft provides a set of PowerShell cmdlets, so it was decided to write all of our PowerShell scripts. Our web service written in .NET is essentially a driver that provides a low-level infrastructure for running scripts and implements the processing of network requests via SOAP , transactionality, logging, and so on. When using PowerShell in many cases, you can make changes to the application logic “on the fly” and avoid rebuilding the entire application, which is convenient for both developers and support services.

Today we will talk about the following:

1. Configuring PowerShell settings for running scripts from .NET applications, specifying how to search for PS modules (PowerShell modules) used by the script, as well as error handling
2. Remote and local script launches from .NET and passing parameters to PowerShell
3. Ways to reduce memory used by Runspace objects

Common Problems Integrating .NET with PowerShell

Permission to run PowerShell scripts

By default, remote execution of PowerShell scripts is prohibited. Execution policy must be set to RemoteSigned or Unrestricted (not recommended for security reasons). What can be done with the following command:

C:\Users\user> "%SystemRoot%\system32\WindowsPowerShell\v1.0\PowerShell.exe" -NoLogo -NoProfile -NonInteractive -inputformat none -Command Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Force -Scope LocalMachine 

We have this action performed by the application installation script.
')

Configuring PowerShell Modules Search Directories

Complex projects, as a rule, contain a large number of modules on PowerShell, placed in different directories. For example, in our application, the directory structure with modules looks like this:


In our application, subsystems for managing end services, such as Exchange, are called providers . They are separated by separate directories. The provider contains a set of PowerShell modules, each of which performs a single function. In the picture above, the directory with the Exchange management provider is highlighted. We try to give the modules “speaking” names:

• CreateMailbox.ps1 - create mailbox
• CreateGlobalAddressList.ps1 - Creating a GAL Exchange Object
• Etc.

In the Common directory we place utilitarian PowerShell modules that are used by several providers. For the provider modules to work constructions of the form:

 Import-Module ProviderUtils Import-Module -Name Utils\ExchangeADUtils.ps1 

You need to set the PSModulePath environment variable correctly . In it you need to add directories that contain the modules used.

Error processing

PowerShell divides errors into terminating and nonterminating. Simply put, terminating errors are errors, after which the normal continuation of the script is impossible, all other errors are non-terminating. For example, syntax errors are terminating, and the script will be completed anyway. The $ ErrorActionPreference variable allows you to specify how to handle nonterminating errors.

When setting a variable to Continue, nonterminating errors will not cause the script to crash. Non-terminating errors can be processed in .NET code as follows:

 result = PowerShell.Invoke(); checkErrors(PowerShell.Streams.Error); ... //    private static void CheckErrors(PSDataCollection<ErrorRecord> error) { if (error.Count == 1) { ErrorRecord baseObject = error[0]; throw baseObject.Exception; } if (error.Count > 1) { foreach (ErrorRecord baseObject in error) { if (baseObject != null) { throw baseObject.Exception; } } } } 

Types of PowerShell command calls

For remote calls, PowerShell uses the WinRM protocol, which is an implementation of the open WS-Management Protocol. In fact, this is an extension of the SOAP protocol and all transport goes via http (s). To serialize objects that are transferred from a remote host to a local PowerShell, xml is used. This also needs to be remembered in the context of performance. According to experience, it takes a lot of time to establish a connection with a remote host via WinRM - hence the desire to somehow cache the Runspace objects that have already established connections with hosts.

Local script call from .NET looks like this:

 //  runspace using (Runspace runspace = RunspaceFactory.CreateRunspace()) { runspace.Open(); //   PowerShell   runspace.SessionStateProxy.SetVariable("ErrorActionPreference", stopOnErrors ? "Stop" : "Continue"); using (PowerShell powershell = PowerShell.Create()) { var command = new PSCommand(); command.AddCommand("Get-Item"); command.AddArgument(@"c:\Windows\*.*"); powershell.Commands = command; powershell.Runspace = runspace; ICollection<PSObject> result = powershell.Invoke(); //     if (!stopOnErrors) CheckErrors(powershell.Streams.Error); foreach (PSObject psObject in result) { Console.WriteLine("Item: {0}", psObject); } } } 

Calling a script or cmdlet on a remote host looks like this:

 var connectionInfo = new WSManConnectionInfo( new UriBuilder("http", server, 5985).Uri, "http://schemas.microsoft.com/powershell/Microsoft.PowerShell", new PSCredential(user, passw)) {AuthenticationMechanism = AuthenticationMechanism.Basic}; using (Runspace runspace = RunspaceFactory.CreateRunspace(connectionInfo)) { //   ,     ... } 

ATTENTION: In the example, for simplicity, I used the http and Basic authentication protocol. In a real application, you must use the https protocol and Kerberos or Digest authentication.

With remote cmdlet calls, for example, Exchange, if you do not have assemblies containing the types of returned objects, you will have to deserialize the objects received from the remote host yourself. In the case of Exchange, for proper deserialization, you must have the Exchange Management Tools installed. In the case when assemblies of serializable types are missing or there is no need for typed deserialization, you can work with objects as with primitive types: strings and arrays of strings. You can do it like this:

 public static string GetPsObjectProperty(PSObject obj, string propName) { //    null     //   null   , ,    //   return obj.Properties[propName].Value == null ? null : obj.Properties[propName].Value.ToString(); } public static string[] GetPSObjectCollection(PSObject obj, string propName) { object psColl = GetPsObjectProperty<object>(obj, propName); //         var psObj = psColl as PSObject; if (psObj == null) { var coll = GetPsObjectProperty<ICollection>(obj, propName); if (coll != null) { var arr = new string[coll.Count]; int idx = 0; foreach (object item in coll) { arr[idx++] = item.ToString(); } return arr; } } else { var collection = (ArrayList) psObj.BaseObject; return (string[]) collection.ToArray(typeof (string)); } return null; } 

Calling scripts on a remote host

Often, on the objects obtained using cmdlet, it is necessary to perform some actions. You can do this on the client, but in terms of performance and reducing network traffic, it is more efficient to do it on the server. For these purposes, you can use the PowerShell feature to run scripts on a remote server. First of all, create a local PowerShell session:

 public class PsRemoteScript : IDisposable { private readonly Runspace _runspace = RunspaceFactory.CreateRunspace(); private bool _disposed; public PsRemoteScript() { _runspace.Open(); } ... 

Passing the input and output parameters of the script is implemented by changing session variables; this is done by calling the Runspace.SessionStateProxy.SetVariable method.

 ... foreach (string varName in variables.Keys) { object varValue = variables[varName]; if (varValue == null) { throw new ArgumentNullException( "variables", String.Format("Variable '{0}' has null value", varName)); } _runspace.SessionStateProxy.SetVariable(varName, varValue); } ... 

Next, you need to execute the PowerShell script in the current session, in which you can create a new remote PowerShell session using the cmdlet New-PSSession , and then import it by calling the cmdlet Import-PSSession :

 private void OpenRemoteSession(WSManConnectionInfo connInfo, string[] importExchangeCommands) { _runspace.SessionStateProxy.SetVariable("_ConnectionUri", connInfo.ConnectionUri.AbsoluteUri); _runspace.SessionStateProxy.SetVariable("_Credential", connInfo.Credential); _runspace.SessionStateProxy.SetVariable("_CommandsToImport", importExchangeCommands); Pipeline pipeline = _runspace.CreatePipeline(); pipeline.Commands.AddScript(@" $_my_session = $null $_my_session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri $_ConnectionUri -Credential $_Credential -Authentication Kerberos Import-PSSession $_my_session -CommandName $_CommandsToImport"); pipeline.Invoke(); } 

Please note that the list of required cmdlets that will be called in a remote session is passed by the parameter when you call Import-PSSession .

Memory troubleshooting

Performance and Memory

PowerShell itself is quite memory intensive, especially for remote calls.
Another bad news: the Runspace object after an explicit call to Dispose () does not completely release the memory. The fact is that Runspace stores executable pieces of PowerShell code, scripts and generated modules ( proxies ) for remote calls in the internal cache. And the links really remain “alive”, because GC.Collect () , called directly in the application, does not reduce the consumption of physical memory. When the profiler analyzes the dumps of the IIS process of the pool in which the application is running, a large number of line objects with run scripts lexemes are detected.
There are at least three ways to combat excessive memory consumption:

1. Configure the application pool to restart in IIS after reaching some unreasonable amount of used memory (see Configure an Maximum Pool Used (IIS 7) )
2. The most difficult way: execute PowerShell commands and scripts in a separate application domain (.NET Application Domain). There are several problems to solve:
   
   • Data transfer between the main domain and the descendant domains in which the scripts will be executed. It is solved using the .NET AppDomain.SetData and / or AppDomain.DoCallBack methods.
   • Configure download paths for assemblies in descendant domains. Solved by creating an AppDomain.AssemblyResolve event handler.
   • Some global things, for example, writing the application log to the file, most likely, will have to be called in the context of the parent domain. We used the Logging Application Block from the Microsoft Enterprise Library - all log entries had to be performed in the context of the main domain.
3. Via .NET Reflection, access the cache and periodically force it to clear

We tried all the ways, only the first solves the problem with memory.

Tip : use Import-PSSession with explicitly specifying the cmdlets you will use in the body of the script. Without explicitly specifying PowerShell, a special PowerShell proxy module will be generated for all cmdlet exported by Snap-In. For example, for Microsoft.Exchange, a proxy is of the order of 2Mb of PowerShell code. For unexplained reasons, PowerShell sometimes does not remove proxy modules. These files gradually accumulate in the directory with temporary files, and it must be periodically cleaned. Moreover, with the accumulation of a large number of proxy modules (several thousand), the execution speed of scripts is significantly reduced. Exchange Management Tools themselves manage the generation of proxies for remote calls, thereby avoiding the generation of proxies for each remote session. For details, see RemoteExchange.cs and ConnectXXX.cs from MS Exchange Management Tools. For Exchange 2013, these files are located in the C: \ Program Files \ Microsoft \ Exchange Server \ V15 \ Bin folder.

For remote calls, if there is a possibility, it is better to call not a set of cmdlets with the subsequent processing of the results on the local host using C #, but transfer the processing to the script on PowerShell and call it. It should be remembered that the script needs to load snapins (i.e., .NET assemblies) with used cmdlets.

Using RunspacePool to cache execution objects

The main purpose of the RunspacePool class is to organize asynchronous calls to cmdlets and scripts. But since execution objects obtained via RunspacePool are not deleted by the environment, but are cached for later use, there is a natural desire to use this mechanism to speed up the “cold” start of PowerShell scripts. In fact, it does not look so bright. Indeed, the execution time of scripts decreases, although problems arise:

• PowerShell does not clear the variable namespace after the Runspace object is returned to the pool. Memory consumption is growing, this is especially noticeable with the frequent release of exceptions from scripts.
• Care should be taken to ensure that after using an instance of RunspacePool (or before starting to use it in a new session), all confidential data, such as passwords, will be destroyed, that is, at least take care of deleting / cleaning instances of variables remaining in the pool after execution of a script.

Outcome or "Will there be sugar after the uprising?"

I hope our experience will be useful for developers who would like to use PowerShell as a scripting language in .NET applications or for configuring various services with it. We covered the main problems of integrating .NET applications and PowerShell, types of calls, and features of working with PowerShell memory. Described some ways to deal with excessive memory consumption by PowerShell, these methods allow you to effectively use PowerShell in a busy system.

That's all. As they say in MSFT: “Happy Scripting!”.

Full source for the article here.
I am ready to answer your questions in the comments.

Special thanks I want to say to Dmitry Maslakov, Alexey Varchenko and Nikita Popov for help in writing the article.

Links

Windows PowerShell Owner's Manual
System.Management.Automation.Runspaces Namespace
Windows remote management
Exchange 2013 cmdlets
Handling in PowerShell
Configure WinRM to Use HTTP