Ozzy Citrix Noobs

Once a while, I am getting users calling “Service Unavaible” after logon to Access Gateway. By checking, found that one of the Application Pools in IIS was down. I have set seperate application pool for each Citrix Web Interface site, so if one application pool is down, only one site is affected. But how do I monitor this? By googling, I found the following from http://forums.webhostautomation.com/showthread.php?t=11141, and I am only showing this here to thank the author for his wonderful idea. I have created a scheduled task to run this vbscript on the two Web Interface servers every 30 minutes for 24 hours, so I will be notified by email fairly quickly once there is a problem. In the script, I am using my own SendEmail to use SMTP server, as there is no Outlook client installed on that IIS server.

===Code Starts==========================================================
‘ Variables that need to be defined for the script to work
‘http://forums.webhostautomation.com/showthread.php?t=11141
Const LogDir = “C:\Source\AppPoolLog”
ToAddress = “user@it.com”
FromAddress = “user@it.com”

showAppPools ‘Runs showAppPools Sub

‘ This will get the list of all app pools on the server
Sub showAppPools
Dim obj,apool
set obj = GetObject("IIS://localhost/W3SVC/apppools")
for each apool in obj
apppoolStatus apool.name
Next
set obj = nothing
End Sub

‘ This will get the status of the app pool
Sub apppoolStatus(apppool)
Dim obj
set obj = GetObject("IIS://localhost/W3SVC/apppools/” & apppool)
Select Case obj.apppoolstate
Case 0
writetoLog apppool,"0″ ‘Do not know what this status is
Case 1
writetoLog apppool,"1″ ‘Do not know what this status is
Case 2
writetoLog apppool,"Up”
Case 3
writetoLog apppool,"3″ ‘Do not know what this status is
Case 4
writetoLog apppool,"DOWN”
startapppool apppool
SendEmail GetComputerName,apppool
writetoLog apppool,"Restarted”
End Select
set obj = nothing
End Sub

‘ This will write information to a log file
Sub writetoLog(apppool,response)
Dim fso
lday = datepart("d",date)
lmonth = datepart("m",date)
lyear = datepart("yyyy",date)
logdate = lyear & lmonth & lday
set fso = CreateObject ("Scripting.FileSystemObject")
Set objFSOwriteline = FSO.OpenTextFile(LogDir & “\” & logdate & “.log", 8,True)
objFSOwriteline.WriteLine(response & “,” & apppool & “,” & now())
objFSOwriteline.close
Set objFSOwriteline = nothing
set fso = nothing
End Sub

‘ This starts the down app pool
Sub startapppool(apppool)
Dim obj
set obj = GetObject("IIS://localhost/W3SVC/apppools/” & apppool)
obj.start
set obj = nothing
End Sub

‘ This send a email regarding the down/restarted app pool
Sub emailSendCDOSYS(apppool)
Dim Mailer
Set Mailer=CreateObject("CDO.Message")
Message = “===============================================” & VbCrLf
Message = Message & “Application Pool: ” & apppool & ” has been restarted” & VbCrLf
Message = Message & “Date/Time: “& now() &"” & VbCrLf
Message = Message & “===============================================” & VbCrLf
Subject = “Application Pool: ” & apppool & ” has been restarted”
Mailer.TextBody = Message
Mailer.Subject = Subject
Mailer.To = ToAddress
Mailer.From = FromAddress
Mailer.Send
set Mailer = Nothing
End Sub

sub SendEmail(ServerName, apppool)

Set objEmail = CreateObject("CDO.Message")
objEmail.From = “user@it.com”
objEmail.To = “user@it.com”
objEmail.Subject = “Application pool: ” & apppool & ” is restarted on ” & ServerName
objEmail.Textbody = “Application Pool error.”
objEmail.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
objEmail.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/smtpserver") = _
“smtp.it.com”
objEmail.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25
objEmail.Configuration.Fields.Update
objEmail.Send

End Sub

Function GetComputerName
Set objWMISvc = GetObject( “winmgmts:\\.\root\cimv2″ )
Set colItems = objWMISvc.ExecQuery( “Select * from Win32_ComputerSystem", , 48 )
For Each objItem in colItems
strComputerName = objItem.Name
Next
GetComputerName = strComputerName
End Function
===Code ends============================================

A chat led to our discussion on the role of Data Store. Question:

When user launches Citrix Web Interface, where does the Web Interface server get the application list?

Answer: Citrix Web Interface server gets a list of applications for the user from the server defined in Web Interface site properties.

Good, your answer is correct. Then next:

When Web Interface server is requesting application list, where does the application server get the data? From Data Store or itself (Local Host Cache)?

My instinct was that Citrix server would use IMA to talk to Datastore to get the application list. Tommo thought another possibility was the server would search it Local Host Cache which is a copy of Data Store. The reason is Citrix Servers are functioning if data store is not available.

Testing:
1. Rename MF20.dsn on Citrix server that is listed in Web Interface site properties: Web Interface still works.
2. Stop SQLServer service: Web Interface still works.
3. Running Wireshark during Web Interface logon and listing applications, the Citrix server does not talk to data store during the period Web Interface server displaying the list of applications.

Answer: Citrix application server is self contained, will search farm data from local host cache first. It will update local host cache by polling interval defined in Registry, default is 30 minutes.

Registry Key related to this post:
HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\IMA\
DWORD: DCNChangePollingInterval
Value: 0x1B7740 (default 1,800,000 milliseconds)
(Reference: http://support.citrix.com/article/CTX111914)

Web Interface 4.x
for MetaFrame Presentation Server

Your Web Interface Session is in an inconsistent state. To log on again, you must close and restart your web browser

Symptoms
Below conditions must be meet:
•WI 4.x Load Balanced with VIP
•NO DNS round robin setup
•Session Persistence or Session Affinity enabled
•Manual authentication on WI site

Once meet, continue;

When the Citrix Web Interface servers are NLB using either of the following productions, and all conditions are meet, user still may receive the about error message when using manual authentication.

Your Web Interface Session is in an inconsistent state. To log on again, you must close and restart your web browser

Description

When Citrix released the new version Web Interface 4.x for Metaframe, they included a new security feature to stop “session fixation attacks”, each time a user hits a WI, it copies a cookie to the local PC, locking the user to that particular session ID. (cookie=”WIauthID). If for any reason the user starts sharing a session with another WI in the cluster the WI will terminate that session.

Resolution

Inform the user to not click the login button twice or press enter twice. This error can be replicated by clicking the login button multiple times on a WI in a clustered environment.

It happened again. The Web Interface is giving the dreadful error:

An internal error occurred. Please contact your system administrator.

We can see and pass the first authentication page, but are getting this right after. This only happens to one site, not on all other sites in the same IIS application pool.

I did make one small change in one of the files: applist.inc yesterday. Changed the word from “silver” to “blue", and tested the Web Interface and was sure it working.

Our usual fix was “repair site” in Access Suite Management Console, but this will remove our customizations, such as logo image, title image and font colors.

From event log, I can see the first error was given at 2:15am.

The first steps taken were:

1. Restart Default Web Site in IIS
2. Restart the Application Pool the site is on

Both of the above actions did not help. In the following folder, I can see the influence of the change I made:

C:\windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\

Some files are compiled yesterday right at the time of the file change. All other files are untouched.

This means my change definitely initiated this error. By looking into the properties of the Application Pool, found that the scheduled time to recycle the Worker Process is 2:00am, which corresponds to the time of the first error captured.

So, it seems that the newly compiled files were accepted until the w2wp.exe was restarted at 2:00am.

Then I changed the web.config customErrors mode="Off", so I can see what ASP.NET error we are getting:

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.IO.FileNotFoundException: Could not load file or assembly ‘App_Web_wavlouee, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null’ or one of its dependencies. The system cannot find the file specified.

In the above assembly folder, I can see a lot files named App_Web_********.dll, but not this one. These App_Web_.dll files must be generated with random names, and when I made a change, one of the dll file was deleted and a new dll file was regenerated. This caused the problem.

The page also recommended to enable debug mode in web.config to see the source error. So I made the following change:

<compilation debug="true"/>

After the change, I reloaded the Web Interface page, was expecting to see more detailed error descriptions, but the site was working! Going back to the assembly folder, I could see all files having the same current modified datetime. Fiddling with customError and debug for a few more times. Found changing customError is petty, changing debug mode will activate a whole new compilation.

Should we just keep this on? I left the debug=true on and added a space in the file applist.inc. The I refreshed the web page, I was logged off and after logging on again, I received the “internal error” again. Switching the debug mode made the ASP.NET recompiled the files and the site came back to life.

Then I changed the applist.inc file again, refresh the web page, nothing happened. After restarting the worker process in the application pool, I got the Internal error again on the Web Interface site. Again switching the debug mode worked.

We can also achieve the above action in:

IIS - Default Web Site - Citrix - MetaFrame - Properties - ASP.NET –>
Edit Configuration - Application - toggle the checkbox “Enable debugging".

From Microsoft, ASP.NET is doing a smart thing to detect the file changes and to compile the assembly. But in Citrix Web Interface, the dependencies have been destroyed by part compilation. How can we define the ASP.NET to compile the whole application for each change? This can be our further topic.