Once a while, I am getting users calling “Service Unavaible” after logon to Access Gateway. By checking, found that one of the Application Pools in IIS was down. I have set seperate application pool for each Citrix Web Interface site, so if one application pool is down, only one site is affected. But how do I monitor this? By googling, I found the following from http://forums.webhostautomation.com/showthread.php?t=11141, and I am only showing this here to thank the author for his wonderful idea. I have created a scheduled task to run this vbscript on the two Web Interface servers every 30 minutes for 24 hours, so I will be notified by email fairly quickly once there is a problem. In the script, I am using my own SendEmail to use SMTP server, as there is no Outlook client installed on that IIS server.

===Code Starts==========================================================
‘ Variables that need to be defined for the script to work
‘http://forums.webhostautomation.com/showthread.php?t=11141
Const LogDir = “C:\Source\AppPoolLog”
ToAddress = “user@it.com”
FromAddress = “user@it.com”

showAppPools ‘Runs showAppPools Sub

‘ This will get the list of all app pools on the server
Sub showAppPools
Dim obj,apool
set obj = GetObject("IIS://localhost/W3SVC/apppools")
for each apool in obj
apppoolStatus apool.name
Next
set obj = nothing
End Sub

‘ This will get the status of the app pool
Sub apppoolStatus(apppool)
Dim obj
set obj = GetObject("IIS://localhost/W3SVC/apppools/” & apppool)
Select Case obj.apppoolstate
Case 0
writetoLog apppool,"0″ ‘Do not know what this status is
Case 1
writetoLog apppool,"1″ ‘Do not know what this status is
Case 2
writetoLog apppool,"Up”
Case 3
writetoLog apppool,"3″ ‘Do not know what this status is
Case 4
writetoLog apppool,"DOWN”
startapppool apppool
SendEmail GetComputerName,apppool
writetoLog apppool,"Restarted”
End Select
set obj = nothing
End Sub

‘ This will write information to a log file
Sub writetoLog(apppool,response)
Dim fso
lday = datepart("d",date)
lmonth = datepart("m",date)
lyear = datepart("yyyy",date)
logdate = lyear & lmonth & lday
set fso = CreateObject ("Scripting.FileSystemObject")
Set objFSOwriteline = FSO.OpenTextFile(LogDir & “\” & logdate & “.log", 8,True)
objFSOwriteline.WriteLine(response & “,” & apppool & “,” & now())
objFSOwriteline.close
Set objFSOwriteline = nothing
set fso = nothing
End Sub

‘ This starts the down app pool
Sub startapppool(apppool)
Dim obj
set obj = GetObject("IIS://localhost/W3SVC/apppools/” & apppool)
obj.start
set obj = nothing
End Sub

‘ This send a email regarding the down/restarted app pool
Sub emailSendCDOSYS(apppool)
Dim Mailer
Set Mailer=CreateObject("CDO.Message")
Message = “===============================================” & VbCrLf
Message = Message & “Application Pool: ” & apppool & ” has been restarted” & VbCrLf
Message = Message & “Date/Time: “& now() &"” & VbCrLf
Message = Message & “===============================================” & VbCrLf
Subject = “Application Pool: ” & apppool & ” has been restarted”
Mailer.TextBody = Message
Mailer.Subject = Subject
Mailer.To = ToAddress
Mailer.From = FromAddress
Mailer.Send
set Mailer = Nothing
End Sub

sub SendEmail(ServerName, apppool)

Set objEmail = CreateObject("CDO.Message")
objEmail.From = “user@it.com”
objEmail.To = “user@it.com”
objEmail.Subject = “Application pool: ” & apppool & ” is restarted on ” & ServerName
objEmail.Textbody = “Application Pool error.”
objEmail.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
objEmail.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/smtpserver") = _
“smtp.it.com”
objEmail.Configuration.Fields.Item _
("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25
objEmail.Configuration.Fields.Update
objEmail.Send

End Sub

Function GetComputerName
Set objWMISvc = GetObject( “winmgmts:\\.\root\cimv2″ )
Set colItems = objWMISvc.ExecQuery( “Select * from Win32_ComputerSystem", , 48 )
For Each objItem in colItems
strComputerName = objItem.Name
Next
GetComputerName = strComputerName
End Function
===Code ends============================================

Step 1: Create AGEE Auditing Policy to send Syslog to MOM

1. Go to AGEE configuration tool, root – System – Auditing – Policies – Policies, add new policy.
• In Server IP Address entry, type MOM Management Server IP
• In Port entry, type 514
• In Log Levels, select Emergency, Alert, Critical, Error, Warning, Notice
• Use Log Facility LOCAL1
2. Right click the new policy, and select “Global Bindings…”, tick the checkbox before the new policy. This will make AGEE to send all alerts to MOM.

Step 2: Create a Syslog Port Provider

1. In the left pane of the MOM Administrator console, right-click Providers.
2. On the context menu, click New Provider, specify Application Log as the data provider type, and then click Next.
3. For the provider name, enter Syslog.
4. For the provider log type, select Syslog port, and then click Finish.

Step 3: Create a collection event rule to collect syslog

1. First create a rule group named AGEE rules and associate this rule group with a computer group “Microsoft Operations Manager 2005 Servers”.
2. In the left pane of the MOM Administrator console, expand the rule group, right click Event Rules, and then click New Event Rule.
3. Select Collect Specific Events (Collection), and then click Next.
4. In the list, select the Syslog and then click Next.
5. Enter Collect Syslogs for the name of the rule, ensure that the Enabled check box is selected, and then click Finish.

Step 4: Create an alert event rule

1. In the left pane of the MOM Administrator console, expand the rule group, right click Event Rules, and then click New Event Rule.
2. Select Alert on or Respond to Event (Event), click Next.
3. In the list, select the Syslog and then click Next.
4. In the Criteria Page, click Advanced button, and choose Parameter 1 in field, choose contains substring and enter the Syslog message level (emerg, alert, crit, error, warning, notice, info, and debug ) in value. Click Add to List. (Note, Parameter1 value is the syslog message priority, Parameter2 value is the syslog message text.
5. Click Close and then click next. Check the Generate Alert in the Alert Page and configure the Alert properties.
6. Enter the name AGEE Syslog Emergency for the rule name. Click Finish.
7. Create additional rules for other syslog message level to generate alerts.

Ref:

HOWTO: Configure Microsoft Operation Manager 2000 or 2005 to monitor Unix devices by using Syslog http://support.microsoft.com/kb/555450

Configuring UNIX Computers to Forward Syslog Messages http://msdn.microsoft.com/en-us/library/aa505293.aspx

I decided to cut the pain short. On Tuesday I called Prometric to book the exam on Friday. 1Y0-309, AGEE exam, not many resources around. The Citrix exam enablement guide does have 10 sample questions, which I was comfortable at 6 of them, gave me a lot of confidence.

I passed the exam at 81%, pass rate 62%. It was not hard, half of the questions were very basic operations, and then some simple scenarios. These were enough for me to pass. I was so confident on the Presentation Server integration part, but only got 61% in this area, very strange.

Anyway, another step in my technical life. The CCEA seems very close to me, I only need to do the Password Manager, which is not a big deal. I have installed on my test server, will need to have a look again.

Fiddling with the Access Gateway Enterprise Edition again. This time I was testing with three scripts from Citrix and the command to reset “clear ns config full".

After I have run restore.sh and reboot, I saw the dreadful message on the console again:

Booting [kernel]…
can’t load ‘kernel’
can’t load ‘kernel.old’
can’t load ‘kernel.work’

I have seen this message twice after I did firmware upgrade from the GUI. In both times, Citrix support just asked us to return the box and replaced it. A post in Citrix Forum says this is a bug when updating the firmware via GUI. Sometime is works, sometimes it doesn’t. But I haven’t done upgrade this time, why?

According to Citrix support, we may be able to do something with the flash card – to copy the /flash directory from working system to the card. So it started:

1. Buy a card reader right away to house the CF card.
2. Plug in to a Linux Fedora VM machine
3. We have got a folder called “NetScaler”

We are stuck here!!! From what we see, nothing match the files in the working system directory. We copied the files to this folder, system did not start.

Finally, thinking the NetScaler is running on a BSD system, we downloaded a VMWare FreeBSD 6 and successful see the familiar files in /flash folder. We are missing the ns_x_xx.gz file. This told us the truth. I thought the file redundant and deleted the file in the flash folder. So on booting, the system could not load the kernel.

Copied the files in /flash from working system into the Compact Flash, the system started to working again.

It seems the CF card has got two partitions, one is in EXT3, another one in UFS.