Radius – Server 2008 R2 NPS

We’ve OLYMPUS DIGITAL CAMERA         been using NPS on Server 2008 for a while now and its been perfect for handling 802.1x authentication (EAPTLS) and radius auth from the HP WESM in the 5400zl. The radius setup for the HP Wireless Edge Services was pretty easy, it only needs radius clients for the Primary WESM and any Redundant WESM’s.

Now that we’re adding another 50-70 E-MSM422 AP’s for the MSM765 controller we need to add radius clients for each AP. After a conversation with Adam (@DJADSA) we worked out that we were going to hit the 50 radius client limit in Server 2008 Standard. Adam showed me a couple of neat tricks with their NPS configuration that would save us a tonne of time and are new additions to R2!

The first trick was adding a subnet range for Radius Clients instead of adding a radius client for AP individually. Adding the IP/CIDR and shared secret will let all devices in the range talk to the NPS server.

radiusclient

The next tip from Adam was with the Accounting in NPS. We’d tried to get SQL logging to behave in Server 2008 a few times and failed miserably. The NPS application in 2008 would connect to a SQL database but wouldn’t create the structure etc. There was a sql script on the web that would create it for you but we didn’t have any luck getting it all to work properly. 2008 R2 has a new wizard for setting up NPS accounting and the final stage of the wizard gives you the option of creating the SQL structure of the database. very tidy

accountingwiz

With SQL logging enabled it give us the option of writing a web part or two for SharePoint to let staff know which users are connected where, and lets us easily run scripts to find client/authentication problems.

RemoteApp: Synergetic access from Home

Finding a decent solution for remote access to Synergetic has always been a problem. The Synergetic loader starts the application from a shared network drive, which is fine when Staff are at School but when they’re offsite, it’s tedious loading Synergetic with a 1Mb upload on the School’s internet link.

RemoteApp was a new feature with Server 2008 and has been refined further with 2008 R2 and Windows 7 with the RemoteApp and Desktop Connections. RemoteApp works similarly to the traditional Windows Terminal Server login used with previous versions of Windows Server, but with more functionality. When you configure a program for RemoteApp, the end user gets the same icon on their desktop or start menu that they would if the application was installed locally. The icon is a shortcut for remote desktop(RDP) that loads a full terminal services login, but hides the session and only shows the application, which is running in the TS session. The user’s printers and mapped drives etc can all be used in the RemoteApp, same as an RP session, but is set per program.

Accessing Synergetic via RemoteApp offsite is as seamless as connecting when at School and doesn’t require a VPN connection. Setting up RemoteApp with signed certificates and opening ports on the firewall is the way to go. Users still have to pass AD authentication, and depending on your Synergetic setup, another username and password to login to Synergetic.

 

Video -  loading Synergetic with RemoteApp

After a brief trial of Synergetic with RemoteApp it looks like we’ll be purchasing the necessary RDS User CAL’s (Check here for changes to TS licensing) and using RemoteApp for Staff access to Synergetic from home and getting them to use this setup for their Academic Report writing and avoid the confusion between Synergetic Network/Stand-Alone and importing/exporting reports.

To setup the trial of Server 2008 R2 and RemoteApp, follow something like the TS RemoteApp Step-by-Step Guide which is pretty straight forward. If you have Windows 7 clients, make sure you check out RemoteApp and Desktop Connections where you can set the Win7 machines to check a URL for a list of available RemoteApps and it will update regularly and automatically put shortcuts on the start menu for users

After you’ve configured the TS services, install Synergetic on the TS box and add it as RemoteApp through the Wizard

wiz1

If your a large Synergetic customer, you probably have multiple databases for different users and need to specify different configuration files with command line arguments.

wiz2

Here’s our test RemoteApps for Synergetic

remoteAppPrograms

Pushing the links for RemoteApps out as *.MSI or RDP files via a script or download makes things nice and easy too

I hadn’t paid much attention to RemoteApp and Microsoft’s VDI offerings, which seems like a mistake on my behalf. With the price of EDU licensing for MS Apps, this is a nice and tidy solution to a problem we’ve had since purchasing Synergetic 10-12 years ago. It might be a good solution for getting applications onto our Student Netbooks too… will see how we go

Random AD User Account Lockout

user-account-control-icon The last few weeks we’ve had a problem with one of our IT Staff user accounts where it would regularly get locked out during the day. We suspected some of the Students were trying to guess the password for the account and were probably hoping to get around our web content filter….

This made me realise, we’ve never really looked for account lockouts and and where/why/how they might be happening on the network. I wasn’t excited about scouring the security event logs on our domain controllers to find the info I needed. A quick search brought up the Account Lockout and Management Tools from Microsoft which has been around since 2003, but was new to me. I probably should have been a bit sharper on that one.

One of the applications in the download is that LockoutStatus. This app will take the AD username and return the lockout status and bad password count on each DC for that user.

lockoutstatus

After finding the lockout status if the user you then use the EventCombMT app to search the event logs of the domain controllers. EventCombMT can search the event logs for any event ID but to find events with login issues, I limited the search to the Security logs on the DC’s with event ID’s 529 644 675 676 and 681. More info on usage here

EventComb

EventCombMT produces a text file for each server with the results of the search:

644,AUDIT SUCCESS,Security,Wed Apr 14 10:21:08 2010,NT AUTHORITY\SYSTEM,User Account Locked Out:     Target Account Name: zdjb     Caller Machine Name: STNB4200ZKLC     Caller User Name: IDM$     Caller Domain: TEACH     Caller Logon ID: (0x0,0x3E7)   

644,AUDIT SUCCESS,Security,Wed Apr 14 10:08:14 2010,NT AUTHORITY\SYSTEM,User Account Locked Out:     Target Account Name: zdjb      Caller Machine Name: STTB2710ZJLW     Caller User Name: IDM$     Caller Domain: TEACH     Caller Logon ID: (0x0,0x3E7)  

Straight away I could see the two machines that were causing the account lockout and after calling them back into the Service Desk so we could see what was going on. We were able to work out that it was the Sophos Auto Update causing the problem because these machines had been using the credentials  of this account, with an old password, for receiving updates instead of the update account we normally use.

Solving the problem of the account lockout has prompted some new strategies for monitoring the network that we hadn’t considered before. Now that we have the tools, we can regularly search the event logs for bad passwords on Administrator accounts and follow up the results. Also, we may have started the ball rolling on a project to create more user accounts in AD for different network services. It was lucky that we’d found Sophos was the culprit, but next time it could be difficult to find old credentials being used in an Altiris job for example. By creating different user accounts for services like Sophos, Altiris Jobs, SQL, Exchange and Intranet applications it will help narrow down a search next time we have a password problem and will make it easier to change these passwords on a regular basis.

Exchange 2007 Edge Server Licence

Earlier this year we migrated our Exchange 2003 setup to Exchange 2007 and all was going well for a while. Until we had noticed that our Edge Server thought it was unlicensed, even though we’d entered all the licence info as part of the initial Exchange config. A quick search found this handy PowerShell command for setting, or resetting, the Product Key for our Edge Server. I’m still impressed with how much PowerShell can do in Exchange/Windows, its definitely worth investing the time to learn.

< Power Shell CMD >
[PS] C:\Windows\System32>get-ExchangeServer

Name                Site                 ServerRole  Edition     AdminDisplayVersion
—-                —-                 ———-  ——-     ————–
JACKAL             BGS                  Mailbox,… Standard    Version 8.1…
CARLOS            BGS                  ClientAc… Standard    Version 8.1…
EDGE                BGS                  Edge        Standard… Version 8.1…
ZORO               BGS                  ClientAc… Standard    Version 8.1…

[PS] C:\Windows\System32>set-ExchangeServer Edge -ProductKey xxxxx-xxxxx-xxxxx-xxxxx-xxxxx
[PS] C:\Windows\System32>get-ExchangeServer

Name                Site                 ServerRole  Edition     AdminDisplayVersion
—-                —-                 ———-  ——-     ————–
JACKAL             BGS                  Mailbox,… Standard    Version 8.1…
CARLOS            BGS                  ClientAc… Standard    Version 8.1…
EDGE                BGS                  Edge        Standard    Version 8.1…
ZORO               BGS                  ClientAc… Standard    Version 8.1…

[PS] C:\Windows\System32>

DL380 – HP Insight Manager Goodness

Over the weekend I had the opportunity to experience the HP Insight Manager goodness. We were on our way to Melbourne to go shopping for they day, early Christmas shopping, and about half way there I received an email from one of our servers, “IDM”, which had detected a drive failure. The drive that failed is part of a RAID5 array so we could replace the disk and it should rebuild successfully, as long as we could replace the drive before another disk failure!

We have seen a few disk failures before on the DL380 servers and and have had no issues with replacing the disks and DL380rebuilding them, but it was only recently that we started updating all the servers with the latest firmware and configuring them to send alerts on failures. So this was the first time we’d seen the email alerts for a disk failure, which obviously meant that we could deal with it straight away instead of waiting for someone to notice the RED light on the failed disk when they were in the server room.

With the drive failure occurring  on the weekend, and as I was an hour or so away, we had to do a quick call out to our IT Staff to see who was available to perform the disk swap. As luck would have it, someone was heading in to catch up on a few hours and was only a few minutes away. This particular member of Staff is our web developer and has pretty good knowledge of hardware but hasn’t had experience with our DL380 servers before. So, over the phone, I talked him through changing the disks over and starting the rebuild process. When we had our first disk failure on a DL380 we found it hard to find documentation on what to do, and this was probably because it was so easy and that we didn’t expect the RAID controller to do so much of the work by itself.

All Jeff had to to was unpack a new drive from its box, removed the failed drive and then insert the new disk. The new disk is already boxed inside its caddy, ready to slide into the server. So there’s no need to find a screwdriver and remove the disk and insert the new one in the caddy. After inserting the new disk the raid controller detects the disk, initialises and then begins the rebuilding process. This particular RAID array wasn’t too large and took less than an hour to rebuild, and as every change in disk status occurred, the server detected the change and sent a notification message. So as Jeff was replacing the disk, I was getting the notification messages instantly on my phone.

I’ve included the emails from the Insight Manager below. At the moment we only a few of our DL380s with the current firmware and Insight Manager, this is because of a problem we found with the SCSI backplane and the newer firmware. The latest update caused a problem where the Status LED’s on the SCSI disks fail to light up, green or red, for one of the disks in the server. We held off on continuing with the firmware updates but may reconsider that for the moment when we get such comprehensive information from the Insight Manger and alert emails it seems like the gains out weigh the inconvenience of LED issues.

Initial email – detected drive failure
—————————————————————————————————————————–
From: <ProLiant@>
Date: Sat, 22 Nov 2008 09:31:01 +1100
Subject: Storage Agents: Physical Drive Status Change

The system has detected the following event:
SNMP Trap:      3046
Date time:      11/22/2008  09:31:00 AM
Computer:       IDM
Source:         Storage Agents
Type:           Error
Category:       (4)
Description:
A ‘Physical Drive Status Change’ trap signifies that the agent has detected a change in the status of a drive array physical drive.
Details:
IDA Physical Drive Status ‘FAILED’
Drive Type 2
Location  ‘SCSI Port 1 Drive 3’
Error Code 13
Bus # 1
Controller Slot # 2
Model ‘COMPAQ  BD14689BB9      ‘
Serial Number ‘DAA1P6909WNS0637’
Firmware Revision ‘HPB1’

Second Email –new disk inserted and initialised ‘OK’
—————————————————————————————————————————–
From: <ProLiant@>
Date: Sat, 22 Nov 2008 11:47:07 +1100
Subject: Storage Agents: Physical Drive Status Change

The system has detected the following event:
SNMP Trap:      3046
Date time:      11/22/2008  11:47:06 AM
Computer:       IDM
Source:         Storage Agents
Type:           Informational
Category:       (4)
Description:
A ‘Physical Drive Status Change’ trap signifies that the agent has detected a change in the status of a drive array physical drive.
Details:
IDA Physical Drive Status ‘OK’
Drive Type 2
Location  ‘SCSI Port 1 Drive 3’
Error Code 0
Bus # 1
Controller Slot # 2
Model ‘COMPAQ  BF14684970      ‘
Serial Number ‘        J4W1PB3C’
Firmware Revision ‘HPB5’

Third Email – new disk is being rebuilt in the RAID5 array
—————————————————————————————————————————–
From: <ProLiant@>
Date: Sat, 22 Nov 2008 11:47:07 +1100
Subject: Storage Agents: Logical Drive Status Change

The system has detected the following event:
SNMP Trap:      3034
Date time:      11/22/2008  11:47:06 AM
Computer:       IDM
Source:         Storage Agents
Type:           Warning
Category:       (4)
Description:
A ‘Logical Drive Status Change’ trap signifies that the agent has detected a change in the status of a drive array logical drive.
Details:
IDA Logical Drive Status ‘REBUILDING’
Logical Drive # 2
Controller Slot # 2

Fourth Email – all done, RAID rebuilding complete and disk is OK, back to normal
—————————————————————————————————————————–From: <ProLiant@>
Date: Sat, 22 Nov 2008 12:37:07 +1100

Subject: Storage Agents: Logical Drive Status Change

The system has detected the following event:
SNMP Trap:      3034
Date time:      11/22/2008  12:37:07 PM
Computer:       IDM
Source:         Storage Agents
Type:           Informational
Category:       (4)
Description:
A ‘Logical Drive Status Change’ trap signifies that the agent has detected a change in the status of a drive array logical drive.
Details:
IDA Logical Drive Status ‘OK’
Logical Drive # 2
Controller Slot # 2

After reading Mick Liubinskas’ post on ‘How I Blog’ I thought I’d try a quick and nasty, two beer post with minimal spell checking and absolutely no grammar checking or proof reading…..

Exchange c1041721 Information Store error

We had a strange problem today with our main Exchange Server. The server is running Exchange 2003, Windows Server 2003 and on decent HP hardware. We noticed the problem when we had a call from Administration to say that emails weren’t leaving their Outbox in Outlook. After experiencing the same problem from our machines we restarted the SMTP service on the Exchange Server. This had no effect on the problem so we looked at the Information Stores in the Exchange System Manager, and received  a popup with something to the effect that the IS wasn’t running and the error code c1041721.

After checking that all auto-start services were running we scoured the Event logs and couldn’t find anything relating to Exchange and the problems we had. We thought that the issue began between 13:12 and 13:15 by talking to a few staff members and checking when email stopped flowing through the server.

After some quick Googling the consensus was to try restarting the Information Store service, even though it was running in the task manager and occupying RAM and CPU. The service timed out during the shutdown and went into an uncontrollable state so we killed the process for store.exe. We were able to start the Information Store service again, without issue and without a reboot. Our Outlook Exchange connections appeared happier, with some machines stating that the connection had been lost and restored, and email flowed immediately.

After the Information Store server started there were some interesting messages in the Event Viewer:

ExchangeSA EventID 9175 where the MAPI call OpenMsgStore failed with the following error: Microsoft Exchange Server computer is not available. Either there are network problems or the Exchange Server is down for maintenance. MAPI Provider failed. Microsoft Exchange Server Information Store ID no: 8004011d-0506-00000000

VSS EventID 8194 Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface. hr = 0x800706ba.

MSExchangeIS EventID 9665 The memory settings for this server are not optimal for Exchange.

ESE Backup EventID 905 Information Store (247004) Server registered: Microsoft Exchange Server / Microsoft Information Store (callback DLL mdbrest.dll, flags 0x103).

ESE EventID 300 Information Store (247004) BG Staff Mail: The database engine is initiating recovery steps.

ESE EventID 301 Information Store (247004) BG Staff Mail: The database engine has begun replaying logfile e:\Program Files\Exchsrvr\Admin Staff\E01.log.

ESE EventID 302 Information Store (247004) BG Staff Mail: The database engine has successfully completed recovery steps.

The VSS Event 8194 had been logged 5-6 times in a row before the ESE recovery events.  The Exchange Server runs ShadowProtect for server snapshots and takes an incremental backup every 15 minutes during the day. The last snapshot had been taken just before midday and a snapshot should have initiated around the time the Exchange IS went, effectively, offline.

Since starting the Information Store service Exchange seems to be running as normal. Although I am presently running the Data Protector backup for Exchange before resuming ShadowProtect backups. While I don’t think ShadowProtect was directly responsible for the problem, I think there was something happening between ShadowProtect/VSS/Exchange that went pear shaped and hung the Information Store? Googling “c1041721” and ShadowProtect or VSS brought up basically no results, I would be interested to hear from anyone else who has had this error, or something similar, come up.

Update: The Data Protector Exchange backup completed successfully and I restarted the ShadowProtect backups and it picked up from the last incremental backup without any problems.