We have a customer with about 200 client boxes (some via Citrix) who is running a bunch of custom programs. The custom programs (which I didn't right btw) mostly work now.
But once in a while.... (2 or 3 times a month) one of the custom programs fails to write information into the databases for the accounts payable system.
If this were easily repeatable we would simply run the program with DbgView running to catch the trace info until we found it. But DbgView (which reads the windows OutputDebugString channel) only works on one box at a time.
I guess we could run 200 copies of DbgView, one on each client, and then collect the data files daily or something. But this a management nightmare from the customers perspective. Blech!
Some Basic Assumptions:
- The customer doesn't know squat about Linux/Unix/Aix etc. They know windows.
- Linux has the syslog facility, which can collect local and remote udp based syslog messages.
- A syslog derivative called "rsyslog" will write incoming messages to a MySQL database.
- Freeware windows based DLLs are available which will allow windows based programs to create udp based syslog messages.
- syslog messages are fire and forget - if there isn't anything on the other end to receive them... not a problem.
- Logging system will only be enabled log enough to find the point(s) of failure
- The custom programs already output detailed logging information via OutputDebugString()
- Using VMWare create a VMWare Image based on Ubuntu Linux Server (no X windows overhead).
- Add the development tools to linux using apt-get.
- Download, build, and install rsyslog on the system.
- Download, build, and install a web based SysLog viewer.
- Get network config info from the customer, and setup the network on the linux system.
- Shutdown the linux system, then close down VMWare.
- Zip up the entire VMWare Image, put it on a DVD with the free VMWare player and send to the customer.
- Rebuild the custom programs with one of the freeware syslog enabling DLLs, and hook into the debug message function, sending a duplicate message to syslog.
- Have the customer deploy the updated programs and DLL (we have in place an auto-update system to make this an easy thing to do).
- Consolidates logging for multiple clients
- Logging data delineated by IP address/Date Time
- Customer doesn't have to learn/know linux they are delivered a preconfigured linux server which is already setup to collect the data.
- Minimal changes to the custom programs.
- Log data is accessible with a web browser; no need to know SQL commands.
- Did I mention free?
Once this is done it becomes a data analysis chore; but a manageable one.
RSyslog - a syslog alternative. There is a ton of installation and configuration information in the Documentation pages.
KLog - a free suite of syslog message sending tools for the Windows platform. This allows you to add syslog message sending capabilities to your applications or batch files. Tools include: Console, Win32-API, COM and .Net components.
phpLogCon - a web based syslog data interface
VMWare - home of the free VMWare player and other VM tools