My blog has moved and can now be found at http://blog.aniljohn.com

No action is needed on your part if you are already subscribed to this blog via e-mail or its syndication feed.

Tuesday, September 14, 2004
« Rack-Mount.. Hmm... | Main | Courtesy & Professionalism.... »

One of the things I am researching a bit these days is how best to go about incorporating logging into a web based application.  I am looking at it from both an application troubleshooting perspective as well as a security perspective. 

Obviously web servers log activity in the W3C format. But those are access logs which tend to get very detailed and large. And it really does not give you a good picture of what is going on in the application. I've been doing a fair bit of reading in preparation for this and have come across some good information.  Hopefully it will spark some conversations from the developers, IA (Information Assurance) folks and other like minded people on what they would like to see in an application log.
 
A source that addressed this question directly was the AppSec FAQ at OWASP:
 
  • Do I need to have logging in my application even if I've W3C logs?

Yes, it's important that your application maintains "application level" logs even when W3C logging is used. As W3C logs contain records for every http request, it is difficult (and, at times impossible) to extract a higher level meaning from these logs. For instance, the W3C logs are cumbersome to identify a specific session of user and the activities that the user performed. It's better that the application keeps a trail of important activities, rather than decode it from W3C logs.

  • What should I log from within my application?

Keep an audit trail of activity that you might want to review while troubleshooting or conducting forensic analysis. Please note that it is inadvisable to keep sensitive business information itself in these logs, as administrators have access to these logs for troubleshooting. Activities commonly kept track of are:

  • Login and logout of users
  • Critical transactions (e.g.. fund transfer across accounts)
  • Failed login attempts
  • Account lockouts
  • Violation of policies
The data that is logged for each of these activities usually include:
  • User ID
  • Time stamp
  • Source IP
  • Error codes, if any
  • Priority
 
Mike Gunderloy, in his book "Coder to Developer" (excellent book!, go buy it now!), also devotes an entire chapter to logging application activity. As compared to OWASP, he looks at it more from an application troubleshooting perspective and advises that as a rule, you should "Log the information that you use when debugging".  In particular he recommends looking at logging the following items:
  • Error messages and information, including a stack trace of the error
  • The state of internal data structures at key points in the application
  • User actions, such as button clicks and menu item selections
  • The time the significant actions were performed
  • Important information about the environment, such as variable settings and available memory
The PAG folks in "Improving Web Application Security" recommend the following when it comes to Auditing and Logging:
  • Audit and log access across application tiers
  • Consider identity flow
  • Log key events
  • Secure log files
  • Back up and analyze log files regularly
Michael Howard in "Writing Security 2" has the following suggestions:
  • Log IP Addresses
  • If you are creating a lot of files, you should have your own log files and not use the OS Application Log
  • Logs should go into a directory that is user configurable and it's best to create a new log file every day
  • Consider creating multiple log files, one for routine events another for extraordinary events
  • Application logs should be writable only by the administrator and the user the service runs under
  • When code fails for security reasons, log the data in a place that only the administrator has access to
One surprise in my research was that this topic was almost completely (beyond a couple of sentences) ignored in "Code Complete 2".
 
Comments?
 

Tags:: Security
9/14/2004 7:46 PM Eastern Daylight Time  |  Comments [10]  |  Disclaimer  |  Permalink   
Sunday, May 8, 2005 12:06:43 AM (Eastern Daylight Time, UTC-04:00)
Consider custom performance counters also. They fall more under the application troubleshooting category, but are a great way to take measurements. Particularly when the event is high frequency and you need to be unobtrusive.
Scott Allen
Sunday, May 8, 2005 12:06:43 AM (Eastern Daylight Time, UTC-04:00)
Scott - Agreed that the perf counters can be very good way to take measurments. The caveat in that case is that we have console access to the machine, right? Or am I missing something?
<br>
<br>Keep in mind that the assumption that I'm making is that this is a production app that is going to be deployed and the logging/instrumenting that I doing is going to be turned on while the app is in production. Add in another very common constraint that the app may be remotely hosted and I simply may not have access to the perf counters.
<br>
Anil John
Sunday, May 8, 2005 12:06:43 AM (Eastern Daylight Time, UTC-04:00)
Ah, for remote hosting performance counters aren't all that useful.
<br>
<br>Are you planning on building a hacme type app? :)
Scott Allen
Sunday, May 8, 2005 12:06:43 AM (Eastern Daylight Time, UTC-04:00)
:-) It's just that I am really gettting into instrumenting prod apps AND I am also running into some heavy duty logging requirements from a security perspective for some some things I am working on.. So am trying to kill two birds with one stone.
Anil John
Sunday, May 8, 2005 12:06:43 AM (Eastern Daylight Time, UTC-04:00)
Anil,
<br>
<br>I've been working on a HTTP security proxy for some time. It turns out implementing logging correctly is extremely difficult to do. For one thing it can be a security problem in and of itself, as many log entries are based on user (or network) input. Generally not a good thing.
<br>
<br>The other problem is performance. If you are really concerned about throughput, logging can dog your application. This is especially bad for Unix based event driven servers. To get resonable performance, logging often has to occur in a seperate thread. You also can have the problem of the network being faster than the disk, especially if you use verbose logging. That basically makes your app bound on local disk write speed.
<br>
<br>Here are some blog entries I've written on the topic:
<br>
<br><a target="_new" href="http://www.baus.net/archives/000127.html">http://www.baus.net/archives/000127.html</a>
<br><a target="_new" href="http://www.baus.net/notvoid">http://www.baus.net/notvoid</a>
<br>
<br>
christopher
Sunday, May 8, 2005 12:06:43 AM (Eastern Daylight Time, UTC-04:00)
Christopher,
<br>
<br>You are obviously approaching this at a much deeper level that I am :-)
<br>
<br>What I ended up going with for my app, and what works rather well for me, is Log4Net (which is a .NET port of Log4J)
Anil John
Sunday, May 8, 2005 12:06:43 AM (Eastern Daylight Time, UTC-04:00)
SecureCoder by Anil John
Sunday, May 8, 2005 12:06:43 AM (Eastern Daylight Time, UTC-04:00)
SecureCoder by Anil John
Sunday, May 8, 2005 12:06:43 AM (Eastern Daylight Time, UTC-04:00)
Playing a devil's advocate, what do you use all the captured data for? Considering that there's way too much data that can be logged, IDS has proven way to complicated to set up and there's way too little motivation to review logs, a desire to log as much as possible may not be the best way forward...
Jiri Ludvik
Sunday, May 8, 2005 12:06:43 AM (Eastern Daylight Time, UTC-04:00)
My primary purpose in logging is twofold.
<br>1) CYA from the from the perspective of a security spill. If need be, I should be able at a minimum to see who did what on the secure portions of a site.
<br>2) Troubleshooting. I should be able to turn on DEBUG mode as needed to troubleshoot the app while it is in production.
Anil John
Comments are closed.