One of the logs I want to process is an error log that contains a lot of java exceptions with stack traces. Here is the sample of the log file data showing two events:
(Note: the first character position on a line is where you see the "<" preceding the date)
<2014-09-29 18:38:08,972> [ERROR] [pool-1-thread-1] (AbstractWinkClientHandler.java:51) -
com.hp.sw.bto.ast.security.lwsso.ws.handlers.SecurityHandlersException
at com.hp.sw.bto.ast.security.lwsso.ws.handlers.wrappers.LWSSOWinkClientHandler.handleException(LWSSOWinkClientHandler.java:42)
at com.hp.sw.bto.ast.generichandlers.wrappers.AbstractWinkClientHandler.handle(AbstractWinkClientHandler.java:39)
at org.apache.wink.client.internal.handlers.HandlerContextImpl.doChain(HandlerContextImpl.java:52)
at org.apache.wink.client.internal.ResourceImpl.invoke(ResourceImpl.java:216)
at org.apache.wink.client.internal.ResourceImpl.invoke(ResourceImpl.java:178)
at org.apache.wink.client.internal.ResourceImpl.get(ResourceImpl.java:288)
at com.hp.sw.bto.ast.security.cm.api.CMRestClientAPIsImpl.getAuthorizedFolderNames(CMRestClientAPIsImpl.java:814)
at com.hp.sw.bto.ast.security.cm.api.CMClientCacheProxy.updateCache(CMClientCacheProxy.java:254)
at com.hp.sw.bto.ast.security.cm.api.CMClientCacheProxy$CacheUpdateThread.run(CMClientCacheProxy.java:232)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
<2014-09-29 18:39:08,988> [ERROR] [pool-1-thread-1] (AbstractWinkClientHandler.java:51) -
com.hp.sw.bto.ast.security.lwsso.ws.handlers.SecurityHandlersException
at com.hp.sw.bto.ast.security.lwsso.ws.handlers.wrappers.LWSSOWinkClientHandler.handleException(LWSSOWinkClientHandler.java:42)
at com.hp.sw.bto.ast.generichandlers.wrappers.AbstractWinkClientHandler.handle(AbstractWinkClientHandler.java:39)
at org.apache.wink.client.internal.handlers.HandlerContextImpl.doChain(HandlerContextImpl.java:52)
at org.apache.wink.client.internal.ResourceImpl.invoke(ResourceImpl.java:216)
at org.apache.wink.client.internal.ResourceImpl.invoke(ResourceImpl.java:178)
at org.apache.wink.client.internal.ResourceImpl.get(ResourceImpl.java:288)
at com.hp.sw.bto.ast.security.cm.api.CMRestClientAPIsImpl.getAuthorizedFolderNames(CMRestClientAPIsImpl.java:814)
at com.hp.sw.bto.ast.security.cm.api.CMClientCacheProxy.updateCache(CMClientCacheProxy.java:254)
at com.hp.sw.bto.ast.security.cm.api.CMClientCacheProxy$CacheUpdateThread.run(CMClientCacheProxy.java:232)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
I've tried the following pattern and it sort of works, except the leading spaces in the exception stack trace seem to be causing the information not to be captured successfully:
The pattern I'm using looks like this: <(2*)\s+(*)>\s\[(INFO|DEBUG|WARN|ERROR|FATAL)\s?\]\s+\[(*)\]\s+\((*):(*)\)\s-(**)
The pattern parses into the following column names: date, time, level, thread, class, lineNo, line, ExceptionName, ExceptionMessage
The columns ExceptionName and the ExceptionMessage aresynthetic fields:
Name:ExceptionName
synth-source: line
synth-expression: (*)Exception
Name: ExceptionMessage
Summary: DISABLED
synth-source: line
synth-expresssion: Exception(**)
Testing it from the Data Types GUI worked at first, until I realized I somehow lost the leading spaces in the exception stack trace that appear in the actual file. Once I got an accurate sample (what I included above), the capture of data for the column line and the two synthetics ExceptionName and ExceptionMessage were empty. What needs to be done to correct the parsing?
I've copied your sample using your pattern and I see the expected values in the Datatype page. The spaces don't appear to be the problem. I notice that the pattern you are using will match a log file that contains only exceptions. Is this the case with your log? If your log contains other lines that don't have a class. line no and exception in them, the type will not process your log file correctly. The sample lines used on the Datatype should use enough lines that are representative of the type of log lines that occur in your file and the pattern must match all the different types. The more varied the lines in your log pattern, the more general your pattern will have to be and the more you will need to rely on synthetics to pick data that does not occur on every line.
I have not been able to reproduce the problem in our lab. Could you provide a screenshot where the error is occurring, I may be missing something.
Thanks. I think it may have something to do with how the adapter is interpreting the end of a message. I've also attached a screenshot from an editor with one of the exceptions showing the hidden characters (end of line & tabs).