Logscape Support

Members Login
Username 
 
Password 
    Remember Me  
Post Info TOPIC: Data Source Filemasks and Data Type File fields


Veteran Member

Status: Offline
Posts: 56
Date:
Data Source Filemasks and Data Type File fields
Permalink  
 


I think my most recent problems were the result of the filesystem running out of space and something getting corrupted.  I uninstalled, deleted any remaining Logscape files, then re-installed Logscape and it is working again.

I have a question about how the values in the fields for Data Source "File Mask" and "file" for Data Types.   For my implementation many different logs go into the same directory, but I don't necessarily want to get all of them, so specifying a file mask of *.log pulls in too much.

 

I've tried putting a list of the files I'm specifically interested in, but it seems like there's an issue with that or with using literal names.  When I looked under the work directory on the manager, I noticed only one or two files.  I went to the Data Source screen and reordered the csv list of file names and boom, more files started to appear on the manager.  

 

The system generating the log4j files also uses a rolling appender based on the size of the file.  In this case when the wrapper.log file reaches 5k, It creates a new wrapper.log file and renames the first one to wrapper.log1 (in the same directory).  There can be as many as 9 per the systems configuration.  I was suprised to see that even though I didn't use any wildcards in the csv list of file names, the forwarder was sending wrapper.log, wrapper.log1, wrapper.log2, wrapper.log3, etc.  Doesn't this mean that same file content gets re-indexed up to 9 times?  Is there a way to specify a literal file name and not have the forwarder pass the rolled logs?

 

Are there any particular rules or limitations for specifying multiple file names in the  Data Source "File Mask" and Data Types "File" fields (csv list with spaces, or don't use spaces, what if spaces are in the filename)?

 

Thanks,

 Thom



__________________


Veteran Member

Status: Offline
Posts: 56
Date:
Permalink  
 

Can I get any feedback on this? Am I missing something?

__________________
ZG


Veteran Member

Status: Offline
Posts: 60
Date:
Permalink  
 


Hi Thom,

Could you send an example of the file list pattern that changed it's behaviour on reordering. I have a brief summary of how  to use exclusions in file masks using an example below. Note that when you use the syntax !server.log it equates to exclude *server.log.

 

Forwarders

Each data source detects rolling files automatically. The same contents of the file are not reindexed when a file rolls. The index associated with the rolling file is renamed and the new contents of the file are indexed as they occur. Since rolling is detected automatically the Forwarder will always send rolled files to the Index Store.  If your rolled files lose their value after a few days the archiving policy on the Data Source can be modified to reflect this. For example your data source could have the following policy. 

Selection_737.png

This will delete any files older than 2 days off the Forwarder and delete files older than 7 days of the Index Store for this data source only. 

 

 


File Masks


The file mask syntax used on the Data types and the data sources is the same. Let's say you have a folder with the following log files:

server.log
webserver.log
fileserver.log
app.log
webapp.log
fileapp.log

You can create two data sources using the exclude syntax to separate the server and app logs.

tag: app-logs
FileMask: *.log,!server.log

tag:server-logs
Filemask: *.log,!app.log

You can create the data sources using an explicit list of files

tag:app-logs
FileMask: app.log,webapp.log,fileapp.log

tag:server-logs
FileMask: server.log,fileserver.log,webserver.log

You can also use regexp for your file pattern, but there is a restriction. A comma separated list of regex file patterns is not supported because of the ambiguity introduced by ',' character.





Attachments
__________________


Veteran Member

Status: Offline
Posts: 56
Date:
Permalink  
 

Hi,

The file name list I referred to is this: wrapper.log,cmdb.dal.log, cmdb.reconciliation.audit.log, cmdb.api.audit.detailed.log, jvm_statistics.log
(note, that I copied that from the GUI and noticed the first two files don't have a space after the comma, and those are the only files that are showing up in search results for the data type).

It is looking like I may just want Logscape to look at a few specific log files and not all of them.

With the automatic file rolling, you're saying that the files don't get re-indexed; however is it to be expected to see them show up in the list of files returned from a search. I tried using _filename.exclude(wrapper.log.\d+) to exclude them, but it didn't exclude them? As a side note, it would be nice if you could make multiple selections when selecting specific items to include or exclude.

With the forwarders, while I don't want the files on the forwarder host to be deleted (it has its own process for that), I would like to limit what is in the index store just to minimize space. My use case is for more near time troubleshooting, not for long term trending data. Is it possible to set the archiving rules as zero (as in never - not deleting the files) on the forwarder host and 5 days to retain for the index store?

__________________


Veteran Member

Status: Offline
Posts: 56
Date:
Permalink  
 

Regarding my comment about spaces in the list of file names; after I removed the spaces the other files started showing up.


__________________


Senior Member

Status: Offline
Posts: 100
Date:
Permalink  
 

Oh thanks for the update. Ill make sure this goes into the next 2.5 update.
Regards,
Na.

__________________


Veteran Member

Status: Offline
Posts: 56
Date:
Permalink  
 

Hi Na, 

There were a couple other questions left unanswered:

With the forwarders, while I don't want the files on the forwarder host to be deleted (it has its own process for that), I would like to limit what is in the index store just to minimize space. My use case is for more near time troubleshooting, not for long term trending data. Is it possible to set the archiving rules as zero (as in never - not deleting the files) on the forwarder host and 5 days to retain for the index store?

From the search screen, when making selections to include or exclude could you make it so you can select multiple items at once?

Additionally, it would be nice when you click on an one of the datatype elements that if it has a list longer than what can currently be displayed, you have an option to see all of them (say when you click 'other' it opens a window with a complete list). 

Thanks,

 Thom



__________________
ZG


Veteran Member

Status: Offline
Posts: 60
Date:
Permalink  
 



Hi Thom,

Disabling Archiving for the Forwarder

You can disable an archiving rule for a component by setting the value to 0. In your case, set the Forwarder Rule to 0 days, and the Index Store to 5 days [delete]

Datatypes (Side Panel)

For the system fields on the search page we could collect more than the top 5 values and bump that number to 20. Anything more than may clutter the UI. 

UI Driven Search

We could make a few minor enhancements around this for the 2.5 release. How about having the search autorun disabled by default? This would allow you to make multiple selections without a search running automatically. When you are ready you would then click the search button to execute the updated search with your selections.  

Regards,

ZG

 

 

 



__________________


Veteran Member

Status: Offline
Posts: 56
Date:
Permalink  
 

Hi,

Thanks for the clarification on configuring the forwarder archiving.

Regarding the Side Panel, having the ability to disable search auto-run via a simple check-box would be very helpful as I find myself going back an forth as I tinker to get the proper query string.

In my case I have a lot of files that require different Data Types and Data Sources. As I work with the tool to do somewhat blind data-mining I would like the option to display the complete field selection list in a separate pop-over window to do my selections. I've noticed in that the list of Discovered Fields can get truncated if the list is long which has also caused a bit of frustration.

Cheers,
Thom

__________________
Page 1 of 1  sorted by
 
Quick Reply

Please log in to post quick replies.



Create your own FREE Forum
Report Abuse
Powered by ActiveBoard