Logscape Support

dajinn · Newbie

Data isn't coming over?

ZG · Veteran Member

Hi dajinn

Could you put a screenshot of

1.) Your Data source for the *.txt log files
2.) The LogServer data source

and then check the LogServer directory on the Manager to see if the logs are actually shipped over to the manager ? The LogServer stores its files in the $LOGSCAPE_HOME/LogServer_SERVER_ folder.

Regards,

ZG.

dajinn · Newbie

Greetings ZG,

What do you mean in 2) the logserver data source?

Here is a screenshot of my data sources with the desired remote log data source pre-selected:

i.imgur.com/EzrSj0H.png

Also, incase you're wondering, when I click "browse" and pick the folder on the remote forwarder, the host name is correct, it just isn't shown there.

and yes, it does appear that the logs are showing up from the correct host location under LogServer_SERVER, they just don't get loaded into the interface.

I think I understand now. Am I supposed to create a data source to pull the logs from a forwarder and then create a data source on the manager to point to that folder once it's started to grab files?

Thanks,

Dajinn

ZG · Veteran Member

Hi Dajinn

? When a Forwarder sends data to an IndexStore or Manager the files are stored in the $LOGSCAPE_HOME/work/LogServer_SERVER_/$SERVER folder.
So your blackboard data would appear in a path like

$LOGSCAPE_HOME/work/LogServer_SERVER_/$SERVER/C/Blackboard

and you sqlerrors in something like

$LOGSCAPE_HOME/work/LogServer_SERVER_/$SERVER/C/sqlerrros

The IndexStore creates a datasource annotating it with information from where the data came from. It looks like this isn't happening in your set up.

What we need from you ?

* What OS is your IndexStore/Manager running on ?

* Are your Forwarders in a different time zone from the Manager

* Could you send an export of your config ( Accessed from the Backup in the Configure section)

You can either attach this to your forum reply or send the attachments to support@logscape.com

Workaround

A workaround for this would be to create an explicit data source to the exact location on the blackboard and sqlerror logs on the Manager/IndexStore. So something like

tag: Blackboard
directory: **/blackboard/logs
file mask: 30

will pick up the logs and allow you to search on them. You will lose the server information with this approach, but at least it will get you access to your logs through the interface.

Kind Regards,

ZG.

dajinn · Newbie

Hi ZG,

Manager and Forwarder are both running on Windows Server 2008 R2 Standard

The forwarder is not in a different time zone.

I've attached the config file in an email and sent it to the email address provided.

Your support format / reply is very professional! I look forward to working further with you on the subject.

Thanks,

Dajinn

Skazal · Veteran Member

I'm having a related problem. I have a simple deployment; manager on one host and a forwarder on another, both Windows Server 2008 R2 Standard. I see the log files from the forwarder showing up on the manager in the folder $LOGSCAPE_HOME/work/LogServer_SERVER_/$SERVER folder but when I try to search, no data from the forwarder system is displayed. What should I look for to troubleshoot?

neilson9 · Senior Member

Hi Guys,

We have found the problem and have provided a fresh 2.5.0 prerelease to the downloads page. However, to save you pain of a full upgrade you are able to download the following files.

http://www.logscape.com/boot.zip

http://www.logscape.com/vs-log-1.0.zip

Upload them on the Deployment page, once all agents have completed the download bounce the system and you should be seeing your data!

Apologies for the inconvenience & best regards,

NA.

Skazal · Veteran Member

Well, I'm now seeing the logscape-audit tagged data, but nothing from the logfiles in the directory I selected in my Data Source definition. I've also modified my config so my environment has 2 Windows Server 2008 R2 Standard (one as manager [dev.manager] and a forwarder [dev.IndexStore.ucmdb] on the other) and I've added a RHEL 6 server as an Index Store [dev.IndexStore]. Any idea why I'm not seeing the data from the logfiles on my forwarder server?

ZG · Veteran Member

Hi Skazal,

The freely available download on the website is for a Manager and as many forwarders as you like ( or can handle). All the data will be forwarded to the Manager.

Configuring additional IndexStores ( apart from the one running when you start the Manager ) will require a license. All scalability options around zoning with Index Stores and shaping the flow of data in a large deployment are features provided with a support license.

If you need any further information on what support and licensing costs may look like with your current deployment please see the pricing page

For an unlimited enterprise licensing please contact sales@logscape.com or support@logscape.com

Note: The freely downloaded version does not put any restrictions on the amount of data you can index and search.

Regards,

ZG

ZG · Veteran Member

Hi Skazal,

If you revert your set up back to the Manager and Forwarders you should then see your data come through. If that is not the case, we'll need to diagnose further by taking
a look at the data sources assigned to your forwarders using the jmx http url found on the Agents page.

Skazal · Veteran Member

Ok, I reverted back to a single host for the manager and a single host for the Forwarder, but my log files (all log4j format) are not showing for searching.

For example, when I go to the Datasource page and click on the action link to search my Forwarder's logs, no results are returned. However, if I do the search "*| _host.equals('forwarder-host-name')" I can drill-down into "_type" and see there are 2 values: 'basic' and 'agent-stats'. When I click on "_tag" it shows a count over 1000 for "logscape-audit".

I invoked the jmx console from the Agents page. What should I be looking at?

On a related note, the log4j files I want to digest have different pattern layouts. Since the Logscape log4j Data Type pattern uses a perl regular expression, is there script/tool/utility that would convert the log4j pattern format and create the perl regex, or is there a different type that would allow me to directly past in the log4j pattern layout from the log4j properties files that create the logs?

neilson9 · Senior Member

Hi Skazal,
Can you access the following JMX page on your manager and send us the output (Replace 10.28.0.74 with its IP address.)?

10.28.0.74:11005/InvokeAction//com.liquidlabs.logscape%3AService%3DIndexer/action=listFwderAutoDataSources?action=listFwderAutoDataSources&p1%2Bjava.lang.String=

The output lists the path being searched by appending the remote datasources to the LogServer_SERVER_/ folder,

Can you also export your config that with the output to us: support@logscape.com

WRT datatypes you can use perl regex, the expression on the log4j type will be converted to a perl regex. You can see the translation by clicking the test button on the RHS of the page.

You could either hijack the existing type or use new one.

Regards,

Na.

ZG · Veteran Member

Hi Skazal,

I am going to open a new thread for your log4j data type mapping

Regards,

ZG

Skazal · Veteran Member

Thank you. I have just sent the requested info to support from my work email.

neilson9 · Senior Member

Hi Skazal,
The fix was not complete, as per the email - we have fix it again (!). The boot.zip link above has been replaced and the 2.5 pre-release also replaced.

Best regards,
Na.