Logscape Support

Members Login
Username 
 
Password 
    Remember Me  
Post Info TOPIC: data sources and tags


Member

Status: Offline
Posts: 10
Date:
data sources and tags
Permalink  
 


Maybe I am missing something (new user of logscape)... 

So, I have "cloned" the syslog-server datasource to a "unix" as a test.  I have reindexed the data source but there simply are no hits on the search.  When I search with

* | _tag.equals(syslog-server) I get tons of data but if I use

* | _tag.equals(unix) I get 0 results.  

What am I doing wrong?  I would love to get this working as I have almost a thousand servers storing syslog data on this server and need to partition the data so only certain people can see certain data.

Thanks in advance for any help.



__________________
ZG


Veteran Member

Status: Offline
Posts: 60
Date:
Permalink  
 

Hi Chad,

In Release 2.3 coming out in a couple of weeks (we're currently in QA), you will be able to do exactly this using DataGroups. DataGroups is a new feature which allows you to partition users to groups of data based on what hosts the data is on. The DataGroups data access model can also be nested, allowing users access permission to reflect hierarchical organisational structures. The hosts also do not need to have any Logscape components running on them.

Prior 2.3 release the method to do this would be:

1.Decide How to partition your data

It's common to partition your data based on environment, application or location. I'll use a familiar example where your data is located across prod,uat, and dev environments. Your hosts are distributed across these three environments.

 

2. Create data sources using the host filter.

Create data sources with a naming scheme that reflects the partition or the environment. e.g

  • syslog-uat
  • syslog-prod
  • syslog-dev

In this example we know we are receiving syslog data, so we can clone the syslog data sources and rename it to something like syslog-uat.  In the syslog-uat datasource update the host filter to reflect all the  hosts that belong in your UAT environment. Repeat this procedure for the other partitions , dev and prod. 

The host filter accepts a comma separated list of hosts substrings or regex expressions. This avoids the problem of explicitly listing all hosts which would be impractical in larger environments. 

By the end of step two, you should have the syslog-uat, syslog-prod and syslog-dev sources set up with their respective host lists. 

IMPORTANT: After you have done this delete the original syslog-server data source that came with Logscape. Data Sources can not overlap. 

 

3.  Setting up Data Groups in 2.2. 

When setting up a user, they can be assigned to a data group. A Data Group represents the data a user has access to. A DataGroup is composed of a list of data sources and/or other Data Groups.

Create a DataGroup for each partition. In the includes section provide the name of the data sources that you created in Step 2. Any user assigned to that Data Group will only see the data defined by the includes part of the form. 

 

 

4. Set up the user. 

Update the DataGroup part of the User details to the partition that the user has access to. 

 

5. When the user logs in and types in the search

 |  _tag.contains(syslog) 

They will only see data from the hosts defined by their data group. The list of hosts displayed in the facets will reflect this.

 

6. When the admin or sysadmin user searches:

They will see all data. To see data from a particular partition you could use a search like

| _tag.contains(uat) 

or 

| _tag.equals(syslog-uat)

 

These searches take advantage of the naming scheme chosen in step 2. 

 

 

 

 

 

 

 

 

 

 

 



__________________
PGM


Newbie

Status: Offline
Posts: 1
Date:
Permalink  
 

Hi ZG,

 

I did try this and it did work really goodbiggrin, but in my case it was a bit too complicated for the config we have.hmm

 

We ended up creating a Data Group that excludes the rest of Data Sources by tag.

 

 



__________________


Member

Status: Offline
Posts: 10
Date:
Permalink  
 

I can't seem to figure out how to tag data that is coming off a non-master indexer.  Can you point me to the some documentation?  

ALSO

   6. dev,SysLogServer (SysLog UDP/514 & TCP/1468 AND Indexer)   - already run on Management node)  <-- two typos ... port 1514 and there is a comma ... I wasted about an hour on the non standard port typo... wink oh well. 

 

 



__________________
ZG


Veteran Member

Status: Offline
Posts: 60
Date:
Permalink  
 

Hi Chad,

Navigate to Configure/Datasource page, click browse, select the host, then navigate to the folder you are interested in, then click OK.
Update the file filter before saving ( default *.log)

This will pull in data even if it located on a Forwarder.

More information can be found here  http://logscape.github.io/ds-add.html



__________________


Member

Status: Offline
Posts: 10
Date:
Permalink  
 

Got it... all works now except one last issue. Most all my servers logs are labeled "UNKNOWN" for facility.  Seems like either syslog/rsyslog clients are not configured correctly OR the syslog4j server is not identifying the facility correctly... Any ideas ??? I have about 1k servers flowing in and this is about the only thing left to fix.

btw,  I am forwarding from syslog-ng (acting as a central collector).  The logs look as they should there but not when they get to the logscape syslog server.  Also, this is over udp.

Thanks in advance!

-Chad



-- Edited by chadpatt on Wednesday 9th of April 2014 03:57:00 AM

__________________


Senior Member

Status: Offline
Posts: 100
Date:
Permalink  
 

Hi Chad,
The facility showing as 'UNKNOWN' is a bug and is fixed in 2.3 which is due out at the end of the month.

Can you change the syslog sending to use TCP and see if that helps?

Regards Neil.

__________________


Member

Status: Offline
Posts: 10
Date:
Permalink  
 

Neil. 

Thanks for letting me know, I have tried udp, tcp, syslog-protocol, transforms from syslog-ng... nothing works.  The following kinda works "logger -p local0.info test" but the file was named mail.log.  So, I then went threw the whole stack of facilities and the only ones that kinda worked were local0 and uucp.  Uucp saved as user, at this point I figured it was a bug and finally gave up (about 2 days worth of work to conclude this... sigh).  

Might you have a patch or work around for this?  I have to give a presentation and every time I show logscape I am always asked about this.

Thanks,

Chad



__________________
ZG


Veteran Member

Status: Offline
Posts: 60
Date:
Permalink  
 

Hi Chad,

Could you email us on support AT logscape.com and we will email you a Syslog Server which should fix the problem.

Regards,

ZG.

__________________


Member

Status: Offline
Posts: 10
Date:
Permalink  
 

Awesome, that worked!  Thanks a ton.



__________________


Senior Member

Status: Offline
Posts: 100
Date:
Permalink  
 

Good stuff! - glad to hear it!
-Neil.

__________________
Page 1 of 1  sorted by
 
Quick Reply

Please log in to post quick replies.



Create your own FREE Forum
Report Abuse
Powered by ActiveBoard