Maybe I am missing something (new user of logscape)...
So, I have "cloned" the syslog-server datasource to a "unix" as a test. I have reindexed the data source but there simply are no hits on the search. When I search with
* | _tag.equals(syslog-server) I get tons of data but if I use
* | _tag.equals(unix) I get 0 results.
What am I doing wrong? I would love to get this working as I have almost a thousand servers storing syslog data on this server and need to partition the data so only certain people can see certain data.
In Release 2.3 coming out in a couple of weeks (we're currently in QA), you will be able to do exactly this using DataGroups. DataGroups is a new feature which allows you to partition users to groups of data based on what hosts the data is on. The DataGroups data access model can also be nested, allowing users access permission to reflect hierarchical organisational structures. The hosts also do not need to have any Logscape components running on them.
Prior 2.3 release the method to do this would be:
1.Decide How to partition your data
It's common to partition your data based on environment, application or location. I'll use a familiar example where your data is located across prod,uat, and dev environments. Your hosts are distributed across these three environments.
2. Create data sources using the host filter.
Create data sources with a naming scheme that reflects the partition or the environment. e.g
syslog-uat
syslog-prod
syslog-dev
In this example we know we are receiving syslog data, so we can clone the syslog data sources and rename it to something like syslog-uat. In the syslog-uat datasource update the host filter to reflect all the hosts that belong in your UAT environment. Repeat this procedure for the other partitions , dev and prod.
The host filter accepts a comma separated list of hosts substrings or regex expressions. This avoids the problem of explicitly listing all hosts which would be impractical in larger environments.
By the end of step two, you should have the syslog-uat, syslog-prod and syslog-dev sources set up with their respective host lists.
IMPORTANT: After you have done this delete the original syslog-server data source that came with Logscape. Data Sources can not overlap.
3. Setting up Data Groups in 2.2.
When setting up a user, they can be assigned to a data group. A Data Group represents the data a user has access to. A DataGroup is composed of a list of data sources and/or other Data Groups.
Create a DataGroup for each partition. In the includes section provide the name of the data sources that you created in Step 2. Any user assigned to that Data Group will only see the data defined by the includes part of the form.
4. Set up the user.
Update the DataGroup part of the User details to the partition that the user has access to.
5. When the user logs in and types in the search
| _tag.contains(syslog)
They will only see data from the hosts defined by their data group. The list of hosts displayed in the facets will reflect this.
6. When the admin or sysadmin user searches:
They will see all data. To see data from a particular partition you could use a search like
| _tag.contains(uat)
or
| _tag.equals(syslog-uat)
These searches take advantage of the naming scheme chosen in step 2.
I can't seem to figure out how to tag data that is coming off a non-master indexer. Can you point me to the some documentation?
ALSO
6. dev,SysLogServer (SysLog UDP/514 & TCP/1468 AND Indexer) - already run on Management node) <-- two typos ... port 1514 and there is a comma ... I wasted about an hour on the non standard port typo... oh well.
Navigate to Configure/Datasource page, click browse, select the host, then navigate to the folder you are interested in, then click OK. Update the file filter before saving ( default *.log)
This will pull in data even if it located on a Forwarder.
Got it... all works now except one last issue. Most all my servers logs are labeled "UNKNOWN" for facility. Seems like either syslog/rsyslog clients are not configured correctly OR the syslog4j server is not identifying the facility correctly... Any ideas ??? I have about 1k servers flowing in and this is about the only thing left to fix.
btw, I am forwarding from syslog-ng (acting as a central collector). The logs look as they should there but not when they get to the logscape syslog server. Also, this is over udp.
Thanks in advance!
-Chad
-- Edited by chadpatt on Wednesday 9th of April 2014 03:57:00 AM
Thanks for letting me know, I have tried udp, tcp, syslog-protocol, transforms from syslog-ng... nothing works. The following kinda works "logger -p local0.info test" but the file was named mail.log. So, I then went threw the whole stack of facilities and the only ones that kinda worked were local0 and uucp. Uucp saved as user, at this point I figured it was a bug and finally gave up (about 2 days worth of work to conclude this... sigh).
Might you have a patch or work around for this? I have to give a presentation and every time I show logscape I am always asked about this.