Starting a new project with Adobe's CQ5...
I'm starting with the access log, as it is straight forward.
I've done field extractions before for another custom log type, worked great. Now, I can't seem to get any of my extractions appear in the Search.
Walkthrough:
- Created an index called adobe_cq5.
- Created various "file" type data inputs using various CQ5 log files, setting the index to adobe_cq5. The one I started with is called cq5-access.
- Go into Search and verify that I'm getting good data, which I am.
- Select the drop down next to the first log line and click Extract Fields.
- Under Generated Pattern, click Edit.
- Put in a basic regex:
^(?P<FIELDNAME>\d+\.\d+\.\d+\.\d+?) - Click "Apply"
- Check several lines to make sure the IP addresses are selected.
- Click "Save"
- In "Save Field Extraction" enter ip_address for the field name.
- Click "Save"
- Click "Close" on "Successfully Saved" dialog.
- Reload the Search page.
- Note that ip_address is not appearing in the log line list as it has for past projects.
- Click "Pick fields"
- Note that ip_address does not appear in Available Fields.
- Go back to Extract Fields
- Enter in:
^(?P<FIELDNAME>\d+\.\d+\.\d+\.\d+?) - Get two errors:
- --Note: the values you want may already be extracted in the 'ip_address' field.
- --Note: This regex already extracts ip_address for cq5-access.
- Close out of Extract Fields
- Browse to Manager » Fields » Field extractions
- Verify extraction: cq5-access : EXTRACT-ip_address
- Click Permissions give Everyone Read Permission and set Object should appear in This app only (search)
- Click Save
- Re-Check the search page, ip_address still does not appear.
- Open up terminal
- cat: /opt/splunk/etc/apps/search/local/props.conf
- Verify extraction: EXTRACT-ip_address =
^(?P<ip_address>\d+\.\d+\.\d+\.\d+?)
For my last project, I simply entered the Extract Fields tool, entered my regex, saved and the data appeared right in the Search.
props.conf for modified extraction
[cq5-access]
EXTRACT-ip_address = ^(?P<ip_address>\d+\.\d+\.\d+\.\d+?)
props.conf with original full extraction
[cq5-access]
EXTRACT-ip_address-username-day-month-year-hour-minute-second-http_type-http_request-http_code-referer-user_agent = ^(?P<ip_address>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s.+?\s(?P<username>.+?)\s(?P<day>\d\d)/(?P<month>\w\w\w)/(?P<year>\d\d\d\d):(?P<hour>\d\d):(?P<minute>\d\d):(?P<second>\d\d)\s.+?\s"(?P<http_type>\w+?)\s(?P<http_request>.+?)\sHTTP.+?"\s(?<http_code>\d+?)\s.+?\s"(?P<referer>.+?)"\s"(?P<user_agent>.+?)"
Sample data:
10.71.40.57 - admin 23/Apr/2013:16:15:14 -0400 "GET /crx/server/crx.default/jcr%3aroot/etc/map/http.1.json?_dc=1366748119022&node=xnode-339 HTTP/1.1" 200 175 "https://twcc-ci01.lab.webapps.rr.com:4602/crx/de/index.jsp" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0"
10.71.40.57 - admin 23/Apr/2013:16:15:13 -0400 "GET /crx/de/icons/crxde_favicon.ico HTTP/1.1" 200 295606 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0"
127.0.0.1 - admin 23/Apr/2013:16:42:31 -0400 "GET /bin/receive?sling:authRequestLogin=1 HTTP/1.1" 200 32 "-" "Jakarta Commons-HttpClient/3.1"