Splunk HTM add-on


#1

Continuing the discussion from Introduce yourself!:

@azanians reminded me in the thread linked above about Splunk! I’ve always thought that HTM would be very fitting for a Splunk add-on service, but we’ve never officially investigated it. Does anyone have Splunk experience and an interest in creating some type of log anomaly system on top of Splunk? I’ve used their services before, and I can tell you they have a great infrastructure that is really hackable and fun to write queries against.


#2

Hi @rhyolight, I wouldn’t mind being involved in the initiative to create an HTM Splunk add-on and perhaps App. I have three years experience of working and can develop Splunk Apps and add-on. I’m however not so clued up with HTM. All I know is a little on what I’ve read and nothing hands-on. So I guess I need to start applying myself a little more on gaining practical experience to I get to understand HTM better, and how to implement solutions based on the technology.

Is there some HTM for dummies/beginners video tutorials I can use to get started? I’d appreciate any guidance on where to get started…


#3

I would love to be involved @azanians, I have experience with splunk and HTM.

FYI: @rhyolight


#4

@azanians @Manpreet_Singh Do either of you have any experience with the Splunk add-on licensing model? Are there restrictions on OS licenses on the Splunk side?


#5

I don’t @rhyolight but can find out


#6

Hi @rhyolight & @Manpreet_Singh ,

We would have to get the detail for licensing of addons from the Splunk vendor itself, but the short of it is that any one can develop an addon that they upload to Splunkbase for use by others for free, or for a fee. So the developer can decide on whether they are doing it just to contribute to the community, or for commercial objectives.

Here’s a link to Splunkbase https://splunkbase.splunk.com/


#7

Splunk-on Bros! :confetti_ball:


#8

Thanks @azanians.

Hi @rhyolight @azanians Let’s get started with this. Shall we open an issue to track this or can be tracked as separate feature.

I am planning to spend some time to put a spike together using htm.java for this. Lets connect @azanians. Thanks.


#9

Hi @Manpreet_Singh,

I’m all for it!
Please also check out the information here as I’d think we should aim to have the app certified by Splunk for credibility’s sake. -> http://dev.splunk.com/view/app-cert/SP-CAAAE2S

I suppose the other decision should be around whether we’re merely contributing to the community, or out to get paid. :wink:

On your question of “issue tracking”. I’ll need your guidance there. I’ve never developed an app through the community so I’m not so familiar with much of the jargon used. :slight_smile:


#10

Sound good @azanians i will check the certification part, thanks for sharing.

i think it will be more of contribution, but let @rhyolight guide us there. I love the other idea as well :slight_smile:

@rhyolight your thoughts on above and best way to get started ?


#11

@Manpreet_Singh @azanians

Unfortunately I can’t help with this as I’m barreled over in to-dos right now, but I’m always available for consultation if desired, just fyi! :slight_smile:


#12

np @cogmission, will definitely consult and get stuff reviewed.


#13

I think you’ll want to start your own repository for this work. And just to be clear, I was suggesting this as a community project. I can advise, but I don’t have time to work on it.

If you are “out to get paid” , you’ll need to eventually read our licensing guidelines.

I think you want to find anomalies in logs. Splunk has great tools to create queries that segregate different logs, and you can aggregate on log counts etc. This would be a good place to start. If there are a ton of logs, you might be able to identify anomalies simply on aggregated log counts.


#14

Thanks @rhyolight will do. Agree and will connect as needed.

I used ML Toolkit App from splunk available with latest version. Pretty cool stuff available to predict different fields, clustering etc.

Have good amount of splunk data to build anomaly detection using HTM. Will keep forum posted. Thanks


#15

Thanks for mentioning the Machine Learning Toolkit App @Manpreet_Singh. I didn’t know about it an now learned a littl from this link -> https://splunkbase.splunk.com/app/2982/

Other information links that may come handy on the Splunk For HTM App initiative:

Event Generator: https://splunkbase.splunk.com/app/1924/#/overview
https://github.com/splunk/eventgen

The Pluggable Auditing System (PAS) app - Splunk Reference App : https://splunkbase.splunk.com/app/1934/

Splunk 6.x Dashboard Examples: https://splunkbase.splunk.com/app/1603/

I hope that helps with some groundwork.

Kind regards,
J. Napo Mokoetle


#16

Hey guys, how did this project go?

I’m a Splunker and long-time HTM fan, and recently came up with a potentially interesting use case for combining the two together. Would love to work with whomever to build it out.