DeepX Professional Services team offers configuration and consulting around Elastic Search stack, in particular Kibana visualizations and Kibana Machine Learning modules.
We can help you set up ELK from scratch and/or configure the Kibana Machine learning tools to provide you with meaningful and easy to grasp insights into your data.
Listed below are some of the tools that we use when harnessing the power of Kibana Machine Learning capabilites.
Kibana Machine Learning – Data Visualizer
Data Visualizer tool would be the first stop after the indexes and data ingestion have been configured correctly.
Data Visualizer enables the operator to carry out a high-level inspection of the data.
This allows us to identify areas which require more in-depth analysis using both ML and non-ML tools
Kibana Machine Learning Data Visualizer enables quick inspection of data distribution by individual parameters.
This is useful, for example, when you need to quickly identify outliers such as:
- certain IP address doing too many requests
- certain unusual error code or parameter showing up (you may identify this by looking at keywords and codes producing lowest count – these are the ones you won’t normally notice by looking at raw logs data)
- unusual country ISO codes
- ports, events, requests and error codes whose statistical distribution is unusual
After having looked at the overall picture in Kibana Data Visualizer you may want to look at raw data or create custom visualizations looking closer at the sources of suspicious data.
Kibana Machine Learning Jobs Management
The Jobs Management interface is an important dashboard or control panel interface effectively providing the Kibana ML operator with the full control over what jobs are running, how many records are being processed, ability to start and stop jobs, ability to clone, re-configure and annotate jobs etc.
This is especially important as there may be multiple Machine Learning jobs running in parallel processing billions of data records, generating significant load over your Machine Learning nodes. While ELK Machine Learning is fully capable of dealing with high load and parallel processing, you do want to keep your cluster busy only with relevant tasks, and to generally have a full overview and control of the Kibana Machine Learning jobs.
Once any particular job has processed enough records you may proceed to review the results via Single Metric Viewer or Anomaly Explorer.
Kibana Machine Learning – Anomaly Detection
Anomaly detection is the crown jewel of the Kibana Machine Learning tools.
Simply put, it applies Machine Learning models to detect outliers in the data which might need your attention.
The underlying indexes and the Kibana ML job shall be configured correctly however, to make sure Anomaly Detection is looking at the right data set.
There are a number of templates to pick from to make your life easier if you work with typical data such, for example, as Winlog (Windows logs), Apache logs, Syslog etc – the existing pre-configured template jobs work great and prepare the optimal ML job configuration for typical SIEM purposes etc.
When creating your own custom job it’s important to get settings right to ensure you correctly configure the job to make use of ML models, select the analyzed parameters and correctly assign the “influencers” in order to have Kibana Machine Learning do its anomaly detection work for you highlighting the data sources which are relevant to you.
Kibana ML Anomaly Detection provides a great visual analysis interface enabling the operator to review time-series data with critical and warning level highlights mapped over time in red, amber, blue etc colour coding and annotated in human readable format in the Anomaly Exporer section such, for example, as “50x higher”, “unexpected value” etc.
The operator can then easily dive in into the specific data source to learn more about the issue causing the anomaly by selecting “View series” leading to Single Metric Viewer interface or “View examples” leading to raw data review interface.
Kibana Machine Learning Anomaly Detection when used correctly therefore is an amazing tool and in our belief shall be the entry point to your discovery and analysis of your data, from which you cascade into more detailed research where required provided by ELK stack.
Kibana Machine Learning – Forecasting
Forecasting is another useful functionality of Kibana ML toolset.
This feature uses Machine Learning models to predict future data patterns effectively allowing us to estimate parameters such as load and vulnerability threats.
For the Forecasting feature to generate accurate results it should obviously be configured correctly and have enough historical data for its ML models to work with.
