In times of big data, predictive analytics and business intelligence, it is becoming increasingly clear how valuable it can be not just to store the large amounts of data that are often collected and stored automatically, but to generate added value from it through analysis. Big data buzzword bingo at punkt.de
Ich liebe es wenn ein Plan funktioniert!
Daniel Lienert
Daniel ist immer auf der Suche nach technologisch innovativen aber dennoch nachhaltig stablilen Lösungen für unsere Kunden.
On many websites, every click, every login, every transaction and every request is now stored. But user profiles, comments, likes, shares and the like also find their way into databases. However, this collected data is often only evaluated by marketing teams using Google Analytics or used by development teams for log file analysis. The information that is really interesting for companies is often not analyzed. All that stands between the collection and understanding of data is its visual presentation.
As it is impossible for the human mind to understand large data sets simply by looking at tabular lists, they must be presented in some other way. In classic data visualization, a whole series of processing steps are identified that are required to present data in an unadulterated and understandable way. Raw data must undergo data analysis, filtering, mapping and rendering before a displayable graphic is created.
This can be solved with data visualization tools such as Kibana. Kibana is a graphical front-end that was developed specifically for displaying data from Elasticsearch. This frontend receives the data from Elasticsearch and offers the user the option of filtering the results as desired. The result is a dynamic, interactive and appealing presentation of the data in real time. The data, which is stored in the document-based structure of Elasticsearch, can be examined exploratively as well as compiled into dashboards in visualizations such as pie charts and bar charts. Trends and relationships can then be revealed here.
As part of my project work on data visualization, I created a proof of concept using Elasticsearch and Kibana. To do this, the data had to be transferred from the original MySQL database to Elasticsearch using a database connector, where the data aggregation takes place. From there, it is automatically transferred to Kibana when the user submits a query in the frontend. This can happen as an explicit search query or as an interaction with the visualizations. The visualizations of a dashboard are then immediately adapted accordingly.
Traditionally, Elasticsearch and Kibana are used as log analysis stacks, especially in conjunction with log shippers such as Logstash or Heka. Access data, HTML responses and error messages are then displayed there. However, there are also many other potential use cases. For example, it is perfectly possible to display business transactions such as sales for the sales department on a map so that research can be carried out into what sells particularly well and where. Forum posts from an online community can just as easily be analyzed to reveal trending topics.
Analyses with Elasticsearch and Kibana thus open up a multitude of possibilities to realize powerful and at the same time intuitively operable analyses of a wide variety of business data. With this solution, we enable our customers to gain even more value from their data.
Elasticsearch Workshop
Our workshop repertoire includes a more general Elasticsearch workshop and one for sophisticated server monitoring.
Thanks to our experience with the tools, we can also put together a workshop tailored to your needs.
Old certainties are crumbling, new questions are emerging. Who actually decides on technology? How independent are we really? And what role does open source still play? We share our observations from a year that has reorganized many things.
Working at punkt.de - getting started and the first few months
I've been at punkt.de for four months now and in that time I've already taken part in the team days, the Just Do It Day, as a helping hand at a conference and experienced a project going live.
