This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter’s approach.
“Life is really simple, but we insist on making it complicated.” The immortal words ofConfuciusresonate with anyone who has ever tried to glean useful information from log data.
There are consensus-drivendefinitionsof what exactly log analysis is, but a simplified, accessible explanation might be:to organize log entries into a human-friendly display and make business decisions based on what you learn.
1GB of log data is the equivalent ofnearly 700,000 pages of text.Sifting through this manually would take nearly three years, or about the same amount of time it would take to readWar and Peace571 times consecutively (debatable as to which fate is worse). And if you consider that at least a few gigs of log data are produced per day even insmall organizations, well… cue the data-parsing apocalypse.
Suffice to say, attempting to make sense of all your logs without some kind of automated assistance will lead to information overload, wasted time and serious staff demoralization.
Investing in a dedicated log analysis platform, be it an in-house build or 3rd-party software, should be seriously considered by the C-level powers.
So the solution appears simple: just find or build an automated analysis platform, right?
Just be careful. Adoption of the wrong platform might just exacerbate the issue. Non-intuitive visualizations and statistical views can lead to hours of screen staring and head scratching.
Log analysis platforms should solve the issue of information overload by breaking down the mass of log entries into digestible, relevant groups, andignoring non-relevant data. Entries might be grouped according to common characteristics or patterns, e.g. the action performed or users performing them. Keep the following functionalities in mind when building your analysis platform or choosing an out-of-the-box solution:
* Broad collection –Make sure that all the log types you need are supported, including logs that originate from different operating systems, device types, and languages.
* Aggregation –Log entries are structured/written in a wide variety of ways, depending on their point of origin. Aggregation helps clear up this chaos by correlating similar logs together based on their shared patterns or characteristics, e.g. the action performed or users performing them. An aggregation feature should have some level of customization, such as saving logs to user-defined groups for ongoing reference.
* Search –A useful search function goes beyond a mere text box that returns matching log data. It’s important that queries return not only search-specific data results, but also allow users to easily access thecontextof those results, i.e. what occurred immediately before and after in the log.
* Alerts– A system should alert users to repeated error occurrences, system anomalies, or the absence of typical events. Make sure alert settings can be adjusted to different delivery methods, time intervals and priority levels.
* Automation– In most log analysis systems, the above features are dependent on manual operation and programming by users. But the growing scale and complexity of log data has pushed the most advanced analysis platforms to embrace machine learning. A productive platform should have the ability to:
- Intelligently learn a business’s routine log flows
- Use that intelligence to automatically catch anomalies and errors as they occur in real time
- Generate alerts independently
- Provide a blueprint for devs to resolve identified issues
- Present data trends in intuitive visualizations
Technologies may change dramatically from year to year, but time still equals money. Less time spent manually hunting bugs and system events leads to earlier and more rapid releases, more time for innovative feature development, and happier users. Those are the essential benefits from the DevOps perspective, but log analysis is a diverse art, and has a surprisingly broad range of uses.
Who does it help beyond DevOps and IT?
A flexible log analysis system could conceivably become the singular source of analytics for an organization, across departments. Here are three (among many) examples:
- Security Engineerscanexploit an Analysis platform’s search toolsto find suspicious actions or breaches. The ideal platform should save search queries for repeated use, and automate the searches for a regular interval of security checks.
- Compliance– Log analysis can help keep your product in line with arange of compliance laws, and a strong automation component can speed up the process of auditing and review.
- Marketing and UX– Tools like Google Analytics and HubSpot might generally be considered irreplaceable for usage and engagement analytics. However, log analysis provides thesame essential data: user traffic, referrals, usage time, click rates – they’re all in the log.
Though some departments will be resistant to experimenting with what they perceive as a dev-only tool, the C-suite’s collective ears might perk up at the mention of paying for just one multi-purpose analytics platform. This cuts down operational costs associated with buying a litany of analytics softwares for different teams.
To build or buy?
市场充斥着外-the-box log analysis systems, varying in levels of price, quality and versatility. Be mindful of the level of technical support offered by each, particularly if you plan to use the platform for departments without a dev background.
For those larger organizations that have the time, human capital and financial resources necessary to build in-house, this approach allows formade-to-measure customizationfor your particular industry and your business’s tech architecture. If this is your first time building in-house software, ponder this statistic during your cost-benefit analysis: the industry of data analytics enterprise software will reachnearly $200 billion in sales volume在接下来的两年。这是一代的象征eral (though not exclusive) preference toward commercial software as opposed to in-house.
Not everyone in your organization has prior familiarity with log analysis in and of itself, but they’re already familiar with its essential goal. In the world of big data, log analysis is merely another reflection of auniversal business needtoday -- that is, to take mass amounts of information our technical environments generate and distill it to the essential info you need in order to constantly improve your product.
As we opened with a quote from a scholarly icon of Eastern philosophy, I’ll close with one from a more mainstream emblem of the East: “It is not a daily increase, but a daily decrease. Hack away at the inessentials.” – Bruce Lee
To learn more visitCoralogix.