The foundation of log management

Optimizing SIEM

syslog-ng is the log management solution that improves the performance of your SIEM solution by reducing the amount and improving the quality of data feeding your SIEM.

Rapid search and troubleshooting

With syslog-ng Store Box, you can find the answer. Search billions of logs in seconds using full text queries with Boolean operators to pinpoint critical logs.

Meeting compliance requirements

syslog-ng Store Box provides secure, tamper-proof storage and custom reporting to demonstrate compliance.

Big data ingestion

syslog-ng can deliver data from a wide variety of sources to Hadoop, Elasticsearch, MongoDB, and Kafka as well as many others.

Universal log collection and routing

syslog-ng flexibly routes log data from X sources to Y destinations. Instead of deploying multiple agents on hosts, organizations can unify their log data collection and management.

Secure data archive

syslog-ng Store Box provides automated archiving, tamper-proof encrypted storage, granular access controls to protect log data. The largest appliance can store up to 10TB of raw logs.

syslog-ng Premium Edition

Optimize your SIEM with syslog-ng

Enterprise class log management software

Whether it’s user activity, performance metrics, network traffic or any other log data, syslog-ng can collect and centralize log data. You can remove data silos and gain full-stack visibility of your IT environment. Depending on its configuration, one syslog-ng server can collect more than half a million log message per second from thousands of log sources.

Key features

Secure transfer and storage

Have confidence in the data underlying your analytics, forensics and compliance efforts

Using local disk buffering, client-side failover and application layer acknowledgement syslog-ng can transfer logs with zero message loss. Encrypted transfer and storage ensure logs cannot be tampered with, preserving the digital chain of custody.

Reliable log transfer

syslog-ng Premium Edition can send and receive log messages in a reliable way over the TCP transport layer using the Reliable Log Transfer Protocol™ (RLTP™).

RLTP™ is a new transport protocol that prevents message loss during connection breaks.

Secure Transfer using TLS

Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng Premium Edition uses the Transport Layer Security (TLS) protocol to encrypt the communication.

TLS also allows the mutual authentication of the host and the server using X.509 certificates.

Secure, Encrypted Log Storage

syslog-ng Premium Edition can store log messages securely in encrypted, compressed, indexed and timestamped binary files, so any sensitive data is available only for authorized personnel who have the appropriate encryption key.

Timestamps can be requested from external timestamping authorities.

Scalable architecture

Scale up your log management

Depending on its configuration, one syslog-ng server can collect more than half a million log message per second from thousands of log sources.

A single central server can collect log messages from more than 5,000 log source hosts. When deployed in a client relay configuration, a single syslog-ng log server can collect logs from tens of thousands of log sources.

Extreme message rate collection

The syslog-ng application is optimized for performance, and can handle an enormous amount of messages.

Depending on its exact configuration, it can process over half a million messages per second in real-time, and over 24 GB of raw logs per hour on standard server hardware.

Secure Transfer using TLS

Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng Premium Edition uses the Transport Layer Security (TLS) protocol to encrypt the communication.

TLS also allows the mutual authentication of the host and the server using X.509 certificates.

Easy monitoring

syslog-ng allows you to granularly select which statistics of syslog-ng you want to monitor. The statistics are available as structured name-value pairs, so you can format the output similarly to other log messages.

That way, you can easily convert the statistics and metrics and send the results into your enterprise monitoring solution (for example, IBM Tivoli Netcool, Riemann, Redis, or Graphite).

Scaling to large networks with syslog-ng

This short video will show you how syslog-ng scales to the largest IT environments, ensuring your log infrastructure can reliably and securely collect and manage log data.

Flexible log routing

Reduce maintenance and deployment costs with universal collection

syslog-ng can be deployed as an agent on a wide variety of hosts and flexibly route logs to multiple analytic tools or databases, eliminating the need to deploy multiple agents on servers.

Tested binary files for the syslog-ng Premium Edition are available for more than 50 server platforms, reducing the time required for installation and maintenance.

Collect from a wide variety of sources, including Windows

syslog-ng Premium Edition can natively collect and process log messages from SQL databases, enabling users to easily manage log messages from a wide variety of enterprise software and custom applications.

The syslog-ng Agent for Windows is an event log collector and forwarder application for Microsoft Windows platforms.

Read log messages from any text file

Some applications use many different log files, and sometimes these files are not even located in the same folder. Automatically generated file and folder names are also often a problem.

To solve these issues, the filenames and paths specifying the log files read by syslog-ng can include wildcards, and syslog-ng can automatically scan entire subfolder trees for the specified files.

The syslog-ng Premium Edition application is also able to process multi-line log messages, for example, Apache Tomcat messages.

Forward to multiple destinations

Many large organizations need to send their logs to multiple log analysis tools. Different groups, including IT operations, IT security and corporate risk and governance, need access to the same log data but have different log analysis goals and tools.

The syslog-ng application can send logs directly to SQL databases, MongoDB and Hadoop Distributed File System (HDFS) nodes, or use the Standard Network Management Protocol (SNMP) and Simple Mail Transfer Protocol (SMTP) for other destinations.

Real time transformation

Optimize your tools with distributed processing

With powerful filtering, parsing, re-writing and classification options, syslog-ng can transform logs on remote hosts, reducing the amount and complexity of log data forwarded to analytic tools like SIEM or APM, reducing their total cost of ownership.

The flexible configuration language allows users to construct powerful, complex log processing systems on remote hosts with simple rules.

Filter, parse, re-write

syslog-ng can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. Directories, files, and database tables can be created dynamically using macros.

Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.

Real time classification

By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can then be used to classify the type of the event described in the log message.

The message classes can be customized, and for example can label the messages as user login, application crash, file transfer, etc. events.

Enrich

syslog-ng can use an external database file to append custom name-value pairs to incoming logs, thus extending, enriching, and complementing the data found in the log message.

You can also correlate and aggregate information from log messages using a few simple filters that are similar to SQL GROUPBY statements.

Additional Features

Disk-based buffering

syslog-ng stores messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is re-established, in the same order the messages were received. The disk buffer is persistent – no messages are lost even if syslog-ng is restarted.

Flow control

Flow control uses a control window to determine if there is free space in the output buffer of syslog-ng for new messages. If the output buffer is full and the destination cannot accept new messages for some reason, for example it’s overloaded or the network connection has become unavailable. In such cases, syslog-ng stops reading messages from the source until some messages have been successfully sent to the destination.

Real time classification

By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and, for example, can label the messages as user login, application crash, file transfer, etc.

Python log parser

The Python log parser allows you to write your own parsers in Python. Practically, that way you can process the log message (or parts of the log message) any way you need. You can also write your own template functions in Python.

Normalize with PatternDB

syslog-ng can compare the contents of the log messages to a database of predefined message patterns.

Read and parse SNMP traps

syslog-ng PE can read these traps from a log file, and extract their content into name-value pairs, making it easy to forward them as a structured log message (for example, in JSON format).

Extract important information

In addition to classifying messages, you can also add different tags which can be used later for filtering messages. For example, to collect messages tagged as user_login to a separate file or to perform conditional post processing on the tagged messages.

Real time event correlation

syslog-ng also makes real time event correlation possible. This can be useful in many different situations, for example important data for a single event is often scattered into multiple syslog messages. Also login and logout events are often logged far away from each other, even in different log files, making log analysis difficult. Using correlation these can be collected into a single new message.

syslog-ng Store Box

syslog-ng Store Box, a turnkey appliance to manage your log data

High performance, enterprise-class log management appliance

The syslog-ng Store Box™ (SSB) is a high-performance, high-reliability log management appliance that builds on the strengths of syslog-ng Premium Edition. With SSB, you can search logs, secure sensitive information with granular access policies, generate reports to demonstrate compliance and forward log data to third-party analysis tools.

Key features

Collect and index

The syslog-ng Store Box’s indexing engine is optimized for performance. Depending on its exact configuration, one syslog-ng Store Box can collect and index up to 100,000 messages per second for sustained periods.

When deployed in a client-relay configuration, a single SSB can collect logs from tens of thousands of log sources

Flexible collection

Every installation of SSB comes with the possibility of using syslog-ng Premium Edition as log collection agents or relay servers at no additional cost.

Installers are available for 50+ platforms, including the most popular Linux distributions, commercial flavors of UNIX and Windows.

Scalable indexing

The syslog-ng Store Box is optimized for performance, and can handle enormous amounts of messages.

Depending on its configuration, it can index over 100,000 messages per second for sustained periods and process over 70 GB of raw logs per hour.

Real time processing

SSB can sort the incoming logs based on their content and various parameters. Directories, files and database tables can be created dynamically using macros.

Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important messages to the selected destinations.

Search and report

With full-text search, you can search through billions of logs in seconds via the web-based user interface. Wildcards and boolean operators allow you to perform complex searches and drill down on the results.

Users can easily create customized reports from the charts and statistics they create on the search interface to demonstrate compliance with standards and regulations such as PCI-DSS, ISO 27001, SOX and HIPAA

Web-based UI

SSB has an intuitive web-based user interface for configuring, searching, drilling down and generating reports. It’s easy to get an overview and quickly identify problems.

This user interface is exclusive to SSB and is not available separately for syslog-ng Premium Edition, which remains a purely command line interface solution.

Content-based alerting

SSB offers an automatic search functionality for quicker detection of anomalies: it is able to perform continuous search on the incoming log data and send alerts when predefined critical events are detected.

The alerts are actionable, so the detailed investigation of the corresponding logs can immediately and easily be started.

Federated search

SSB collects and indexes logs in virtual containers called logspaces that enable organizations to segment their log data based on any number of criteria and restrict access to logs based on user profiles.

With the federated search feature, you can search in multiple logspaces whether on the same SSB appliance or located on a different appliance even at a remote location.

Store and forward

You can store large amounts of log data, create automated retention policies, and backup data to remote servers.

The largest appliance can store up to 10 terabytes of uncompressed data.

You can also forward logs to 3rd party analysis tools or fetch data from syslog-ng Store Box via its REST API.

Automated backup

SSB provides automated data archiving to remote servers. The data on the remote server remains accessible and searchable.

SSB uses the remote server as a network drive via the Network File System (NFS) or the Server Message Block (SMB/CIFS) protocol.

REST API

SSB can forward logs to 3rd party analysis tools or fetch data from SSB via its REST API.

You can access the API using a RESTful protocol over HTTPS, meaning that you can use any programming language that has access to a RESTful HTTPS client to integrate SSB into your environment, including popular languages such as Java and Python.

Secure log data

Log data frequently contains sensitive information. SSB can store log data in encrypted, compressed, and time-stamped binary files restricting access to authorized personnel only.

Authentication, Authorization and Accounting settings can restrict access to the SSB configuration and stored logs based on usergroup privileges and can be integrated with LDAP and Radius databases.

Granular access control

Authentication, authorization and accounting settings provide granular access control restricting access to the SSB configuration and stored logs based on usergroup privileges.

SSB can be integrated with LDAP and Radius databases.

Encrypted log store

SSB’s logstore stores log data in encrypted, compressed, and timestamped binary files, restricting access to authorized personnel only.

The largest SSB appliance can store up to 10 terabytes of uncompressed data.

Secure transfer

syslog-ng Premium Edition ensures that messages cannot be accessed by third parties by using the Transport Layer Security (TLS) protocol to encrypt the communication between the agents and syslog-ng Store Box.

It is possible to use one-way or mutual authentication between clients and the server using X.509 certificates.

Additional Features

Parse key-value pairs

syslog-ng Store Box can separate a message consisting of whitespace or comma-separated key-value pairs (for example firewall logs) into name-value pairs.

Normalize with PatternDB

The syslog-ng application can compare the contents of the log messages to a database of predefined message patterns.

Real time classification

By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and, for example, can label the messages as user login, application crash, file transfer, etc.

Message Rate Alerting

SSB can be configured to send alerts based on the number of messages being received from sources. Minimum and maximum log message thresholds for specified time periods can be set to monitor the log management infrastructure for any performance issues.

Parse sudo log messages

Privileged user accounts represent the highest security risk, as they allow access to the most sensitive data and resources. The sudo parser enables you to enrich your log message data with details of privilege escalation events.

Extract important information

In addition to classifying messages, you can also add different tags which can be used later for filtering messages, for example, to collect messages tagged as user_login to a separate file or to perform conditional post processing on the tagged messages.

Real time event correlation

syslog-ng also makes real-time event correlation possible. This can be useful in many different situations. For example, important data for a single event is often scattered into multiple syslog messages. Also, login and logout events are often logged far away from each other, even in different log files, making log analysis difficult. Using correlation these can be collected into a single new message.

Cloud-ready

You can run your virtual SSB instances both in Amazon Web Services and in Microsoft Azure.