The foundation of log management
Optimizing SIEM
syslog-ng is the log management solution that improves the performance of your SIEM solution by reducing the amount and improving the quality of data feeding your SIEM.
Rapid search and troubleshooting
Meeting compliance requirements
Big data ingestion
syslog-ng can deliver data from a wide variety of sources to Hadoop, Elasticsearch, MongoDB, and Kafka as well as many others.
Universal log collection and routing
Secure data archive
syslog-ng Premium Edition
Optimize your SIEM with syslog-ng
Enterprise class log management software
Whether it’s user activity, performance metrics, network traffic or any other log data, syslog-ng can collect and centralize log data. You can remove data silos and gain full-stack visibility of your IT environment. Depending on its configuration, one syslog-ng server can collect more than half a million log message per second from thousands of log sources.
Key features
Secure transfer and storage
Have confidence in the data underlying your analytics, forensics and compliance efforts
Using local disk buffering, client-side failover and application layer acknowledgement syslog-ng can transfer logs with zero message loss. Encrypted transfer and storage ensure logs cannot be tampered with, preserving the digital chain of custody.
Reliable log transfer
syslog-ng Premium Edition can send and receive log messages in a reliable way over the TCP transport layer using the Reliable Log Transfer Protocol™ (RLTP™).
RLTP™ is a new transport protocol that prevents message loss during connection breaks.
Secure Transfer using TLS
Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng Premium Edition uses the Transport Layer Security (TLS) protocol to encrypt the communication.
TLS also allows the mutual authentication of the host and the server using X.509 certificates.
Secure, Encrypted Log Storage
syslog-ng Premium Edition can store log messages securely in encrypted, compressed, indexed and timestamped binary files, so any sensitive data is available only for authorized personnel who have the appropriate encryption key.
Timestamps can be requested from external timestamping authorities.
Scalable architecture
Scale up your log management
Depending on its configuration, one syslog-ng server can collect more than half a million log message per second from thousands of log sources.
A single central server can collect log messages from more than 5,000 log source hosts. When deployed in a client relay configuration, a single syslog-ng log server can collect logs from tens of thousands of log sources.
Extreme message rate collection
The syslog-ng application is optimized for performance, and can handle an enormous amount of messages.
Depending on its exact configuration, it can process over half a million messages per second in real-time, and over 24 GB of raw logs per hour on standard server hardware.
Secure Transfer using TLS
Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng Premium Edition uses the Transport Layer Security (TLS) protocol to encrypt the communication.
TLS also allows the mutual authentication of the host and the server using X.509 certificates.
Easy monitoring
syslog-ng allows you to granularly select which statistics of syslog-ng you want to monitor. The statistics are available as structured name-value pairs, so you can format the output similarly to other log messages.
That way, you can easily convert the statistics and metrics and send the results into your enterprise monitoring solution (for example, IBM Tivoli Netcool, Riemann, Redis, or Graphite).
Scaling to large networks with syslog-ng
This short video will show you how syslog-ng scales to the largest IT environments, ensuring your log infrastructure can reliably and securely collect and manage log data.
Flexible log routing
Reduce maintenance and deployment costs with universal collection
syslog-ng can be deployed as an agent on a wide variety of hosts and flexibly route logs to multiple analytic tools or databases, eliminating the need to deploy multiple agents on servers.
Tested binary files for the syslog-ng Premium Edition are available for more than 50 server platforms, reducing the time required for installation and maintenance.
Collect from a wide variety of sources, including Windows
syslog-ng Premium Edition can natively collect and process log messages from SQL databases, enabling users to easily manage log messages from a wide variety of enterprise software and custom applications.
The syslog-ng Agent for Windows is an event log collector and forwarder application for Microsoft Windows platforms.
Read log messages from any text file
Some applications use many different log files, and sometimes these files are not even located in the same folder. Automatically generated file and folder names are also often a problem.
To solve these issues, the filenames and paths specifying the log files read by syslog-ng can include wildcards, and syslog-ng can automatically scan entire subfolder trees for the specified files.
The syslog-ng Premium Edition application is also able to process multi-line log messages, for example, Apache Tomcat messages.
Forward to multiple destinations
Many large organizations need to send their logs to multiple log analysis tools. Different groups, including IT operations, IT security and corporate risk and governance, need access to the same log data but have different log analysis goals and tools.
The syslog-ng application can send logs directly to SQL databases, MongoDB and Hadoop Distributed File System (HDFS) nodes, or use the Standard Network Management Protocol (SNMP) and Simple Mail Transfer Protocol (SMTP) for other destinations.
Real time transformation
Optimize your tools with distributed processing
With powerful filtering, parsing, re-writing and classification options, syslog-ng can transform logs on remote hosts, reducing the amount and complexity of log data forwarded to analytic tools like SIEM or APM, reducing their total cost of ownership.
The flexible configuration language allows users to construct powerful, complex log processing systems on remote hosts with simple rules.
Filter, parse, re-write
syslog-ng can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. Directories, files, and database tables can be created dynamically using macros.
Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.
Real time classification
By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can then be used to classify the type of the event described in the log message.
The message classes can be customized, and for example can label the messages as user login, application crash, file transfer, etc. events.
Enrich
syslog-ng can use an external database file to append custom name-value pairs to incoming logs, thus extending, enriching, and complementing the data found in the log message.
You can also correlate and aggregate information from log messages using a few simple filters that are similar to SQL GROUPBY statements.
Additional Features
Disk-based buffering
syslog-ng stores messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is re-established, in the same order the messages were received. The disk buffer is persistent – no messages are lost even if syslog-ng is restarted.
Flow control
Flow control uses a control window to determine if there is free space in the output buffer of syslog-ng for new messages. If the output buffer is full and the destination cannot accept new messages for some reason, for example it’s overloaded or the network connection has become unavailable. In such cases, syslog-ng stops reading messages from the source until some messages have been successfully sent to the destination.
Real time classification
By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and, for example, can label the messages as user login, application crash, file transfer, etc.
Python log parser
The Python log parser allows you to write your own parsers in Python. Practically, that way you can process the log message (or parts of the log message) any way you need. You can also write your own template functions in Python.
Normalize with PatternDB
syslog-ng can compare the contents of the log messages to a database of predefined message patterns.
Read and parse SNMP traps
syslog-ng PE can read these traps from a log file, and extract their content into name-value pairs, making it easy to forward them as a structured log message (for example, in JSON format).
Extract important information
In addition to classifying messages, you can also add different tags which can be used later for filtering messages. For example, to collect messages tagged as user_login to a separate file or to perform conditional post processing on the tagged messages.
Real time event correlation
syslog-ng also makes real time event correlation possible. This can be useful in many different situations, for example important data for a single event is often scattered into multiple syslog messages. Also login and logout events are often logged far away from each other, even in different log files, making log analysis difficult. Using correlation these can be collected into a single new message.
syslog-ng Store Box
syslog-ng Store Box, a turnkey appliance to manage your log data
High performance, enterprise-class log management appliance
The syslog-ng Store Box™ (SSB) is a high-performance, high-reliability log management appliance that builds on the strengths of syslog-ng Premium Edition. With SSB, you can search logs, secure sensitive information with granular access policies, generate reports to demonstrate compliance and forward log data to third-party analysis tools.
Key features
Collect and index
The syslog-ng Store Box’s indexing engine is optimized for performance. Depending on its exact configuration, one syslog-ng Store Box can collect and index up to 100,000 messages per second for sustained periods.
When deployed in a client-relay configuration, a single SSB can collect logs from tens of thousands of log sources
Flexible collection
Every installation of SSB comes with the possibility of using syslog-ng Premium Edition as log collection agents or relay servers at no additional cost.
Installers are available for 50+ platforms, including the most popular Linux distributions, commercial flavors of UNIX and Windows.
Scalable indexing
The syslog-ng Store Box is optimized for performance, and can handle enormous amounts of messages.
Depending on its configuration, it can index over 100,000 messages per second for sustained periods and process over 70 GB of raw logs per hour.
Real time processing
SSB can sort the incoming logs based on their content and various parameters. Directories, files and database tables can be created dynamically using macros.
Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important messages to the selected destinations.
Search and report
With full-text search, you can search through billions of logs in seconds via the web-based user interface. Wildcards and boolean operators allow you to perform complex searches and drill down on the results.
Users can easily create customized reports from the charts and statistics they create on the search interface to demonstrate compliance with standards and regulations such as PCI-DSS, ISO 27001, SOX and HIPAA
Web-based UI
SSB has an intuitive web-based user interface for configuring, searching, drilling down and generating reports. It’s easy to get an overview and quickly identify problems.
This user interface is exclusive to SSB and is not available separately for syslog-ng Premium Edition, which remains a purely command line interface solution.
Content-based alerting
SSB offers an automatic search functionality for quicker detection of anomalies: it is able to perform continuous search on the incoming log data and send alerts when predefined critical events are detected.
The alerts are actionable, so the detailed investigation of the corresponding logs can immediately and easily be started.
Federated search
SSB collects and indexes logs in virtual containers called logspaces that enable organizations to segment their log data based on any number of criteria and restrict access to logs based on user profiles.
With the federated search feature, you can search in multiple logspaces whether on the same SSB appliance or located on a different appliance even at a remote location.
Store and forward
You can store large amounts of log data, create automated retention policies, and backup data to remote servers.
The largest appliance can store up to 10 terabytes of uncompressed data.
You can also forward logs to 3rd party analysis tools or fetch data from syslog-ng Store Box via its REST API.
Automated backup
SSB provides automated data archiving to remote servers. The data on the remote server remains accessible and searchable.
SSB uses the remote server as a network drive via the Network File System (NFS) or the Server Message Block (SMB/CIFS) protocol.
REST API
SSB can forward logs to 3rd party analysis tools or fetch data from SSB via its REST API.
You can access the API using a RESTful protocol over HTTPS, meaning that you can use any programming language that has access to a RESTful HTTPS client to integrate SSB into your environment, including popular languages such as Java and Python.
Secure log data
Log data frequently contains sensitive information. SSB can store log data in encrypted, compressed, and time-stamped binary files restricting access to authorized personnel only.
Authentication, Authorization and Accounting settings can restrict access to the SSB configuration and stored logs based on usergroup privileges and can be integrated with LDAP and Radius databases.
Granular access control
Authentication, authorization and accounting settings provide granular access control restricting access to the SSB configuration and stored logs based on usergroup privileges.
SSB can be integrated with LDAP and Radius databases.
Encrypted log store
SSB’s logstore stores log data in encrypted, compressed, and timestamped binary files, restricting access to authorized personnel only.
The largest SSB appliance can store up to 10 terabytes of uncompressed data.
Secure transfer
syslog-ng Premium Edition ensures that messages cannot be accessed by third parties by using the Transport Layer Security (TLS) protocol to encrypt the communication between the agents and syslog-ng Store Box.
It is possible to use one-way or mutual authentication between clients and the server using X.509 certificates.
Additional Features
Parse key-value pairs
syslog-ng Store Box can separate a message consisting of whitespace or comma-separated key-value pairs (for example firewall logs) into name-value pairs.
Normalize with PatternDB
The syslog-ng application can compare the contents of the log messages to a database of predefined message patterns.
Real time classification
By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and, for example, can label the messages as user login, application crash, file transfer, etc.
Message Rate Alerting
SSB can be configured to send alerts based on the number of messages being received from sources. Minimum and maximum log message thresholds for specified time periods can be set to monitor the log management infrastructure for any performance issues.
Parse sudo log messages
Privileged user accounts represent the highest security risk, as they allow access to the most sensitive data and resources. The sudo parser enables you to enrich your log message data with details of privilege escalation events.
Extract important information
In addition to classifying messages, you can also add different tags which can be used later for filtering messages, for example, to collect messages tagged as user_login to a separate file or to perform conditional post processing on the tagged messages.
Real time event correlation
syslog-ng also makes real-time event correlation possible. This can be useful in many different situations. For example, important data for a single event is often scattered into multiple syslog messages. Also, login and logout events are often logged far away from each other, even in different log files, making log analysis difficult. Using correlation these can be collected into a single new message.
Cloud-ready
You can run your virtual SSB instances both in Amazon Web Services and in Microsoft Azure.