> For the complete documentation index, see [llms.txt](https://docs.enginsight.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.enginsight.com/docs/manual/english/platform-usage/siem/data-lake.md). # Data Lake The Data Lake serves as a central data source that consolidates, indexes, and groups all raw data collected from various logs by collectors. *** :compass: Navigate to **SIEM** → **Data Lake** to view all logs sent to the SIEM. The data lake view is divided into two sections. On the [left side](#data-lake-log-fields-list), you can find a list of all logs whose data is forwarded to the SIEM, including the available log fields. You can search for and filter specific fields here. On the [right side](#data-lake-log-overview), you can find the actual log entries based on the selected time range and applied filters. You can also [export](#export-logs) the Data Lake overview, apply filters based on [predefined streams](#open-stream), and [save manually applied filters as streams](#save-stream).
*** ## Navigation :computer-mouse: Click the **Refresh** icon :arrows-rotate-reverse: above the log list to refresh all displayed log data based on the applied filters. :computer-mouse-scrollwheel: Scroll down the log list or adjust the number of results displayed per page ![](https://2423388680-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FV7OrHgKXA0t1jankQBYf%2Fuploads%2FfjKACCt2SKXcFd3S7Xb5%2FEnginsight-EN-Results%20per%20Page.jpg?alt=media\&token=e1650160-7a41-4965-aaca-6975db758dbe) to view additional logs. Use the page navigation ![](https://2423388680-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FV7OrHgKXA0t1jankQBYf%2Fuploads%2FxhUBkkR24YsCC9GSAa9D%2FEnginsight-EN-Results%20from.jpg?alt=media\&token=7f803007-9768-4377-a884-fd1b08f5cd27) to navigate between multiple pages. ### Defining Start and End Time :computer-mouse: Click :calendar-days: **Start Time** or :calendar-days: **End Time** in the upper-right corner to define the start or end of the period to be analyzed. The date picker opens. {% tabs %} {% tab title="Relative Time Period" %} Define a **Relative Time Period** manually. You can also select a relative time range from one of the available **Templates**.
:computer-mouse: Click the field with the default setting **Days Ago** to expand a list of all available time ranges. Select a time range. In the number field, enter the number of seconds, minutes, hours, days, or weeks you want to show. The default value is **1**. :computer-mouse: Click **Apply** to confirm your settings. {% endtab %} {% tab title="Absolute Time Period" %} Select a date from the calendar view.
:computer-mouse: Optionally, click :clock: **Time** to set an exact time in the 24-hour format using the clock face. First, select an hour. The clock face then switches to the minute view. Select the required minutes. :computer-mouse: Click **Apply** to confirm your settings. {% endtab %} {% endtabs %} :computer-mouse: Click :arrow-rotate-left: **Reset** in the upper-right corner to reset the settings for the selected time range. ### Full-Screen Mode :computer-mouse: Click the **Full Screen** icon :expand: above the log list on the right to switch to full-screen mode. When this mode is active, only the right side of the Data Lake view is shown, without the available filter options. :computer-mouse: Click the **Full Screen** icon :expand: again or press **ESC** to exit the full-screen mode. :computer-mouse: Click the **Refresh** icon :arrow-rotate-right: to refresh all shown log data based on the applied filters. ### Customizing the Log Table :computer-mouse: Click the **Table** icon :table-cells-columns: above the log list on the right to open a new window where you can customize the amount of information that is shown in each table row.
The following options are available:
OptionDescription
Row Height

Define whether the log information preview should show one line or all available log lines within a table row.

:computer-mouse: Click the desired option to select it.

The following options are available:

  • Single Line: The log information preview includes only one line.
  • Autofit: The height of a table row is adjusted to the length of the log. All available information is shown.
Lines per Row

Define the exact number of log lines that you want to show within a table row.

The default line number is 4.

:computer-mouse: Click and drag the slider :slider: until the required number of log lines is shown in the field next to it, or manually change the value in the field next to Lines per Row.

Shorten Field Label

Define how the names of fields should be displayed.

This feature is enabled by default.

When enabled, the displayed field name is shortened to the final group after the last period. When disabled, the complete field names are displayed.

:computer-mouse: Click the toggle :slider: to enable or disable the option.

:computer-mouse: Click **Reset** to reset all settings to their default values. :computer-mouse: Click **Close** to save the settings and close the window. *** ## Data Lake: Log List The left side of the data lake view contains a list of all logs that are forwarded to the SIEM. Each log contains a list of fields that you can use to filter the [data lake overview](#data-lake-log-overview) on the right more precisely.
### Searching for Fields :computer-mouse: Click the :magnifying-glass: **Search for Fields** field above the log list and enter a complete or partial field name to find a field that you want to use to filter the [data lake overview](#data-lake-log-fields-list) more quickly. By default, the log list then shows only the logs and associated fields that match the search term.
### Filter Query Settings :computer-mouse: Click the **Settings** icon :sliders: next to the [search field](#search-for-fields) above the log list to open a new window where you can configure filter query settings.
The following options are available:
OptionDescription
Filter-Based Search

Enable filter-based search to restrict the values listed for each field to those that match the filters currently applied in the data lake overview.

Values that do not match the applied filters are not shown in the listed field values.

This option is enabled by default.

Show Only Result Fields

Enable this option to suggest only fields in the free-text search that actually occur in the log entries in the data lake overview.

This option is enabled by default.

:computer-mouse: Click **Save** to apply the configured settings. :computer-mouse: Click **Close** to return to the data lake overview without saving. ### Field Favorites At the beginning of the log list, you can see various field names marked with a **star** icon :star:.
These are managed favorite fields that allow you to filter more quickly. ### Hiding and Showing Log Fields You can hide log fields from the list and show them again. Hidden log fields are located at the end of the log list under **Hidden Fields**. :arrow-pointer: Hover over a log name to show the **Hide** icon :eye-slash:. Click it to remove the log and all associated fields from the log list.
:computer-mouse: Navigate to the end of the log list and click the grayed-out **Hidden Fields** field to show all hidden log fields. :arrow-pointer: Hover over a log name to show the **Show** icon :eye:. Click it to show the log and all associated fields in the log list again.
### Filtering by Fields :computer-mouse: Click the **Expand** icon :angle-down: next to a log name to display all fields associated with the log. Click the **Collapse** icon :angle-up: to collapse all fields again.
You now have two ways to filter by specific fields. You can [show selected fields as table columns](#option-1-display-filtered-fields-as-table-columns) in the data lake overview. You can also use operators to [create granular field value filters](#option-2-create-field-value-filters). #### Option 1: Displaying Filtered Fields as Table Columns :arrow-pointer: Hover over a field name to display the **Add as Column** icon :table-layout:. Click it to add the field as a column in the data lake overview.
You can repeat this process with any number of additional fields to add more columns to the view.
If the selected field exists in a log entry, the corresponding field value is shown in the column.
:computer-mouse: Click a field value to open the [advanced filter settings](#advanced-filter-settings), where the selected value is already entered. Keep the existing settings or adjust further settings, then click **Add Filter** to filter the data lake overview by the corresponding field value.

---------------

If the selected field does not exist in a log entry, the following is shown:
:computer-mouse: Click **Add Field Filter** to show only log entries in which the selected field exists.

---------------

Fields that you have added to the [data lake overview](#data-lake-log-overview) as columns are listed below the search field under **Filtered Fields**.
:computer-mouse: Click the **Delete** icon :trash: next to a field name to remove the corresponding column from the log view. #### Option 2: Creating Field Value Filters :computer-mouse: Click a field that you want to use to filter the [data lake overview](#data-lake-log-overview) more precisely. A window opens where all values that occur in the corresponding field across all log entries are listed.
The window contains the following information and options:
OptionDescription
Field NameShows the complete name of the selected field.
:chart-column::computer-mouse: Click the Add as Chart Field icon :signal-bars: to add the field as a chart field.
:table-layout::computer-mouse: Click the Insert as Column icon :table-layout: to add the field as a table column in the log entries list.
:filter::computer-mouse: Click the Filter icon :filter: to open the advanced filter settings.
Filter-Based Search ActivatedShows whether Filter-Based Search is enabled or disabled in the filter query settings.
Start and End TimeShows the start and end time of the displayed log entries as configured by you.
:plus::computer-mouse: Click the Plus icon :plus: next to a listed value to filter the data lake overview directly for all log entries containing that value.
:minus::computer-mouse: Click the Minus icon :minus: next to a listed value to exclude all log entries containing that value from the data lake overview.
#### Advanced Filter Settings There are two ways to open the advanced filter settings. **Option 1: Log List** :computer-mouse: Select a specific field in the [log list](#data-lake-log-fields-list), then click the **Filter** icon :filter: in the window that opens.
**Option 2: Log Entries List** :computer-mouse: Select a specific field in the [log entries list](#log-entries-list), then click the **Filter** icon :filter: in the window that opens.
The **Add Filter** window opens. Here you can configure granular settings for field values and field value combinations to filter the [data lake overview](#data-lake-log-overview) accordingly. A **Result Preview** is also shown on the right side of the window. It shows the values that are available for the selected field.
:computer-mouse: Click the **List** icon :bars-staggered: above the result preview on the right to enable filter-based search and show only values that are available within the currently applied filter rules. :computer-mouse: Click the **Globe** icon :earth-americas: above the result preview on the right to disable filter-based search and show all available values for the selected field. The following filter options are available on the left side of the window:
OptionDescription
Field NameShows the complete field name and the field type of the field for which you are configuring the settings.
Operator

Select the operator that should apply to the filter.

The default operator is Equals.

:computer-mouse: Click the Operator field to expand a list of all available options.

The following options are available:

  • Equal: Shows only log entries in which the value in the field exactly matches the specified value.
  • Unequal: Shows only log entries in which the value in the field does not match the specified value.
  • Exists: Shows only log entries in which the field exists.
  • Not Exists: Shows only log entries in which the field does not exist.
:eye: The following fields are shown only if you select **Equal** or **Unequal** as the **Operator**.
OptionDescription
Value

Enter the field value to which the operator should be applied.

:computer-mouse: Click the Plus icon :plus: next to the Value field to add additional field values to the filter.

OR

:computer-mouse: Click an available value in the Result Preview on the right to add it directly to the filter on the left. Click a value again to remove it from the selection. Added values are indicated by a pink bar.

:computer-mouse: Click the :trash: Delete icon next to an added value in the Value field to remove it.

Logical Link

:eye: This option is visible only if you specify two or more values.

Select how the specified values should be combined.

The default logical link is Or.

The following options are available:

  • Or: At least one of the values entered above must exist in the corresponding field of a log entry.
  • And: All values entered above must be exist in the corresponding field of a log entry.
Exact Match

Define whether the spelling of the values entered above must exactly match the values in the actual log entries, including capitalization.

When this option is enabled, only log entries with an exact, case-sensitive match are shown.

This option is enabled by default.

Case Insensitive

Define whether the spelling of the values entered above should match the values in the actual log entries without considering capitalization.

When this option is enabled, only log entries that match regardless of capitalization are shown.

This option is disabled by default.

Regular Expression

Define whether you want to use regular expressions to define field values for filtering.

When this option is enabled, you can use regular expressions in the Value field.

This option is disabled by default.

:computer-mouse: Click **Add Filter** to apply the filter. :computer-mouse: Click **Close** to return without applying the filter. ### Editing Applied Filters If you selected a managed or custom [stream](/docs/manual/english/platform-usage/siem/advanced-settings/streams.md) as a filter or applied a filter using the [log list](#data-lake-log-fields-list) on the left, the filter is shown as a string above the bar chart in the [data lake overview](#data-lake-log-overview).
You have several options for editing it. #### Option 1: Editing the Filter Operator :computer-mouse: Click an operator in the filter string to change it.
The following options are available:
OperatorDescription

= (Equal)

or

≠ (Unequal)

:computer-mouse: Click the operator to expand a list of all available options, then select the desired option.

Only operators that are not currently applied to the filter are shown.

The following options are available:

  • = (Equal): Shows only log entries in which the value in the field exactly matches the defined value.
  • ≠ (Unequal): Shows only log entries in which the value in the field does not match the defined value.
  • Other: Select this option to choose an operator other than = (equal) or ≠ (unequal). The advanced filter settings window opens, where you can edit the configured filter settings.

Exists

or

Does Not Exist

:computer-mouse: Click the Exists or Does Not Exist operator to open the advanced filter settings window, where you can edit the configured filter settings.
#### Option 2: Editing the Filter Connection :eye: A logical operator is visible only if you have applied at least two filters. :computer-mouse: Click the logical operator between two filters to expand the available options, then select an option. {% hint style="info" %} The default filter combination is **AND**. {% endhint %}
The following logical operators are available: * **AND**: All applied filters must match a log entry for it to be shown. * **OR**: At least one of the applied filters must match a log entry for it to be shown. #### Option 3: Editing the Filter Expression :computer-mouse: Click the **Edit** icon :pencil: on the right side of the filter string.
You can then edit the filter expression manually using raw query syntax.
:computer-mouse: After making all changes, click **Apply** to apply the filter. {% hint style="warning" %} **Please note:** When you combine more than two filters and switch between **AND** and **OR**, place the filter logic that should be evaluated first in parentheses to define the evaluation order. In the following example, the search first checks for SSH and security notifications before evaluating the date: `(gen.product:"SSH" AND gen.facility:"security/authorization messages") OR gen.timestamp:"2026-01-28"` {% endhint %} #### Option 4: Removing the Filter :computer-mouse: Click the **Remove** icon :xmark: next to an applied filter to delete it. ### Adding a Chart Field Here, you can define a field whose associated values affect how the log overview is shown. There are two ways to select and add a chart field. #### Option 1: Log List :computer-mouse: Click a field name in the [log list](#data-lake-log-list). In the window that opens, click the **Insert as Chart Field** icon :chart-column:.
#### Option 2: Log Entries List :computer-mouse: Select a specific field in the [log entries list](#log-entries-list). In the window that opens, click the **Insert as Chart Field** icon :chart-column:.
#### Changes Caused by Chart Fields Setting a chart field affects several areas of the data lake view.
The changes apply to the following areas:
No.Data Lake AreaChanges
Log List

Creates a new Chart Fields section in the log list. The selected field is shown in this section together with all associated values and the number of log entries for each field value.

The color assigned to each field value corresponds to its color in the bar chart.

:computer-mouse: Click the Delete icon :trash: next to a field name in the Chart Fields section to remove the chart field. The bar chart and log entries list are reset according to the applied filters.

Log Entries: TimelineThe bar chart shows the total number of log entries for the selected field, broken down by field value. Each field value is assigned a separate color.
Log Entries: List

Pins the selected field to the beginning of each log entry.

The color assigned to a field value corresponds to the color used in the bar chart and the log list.

*** ## Data Lake: Log Overview The data lake overview consists of two main sections: a [bar chart](#log-entries-timeline) and the actual [log entries](#log-entries-list). Applied filters are also shown as a string above the bar chart. ### Free-Text Search In addition to the **Search for Fields** field in the log list, you can also search for fields in the data lake overview. :computer-mouse: Click the **Text Search** field above the bar chart and enter any search term. All fields that match the search term and have matching results in the log overview are shown.
:computer-mouse: Click a field name to add an advanced filter for that field. The [add filter window](#advanced-filter-settings) opens with the selected field already entered. ### Log Entries: Timeline The bar chart visualizes the number of log entries recorded in the data lake within the [configured time range](#define-start-and-end-time) and based on the [applied filters](#advanced-filter-settings).
You can also use [added chart fields](#add-chart-field) to show the number of all log entries for a specific field, grouped by field value and shown in different colors. Each field value is assigned its own color. :arrow-pointer: Hover over a bar to show the exact time of the measurement and the exact number of log entries recorded since the previous measurement, based on the applied filters. ### Log Entries: List The list shows log entries from all connected sources within the [configured time range](#define-start-and-end-time) and based on the [applied filters](#advanced-filter-settings).
The table provides the following information:
ColumnDescription
Created AtShows the exact date and time when the log entry was recorded and the corresponding time zone.

:computer-mouse: Click the Expand icon :arrow-up-right-and-arrow-down-left-from-center: to open the log detail view.
DocumentShows a preview of the log entry. The number of log lines displayed per table row depends on the log table settings.
:computer-mouse: Click a field value within a log entry to show the available options.
:computer-mouse: Click the **Bar Chart** icon :chart-column: to [add the field as a chart field](#add-chart-field). :computer-mouse: Click the **Filter** icon :filter: to add an advanced filter. The [add filter window](#advanced-filter-settings) opens with the selected value already entered. ### Log Detail View :computer-mouse: Click the **Expand** icon :arrow-up-right-and-arrow-down-left-from-center: in a log entry table row to show all fields and field values of the log entry in JSON format.
:computer-mouse: Click **Copy to Clipboard** above the log entry on the right to copy all information to your clipboard. :computer-mouse: Click the **Lines** icon :align-justify: above the log entry on the right to show all parameters and values in CSV format. :computer-mouse: Clicke the **Tree** icon :diagram-project: above the log entry on the right to show all parameters and values in JSON format again.

---------------

Use the AI analysis feature on the left side of the window to summarize or analyze the shown log entry based on your instructions. {% hint style="danger" %} **Please note**: This feature is available only if you have connected a [Large Language Model (LLM) provider](/docs/manual/english/platform-usage/siem/orchestration-soar-beta/llm-providers/add-llm-provider.md) to your SIEM. {% endhint %}
:computer-mouse: Click one of the suggested prompts, **Assess**, **Summarize the Log** or **Explain the Log Fields**, or enter your own prompt in the **Ask Me Anything** field and press **Enter**. :computer-mouse: Click the **Settings** icon :sliders-up: next to the **Ask Me Anything** field to show further options.
The following option is available:
OptionDescription
Enable Tool Calling

Allow the connected LLM to automatically call external sources to get more precise answers.

This option is enabled by default.

:computer-mouse: Select or deselect the checkbox :square: next to Enable Tool Calling to enable or disable the option.

*** ## Further Actions ### Exporting Logs :computer-mouse: Click :down-to-dotted-line: **Export** in the upper-right corner to open the **Create Export** window. Here you can export the shown log entries for the [configured time range](#define-start-and-end-time) and based on the [applied filters](#advanced-filter-settings).
The following options are available:
OptionDescription
NameEnter a meaningful name for the export.
Format

Select the format in which the export should be created.

The following formats are available:

  • CSV (Comma-Separated Values): Download the export as a .csv file.
  • JSON (JavaScript Object Notation): Download the export as a .json file.
  • XML (Extensible Markup Language): Download the export as an .xml file.

The default format is CSV.

:computer-mouse: Click the Format field to expand a list of all available options, then select a format.

Expires

Define the date until which the created export should be available for download under Exports.

:computer-mouse: Click the Calendar icon :calendar: to open the date picker. Use the navigation to select the required date. Click Close to apply the selected date.

Logs

Shows the total number of log entries that will be exported for the configured time range and based on the applied filters.

This field cannot be edited.

Please note: We recommend that you export no more than 1 million log entries to avoid resource bottlenecks.

I Am Aware of the Risk and Would Like to Continue

:eye: This option is only visible if you want to export more than 1 million log entries.

Enable this option to confirm that you understand the risk of an export of this size and still want to proceed.

:computer-mouse: Select or deselect the checkbox :square: next to the option to enable or disable it.

Please note: This field is required. The export can be started only after you confirm that you understand the risk of excessive resource usage caused by the export.

:computer-mouse: Click **Create Export** to create the export asynchronously in the background. Once the export has been created, you can view and download it under **SIEM** → [Exports](/docs/manual/english/platform-usage/siem/exports.md). ### Opening a Stream :computer-mouse: Click :down-to-line: **Open Stream** in the upper-right corner to select an existing managed or custom stream and apply it as a data lake filter. The **Select Stream** window opens. You can use the tabs to navigate between three different views: * [Tab: Managed Catalogs](#tab-1-managed-catalogs): Shows all managed streams from managed stream catalogs. * [Tab: My Catalogs](#tab-2-my-catalogs): Shows all streams from custom catalogs and imported managed stream catalogs. * [Tab: My Streams](#tab-3-my-streams): Shows all custom streams. {% hint style="info" %} By default, the **Select Stream** window opens on the **My Streams** tab. {% endhint %} #### Navigation :magnifying-glass: Use the **Text Search** to find streams in the respective list more quickly. :computer-mouse: Click the **Sort** icon :bars-sort: above the stream list to adjust the priority of the applied sorting criteria. A menu opens that shows all columns that are currently used to sort the list. Move the shown items using drag and drop to change the sorting priority. :computer-mouse-scrollwheel: Scroll down the stream list or adjust the number of results shown per page ![](https://2423388680-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FV7OrHgKXA0t1jankQBYf%2Fuploads%2FfjKACCt2SKXcFd3S7Xb5%2FEnginsight-EN-Results%20per%20Page.jpg?alt=media\&token=e1650160-7a41-4965-aaca-6975db758dbe) to view additional entries. Use the page navigation ![](https://content.gitbook.com/content/V7OrHgKXA0t1jankQBYf/blobs/wnxVw06QGoSsYjhwMJRF/Enginsight-Seitennavigation-2.jpg) to switch between pages. #### Tab: Managed Catalogs The **Managed Catalogs** tab shows a list of all available stream catalogs managed by Enginsight on the left.
:computer-mouse: Click a stream catalog to show a list of all streams contained in the corresponding catalog on the right. :computer-mouse: Click a stream in the list to select it. #### Tab 2: My Catalogs The **My Catalogs** tab shows a list of all custom stream catalogs and imported, managed stream catalogs on the left.
:computer-mouse: Click a stream catalog to show a list of all streams contained in the corresponding catalog on the right. :computer-mouse: Click a stream in the list to select it. #### Tab 3: My Streams The **My Streams** tab shows a list of all custom streams. These can include streams that you added under **SIEM** → [Streams](/docs/manual/english/platform-usage/siem/advanced-settings/streams.md) or streams that you [saved from the data lake view](/docs/manual/english/platform-usage/siem/data-lake.md#saving-a-stream).
:computer-mouse: Click a stream in the list to select it. #### Opened Streams Open streams are shown as tabs above the data lake overview.
:computer-mouse: Click a tab to switch to the corresponding filtered data lake view. :computer-mouse: Click the **Remove** icon :xmark: next to a stream name to close the open tab. :computer-mouse: Click the **Open** icon :arrow-up-right-from-square: next to a stream name to navigate to the corresponding [edit view](/docs/manual/english/platform-usage/siem/advanced-settings/streams.md#stream-edit-view) of the stream. ### Saving a Stream You can save filters that you applied manually in the data lake as streams so that you can access them more quickly or use them for other SIEM functions. :computer-mouse: Click :floppy-disk: **Save Stream** in the upper-right corner to open a new window.
The following options are available:
OptionDescription
Add Stream

Save the current filter as a new stream with a defined severity for SIEM incidents that are triggered by this stream.

:computer-mouse: Click the Name field and enter a meaningful name for the new stream.

:computer-mouse: Click the Severity field to expand a list of all available options and select a severity.

The default severity is Medium.

The following severity levels are available:

  • Informational: An incident triggered by this stream is for informational purposes only.
  • Low: An incident triggered by this stream has a low criticality.
  • Medium: An incident triggered by this stream has a medium criticality.
  • High: An incident triggered by this stream has a high criticality.
  • Critical: An incident triggered by this stream has a very high criticality.
Overwrite

Select an existing stream whose filter you want to update.

Please note: If you overwrite an existing stream, all filter settings configured in that stream will be overwritten.

:computer-mouse: Click **Add Stream** to save the new stream. You can now find the newly created stream under **SIEM** → [Streams](/docs/manual/english/platform-usage/siem/advanced-settings/streams.md). :computer-mouse: Click **Close** to return to the data lake overview without saving. ***