SIEM

Use our SIEM to collect and analyze information about the security posture of your IT network.

PreviousLoad Balancing NextDeinstallation

Last updated 17 days ago

Was this helpful?

SIEM

Use our SIEM to collect and analyze information about the security posture of your IT network.

Secure your data with our powerful SIEM: contact our experts at sales@enginsight.com to get started as quickly as possible and get support with your installation!

Structure

Preparation

You will need the following components to install the SIEM

Component

Requirement

1x SIEM Management Server

4CPU, 8GB RAM, 200GB Disk

1X SIEM Index Server

4CPU, 8GB RAM, 200GB Disk

This setup is designed exclusively for a workload of 10 GB per day! If you have higher requirements, please contact us for individual advice.

The following firewall rules must also be enabled:

NGS APP Server -> SIEM Management Server - Port 443
SIEM Index Server <- SIEM Management Server - Port 8983
INDEX-Server -> SIEM-Management TCP/2181

Create a swap on all VMs if you do not already have one.

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo cp /etc/fstab /etc/fstab.bak
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
echo 'vm.vfs_cache_pressure=10' | sudo tee -a /etc/sysctl.conf

Please do not change the Docker configurations provided below, as they are coordinated with each other.

Installation SIEM Management Server

The SIEM Management Server receives the logs via the API and sends them to the SIEM Index Server, where they are processed.

The following components are installed for this purpose:

nginx
docker-ce
docker-ce-cli
containerd.io
docker-buildx-plugin
docker-compose-plugin
Enginsight Loggernaut

Install Docker

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
    "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
    "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Regulate the memory requirements of the Docker logs by limiting them to 100 Mb as follows:

nano /etc/docker/daemon.json

Insert the following:

{  
  "log-driver": "json-file",  
  "log-opts": {
    "max-size": "50m",
    "max-file": "30"  
  }
}

Restart Docker service:

systemctl restart docker

Create a folder for Docker and navigate to it

sudo mkdir /opt/enginsight/enterprise -p
cd /opt/enginsight/enterprise

Customize the docker-compose.yml

sudo nano docker-compose.yml

Add and complete the following configuration

version: '3'
services:
  zoo:
    image: zookeeper:3.9
    container_name: zookeeper
    restart: always
    volumes:
      - /var/zookeeper_data:/data
      - /var/zookeeper_logs:/logs
      - /var/zookeeper_datalog:/datalog
    ports:
      - 10.1.0.4:2181:2181
    environment:
      ZOO_AUTOPURGE_PURGEINTERVAL: 24
      ZOO_SERVERS: server.1=10.1.0.4:2888:3888;2181
      ZOO_4LW_COMMANDS_WHITELIST: mntr,conf,ruok

The corresponds to the internal IP address of the SIEM management server. This must be accessible both from the management server and from the other SIEM index server(s).

Create the directories that are defined in the docker-compose:

sudo mkdir /var/zookeeper_data
sudo mkdir /var/zookeeper_datalog
sudo mkdir /var/zookeeper_logs

Start the Docker container

sudo docker compose up -d

Check whether your Docker container is running

sudo docker ps

Install nginx

Install nginx

sudo apt install nginx -y

Create a user name and password for authentication on the SIEM Management Server.

curl -sSL https://get.enginsight.com/siem/scripts/basicauth.sh | sudo -E bash -s

The user name and password are generated automatically (BasicAuth).

You will receive the following output:

{
  "siem": {
    "basicAuth": {
      "username":"<YourUsername>",
      "password":"<YourPassword>"
    },
    ...
  }
}

Customize the nginx configuration:

To do this, open the configuration:

sudo nano /etc/nginx/sites-available/default

and adjust them as follows:

upstream backend {
   server <IpOfTheSiemIndexServer1>:8983;
   server <IpOfTheSiemIndexServer2>:8983;
}


# only relevant with LetsEncrypt  
server {
    listen 80;
 
    location ~ /.well-known {
        allow all;
    }
 
    location / {
        return 302 https://<IhreSiemUrl>/solr/;
    }
 
    root /var/www/<YourSiemUrl>;
 
    server_name <YourSiemUrl>;
}

server {
    listen 443 ssl http2;

    server_name <YourSiemUrl>;
 
    ssl_protocols TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_ecdh_curve secp384r1;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;
    
    #$ mkdir /etc/nginx/ssl -p
    #$ openssl dhparam -out /etc/nginx/ssl/dhparam.pem 2048 
    
    ssl_dhparam /etc/nginx/ssl/dhparam.pem;
    ssl_certificate /etc/letsencrypt/live/<IhreSiemUrl>/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/<IhreSiemUrl>/privkey.pem;
 
    client_max_body_size 1024m;
 
    auth_basic "Restricted Content";
    auth_basic_user_file /etc/nginx/.htpasswd;
 
    location ~* "^/v1/" {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-Proto "https";
        proxy_set_header X-Forwarded-Ssl   "on";
    }
 
    location / {
        proxy_pass http://backend;
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-Proto "https";
        proxy_set_header X-Forwarded-Ssl   "on";
    }
}

Check the nginx configuration:

sudo nginx -t

Restart the nginx to apply the configuration:

sudo systemctl restart nginx

Installation SIEM Index Server

Install Docker

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
    "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
    "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Regulate the memory requirements of the Docker logs by limiting them to 100 Mb as follows:

nano /etc/docker/daemon.json

Insert the following:

{  
  "log-driver": "json-file",  
  "log-opts": {
    "max-size": "50m",
    "max-file": "30"  
  }
}

Restart Docker service:

systemctl restart docker

Create a folder for Docker and navigate to it:

sudo mkdir /opt/enginsight/enterprise -p
cd /opt/enginsight/enterprise

Customize the docker-compose.yml:

sudo nano docker-compose.yml

Add and complete the following configuration:

version: '3'
services:
 solr:
    restart: always
    image: solr:9.3
    ports:
      - "<IPvomSIEMIndexServer>:8983:8983"
    volumes:
      - /var/solr:/var/solr/data
    environment:
      - ZK_HOST=<IPvomSIEMManagementServer>:2181
      - SOLR_HOST=<IPvomSIEMIndexServer>
      - SOLR_OPTS=-XX:G1HeapRegionSize=32M -Dpkiauth.ttl=5000 -Dsolr.log.level=WARN
      - SOLR_JAVA_MEM=-Xms<40%RAM in gb>g -Xmx<50%RAM in gb>g
    command: solr -f -cloud

<IPfromSIEMIndexServer> corresponds to the IP of the SIEM Index Server

<IPfromSIEMManagementServer> corresponds to the IP of the SIEM Management Server

Create directories and assign rights to access them:

sudo mkdir /var/solr 
sudo chown 8983:8983 -R /var/solr

Start the Docker container:

sudo docker compose up -d

Check whether the port is only open for the internal address. Here you can install dienet-tools (sudo apt install net-tools).

netstat -tulpen

The result should look like this:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name       
tcp        0      <interneIP>:8983          0.0.0.0:*               LISTEN      0

Make sure that the correct IP address appears under Host.

Customization APP Server

Customize the Enginsight APP Server configuration and access to communicate with your SIEM Management Server.

Switch to the configuration file:

sudo nano /opt/enginsight/enterprise/conf/services/config.json

add the following section to the configuration

"siem": {
  "basicAuth": {
    "username": "",
    "password": ""
  },
  "url": "",
  "numShards": 2,
  "replicationFactor": 1,
  "management": {
    "organisation": "YourOrganisation"
  }
},

Fill in "basicAuth" with the credentials you have stored in the proxy. For the URL, enter the URL of the SIEMs server or its IP address.

numShards are responsible for the parallel processing of requests. At least 2 should be entered here. For larger instances, half of the cores should be entered. For a server with 8 cores, correspondingly 4.

The replicationFactor is responsible for replication. The data is distributed to several servers. The advantage is that the other index servers can intervene in the event of a failure. However, the consumption of memory increases. If desired, you can enter the number of your index servers for replication. Alternatively, leave this at 1. The organization corresponds to the organization ID that manages the SIEM clusters. Select an organization for this (we recommend your "main org").

In the configuration under "api", add:

"url": "",

So that your configuration looks like this:

"siem": {
  "basicAuth": {
    "username": "<IhrUsername>",
    "password": "<IhrPasswort>"
  },
  "url": "<IhreSIEMURl>",
  "numShards": 2,
  "replicationFactor": 1,
  "management": {
    "organisation": "YourOrganisation"
  }
},
"api": {
  "url": "<IhreAPIURL>",
  ...

Change to the Enginsight directory and start setup.sh

cd /opt/enginsight/enterprise
sudo ./setup.sh

Regulate the memory requirements of the Docker logs by limiting them to 100 Mb as follows:

nano /etc/docker/daemon.json

Insert the following:

{  
  "log-driver": "json-file",  
  "log-opts": {
    "max-size": "100m",
    "max-file": "30"  
  }
}

Restart the Docker service:

systemctl restart docker

Create an access key for the SIEM Mangagement Server from

Install Loggernaut

The Loggernaut is used to receive and process the logs.

Log in to your SIEM management server
Install the Loggernaut as follows:

curl -sSL https://get.enginsight.com/loggernaut/latest/setup.sh | sudo -E bash -s \
    api=https://<apiurl> \
    accessKeyId=<accessKeyId> \
    accessKeySecret=<accessKeySecret> \
    indices=http://<SiemIndexServer>:8983 \
    username=<IhrUsername> \
    password=<IhrPasswort>

Switch to the Enginsight instance and click on the "SIEM" tab. The interface establishes a connection to the servers, which may take a few seconds.

BETA-Installation SIEM KI

To be able to use the functionalities in the SIEM AI, some configurations are required.

This is a beta functionality. Please note that errors may occasionally occur when predicting the metrics and the resulting anomaly detection.

Update affected components.
```
- ui-m1 
- server-m2 
- loggernaut-m47 
```

Activate experimental ML mode in Loggernaut.
To do this, customize the configuration file under:
```
/opt/enginsight/loggernaut/config.json
```
by adding the following line at root level:
```
  "experimental": ["traicer"]
```
Then restart the Loggernaut to apply the configuration.
```
sudo systemctl restart ngs-loggernaut.service
```

Install Docker.

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
    "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
    "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Regulate the memory requirements of the Docker logs by limiting them to 100 Mb as follows:

nano /etc/docker/daemon.json

Insert the following:

{  
  "log-driver": "json-file",  
  "log-opts": {
    "max-size": "50m",
    "max-file": "30"  
  }
}

Restart Docker service:

systemctl restart docker

Create Docker Compose.

The following must be included in docker-compose.yml:

version: '3'
services:
  traicer:
    image: registry.enginsight.com/enginsight/traicer-m50:0.0.1
    restart: always
    volumes:
      - /var/traicer/data:/traicer/data
    environment:
      ADDRESS: 'wss://<SIEM-Management-Adresse>/v1/traicer'
      BASIC_AUTH_USER: '<SIEM-BasicAuth-User>'
      BASIC_AUTH_PASSWORD: '<SIEM-BasicAuth-Passwort>'

If the machine learning runs on the same machine as the Loggernaut and generates too much CPU load so that the Loggernaut can no longer handle the log throughput, we recommend reducing the CPU shares of the container so that other processes have a higher priority. This adjustment to docker-compose.yml is necessary for this:

version: '3'
services:
  traicer:
    cpu_shares: 512

Adjustments to Nginx Config from the Management Server for Websocket Support. Modify the following file: /etc/nginx/sites-available/default

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

...

server {
  ...
  
  proxy_http_version 1.1;

  ...

  location ~* "^/v1/" {
    ...
    
    proxy_set_header Upgrade           $http_upgrade;
    proxy_set_header Connection        $connection_upgrade;

    ...
  }
}

Check the config for errors:

sudo nginx -t

Restart nginx to apply the config:

sudo systemctl restart nginx

Make sure that you have root rights. If this is not the case, log in as superuser.
```
su - root
```
```
docker version
```
```
docker-compose -v
```

Do not install Docker via Snap or when installing the operating system, but only install Docker from the package sources in the official instructions.

Log in to Docker.
```
docker login registry.enginsight.com
```

You will receive the access data from us.

If there are problems with authentication, check whether a current gpg2 key is available and generate it automatically if necessary:

apt install gnupg2

gpg2 --gen-key

Start the container with the adjustments you have just made.
```
sudo docker compose up -d
```

Potential problems

If the platform indicates that it was unable to establish a connection, please check the following points:

Can the APP server reach the SIEM management server? Test this using curl. Status code 200 must be returned.

curl -v -u "<username>:<password>" <IhreSIEMDomain>/v1/collections

Can the SIEM server reach the APP server?
Have all firewall permissions from the Preparation section been set?

Self-signed certificates

If you use a self-signed certificate or a Windows PKI for your SIEM URL, you must first trust the root certificate on the SIEM management and on the app server. It also requires an adjustment in the docker-compose.yml of the App Server.

Trust certificate:

Copy your root certificate on the SIEM Managemt to /usr/local/share/ca-certificates/ and update the CA Store

sudo cp foo.crt /usr/local/share/ca-certificates/foo.crt

sudo update-ca-certificates

Repeat this for the App Server
customize docker-compose.yml on the App Server:

Navigate to the /opt/enginsight/enterprise folder on the app server and open the file using :

sudo nano docker-compose.yml

Add the following under enviroment:

environment:
       NODE_EXTRA_CA_CERTS: /usr/local/share/ca-certificates/ca.crt
       NODE_OPTIONS: --use-openssl-ca
    volumes:
      - "/usr/local/share/ca-certificates/ca.crt:/usr/local/share/ca-certificates/ca.crt"
      - "./conf/services/config.json.production:/etc/enginsight/server-m2/config.json"

Please replace ca.crt with the name of your root certificate.

Traicer Troubleshooting

If you use self-signed certificates or an error such as the following occurs in the Traicer log:

manager.py:468: failed connecting to wss://<URL>/v1/traicer due to [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000). Retrying in 0:00:12

you must explicitly transfer the certificates to the Traicer container. To do this, follow the next steps:

Customization of the docker-compose.yml
Add the docker-compose.yml to mount the certificate in the container and set the necessary environment variable. An example of the configuration of the Traicer service:
```
traicer:
  image: registry.enginsight.com/enginsight/traicer-m50:0.0.2
  restart: always
  volumes:
    - /var/traicer/data:/traicer/data
    - /path/to/your/certificate.pem:/etc/ssl/cert.pem
  environment:
    ADDRESS: 'wss://<url>/v1/traicer'
    BASIC_AUTH_USER: '<user>'
    BASIC_AUTH_PASSWORD: '<pw>'
    EXTRA_CA_CERTS: /etc/ssl/cert.pem
```
Explanation:
- <path/to/your/certificate.pem>: Replace this with the path to your certificate.
- EXTRA_CA_CERTS: This environment variable specifies where the additional certificates are located in the container.
- ADDRESS without double https://: Make sure that the value is correct. A common error is: False: wss://https://<url>/v1/traicer Right: wss://<url>/v1/traicer
Restart container
After customization, execute the following command to apply the changes:
```
sudo docker-compose up -d
```

Add the first logs

You will notice at the beginning that the SIEM is already filled with logs. These come from the IDS, IPS, FIM, etc. as standard. You can view the sources under SIEM -> Data Lake -> ngs.source.

Agent logs

To receive the logs, such as Syslog (Linux), EventLog (Windows) and Unified Logs (MacOs), you must first allow this in the Policy Manager. To do this, navigate to Hosts -> Policy Manager and create a new policy called SIEM.

Use tags to determine which host is allowed to send logs and activate "Allow evaluation of system logs" under Advanced settings.

Now navigate back to the SIEM tab. You will find the individual operating systems under Integrated collectors. Here you can add a new collector in the top right-hand corner.

Assign a name for the respective operating system that is to be logged. Activate logging and assign it to a tag. You can then select the channels via which the logs are to be collected.

Connection of devices without agent (e.g. firewall)

You can also connect devices to the SIEM that do not have an agent installed. To do this, define an agent to collect the logs. To do this, navigate to SIEM -> Event relay.

Select a name with which you can clearly identify your collector. Activate logging again and define a client or server under Host which is to receive the logs from the device (e.g. firewall). Also select the format and decide via which port and which protocol the data should be sent.

It is possible to configure multiple collectors. This is particularly useful for multiple networks so that no new firewall rules need to be configured, as one collector can be used per network.

PreviousLoad Balancing NextDeinstallation

Last updated 17 days ago

Was this helpful?