Synchronize logs to SeaTable to make logs better visualized

SeaTable provides Restful API, so we can easily write data through HTTP protocol from anywhere. In this article, we mainly introduce how to use SeaTable to record the error logs of various services on the server, realize the visualization of the logs, and prevent the error messages from being ignored.

The main process is as following:

             Filebeat               Python Script
Log files -------------> Redis ------------------> SeaTable

Features include

  • The log is placed in a code block in Markdown format and recorded in the long text column of SeaTable
  • We can use SeaTable’s multi-view function to create multiple different views, each view includes specific filter conditions, which is convenient for us to quickly view data

Let’s introduce the construction process in detail below.

Use filebeat to send logs to Redis

Take Ubuntu 20.04 as an example, use the following command to install filebeat

apt update && apt install filebeat -y

Config filebeat (You can find an example at seatable-syncer/filebeat.yml at main · seatable/seatable-syncer · GitHub):

vim /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    paths:
      - /opt/seatable/shared/seatable/logs/dtable_web.log
    tags: ["dtable-web"]       
    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}|^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
    multiline.negate: true
    multiline.match: after

  - type: log
    paths:
      - /opt/seatable/shared/seatable/logs/dtable-server.log
    tags: ["dtable-server"]
    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}|^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
    multiline.negate: true
    multiline.match: after

output.redis:
  hosts: ["redis host"]
  password: "redis password"
  db: 0
  timeout: 5
  keys:
    - key: "seatable-error-logs" 
      when.contains:
        message: "[ERROR]"

Note: we use multiline match pattern to match multiple lines as a single log item,
the beginning of a log item is 2021-10-12 or [2021-10-12. In the last line message: "[ERROR]", we only send log items containing string “[ERROR]” to Redis.

Finally, start filebeat:

service filebeat start

Reading logs from Redis and save to SeaTable using Python script

Preparing SeaTable base

  1. First, generate an API Token from the base
  2. Create a table with following columns
Name Is first column Type
Service Text column or single select column
Time x Date column with minutes
Log x Long text column

Preparing the script

Download and install the script to /opt/

cd /opt
git clone https://github.com/seatable/seatable-syncer.git

Install Python libraries

cd seatable-syncer/log-sync/
pip3 install -r requirements.txt

Modify the configuration file log_syncer_settings.py

# SeaTable
server_url = 'Your SeaTable server URL'
api_token = 'the API token you created'
table_name = 'table name'

# Redis
redis_host = 'your redis host'
redis_db = 0
redis_port = 6379
redis_password = None

Starting the script

python3 log_syncer.py

log_syncer.py can handle log items start with following strings:

2020-01-01 01:02:02
2020-01-01T01:02:02
[2020-01-01 01:02:02
[2020-01-01T01:02:02

If you have logs in other format, you can modify the script accordingly.

That’s it

With filebeat, Redis and Python scripts, we can collect logs from multiple servers into a base in SeaTable. Through the filtering conditions of the table and the multi-view function, we can easily analyze and view the log.

3 Likes