Your Setup:
-
Hardware or environment you use:
Docker setup -
SeaTable Edition (Developer or Enterprise)
Developer -
SeaTable Version (only necessary for self-hosted):
{"version":"5.1.0","edition":"developer edition"}
I have a Seatable instance which I use from time to time with not much data (maybe a a couple of thousands of lines).
Today it suddenly stopped working. I have daily n8n backups for Seatable and I noticed that the backup failed today.
I can login without a problem but as soon as I open a Base, it shows me the following error message
I did try to troubleshoot myself but no luck. Here are some error messages that might be relevant.
dtable-server.error.log
2025/02/05 12:38:09 [error] 60#60: *1766 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET /dtable-server/dtables/3d1eb410-9228-4b19-a209-835af6592bde?lang=en HTTP/1.1", upstream: "http://127.0.0.1:5000/dtables/3d1eb410-9228-4b19-a209-835af6592bde?lang=en", host: "tables.domain.com", referrer: "https://tables.domain.com/workspace/4/dtable/Administrim%202025/"
2025/02/05 12:38:17 [error] 60#60: *1774 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET /dtable-server/dtables/50dac823-6a03-44ee-b12d-42df3258fbd9?lang=en HTTP/1.1", upstream: "http://127.0.0.1:5000/dtables/50dac823-6a03-44ee-b12d-42df3258fbd9?lang=en", host: "tables.domain.com", referrer: "https://tables.domain.com/workspace/4/dtable/Administrim%202024/"
2025/02/05 12:40:26 [error] 58#58: *1798 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET /dtable-server/dtables/3d1eb410-9228-4b19-a209-835af6592bde?lang=en HTTP/1.1", upstream: "http://127.0.0.1:5000/dtables/3d1eb410-9228-4b19-a209-835af6592bde?lang=en", host: "tables.domain.com", referrer: "https://tables.domain.com/workspace/4/dtable/Administrim%202025/"
2025/02/05 12:40:29 [error] 53#53: *1808 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET /dtable-server/dtables/50dac823-6a03-44ee-b12d-42df3258fbd9?lang=en HTTP/1.1", upstream: "http://127.0.0.1:5000/dtables/50dac823-6a03-44ee-b12d-42df3258fbd9?lang=en", host: "tables.domain.com", referrer: "https://tables.domain.com/workspace/4/dtable/Administrim%202024/"
dtable-web.error.log
2025/02/05 12:11:27 [error] 53#53: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8000/favicon.ico", host: "tables.domain.com", referrer: "https://tables.domain.com/"
2025/02/05 12:11:27 [error] 53#53: *19 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "tables.domain.com", referrer: "https://tables.domain.com/sys/users/"
2025/02/05 12:11:27 [error] 53#53: *21 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8000/favicon.ico", host: "tables.domain.com", referrer: "https://tables.domain.com/"
2025/02/05 12:11:27 [error] 53#53: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8000/favicon.ico", host: "tables.domain.com", referrer: "https://tables.domain.com/"
2025/02/05 12:11:27 [error] 53#53: *25 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "tables.domain.com", referrer: "https://tables.domain.com/sys/users/"
2025/02/05 12:11:27 [error] 53#53: *27 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8000/favicon.ico", host: "tables.domain.com", referrer: "https://tables.domain.com/"
2025/02/05 12:11:27 [error] 53#53: *29 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8000/favicon.ico", host: "tables.domain.com", referrer: "https://tables.domain.com/"
2025/02/05 12:11:27 [error] 53#53: *31 connect() failed (111: Connection refused) while connecting to upstream, client: 172.23.0.1, server: 127.0.0.1, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "tables.domain.com", referrer: "https://tables.domain.com/sys/users/"
dtable_events_io.log
Traceback (most recent call last):
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/urllib3/connectionpool.py", line 416, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/urllib3/connection.py", line 244, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 951, in send
self.connect()
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9251054fa0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/urllib3/connectionpool.py", line 799, in urlopen
retries = retries.increment(
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /dtables/ee67013a-dc86-48c2-b498-27e61f96eb7f?from=dtable_events (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9251054fa0>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/seatable/seatable-server-latest/dtable-events/dtable_events/dtable_io/__init__.py", line 77, in get_dtable_export_content
prepare_dtable_json_from_memory(workspace_id, dtable_uuid, username)
File "/opt/seatable/seatable-server-latest/dtable-events/dtable_events/dtable_io/utils.py", line 136, in prepare_dtable_json_from_memory
json_content = dtable_server_api.get_base()
File "/opt/seatable/seatable-server-latest/dtable-events/dtable_events/utils/dtable_server_api.py", line 169, in get_base
response = requests.get(url, headers=self.headers, timeout=self.timeout)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /dtables/ee67013a-dc86-48c2-b498-27e61f96eb7f?from=dtable_events (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9251054fa0>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/seatable/seatable-server-latest/dtable-events/dtable_events/dtable_io/task_manager.py", line 417, in handle_task
task[0](*task[1])
File "/opt/seatable/seatable-server-latest/dtable-events/dtable_events/dtable_io/__init__.py", line 80, in get_dtable_export_content
raise Exception('prepare dtable json failed. ERROR: {}'.format(e))
Exception: prepare dtable json failed. ERROR: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /dtables/ee67013a-dc86-48c2-b498-27e61f96eb7f?from=dtable_events (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9251054fa0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2025-02-05 11:00:01,100 [ERROR] Failed to handle task 7b825c33-6293-4e8d-b0ef-d8e22f25aa7b <function get_dtable_export_content at 0x7f9256079f70>, error: prepare dtable json failed. ERROR: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /dtables/ee67013a-dc86-48c2-b498-27e61f96eb7f?from=dtable_events (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9251054fa0>: Failed to establish a new connection: [Errno 111] Connection refused'))
dtable_web.log
Traceback (most recent call last):
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/urllib3/connectionpool.py", line 799, in urlopen
retries = retries.increment(
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /api/v1/admin/sys-info/?from=dtable_web (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f320740d580>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/seatable/seatable-server-latest/dtable-web/seahub/api2/endpoints/admin/utils.py", line 32, in get_dtable_server_info
sys_info = dtable_server_api.get_sys_info()
File "/opt/seatable/seatable-server-latest/dtable-web/seahub/dtable_apps/dtable_server_api.py", line 501, in get_sys_info
response = requests.get(url, headers=admin_headers)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /api/v1/admin/sys-info/?from=dtable_web (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f320740d580>: Failed to establish a new connection: [Errno 111] Connection refused'))
2025-02-05 12:11:37 [WARNING] django.request[241] - Not Found: /favicon.ico
2025-02-05 12:11:38 [WARNING] django.request[241] - Not Found: /favicon.ico
2025-02-05 12:11:38 [WARNING] django.request[241] - Not Found: /favicon.ico
2025-02-05 12:18:22 [WARNING] django.request[241] - Not Found: /api/v1/admin/sys-info/
2025-02-05 12:19:15 [WARNING] django.request[241] - Not Found: /workspace/4/dtables/Administrim 2024/
2025-02-05 12:25:03 [ERROR] seahub.api2.views[247] - scripts_running_count error: HTTPConnectionPool(host='python-scheduler', port=80): Max retries exceeded with url: /scripts-running-count/?username=8345ad0307a043a2b9560f66a4fc7bf2%40auth.local (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fde19ffb9a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
2025-02-05 12:31:57 [ERROR] seahub.api2.endpoints.admin.dtables[826] - HTTPConnectionPool(host='127.0.0.1', port=6000): Max retries exceeded with url: /add-export-task?username=8345ad0307a043a2b9560f66a4fc7bf2%40auth.local&table_name=Administrim+2023&repo_id=d9995dbc-cd23-4c30-9cd2-f6de9dc7f212&workspace_id=4&dtable_uuid=ee67013a-dc86-48c2-b498-27e61f96eb7f&ignore_asset=False (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fde19f2da90>: Failed to establish a new connection: [Errno 111] Connection refused'))
2025-02-05 12:31:57 [ERROR] django.request[241] - Internal Server Error: /api/v2.1/admin/dtables/ee67013a-dc86-48c2-b498-27e61f96eb7f/synchronous-export/export-dtable/
2025-02-05 12:35:27 [WARNING] django.request[241] - Not Found: /favicon.ico
2025-02-05 12:35:27 [WARNING] django.request[241] - Not Found: /favicon.ico
dtable_server_config.json
{
"host": "mariadb",
"user": "xxxxx",
"password": "xxxxxx",
"database": "xxxxxxx",
"port": 3306,
"private_key": "xxxxxxxxx",
"redis_host": "redis",
"redis_port": 6379,
"redis_password": "",
"dtable_web_service_url": "https://tables.domain.com/",
"rows_api_hour_max": 600000,
"rows_api_minute_max": 10000,
"api_req_max": 60000,
"api_base_day_max": 500000,
"rows_api_max_limit": 1000000
}
I bumped up the rate limit settings just in case there was a rate limit somehow but it didn’t fix anything.
Is there any way to check if something got corrupted in the DB?
Let me know if there is anything else I can provide.
Thanks in advance!