Common Dataset Won't Import

I’m trying to import a common dataset into a base. I get the attached messages, but the table never appears in the base, even after hours. The common dataset only has 190 rows in it, so wouldn’t expect the import to take long.

In case it matters… I’m using the developer edition and have just updated from 2.2 to 2.7. I did the DB upgrade for each version and added the extra changes for 2.3 and 2.7. Everything else in Seatable seems to be working fine.

I’ve tried other datasets that existed before the upgrade and they won’t import either. Are there logs somewhere that might help troubleshoot this?

image

1 Like

I notice in dtable_web.log there are some errors that appear following a common dataset import. Should the API URL be /api/v2.7 ?
v2.1 is the first version I started on. Upgraded to v2.2 and common datasets were working (I think) before upgrading to v2.7 today.

I’ve created a fresh Docker instance of seatable-developer:2.7.0 to test this and am now getting an error when importing the common dataset. I notice there are some fixes to common dataset synchronisation in the changelog, could they be what is causing this?

The developer edition seems to only have version 2.7.0 available so not sure how to upgrade this to latest version.

image

Your log is full of memcache errors. With memcache being a critical component, I am not surprised that you experience errors in the app.

I cannot give you any specific advice without further info. But your setup seems has problems.

Having this same issue where I will import and nothing will appear. Upgraded from 2.6.2 to latest
image

Please provide more info: which edition do you use - DE or EE? Do you also find error messages in your log?

I’ve spent some more time experimenting with this issue. Please ignore the memcache errors, I tried running a second docker compose to experiment with and I think my container names caused memcache not to work. The original problem I saw showed no memcache errors.
On further investigation I found the below errors appearing in dtable-events.log whenever I tried to import a common dataset to a base.

[2022-02-09 09:04:55,319] sync_common_dataset.py[line:712] [ERROR] request src_dtable: e7f8d308-d9df-4ae9-bec1-bd8c5e0e6a9c params: {‘table_name’: ‘Table1’, ‘view_name’: ‘Default View’, ‘use_dtable_db’: True, ‘start’: 0, ‘limit’: 10000} view-rows error: HTTPSConnectionPool(host=‘seatable.redacted.local’, port=443): Max retries exceeded with url: /dtable-server/api/v1/internal/dtables/e7f8d308-d9df-4ae9-bec1-bd8c5e0e6a9c/view-rows/?from=dtable_events&table_name=Table1&view_name=Default+View&use_dtable_db=True&start=0&limit=10000 (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)’)))

I forgot to mention that the one other change I made at the time of upgrading was to put Seatable behind our internal reverse proxy (Traefik) which uses a self-signed certificate and a *.local domain name. It seems maybe the import is failing due to an error with the self signed certificate within the requests library of Python? From what I’ve read I need to add my self signed CA to the trusted CAs that this library uses. Dropping the certificate into the ssl folder didn’t seem to help. Any guidance on how I can do this?

@jvibe89 any chance you are also running Seatable using a self signed certificate?

@rdb I’m using the developer edition at present, v2.7. There doesn’t seem to be any newer versions of the DE available? Do updates only get published for EE?

I did some more experimenting again this morning. I changed Seatable hostname back to the IP address and bypassed the reverse proxy altogether. Importing common dataset worked again straight away. So I’m sure the issue has something to do with our self signed certificate, as my post above details.

Can you recommend any ways to add our CA to the list of approved CAs that the Python requests library uses in the container? I tried putting the certificate in the Seatable ssl folder but that had no effect. I’m wondering whether Python might be looking somewhere else in the container’s filesystem for approved CAs?

I found these discussions about the error that might help:
Python SSL certificate verify error - Stack Overflow
How to get Python requests to trust a self signed SSL certificate? - Stack Overflow

I am also facing this issue. My setup is Seatable version 2.7 installed on a self hosted server. As there are other services that are activated on the same server I am using Caddy as a reverse proxy. The host port of 4431 is redirected to 443 port of the seatable server container. I can access seatable and create bases / tables etc without issues. I can also create common datasets - however, they will not get imported.
I am not using the nginx reverse proxy in the seatable container to ask for SSL certificate as Caddy provides the SSL certificate automatically. I have configured the nginx.conf file as below :


log_format seatableformat '[$time_iso8601] $http_x_forwarded_for $remote_addr "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $upstream_response_time';

upstream dtable_servers {
    server 127.0.0.1:5000;
    keepalive 15;
}

server {
    if ($host = seatable.XXX) {
        return 301 https://$host$request_uri;
    }
    listen 80;
    server_name seatable.XXX;
    return 404;
}

server {
    server_name seatable.XXX;
    listen 443;



    # for letsencrypt
    location /.well-known/acme-challenge/ {
        alias /var/www/challenges/;
        try_files $uri =404;
    }

    proxy_set_header X-Forwarded-For $remote_addr;

    location / {
        add_header Access-Control-Allow-Origin *;
        add_header Access-Control-Allow-Methods GET,POST,PUT,DELETE,OPTIONS;
        add_header Access-Control-Allow-Headers "deviceType,token, authorization, content-type";
        if ($request_method = 'OPTIONS') {
            add_header Access-Control-Allow-Origin *;
            add_header Access-Control-Allow-Methods GET,POST,PUT,DELETE,OPTIONS;
            add_header Access-Control-Allow-Headers "deviceType,token, authorization, content-type";
            return 204;
        }
        proxy_pass         http://127.0.0.1:8000;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
        proxy_read_timeout  1200s;
        
        # used for view/edit office file via Office Online Server
        client_max_body_size 0;
        
        access_log      /opt/nginx-logs/dtable-web.access.log seatableformat;
        error_log       /opt/nginx-logs/dtable-web.error.log;
    }

    location /seafhttp {
        rewrite ^/seafhttp(.*)$ $1 break;
        proxy_pass http://127.0.0.1:8082;

        client_max_body_size 0;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_request_buffering off;
        proxy_connect_timeout  36000s;
        proxy_read_timeout  36000s;
        proxy_send_timeout  36000s;

        send_timeout  36000s;

        access_log      /opt/nginx-logs/seafhttp.access.log seatableformat;
        error_log       /opt/nginx-logs/seafhttp.error.log;

    }

    location /media {
        root /opt/seatable/seatable-server-latest/dtable-web;
    }

    location /socket.io {
        proxy_pass http://dtable_servers;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_redirect off;

        proxy_buffers 8 32k;
        proxy_buffer_size 64k;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
    
        access_log      /opt/nginx-logs/socket-io.access.log seatableformat;
        error_log       /opt/nginx-logs/socket-io.error.log;

    }

    location /dtable-server {
        add_header Access-Control-Allow-Origin *;
        add_header Access-Control-Allow-Methods GET,POST,PUT,DELETE,OPTIONS;
        add_header Access-Control-Allow-Headers "deviceType,token, authorization, content-type";
        if ($request_method = 'OPTIONS') {
            add_header Access-Control-Allow-Origin *;
            add_header Access-Control-Allow-Methods GET,POST,PUT,DELETE,OPTIONS;
            add_header Access-Control-Allow-Headers "deviceType,token, authorization, content-type";
            return 204;
        }
        rewrite ^/dtable-server/(.*)$ /$1 break;
        proxy_pass         http://dtable_servers;
        proxy_redirect     off;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host  $server_name;
        proxy_set_header   X-Forwarded-Proto $scheme;

        # used for import excel
        client_max_body_size 100m;

        access_log      /opt/nginx-logs/dtable-server.access.log seatableformat;
        error_log       /opt/nginx-logs/dtable-server.error.log;
    }

    location /dtable-db/ {
        add_header Access-Control-Allow-Origin *;
        add_header Access-Control-Allow-Methods GET,POST,PUT,DELETE,OPTIONS;
        add_header Access-Control-Allow-Headers "deviceType,token, authorization, content-type";
        if ($request_method = 'OPTIONS') {
            add_header Access-Control-Allow-Origin *;
            add_header Access-Control-Allow-Methods GET,POST,PUT,DELETE,OPTIONS;
            add_header Access-Control-Allow-Headers "deviceType,token, authorization, content-type";
            return 204;
        }
        proxy_pass         http://127.0.0.1:7777/;
        proxy_redirect     off;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host  $server_name;
        proxy_set_header   X-Forwarded-Proto $scheme;

        access_log      /opt/nginx-logs/dtable-db.access.log seatableformat;
        error_log       /opt/nginx-logs/dtable-db.error.log;
    }

}

Note that XXX refers to my domain name.

My dtable-websettings.conf is as follows:


DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'HOST': 'db',
        'PORT': '3306',
        'USER': 'root',
        'PASSWORD': 'PASSWORD',
        'NAME': 'dtable_db',
        'OPTIONS': {
            'charset': 'utf8mb4',
        },
    }
}

CACHES = {
    'default': {
        'BACKEND': 'django_pylibmc.memcached.PyLibMCCache',
        'LOCATION': 'memcached',
    },
    'locmem': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
    },
}
COMPRESS_CACHE_BACKEND = 'locmem'

SECRET_KEY = 'YYY'

# for dtable-server
DTABLE_PRIVATE_KEY = 'YYY'
DTABLE_SERVER_URL = 'https://seatable.XXX/dtable-server/'
DTABLE_SOCKET_URL = 'https://seatable.XXX/'

# for dtable-web
DTABLE_WEB_SERVICE_URL = 'https://seatable.XXX/'

# for dtable-db
DTABLE_DB_URL = 'https://seatable.XXX/dtable-db/'

# for seaf-server
FILE_SERVER_ROOT = 'https://seatable.XXX/seafhttp/'

ENABLE_USER_TO_SET_NUMBER_SEPARATOR = True


PLUGINS_REPO_ID='a1e654f5-ebe1-49f7-86ca-4e5181d2b21c'

I have also ensured that the same URL is provided in the ccnet.conf file

I am getting the following error in the dtable-events.log

2022-03-14 01:36:21,306 [ERROR] request src_dtable: 74848d8e-c4e2-4b68-8406-bcd5f62befbd params: {'table_name': 'Table1', 'view_name': 'Default View', 'use_dtable_db': True, 'start': 0, 'limit': 10000} view-rows error: HTTPSConnectionPool(host='seatable.XXX', port=443): Max retries exceeded with url: /dtable-server/api/v1/internal/dtables/74848d8e-c4e2-4b68-8406-bcd5f62befbd/view-rows/?from=dtable_events&table_name=Table1&view_name=Default+View&use_dtable_db=True&start=0&limit=10000 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4552f83350>: Failed to establish a new connection: [Errno 110] Connection timed out'))
2022-03-14 01:36:21,306 [ERROR] request src_dtable: 74848d8e-c4e2-4b68-8406-bcd5f62befbd params: {'table_name': 'Table1', 'view_name': 'Default View', 'use_dtable_db': True, 'start': 0, 'limit': 10000} view-rows error: HTTPSConnectionPool(host='seatable.XXX', port=443): Max retries exceeded with url: /dtable-server/api/v1/internal/dtables/74848d8e-c4e2-4b68-8406-bcd5f62befbd/view-rows/?from=dtable_events&table_name=Table1&view_name=Default+View&use_dtable_db=True&start=0&limit=10000 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4552f83350>: Failed to establish a new connection: [Errno 110] Connection timed out'))
2022-03-14 01:36:21,306 [INFO] Run task success: 1647201850555 <function import_common_dataset at 0x7f4552fd73b0> cost 130s 

Would be happy to have any assistance to solve this issue.

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.