My seatable DE is down after upgrade to 3.4

Admin Talk Troubleshooting
1

Hi,
my seatable DE was running smoothly.
I made the upgrade following the usual Upgrade Manual - SeaTable Admin Manual
All docker appear working in portainer, but I reach a :

Page unavailable
Sorry, but the requested page is unavailable due to a server hiccup.

Our engineers have been notified, so check back later.

Starting seatable-redis     ... done
Starting seatable-memcached ... done
Starting seatable-mysql     ... done
Starting seatable           ... done
Attaching to seatable-redis, seatable-mysql, seatable-memcached, seatable
seatable-redis | 1:C 21 Mar 2023 14:28:39.705 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
seatable-redis | 1:C 21 Mar 2023 14:28:39.705 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=1, just started
seatable-redis | 1:C 21 Mar 2023 14:28:39.705 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
seatable-redis | 1:M 21 Mar 2023 14:28:39.706 * Running mode=standalone, port=6379.
seatable-redis | 1:M 21 Mar 2023 14:28:39.706 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
seatable-redis | 1:M 21 Mar 2023 14:28:39.706 # Server initialized
seatable-redis | 1:M 21 Mar 2023 14:28:39.706 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
seatable-redis | 1:M 21 Mar 2023 14:28:39.706 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
seatable-redis | 1:M 21 Mar 2023 14:28:39.706 * DB loaded from disk: 0.000 seconds
seatable-redis | 1:M 21 Mar 2023 14:28:39.706 * Ready to accept connections
seatable-mysql | 2023-03-21 14:28:39+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.13+maria~focal started.
seatable-mysql | 2023-03-21 14:28:40+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
seatable-mysql | 2023-03-21 14:28:40+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.13+maria~focal started.
seatable-mysql | 2023-03-21 14:28:40 0 [Note] mysqld (mysqld 10.5.13-MariaDB-1:10.5.13+maria~focal) starting as process 1 ...
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Uses event mutexes
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Number of pools: 1
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
seatable-mysql | 2023-03-21 14:28:40 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Using Linux native AIO
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Completed initialization of buffer pool
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: 128 rollback segments are active.
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: 10.5.13 started; log sequence number 333504173; transaction id 507852
seatable-mysql | 2023-03-21 14:28:40 0 [Note] Plugin 'FEEDBACK' is disabled.
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
seatable-mysql | 2023-03-21 14:28:40 0 [Note] Server socket created on IP: '::'.
seatable-mysql | 2023-03-21 14:28:40 0 [Warning] 'proxies_priv' entry '@% root@32c6523cdf56' ignored in --skip-name-resolve mode.
seatable-mysql | 2023-03-21 14:28:40 0 [Note] Reading of all Master_info entries succeeded
seatable-mysql | 2023-03-21 14:28:40 0 [Note] Added new Master_info '' to hash table
seatable-mysql | 2023-03-21 14:28:40 0 [Note] mysqld: ready for connections.
seatable-mysql | Version: '10.5.13-MariaDB-1:10.5.13+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
seatable-mysql | 2023-03-21 14:28:40 0 [Note] InnoDB: Buffer pool(s) load completed at 230321 14:28:40
seatable     | *** Running /etc/my_init.d/01_init.sh...
seatable     | *** Booting runit daemon...
seatable     | *** Runit started as PID 15
seatable     | *** Running /templates/enterpoint.sh...
seatable     | 2023-03-21 14:28:40 Conf exists 
seatable     | 2023-03-21 14:28:40 Nginx ready 
seatable     | 2023-03-21 14:28:40 This is an idle script (infinite loop) to keep container running.

I’m stuck …
help

2

Hi @Ben.

Are you sure you started the seatable process with the following command?
I forgot this all the time until I started docker with a script :slight_smile:

docker exec -d seatable /shared/seatable/scripts/seatable.sh restart
3

Hi,

yes, i started it.

And to be sure, i remade the whole process, same thing.

Then I took my snapshot from this morning, no glitches, it worked.
And tried again the whole upgrade process, and same issue appeard.

I’ve got free space on all partitions. I don’t know what is happening.
Rolled back to my snapshot on 3.3 until I sort that out

4

And you ran the database upgrade?
https://manual.seatable.io/upgrade/upgrade_manual/#major-or-minor-version-upgrade

5

yes, and the message was the usual one after the upgrade.

6

There are some more log files in the sub directories. Have you had a look into that?

7

yes, and I did not find anything suscpicious. no strong error messages.

8

Hm, strange. Hard to analyse without direct access

9

I’m trying on a new server to see if I reproduce on a different setup

10

I gave a try on my other server.

One info: both are currently 3.1 version

I’ve performed all steps of upgrade, including sql upgrade for each 3.2 then 3.3 then 3.4

and I’m at the same state, it’s down.
So going from 3.1 to 3.4 seems to break my install, in two different setups.

Shall I go step by step for each version, like 3.2 in docker-compose and pull the image, then 3.3… ?

11

Did you run all the database upgrades in order? 3.1=>3.2, 3.2=>3.3, 3.3=>3.4? Yes, you stated that, but wanted to make sure. And I guess that the safest thing is indeed to do a minor by minor version update, checking that each version runs before applying the next upgrade.

Just updated two servers from 3.3.7 to 3.4.8, no problems.

12

yes, all db upgrade scritps were done.
Shall I got through each docker image version ?

13

Hey Ben,
you don’t have to download all SeaTable docker images. You can switch, for example, go from SeaTable 3.1 to 3.4 without a problem. But of course, you have to execute all the update scripts. So, here are the calls you have to do to upgrade from 3.1 to 3.4:

(I write this out of my head, there might be wrong paths… But it should give you the idea.)

# 1) pull the new image
docker pull seatable/seatable-enterprise:3.4.x

# 2) update docker-compose.yml (replace version number)
nano /opt/seatable/docker-compose.yml

# 3) start new SeaTable Server version
docker-compose down
docker-compose up -d

# 4) execute database updates
docker exec -it seatble bash
cd /opt/seatable/seatabe-server/latest/sql/mysql/upgrade
mysql -h$DB_HOST -p$DB_ROOT_PASSWD dtable_db < ./3.2/dtable.sql
mysql -h$DB_HOST -p$DB_ROOT_PASSWD dtable_db < ./3.3/dtable.sql
mysql -h$DB_HOST -p$DB_ROOT_PASSWD dtable_db < ./3.4/dtable.sql

# 5) start seatable
seatable.sh restart

# 6) leave docker container + leave shell
exit
exit
14

that’s what I made (but with DE edition)
and at the end, the webpage was not accessible at all.

15

The docker logs you posted show no signs of a problem. Please check the log files located at /opt/seatable/seatable-data/seatable/logs for any issues.

16

I’ve located the issue but I don’t know how to fix it.

In my dtable_web.log, I have a mention of a missing column : profile_profile.sms_2fa

I guess it’s related to an upgrade sql script, but I made all of them ?

Which step did I miss ?

Traceback (most recent call last):
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/utils/deprecation.py", line 116, in __call__
    response = self.process_request(request)
  File "/opt/seatable/seatable-server-latest/dtable-web/seahub/base/middleware.py", line 48, in process_request
    lang = Profile.objects.get_user_language(request.user.username)
  File "/opt/seatable/seatable-server-latest/dtable-web/seahub/profile/models.py", line 178, in get_user_language
    profile = self.get(user=username)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/opt/seatable/seatable-server-latest/dtable-web/thirdpart/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1054, "Unknown column 'profile_profile.sms_2fa' in 'field list'")
18

Step 2 is not in your manual. Upgrade Manual - SeaTable Admin Manual … Is there a different instruction page for enterprise?

19

the manual is correct and contains “my step 2”:

image
image875×490 33 KB

20

FIXED

i had to do all upgrade sql steps from 2.7 to latest.
I was running 3.1. That might be because I jumped over one or two versions before and I forgot to make eack intermediate sql request.

Anyway, it’s working. Sorry for the bug.

BTW, this upgrade process could really gain some smoothing

21

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.