Hi,
I use Python runner in a self-hosted context. For my particular context, I needed to run seatable-python-runner in a dockerized manner (mainly related to orchestrator). I want to share here my problem and the way I solved it to benefit to all.
So I was confronted to some problems:
- seatable-python-runner needs to run docker commands (docker run): solved by mounting the docker socker into the container.
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- Second problem : seatable-python-runner uses volumes to mount both input(context) and script to execute. This caused a problem since the container running docker commands is NOT docker server so volume mounts don’t work. So I managed to transmits script and input thru environment variable in base64.
This is the resulting docker image :
FROM alpine:latest
RUN apk add unzip py3-pip uwsgi uwsgi-python3 python3-dev docker patch && \
wget -O /tmp/seatable-python-runner.zip https://github.com/seatable/seatable-admin-docs/releases/download/seatable-python-runner-2.0.1/seatable-python-runner-2.0.1.zip && \
cd /opt && \
unzip /tmp/seatable-python-runner.zip && rm /tmp/seatable-python-runner.zip
# Installing python deps & build uwsgi
RUN set -e; \
apk add --no-cache --virtual .build-deps \
gcc \
libc-dev \
linux-headers \
; \
pip3 install -r /opt/seatable-python-runner/server_requirements.txt; \
apk del .build-deps;
COPY function.py.diff /tmp/function.py.diff
RUN cd /opt/seatable-python-runner/ && \
sh ./init.sh && \
patch function.py /tmp/function.py.diff && \
rm -f /tmp/function.py.diff
# Runtime image
ENV IMAGE=registry.adhocratie.com/seatable-python-runner-container:latest
WORKDIR /opt/seatable-python-runner/
CMD ["/usr/bin/uwsgi", "--ini", "conf/seatable_python_runner.ini"]
And this the required patch to avoid volume mounts :
--- function.py.orig 2022-01-19 14:00:11.454240210 +0100
+++ function.py.new 2022-01-19 13:30:37.351011739 +0100
@@ -10,6 +10,7 @@
from uuid import uuid4
from flask import Flask, request, make_response
+from base64 import b64encode
import settings
@@ -100,19 +101,16 @@
container_name = 'python-runner' + dir_id
file_name = 'index.py'
os.makedirs(dir_id)
- # save script
- with open(os.path.join(dir_id, file_name), 'wb') as f:
- f.write(resp.content)
+ # save script to env in b64
+ script_content = resp.content
+ env['SCRIPT_CONTENT_B64'] = str(b64encode(script_content), 'utf-8')
+ env['SCRIPT_INPUT_B64'] = str(b64encode(bytes(context_data, 'utf-8')), 'utf-8')
# save env
env_file = os.path.join(dir_id, 'env.list')
with open(env_file, 'w') as f:
if env:
envs = '\n'.join(['%s=%s' % (key, value) for key, value in env.items()])
f.write(envs)
- # save arguments as file to stdin
- with open(os.path.join(dir_id, 'input'), 'w') as f:
- if context_data:
- f.write(context_data)
return_code, output = None, None # init output
@@ -120,8 +118,7 @@
scripts_path = os.path.join(os.getcwd(), dir_id)
# mount volumes and set env
command = ['docker', 'run', '--name', container_name,
- '--env-file', env_file,
- '-v', '{}:/scripts'.format(scripts_path)]
+ '--env-file', env_file, '--rm', '-i']
# limit memory and cpus
if settings.CONTAINER_MEMORY:
command.append('--memory={}'.format(settings.CONTAINER_MEMORY))
@@ -149,14 +146,10 @@
send_to_scheduler(False, None, None, None, data)
return
else:
- output_file_path = os.path.join(dir_id, 'output')
- if os.path.isfile(output_file_path):
- with open(output_file_path, 'r') as f:
- output = f.read()
+ output = result.stdout.decode()
return_code = result.returncode
if return_code == 137: # OOM
output += 'out-of-memory(OOM) error!\n'
- output += result.stdout.decode()
finally:
spend_time = time.time() - start_at
try:
And of course, we need to update the runner image also :
FROM seatable/python-runner:latest
ENTRYPOINT ["/bin/sh", "-c", "echo $SCRIPT_CONTENT_B64 | base64 -d > index.py ;echo $SCRIPT_INPUT_B64 | base64 -d | python index.py"]
This works fully on my environment, and I think it’s a better approach for many reasons (including security ones).
I Hope it helps
