Is our use case too much for SeaTable?

[joe-humcrush]

Our Setup:

• SeaTable Cloud (Enterprise, 3 users)

Problem

• We are small team (only 1 full time developer) building a music application.
• The primary object (table) in our system is a “Song”. Songs are related to artists, albums, etc.
• Our editorial team (3 users) will ultimately need to manage roughly 200,000 songs. (Currently we have less than 20,000.)

We started using Airtable, as it was a solution I was very familiar with, but quickly realized it couldn’t scale as we needed it. I was quite excited when I discovered SeaTable and saw that the Enterprise tier had no limits to record size, and was very reasonably priced. But, we’re just getting started and we’re already running into scalability issues again.

• Views onto our Songs table are already becoming very slow with +12,000 rows. Now, upon closer reading, it looks like there is indeed an upper max to the number of records in a single table.
• We need to programmatically add metadata to these songs via the API. It takes roughly 50 API calls per song.
• We have no problem sticking to the per minute rate limit, but there is no way we can work within the 5,000 total calls per day limit.

Questions

• Can we have roughly 200,000 records in one table? Will the UI be performant?
• If we limited ourselves to processing one song a minute, that would add up to 72,000 API calls a day. (This is a one-time need - after the bulk of our songs are processed, our normal daily use would probably fit within the 5,000 calls limit.) Is it possible to temporarily push the daily limit?
• The “dedicated” cloud option seems to remove this limit, but requires 100 users. Pricing isn’t mentioned, but I’m guessing it would be prohibitive. Would it even meet our need?
• If we were to move to self-hosted and could set our own API limits, would SeaTable’s UI be performant with this many rows in a table?

Thanks in advance for any advise.

[rdb]

Interesting use case.

Is SeaTable suitable for your use case? I would say “yes”, but it depends on the details.

@joe-humcrush

• Can we have roughly 200,000 records in one table? Will the UI be performant?

SeaTable Cloud, our public cloud, has a limit of 100k rows per base. In SeaTable Server (self-hosted) and SeaTable Dedicated (private cloud), this limit can be increased.

Additionally, the enterprise plan of SeaTable Cloud supports the big data backend. The big data backend allows you to store millions of rows in one base.

The difference between storing records in the base vs. in the big data backend:

• Records in the base are loaded into memory when you open the base. This allows real-time collaboration. When you have lots of records in your base and/or lots of columns, the loading time may become substantial.
• Records in the big data backend are loaded on demand. Real-time collaboration is not supported.

Important: You can add/modify/delete records in the base and the big data backend in SeaTable’s base editor.

If you access the data via API, it does not really make a difference if you store the records in the base or in big data.

@joe-humcrush

If we limited ourselves to processing one song a minute, that would add up to 72,000 API calls a day. (This is a one-time need - after the bulk of our songs are processed, our normal daily use would probably fit within the 5,000 calls limit.) Is it possible to temporarily push the daily limit?

72k API calls a day would be an undue load on a public system - not temporarily, let alone permanently.

In keeping with what I said above about SeaTable Server and SeaTable Dedicated, the limits on private systems can be increased significantly.

This said, I suggest you try to use batch calls. In doing so, you should be able to reduce the number of API calls per song significantly.

@joe-humcrush

The “dedicated” cloud option seems to remove this limit, but requires 100 users. Pricing isn’t mentioned, but I’m guessing it would be prohibitive. Would it even meet our need?

First off, based on what I know about your use case, I’d argue that you don’t need a dedicated system. Please send a trial request for SeaTable Cloud Enterprise. (You can do this via SeaTable Cloud’s team administration > Subscription > “Request a trial”.) Then you can run a POC using the big data backend and optimize your meta data handling using batch API calls.

Secondly, SeaTable Dedicated allows significantly higher row and API limits. So, yes, it could meet your needs if SeaTable Cloud Enterprise does not. I can reassure you that pricing is not prohibitive. Yes, it is our premium product and hence the most expensive product, but it is a steal compare to Airtable.

@joe-humcrush

If we were to move to self-hosted and could set our own API limits, would SeaTable’s UI be performant with this many rows in a table?

Once a base has been loaded, the data sits in your machine’s memory. As a consequence, you should be able to work in your base smoothly (provided you have an up-to-date processor and sufficient RAM).

(Could you tell me more about the performance issues that you observed? Can you also tell me a bit more about the data that you store in the base - how many columns, what type of columns?)

If you store your data the big data backend (or an app for that matter), the performance should be great since you do not work on the full dataset.

[joe-humcrush]

First off, based on what I know about your use case, I’d argue that you don’t need a dedicated system. Please send a trial request for SeaTable Cloud Enterprise

We are on Cloud Enterprise.

Once a base has been loaded, the data sits in your machine’s memory. As a consequence, you should be able to work in your base smoothly (provided you have an up-to-date processor and sufficient RAM).

(Could you tell me more about the performance issues that you observed? Can you also tell me a bit more about the data that you store in the base - how many columns, what type of columns?)

We have a table with < 13,000 rows. It takes many seconds to initially load. If that delay scales linearly, at 100,000 or 200,000 rows, the wait would be unworkable.

72k API calls a day would be an undue load on a public system - not temporarily, let alone permanently.

In keeping with what I said above about SeaTable Server and SeaTable Dedicated, the limits on private systems can be increased significantly.

This said, I suggest you try to use batch calls. In doing so, you should be able to reduce the number of API calls per song significantly.

I’m fairly confident given the nature of what we’re doing, we have the number of API calls pretty optimized. We might be able to marginally improve, but we could never get to the 5,000 a day limit.

Who can I reach out to to get pricing for Dedicated hosting?

[rdb]

@joe-humcrush

We have a table with < 13,000 rows. It takes many seconds to initially load. If that delay scales linearly, at 100,000 or 200,000 rows, the wait would be unworkable.

The download time is dependent upon the size of your base / the amount of data you store in your base. SeaTable uses compression, but proportionality between download time and download time is a workable, reasonable assumption.

Have you tried the big data backend?

@joe-humcrush

I’m fairly confident given the nature of what we’re doing, we have the number of API calls pretty optimized.

All I know is that you “programmatically add metadata to these songs via the API.” For this job, 50 API calls sounds like a lot.

Maybe you are doing much much which I don’t know. If you share a bit more, we may be able to provide valuable insights.

@joe-humcrush

Who can I reach out to to get pricing for Dedicated hosting?

Please contact sales@seatable.io

[joe-humcrush]

Thanks for the response.

We’re currently testing SeaTable Server on a Digital Ocean droplet, and experimenting with the Big Data store. I think the combination of being on our own server and moving processed records into Big Data will likely work for us.

If a record is in Big Data, can we establish a link to it when creating or modifying an “active” record?

[joe-humcrush]

@rdb

All I know is that you “programmatically add metadata to these songs via the API.” For this job, 50 API calls sounds like a lot.

Maybe you are doing much much which I don’t know. If you share a bit more, we may be able to provide valuable insights.

We’re aggregating data from 10+ external sources and creating or updating records in at least 4 related tables.

[rdb]

@joe-humcrush

We’re aggregating data from 10+ external sources and creating or updating records in at least 4 related tables.

Just to make sure: Calling the API endpoints of external sources does NOT count against SeaTable’s limit. The limit is for calling SeaTable’s API endpoints.

Assuming that you update records in 5 related tables and you run batch updates, the API calls relevant for the daily limit are six: 1x authentication + 5 separate batch updates.