Thus, with 32 shards, an entire month of fares is available with one query per shard. Each night, the fares may be changed as the result of a proactive search.The search will really only change one thing -- the price, but will not change other aspects of the fare.
Of course, workers must have trusted access to the coordinator.
Citus changes the DDL for CREATE TABLE to allow for any number of shards to be placed on workers.
One solution is to break the data up into sub-tables.
An algorithm could select which sub-table to query, or query many of them if required.
Here is how to defining a data model and produce a method that can ingest and search at scale.
The elements of an airline fare are as follows: Note that the model omits several details, such as the exact airline and times.
The other is the worker, which holds the data and executes the queries.
A worker is fundamentally a Postgre SQL server, thus all server tuning and management techniques apply.
Ideally, that scaling should take place with no application changes and no downtime.