[DOC-860] [DOC-832] Add details of YSQL table inheritance support (yugabyte#27830)

* [DOC-860] Add details of YSQL table inheritance support

* Also update cat cache docs for incremental cache invalidation.

* rename to right extension

* Minor edits

* add incremental inval gflag

* Minor fixup

* edits

* Copy changes

* format

---------

Co-authored-by: Dwight Hodge <ghodge@yugabyte.com>

Author: iSignal

docs/content/preview/best-practices-operations/ysql-catalog-cache-tuning-guide.md

@@ -439,9 +439,11 @@ You can customize this tradeoff to control preloading of entries into PostgreSQL
 
 ### Default behavior
 
-By default, YSQL backends do not preload any catalog entries, except right after a schema change (DDL).
+By default, YSQL backends do not preload any catalog entries.
 
-On most schema changes, running backends discard their catalog cache, and are then refreshed from either the YB-Master leader or an intermediate response cache on the local YB-TServer. This refresh causes a latency hit on running queries while they wait for this process to complete. There is also a memory increase because the cache is now preloaded with all rows of these catalog tables (as opposed to just the actively used entries that it had before).
+On most schema changes, running backends invalidate their caches incrementally and refresh such invalidated entries directly from the leader yb-master. This can result in a slight effect on the latency and memory usage of running queries but it should not be significant in most cases.
+
+In rare cases, a full catalog refresh may be needed (this typically happens through an intermediate response cache on the local YB-TServer to minimize load on the yb-master). Such a rare full refresh may cause a more pronounced latency effect on running queries as well as a noticeable spike in memory consumption. If you encounter such cases commonly, report these at {{<issue 23785>}}.
 
 ## Problem scenarios and recommendations
 
@@ -472,11 +474,9 @@ To confirm that catalog caching is the cause of this, see [Confirm that catalog
 - [Use connection pooling](#connection-pooling) to reuse existing connections.
 - [Preload additional system tables](#preload-additional-system-tables).
 
-### Memory spikes on PostgreSQL backends or out of memory (OOM) events
-
-On the flip side, automatic preloading of caches after a DDL change may cause memory spikes on PostgreSQL backends or out of memory (OOM) events.
+### High memory usage on PostgreSQL backends
 
-To confirm that catalog caching is the cause of this, correlate the time when DDLs were run (Write RPCs on YB-Master) to the time of the OOM event or a spike in PostgreSQL RSS metrics.
+On the flip side, automatic preloading of caches may result in higher memory usage of PostgreSQL backends than desired.
 
 **Possible solution**
 
@@ -521,7 +521,7 @@ For example:
 
 ### Minimal catalog cache preloading {#minimal-catalog-cache-preloading}
 
-When enabled, only a small subset of the catalog cache entries is preloaded. This reduces the memory spike that results, but can increase the warm up time for queries after a DDL change, as well as the initial query latency when [additional tables are preloaded](#preload-additional-system-tables).
+When enabled, only a small subset of the catalog cache entries is preloaded. This reduces the memory spike that results, but can increase the initial query latency when [additional tables are preloaded](#preload-additional-system-tables).
 
 To enable minimal catalog cache preloading, set the [YB-TServer flag](../../reference/configuration/yb-tserver/#catalog-flags) `--ysql_minimal_catalog_caches_preload=true`.
 

docs/content/preview/explore/ysql-language-features/advanced-features/_index.md

@@ -82,3 +82,11 @@ Snapshot synchronization ensures that two or more independent, concurrently runn
 {{<lead link="snapshot-synchronization/">}}
 To learn how different transactions can maintain a consistent view of the data, see [Synchronize snapshots](snapshot-synchronization/)
 {{</lead>}}
+
+## Inheritance
+
+Table inheritance lets you create child tables that automatically share columns and constraints from a parent table.
+
+{{<lead link="inheritance/">}}
+To see how to create tables that inherit from a parent table, see [Inheritance](inheritance/)
+{{</lead>}}

docs/content/preview/explore/ysql-language-features/advanced-features/inheritance.md

@@ -0,0 +1,103 @@
+---
+title: Table inheritance
+linkTitle: Table inheritance
+description: Table inheritance in YSQL
+menu:
+  preview:
+    identifier: advanced-features-inheritance
+    parent: advanced-features
+    weight: 900
+tags:
+  feature: tech-preview
+type: docs
+---
+
+YSQL supports table inheritance using the INHERITS keyword, which is a [PostgreSQL feature](https://www.postgresql.org/docs/current/ddl-inherit.html) that allows you to create child tables that inherit columns and certain constraints from one or more parent tables.
+
+## Example
+
+To illustrate with a basic example, create the following tables:
+
+```sql
+-- Columns common to all account types
+CREATE TABLE accounts (
+    account_id INTEGER PRIMARY KEY,
+    balance NUMERIC NOT NULL CHECK (balance >= 0),
+    profit NUMERIC DEFAULT 0
+);
+
+-- Child table for investment accounts
+CREATE TABLE investment_accounts (
+    investment_type TEXT NOT NULL CHECK (investment_type IN ('stocks', 'bonds', 'funds')),
+    CHECK (balance >= 5000),
+    PRIMARY KEY (account_id, investment_type) 
+) INHERITS (accounts);
+
+-- Child table for savings accounts
+CREATE TABLE savings_accounts (
+    interest_rate NUMERIC NOT NULL CHECK (interest_rate >= 0 AND interest_rate <= 0.1),
+    CHECK (balance >= 100),
+    PRIMARY KEY (account_id) 
+) INHERITS (accounts);
+```
+
+```sql
+testdb=# \d investment_accounts
+```
+
+```output
+             Table "public.investment_accounts"
+     Column      |  Type   | Collation | Nullable | Default
+-----------------+---------+-----------+----------+---------
+ account_id      | integer |           | not null |
+ balance         | numeric |           | not null |
+ profit          | numeric |           |          | 0
+ investment_type | text    |           | not null |
+Indexes:
+    "investment_accounts_pkey" PRIMARY KEY, lsm (account_id HASH, investment_type ASC)
+Check constraints:
+    "accounts_balance_check" CHECK (balance >= 0::numeric)
+    "investment_accounts_balance_check" CHECK (balance >= 5000::numeric)
+    "investment_accounts_investment_type_check" CHECK (investment_type = ANY (ARRAY['stocks'::text, 'bonds'::text, 'funds'::text]))
+Inherits: accounts
+```
+
+This schema allows for certain queries to be performed over all accounts while still preserving features unique to each account type. For example:
+
+```sql
+SELECT SUM(balance) FROM accounts WHERE account_id = 10;
+```
+
+Any columns added to or dropped from the parent `accounts` table are propagated to child tables so that such queries on the parent accounts table are always well formed.
+
+However, there are certain caveats to keep in mind:
+
+1. The parent table `accounts` may have its own rows that are not part of any child tables.
+1. The primary key for `account_id` on the parent `accounts` table does not propagate to children and has to be redefined for each child table. This is also the behavior for foreign key constraints and non-primary key unique constaints. You need to take special care to maintain such constraints across parent-child hierarchies.
+
+Table inheritance can lead to complex hierarchies similar to class inheritance in object-oriented programming because a specific table can inherit from multiple parent tables and can itself be a parent table for other child tables.
+
+### Queries and updates on data
+
+SELECT and UPDATE queries on the parent table operate on a union of the parent and all child tables in the hierarchy. To restrict queries to just the specific table, use the ONLY keyword:
+
+```sql
+SELECT SUM(balance) FROM ONLY accounts WHERE account_id = 10;
+
+UPDATE ONLY accounts SET balance = balance + 100 WHERE account_id = 1;
+```
+
+## Schema changes
+
+1. Adding columns to or dropping columns from  or altering columns on the parent table propagates to all children in the hierarchy. The behavior of such operations can be more complex when a child table has a column of the same name and type before establishing inheritance or when a child table inherits from multiple parent tables with slightly differing definitions of a column (for example, NULL vs NOT NULL constraints on the column). For more details on the allowed differences in column definitions and the resolution of such differences in inheritance, [consult the PostgreSQL documentation](https://www.postgresql.org/docs/current/ddl-inherit.html).
+2. Adding or dropping certain kinds of constraints propagates to all children in the hierarchy. The exceptions are primary key constrains, unique constraints and foreign key constraints. For more details, consult the [PostgreSQL documentation](https://www.postgresql.org/docs/current/ddl-inherit.html).
+3. For certain schema changes, the `ONLY` keyword can be used to restrict the schema change to just the parent table. For example, `ALTER TABLE ONLY accounts DROP COLUMN profit` drops the column from the parent table alone while leaving it on the child tables.
+
+## Limitations
+
+Table inheritance is {{<tags/feature/tp idea="2158">}} - report any problems at using issue {{<issue 5956>}}.
+
+- Dropping or adding a column to a parent fails when "local" column on child table exists. (Issue {{<issue 26094>}})
+- Crash while obtaining a row lock on a parent inheritance table with a child file_fdw table. (Issue {{<issue 27105>}})
+
+For an up-to-date list, see issue {{<issue 5956>}}.

docs/content/preview/reference/configuration/yb-master.md

@@ -1032,6 +1032,12 @@ expensive when the number of YB-TServers, or the number of databases goes up.
 
 {{< /note >}}
 
+##### --ysql_yb_enable_invalidation_messages
+
+Enables YSQL backends to generate and consume invalidation messages incrementally for schema changes. When enabled (true), invalidation messages are propagated via the `pg_yb_invalidation_messages` per-database catalog table. Details of the invalidation messages generated by a DDL are also logged when [ysql_log_min_messages](../yb-tserver/#ysql-log-min-messages) is set to `DEBUG1` or when `yb_debug_log_catcache_events` is set to true. When disabled, schema changes cause a full catalog cache refresh on existing backends, which can result in a latency and memory spike on existing YSQL backends.
+
+Default: `true`
+
 ## Advisory lock flags
 
 Support for advisory locks is {{<tags/feature/tp idea="812">}}.

docs/content/preview/reference/configuration/yb-tserver.md

@@ -1485,6 +1485,16 @@ Specifies the threshold (in bytes) beyond which catalog tuples will get compress
 
 To minimize performance impact when enabling this flag, set it to 2KB or higher.
 
+##### --ysql_yb_enable_invalidation_messages
+
+{{% tags/wrap %}}
+
+
+Default: `true`
+{{% /tags/wrap %}}
+
+Enables YSQL backends to generate and consume invalidation messages incrementally for schema changes. When enabled (true), invalidation messages are propagated via the `pg_yb_invalidation_messages` per-database catalog table. Details of the invalidation messages generated by a DDL are also logged when [ysql_log_min_messages](#ysql-log-min-messages) is set to `DEBUG1` or when `yb_debug_log_catcache_events` is set to true. When disabled, schema changes cause a full catalog cache refresh on existing backends, which can result in a latency and memory spike on existing YSQL backends.
+
 ## Performance Tuning
 
 ### Memory division flags

docs/content/v2025.1/best-practices-operations/ysql-catalog-cache-tuning-guide.md

@@ -439,9 +439,11 @@ You can customize this tradeoff to control preloading of entries into PostgreSQL
 
 ### Default behavior
 
-By default, YSQL backends do not preload any catalog entries, except right after a schema change (DDL).
+By default, YSQL backends do not preload any catalog entries.
 
-On most schema changes, running backends discard their catalog cache, and are then refreshed from either the YB-Master leader or an intermediate response cache on the local YB-TServer. This refresh causes a latency hit on running queries while they wait for this process to complete. There is also a memory increase because the cache is now preloaded with all rows of these catalog tables (as opposed to just the actively used entries that it had before).
+On most schema changes, running backends invalidate their caches incrementally and refresh such invalidated entries directly from the leader yb-master. This can result in a slight effect on the latency and memory usage of running queries but it should not be significant in most cases.
+
+In rare cases, a full catalog refresh may be needed (this typically happens through an intermediate response cache on the local YB-TServer to minimize load on the yb-master). Such a rare full refresh may cause a more pronounced latency effect on running queries as well as a noticeable spike in memory consumption. If you encounter such cases commonly, report these at {{<issue 23785>}}.
 
 ## Problem scenarios and recommendations
 
@@ -472,11 +474,9 @@ To confirm that catalog caching is the cause of this, see [Confirm that catalog
 - [Use connection pooling](#connection-pooling) to reuse existing connections.
 - [Preload additional system tables](#preload-additional-system-tables).
 
-### Memory spikes on PostgreSQL backends or out of memory (OOM) events
-
-On the flip side, automatic preloading of caches after a DDL change may cause memory spikes on PostgreSQL backends or out of memory (OOM) events.
+### High memory usage on PostgreSQL backends
 
-To confirm that catalog caching is the cause of this, correlate the time when DDLs were run (Write RPCs on YB-Master) to the time of the OOM event or a spike in PostgreSQL RSS metrics.
+On the flip side, automatic preloading of caches may result in higher memory usage of PostgreSQL backends than desired.
 
 **Possible solution**
 
@@ -521,7 +521,7 @@ For example:
 
 ### Minimal catalog cache preloading {#minimal-catalog-cache-preloading}
 
-When enabled, only a small subset of the catalog cache entries is preloaded. This reduces the memory spike that results, but can increase the warm up time for queries after a DDL change, as well as the initial query latency when [additional tables are preloaded](#preload-additional-system-tables).
+When enabled, only a small subset of the catalog cache entries is preloaded. This reduces the memory spike that results, but can increase the initial query latency when [additional tables are preloaded](#preload-additional-system-tables).
 
 To enable minimal catalog cache preloading, set the [YB-TServer flag](../../reference/configuration/yb-tserver/#catalog-flags) `--ysql_minimal_catalog_caches_preload=true`.
 

docs/content/v2025.1/explore/ysql-language-features/advanced-features/_index.md

@@ -82,3 +82,11 @@ Snapshot synchronization ensures that two or more independent, concurrently runn
 {{<lead link="snapshot-synchronization/">}}
 To learn how different transactions can maintain a consistent view of the data, see [Synchronize snapshots](snapshot-synchronization/)
 {{</lead>}}
+
+## Inheritance
+
+Table inheritance lets you create child tables that automatically share columns and constraints from a parent table.
+
+{{<lead link="inheritance/">}}
+To see how to create tables that inherit from a parent table, see [Inheritance](inheritance/)
+{{</lead>}}

docs/content/v2025.1/explore/ysql-language-features/advanced-features/inheritance.md

@@ -0,0 +1,103 @@
+---
+title: Table inheritance
+linkTitle: Table inheritance
+description: Table inheritance in YSQL
+menu:
+  v2025.1:
+    identifier: advanced-features-inheritance
+    parent: advanced-features
+    weight: 900
+tags:
+  feature: tech-preview
+type: docs
+---
+
+YSQL supports table inheritance using the INHERITS keyword, which is a [PostgreSQL feature](https://www.postgresql.org/docs/current/ddl-inherit.html) that allows you to create child tables that inherit columns and certain constraints from one or more parent tables.
+
+## Example
+
+To illustrate with a basic example, create the following tables:
+
+```sql
+-- Columns common to all account types
+CREATE TABLE accounts (
+    account_id INTEGER PRIMARY KEY,
+    balance NUMERIC NOT NULL CHECK (balance >= 0),
+    profit NUMERIC DEFAULT 0
+);
+
+-- Child table for investment accounts
+CREATE TABLE investment_accounts (
+    investment_type TEXT NOT NULL CHECK (investment_type IN ('stocks', 'bonds', 'funds')),
+    CHECK (balance >= 5000),
+    PRIMARY KEY (account_id, investment_type) 
+) INHERITS (accounts);
+
+-- Child table for savings accounts
+CREATE TABLE savings_accounts (
+    interest_rate NUMERIC NOT NULL CHECK (interest_rate >= 0 AND interest_rate <= 0.1),
+    CHECK (balance >= 100),
+    PRIMARY KEY (account_id) 
+) INHERITS (accounts);
+```
+
+```sql
+testdb=# \d investment_accounts
+```
+
+```output
+             Table "public.investment_accounts"
+     Column      |  Type   | Collation | Nullable | Default
+-----------------+---------+-----------+----------+---------
+ account_id      | integer |           | not null |
+ balance         | numeric |           | not null |
+ profit          | numeric |           |          | 0
+ investment_type | text    |           | not null |
+Indexes:
+    "investment_accounts_pkey" PRIMARY KEY, lsm (account_id HASH, investment_type ASC)
+Check constraints:
+    "accounts_balance_check" CHECK (balance >= 0::numeric)
+    "investment_accounts_balance_check" CHECK (balance >= 5000::numeric)
+    "investment_accounts_investment_type_check" CHECK (investment_type = ANY (ARRAY['stocks'::text, 'bonds'::text, 'funds'::text]))
+Inherits: accounts
+```
+
+This schema allows for certain queries to be performed over all accounts while still preserving features unique to each account type. For example:
+
+```sql
+SELECT SUM(balance) FROM accounts WHERE account_id = 10;
+```
+
+Any columns added to or dropped from the parent `accounts` table are propagated to child tables so that such queries on the parent accounts table are always well formed.
+
+However, there are certain caveats to keep in mind:
+
+1. The parent table `accounts` may have its own rows that are not part of any child tables.
+1. The primary key for `account_id` on the parent `accounts` table does not propagate to children and has to be redefined for each child table. This is also the behavior for foreign key constraints and non-primary key unique constaints. You need to take special care to maintain such constraints across parent-child hierarchies.
+
+Table inheritance can lead to complex hierarchies similar to class inheritance in object-oriented programming because a specific table can inherit from multiple parent tables and can itself be a parent table for other child tables.
+
+### Queries and updates on data
+
+SELECT and UPDATE queries on the parent table operate on a union of the parent and all child tables in the hierarchy. To restrict queries to just the specific table, use the ONLY keyword:
+
+```sql
+SELECT SUM(balance) FROM ONLY accounts WHERE account_id = 10;
+
+UPDATE ONLY accounts SET balance = balance + 100 WHERE account_id = 1;
+```
+
+## Schema changes
+
+1. Adding columns to or dropping columns from  or altering columns on the parent table propagates to all children in the hierarchy. The behavior of such operations can be more complex when a child table has a column of the same name and type before establishing inheritance or when a child table inherits from multiple parent tables with slightly differing definitions of a column (for example, NULL vs NOT NULL constraints on the column). For more details on the allowed differences in column definitions and the resolution of such differences in inheritance, [consult the PostgreSQL documentation](https://www.postgresql.org/docs/current/ddl-inherit.html).
+2. Adding or dropping certain kinds of constraints propagates to all children in the hierarchy. The exceptions are primary key constrains, unique constraints and foreign key constraints. For more details, consult the [PostgreSQL documentation](https://www.postgresql.org/docs/current/ddl-inherit.html).
+3. For certain schema changes, the `ONLY` keyword can be used to restrict the schema change to just the parent table. For example, `ALTER TABLE ONLY accounts DROP COLUMN profit` drops the column from the parent table alone while leaving it on the child tables.
+
+## Limitations
+
+Table inheritance is {{<tags/feature/tp idea="2158">}} - report any problems at using issue {{<issue 5956>}}.
+
+- Dropping or adding a column to a parent fails when "local" column on child table exists. (Issue {{<issue 26094>}})
+- Crash while obtaining a row lock on a parent inheritance table with a child file_fdw table. (Issue {{<issue 27105>}})
+
+For an up-to-date list, see issue {{<issue 5956>}}.