[FIX] util/pg: get only common name with same type #359

sagu-odoo · 2025-11-29T05:11:16Z

This commit Add param type_check get the common columns info with same typcategory or same data type and skipp the column with different datatype and different type
all column name should be returned

Traceback (most recent call last):
  File "/home/odoo/src/odoo/19.0/odoo/service/server.py", line 1514, in preload_registries
    registry = Registry.new(dbname, update_module=update_module, install_modules=config['init'], upgrade_modules=config['update'], reinit_modules=config['reinit'])
  File "/home/odoo/src/odoo/19.0/odoo/tools/func.py", line 88, in locked
    return func(inst, *args, **kwargs)
  File "/home/odoo/src/odoo/19.0/odoo/orm/registry.py", line 186, in new
    load_modules(
  File "/home/odoo/src/odoo/19.0/odoo/modules/loading.py", line 493, in load_modules
    migrations.migrate_module(package, 'end')
  File "/home/odoo/src/odoo/19.0/odoo/modules/migration.py", line 220, in migrate_module
    exec_script(self.cr, installed_version, pyfile, pkg.name, stage, stageformat[stage] % version)
  File "/home/odoo/src/odoo/19.0/odoo/modules/migration.py", line 257, in exec_script
    mod.migrate(cr, installed_version)
  File "/tmp/tmp7yriz79e/migrations/hr/saas~18.4.1.1/end-migrate.py", line 87, in migrate
    cr.execute(query, [e[1] for e in required_default_values])
  File "/home/odoo/src/odoo/19.0/odoo/sql_db.py", line 426, in execute
    self._obj.execute(query, params)
psycopg2.errors.DatatypeMismatch: column "fondo_ahorro" is of type boolean but expression is of type double precision
LINE 16: ..."."distance_home_work_unit", "e"."employee_type", "e"."fondo...
                                                              ^
HINT:  You will need to rewrite or cast the expression.

select id,name,model,ttype,store from ir_model_fields where name='fondo_ahorro';
  id   |     name     |         model         |  ttype  | store
-------+--------------+-----------------------+---------+-------
 28361 | fondo_ahorro | hr.employee           | float   | t
 28687 | fondo_ahorro | calculo.liquidaciones | float   | t
 28379 | fondo_ahorro | hr.version            | boolean | t
(3 rows)

upg-3444635
opw-5260147

robodoo · 2025-11-29T05:11:18Z

sagu-odoo · 2025-11-29T06:28:07Z

upgradeci retry with always only base

aj-fuentes · 2025-12-01T07:53:13Z

You should output a warning when a common column has a different type and it is skipped.
IMO current patch is too specific for custom DBs, if you output a warning you would ensure that this mistake doesn't make us leave columns behind that should have been moved.
@KangOl ?

sagu-odoo · 2025-12-01T08:21:06Z

But getting info of column with respect of column_type make sense (if the defination is not matching). So, what is the meaning of moving them ?.

aj-fuentes · 2025-12-01T08:30:24Z

I don't understand your question. I will clarify my point instead.

You are silently skipping the moving of columns that have compatible types (think float and numeric or int). This could lead to unexpectedly now loosing some information that was previously moved correctly. If you put a warning line in the logs this will achieve a twofold goal. First we block the CI, standard data won't have a silent change of behavior. Second, on actual upgrade requests we leave a trace in the logs to debug if this issue affects clients.

sagu-odoo · 2025-12-01T11:06:28Z

Hello @aj-fuentes , what about this if we use this to categories data type and then find common column?.

EDIT:

test=# \d test_1
                   Table "public.test_1"
 Column |       Type       | Collation | Nullable | Default 
--------+------------------+-----------+----------+---------
 id     | double precision |           |          | 
 name   | text             |           |          | 

test=# \d test_2
                    Table "public.test_2"
 Column |       Type        | Collation | Nullable | Default 
--------+-------------------+-----------+----------+---------
 id     | integer           |           |          | 
 name   | character varying |           |          | 

test=# WITH _cols AS (
    SELECT  c.table_name, c.column_name, t.typcategory
      FROM information_schema.columns c
      JOIN pg_type t ON c.udt_name = t.typname
     WHERE table_schema = 'public'
       AND table_name IN ('test_1', 'test_2')
),                              
_common AS (
    SELECT column_name
      FROM _cols
     GROUP BY column_name, typcategory
    HAVING COUNT(table_name) = 2
) select * from _common;
 column_name 
-------------
 id
 name
(2 rows)

aj-fuentes · 2025-12-01T11:32:16Z

Yes, using typcategory seems better, but I would keep a warning.

If the types are incompatible (different category) -> warn we skipped them
It the types are different but compatible -> warn about the different type (we still return them in the list of common columns)

I think both cases may require attention when they happen. At the very least we are sure we get a trace in the logs.

src/util/pg.py

aj-fuentes

I think we can make the query much simpler and do the filtering in Python.

src/util/pg.py

aj-fuentes · 2025-12-02T08:04:58Z

upgradeci retry with always *hr* *mrp* in versions 18.0 19.0

aj-fuentes · 2025-12-02T11:05:19Z

upgradeci skip

aj-fuentes · 2025-12-02T11:08:02Z

upgradeci retry with always l10n_*_hr_* hr_* *_mrp in versions 18.0 19.0

aj-fuentes · 2025-12-02T11:08:59Z

upgradeci skip

aj-fuentes · 2025-12-02T11:09:30Z

upgradeci retry with l10n_*_hr_* hr_* mrp in versions 18.0 19.0

KangOl · 2025-12-02T13:55:43Z

src/util/pg.py

+            SELECT column_name,
+                   quote_ident(column_name),
+                   count(DISTINCT t.oid) = 1 AS same_type,
+                   count(DISTINCT t.typcategory) = 1 AS compatible


Suggested change

SELECT column_name,

quote_ident(column_name),

count(DISTINCT t.oid) = 1 AS same_type,

count(DISTINCT t.typcategory) = 1 AS compatible

SELECT column_name,

quote_ident(column_name),

count(DISTINCT t.oid) = 1 AS same_type,

count(DISTINCT t.typcategory) = 1 AS compatible

Suggested change

SELECT column_name,

quote_ident(column_name),

count(DISTINCT t.oid) = 1 AS same_type,

count(DISTINCT t.typcategory) = 1 AS compatible

SELECT c.column_name,

quote_ident(c.column_name),

count(DISTINCT t.oid) = 1 AS same_type,

count(DISTINCT t.typcategory) = 1 AS compatible

src/util/pg.py

KangOl · 2025-12-02T14:09:19Z

src/util/pg.py

    :param str table2: second table name whose columns are retrieved
    :param list(str) ignore: list of column names to ignore in the returning list
-    :return: a list of column names present in both tables
+    :return: a list of column names present in both tables with compatible data type


Maybe add an option to return all the common columns, regardless of their types?

Suggested change

:return: a list of column names present in both tables with compatible data type

:param bool type_check: only returns common columns with compatible data type

:return: a list of column names present in both table

done with suggested changes. could you please have a another look ?.

aj-fuentes · 2025-12-04T09:06:32Z

src/util/pg.py



-def get_common_columns(cr, table1, table2, ignore=("id",)):
+def get_common_columns(cr, table1, table2, ignore=("id",), type_check=False):


I think this should be True by default. @KangOl ?

Suggested change

def get_common_columns(cr, table1, table2, ignore=("id",), type_check=False):

def get_common_columns(cr, table1, table2, ignore=("id",), type_check=True):

Yes, indeed.

KangOl · 2025-12-04T09:42:01Z

src/util/pg.py

+               AND c.column_name != ALL(%s)
+          GROUP BY c.column_name
+            HAVING count(c.table_name) = 2
+             ORDER BY c.column_name


Correct indent

Suggested change

ORDER BY c.column_name

ORDER BY c.column_name

KangOl · 2025-12-04T10:48:36Z

src/util/pg.py

+    common_columns = [
+        name for name, _, same_type, compatible in cols_info if compatible or (type_check and not same_type)
+    ]
+    quoted_common_columns = [
+        qname for _, qname, same_type, compatible in cols_info if compatible or (type_check and not same_type)
+    ]


The filtering condition is wrong, it should be not type_check or compatible
To avoid duplication, we can use zip

Suggested change

common_columns = [

name for name, _, same_type, compatible in cols_info if compatible or (type_check and not same_type)

]

quoted_common_columns = [

qname for _, qname, same_type, compatible in cols_info if compatible or (type_check and not same_type)

]

common_columns, quoted_common_columns = zip(

*((name, qname) for name, qname, _, compatible in cols_info if not type_check or compatible)

)

but I have checked it was working the same but this one is straight forward. Thanks for the review

1. This commit Add param ``type_check`` get the common columns info with same typcategory or same data type and skipp the column with different datatype and different type 2. all column name should be returned ``` Traceback (most recent call last): File "/home/odoo/src/odoo/19.0/odoo/service/server.py", line 1514, in preload_registries registry = Registry.new(dbname, update_module=update_module, install_modules=config['init'], upgrade_modules=config['update'], reinit_modules=config['reinit']) File "/home/odoo/src/odoo/19.0/odoo/tools/func.py", line 88, in locked return func(inst, *args, **kwargs) File "/home/odoo/src/odoo/19.0/odoo/orm/registry.py", line 186, in new load_modules( File "/home/odoo/src/odoo/19.0/odoo/modules/loading.py", line 493, in load_modules migrations.migrate_module(package, 'end') File "/home/odoo/src/odoo/19.0/odoo/modules/migration.py", line 220, in migrate_module exec_script(self.cr, installed_version, pyfile, pkg.name, stage, stageformat[stage] % version) File "/home/odoo/src/odoo/19.0/odoo/modules/migration.py", line 257, in exec_script mod.migrate(cr, installed_version) File "/tmp/tmp7yriz79e/migrations/hr/saas~18.4.1.1/end-migrate.py", line 87, in migrate cr.execute(query, [e[1] for e in required_default_values]) File "/home/odoo/src/odoo/19.0/odoo/sql_db.py", line 426, in execute self._obj.execute(query, params) psycopg2.errors.DatatypeMismatch: column "fondo_ahorro" is of type boolean but expression is of type double precision LINE 16: ..."."distance_home_work_unit", "e"."employee_type", "e"."fondo... ^ HINT: You will need to rewrite or cast the expression. ``` ``` select id,name,model,ttype,store from ir_model_fields where name='fondo_ahorro'; id | name | model | ttype | store -------+--------------+-----------------------+---------+------- 28361 | fondo_ahorro | hr.employee | float | t 28687 | fondo_ahorro | calculo.liquidaciones | float | t 28379 | fondo_ahorro | hr.version | boolean | t (3 rows) ``` upg-3444635 opw-5260147

sagu-odoo requested review from a team and diagnoza November 29, 2025 05:37

sagu-odoo changed the title ~~[FIX] util/pg: get only common name with type~~ [FIX] util/pg: get only common name with same type Nov 29, 2025

sagu-odoo force-pushed the master-fix_get_column_name_by_type-sagu branch 2 times, most recently from 6e0d631 to 4462ff8 Compare December 1, 2025 13:30

sagu-odoo commented Dec 1, 2025

View reviewed changes

src/util/pg.py Outdated Show resolved Hide resolved

aj-fuentes reviewed Dec 1, 2025

View reviewed changes

src/util/pg.py Outdated Show resolved Hide resolved

src/util/pg.py Outdated Show resolved Hide resolved

src/util/pg.py Outdated Show resolved Hide resolved

src/util/pg.py Outdated Show resolved Hide resolved

sagu-odoo force-pushed the master-fix_get_column_name_by_type-sagu branch from 4462ff8 to 202e591 Compare December 2, 2025 04:33

aj-fuentes reviewed Dec 2, 2025

View reviewed changes

src/util/pg.py Outdated Show resolved Hide resolved

sagu-odoo force-pushed the master-fix_get_column_name_by_type-sagu branch from 202e591 to a05a92c Compare December 2, 2025 08:16

KangOl requested changes Dec 2, 2025

View reviewed changes

sagu-odoo force-pushed the master-fix_get_column_name_by_type-sagu branch 5 times, most recently from 8ca3c17 to b252009 Compare December 3, 2025 10:59

aj-fuentes reviewed Dec 4, 2025

View reviewed changes

KangOl reviewed Dec 4, 2025

View reviewed changes

sagu-odoo force-pushed the master-fix_get_column_name_by_type-sagu branch from b252009 to e321d93 Compare December 4, 2025 09:44

KangOl reviewed Dec 4, 2025

View reviewed changes

sagu-odoo force-pushed the master-fix_get_column_name_by_type-sagu branch from e321d93 to 54bf0e6 Compare December 4, 2025 11:50

	:return: a list of column names present in both tables with compatible data type
	:param bool type_check: only returns common columns with compatible data type
	:return: a list of column names present in both table



		def get_common_columns(cr, table1, table2, ignore=("id",)):
		def get_common_columns(cr, table1, table2, ignore=("id",), type_check=False):

[FIX] util/pg: get only common name with same type #359

Are you sure you want to change the base?

[FIX] util/pg: get only common name with same type #359

Uh oh!

Conversation

sagu-odoo commented Nov 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robodoo commented Nov 29, 2025

Uh oh!

sagu-odoo commented Nov 29, 2025

Uh oh!

aj-fuentes commented Dec 1, 2025

Uh oh!

sagu-odoo commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aj-fuentes commented Dec 1, 2025

Uh oh!

sagu-odoo commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aj-fuentes commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

aj-fuentes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aj-fuentes commented Dec 2, 2025

Uh oh!

aj-fuentes commented Dec 2, 2025

Uh oh!

aj-fuentes commented Dec 2, 2025

Uh oh!

aj-fuentes commented Dec 2, 2025

Uh oh!

aj-fuentes commented Dec 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sagu-odoo commented Nov 29, 2025 •

edited

Loading

sagu-odoo commented Dec 1, 2025 •

edited

Loading

sagu-odoo commented Dec 1, 2025 •

edited

Loading

aj-fuentes commented Dec 1, 2025 •

edited

Loading