psycopg2 copy_from raises asynchronous ProgrammingError with sync connection in celery task

Question

psycopg2 copy_from raises asynchronous ProgrammingError with sync connection in celery task

asynchronous celery psycopg2 python sqlbulkcopy

Question

My problem is I am facing

ProgrammingError: copy_from cannot be used with an asynchronous callback.

while trying to copy_from without async connection. This must be stated, I am creating connection from a celery task. Can someone give me a clue how sqlalchemy or celery or whatever forces my psycopg2 connection behave like async?

conn = psycopg2.connect(con_string)
conn.async
>>0
cur = conn.cursor()    
data = BytesIO()
data.write('\n'.join(['Tom\tJenkins\t37',
                  'Madonna\t\N\t45',
                  'Federico\tDi Gregorio\t\N']))
data.seek(0)
curs.copy_from(data, 'test_copy')

Phil Matiash

View more on Stack Overflow

Catherine Devlin · Accepted Answer · 6/3/2016 5:25:29 AM

We were facing this error in pgcli; in that case it turned out that a wait_callback was making the connection behave as though it were asynchronous from psycopg2's point of view. This helped:

from contextlib import contextmanager

@contextmanager
def _paused_thread():
    try:
        thread = psycopg2.extensions.get_wait_callback()
        psycopg2.extensions.set_wait_callback(None)
        yield
    finally:
        psycopg2.extensions.set_wait_callback(thread)

with _paused_thread():
    cursor.copy_expert('copy mytable to STDOUT', file)

psycopg2 copy_from raises asynchronous ProgrammingError with sync connection in celery task

Question

Fastest Entity Framework Extensions

Popular Answer

Performance

Expression Evaluator