[dscng-dev] DSCng became very slow in processing data.

Valentin Mitoiu valentin.mitoiu at univie.ac.at
Tue Jul 15 10:44:04 CEST 2014


Hi guys,

In the last month dscng became slower and slower in processing messages.
I enabled logging in postgres to track the slow queries and these seem 
to be the ones which are slowing the inserting

-----------
postgres[22278]: [53-1] 2014-07-15 10:37:34 CEST LOG:  duration: 
5729.992 ms  statement: SELECT original,remap
dscng postgres[22278]: [53-2] #011            FROM dscng_str_to_int_remap
dscng postgres[22278]: [53-3] #011            WHERE dimension_id=19
-----------

The database size is 54 GB for now.

The output from the update daemon is the following:
-----------
DEBUG:dscng.updater:DB pusher stats: 330 bundles in 972.4 s; 0.34 bps; 
wall clock time 976.9 s; utilization 99.5%; Queue fill 100/100
DEBUG:dscng.updater:Removed journal file 
'/var/cache/dscng-update-journal/dscXSvzSr.bundle'
-----------

The code responsible for this select is the following
-----------
https://gitlab.labs.nic.cz/labs/dscng/blob/master/fill_default_value_remaps.py
-----------

 From my point of view seems like faulty logic in programming and not 
feasible for large amounts of data.

Any suggestions on how to keep the data for a long period of time and 
still have good performance ?







More information about the dscng-dev mailing list