[dscng-dev] DSCng became very slow in processing data.

Valentin Mitoiu valentin.mitoiu at univie.ac.at
Tue Jul 15 10:44:04 CEST 2014

Hi guys,

In the last month dscng became slower and slower in processing messages.
I enabled logging in postgres to track the slow queries and these seem 
to be the ones which are slowing the inserting

postgres[22278]: [53-1] 2014-07-15 10:37:34 CEST LOG:  duration: 
5729.992 ms  statement: SELECT original,remap
dscng postgres[22278]: [53-2] #011            FROM dscng_str_to_int_remap
dscng postgres[22278]: [53-3] #011            WHERE dimension_id=19

The database size is 54 GB for now.

The output from the update daemon is the following:
DEBUG:dscng.updater:DB pusher stats: 330 bundles in 972.4 s; 0.34 bps; 
wall clock time 976.9 s; utilization 99.5%; Queue fill 100/100
DEBUG:dscng.updater:Removed journal file 

The code responsible for this select is the following

 From my point of view seems like faulty logic in programming and not 
feasible for large amounts of data.

Any suggestions on how to keep the data for a long period of time and 
still have good performance ?

More information about the dscng-dev mailing list