Date: Mon, 21 Feb 2000 18:06:39 +0100 (CET)
From: Szokoli Gabor <szocske@vaskutya.sch.bme.hu>
Precedence: bulk
On Thu, 17 Feb 2000, Eddie Corns wrote:
> The major limit
> in this instance turned out to be the unexpected (to me) serial update problem
> for SQL (as I believe it's technically known).
You scared me.
I'm not good at RDBMSs either, what's this problem?
Szocske
This is how I understand it. Imagine you were creating a database to hold
this sort of data, something that would surely spring to mind would be create
a sequential list of records such that as you do each poll you just write into
the _next_ record. This would be the fastest you could do. However when
dealing with SQL (probably similar on other RDBMSs) a) you have no idea
whether it will put sequential records next to each other and b) you have no
way of telling it you are doing things in sequence anyway. This means that
for every single record it has to actually search for the place to insert the
record.
In practical terms, my first attempt to update the database by doing an UPDATE
command for each poll required several hours for each iteration, however once
I converted to outputting the data to a file then importing it, it only took a
few seconds! I assume Stuart was noticing similarly huge differences.
I've just discovered that most of the time now (of the 2 minutes per
iteration) is between the SNMP walk command (that gets the ifDescr) and
timeouts for hosts that don't respond. I've just recoded a test version of
the walk code to do the parallel SNMP gets that I do for the main data
gathering and have reduced the time from 1 minute 20 seconds to about 8
seconds. I also tried various other versions including sending out every
single SNMP get request asynchronously to all ifDescr.X for all 80+ hosts
(after extracting X from ifNumber.0), then waiting for all the results,
that one was still 45 seconds.
Eddie
-- !! This message is brought to you via the `tkined & scotty' mailing list. !! Please do not reply to this message to unsubscribe. To subscribe or !! unsubscribe, send a mail message to <tkined-request@ibr.cs.tu-bs.de>. !! See http://wwwsnmp.cs.utwente.nl/~schoenw/scotty/ for more information.
This archive was generated by hypermail 2b29 : Mon Jan 08 2001 - 15:27:37 MET