I'm talking to two people at the same time here, trying to cut down on my
traffic.
On Tue, 22 Feb 2000, Stuart wrote:
> On Tue, 22 Feb 2000 17:36:56 Eddie Corns wrote:
> <snip>
> >This is how I understand it. Imagine you were creating a database to hold
> >this sort of data, something that would surely spring to mind would be create
> >a sequential list of records such that as you do each poll you just
> >Bwrite into
> >the _next_ record. This would be the fastest you could do. However when
> >dealing with SQL (probably similar on other RDBMSs) a) you have no idea
> >whether it will put sequential records next to each other and b) you have no
> >way of telling it you are doing things in sequence anyway. This means that
> >for every single record it has to actually search for the place to insert the
> >record.
> Actually you don't know or care which order it puts the data into the
> database table - you only care about the order it comes out when you
> need to process or report on it.
I get Eddie's originial point: records could be saved one after another,
in the order they arrive, and that would keep things ordered properly,
so there would be no need for an ORDER BY in the query, and everyone would
be content.
With a sophisticated SQL DB, we get the same results, but a piece of
information is lost, that would allow major optimalisations, even if the
table is indexed properly (I'm not completely familiar with SQL terms
here) the DB will find out at each and every INSERT by hard work, that
"heck, this new reckord goes after the last one, and not in between
somewhere!"
But we knew it all the time, and if we could say in SQL "this table will
be loaded in the order the index wants it", we could save the trouble.
I mean, the results would be the same of course, but it would take less
time.
> >In practical terms, my first attempt to update the database by doing
> >an UPDATE
> >command for each poll required several hours for each iteration, however once
What exactly were you doing whith what amount of records in which RDBMS?
Are we talking about simply adding new records to the DB?
> >I converted to outputting the data to a file then importing it, it
> >only took a
> >few seconds! I assume Stuart was noticing similarly huge differences.
> >
> Not so huge as that, but as someone pointed out the SQL*Loader is the
> fastest way to get stuff into a table.
I'm showing my lack of experience again:
is SQL*Loader a DBvendor independent way to load a table from a file?
> >I've just discovered that most of the time now (of the 2 minutes per
> >iteration) is between the SNMP walk command (that gets the ifDescr) and
> >timeouts for hosts that don't respond. I've just recoded a test version of
> >the walk code to do the parallel SNMP gets that I do for the main data
> >gathering and have reduced the time from 1 minute 20 seconds to about 8
> >seconds. I also tried various other versions including sending out every
> >single SNMP get request asynchronously to all ifDescr.X for all 80+ hosts
> >(after extracting X from ifNumber.0), then waiting for all the results,
> >that one was still 45 seconds.
> >
> This is intriguing. Would you be willing to post a snippet of code
> where you do the parallel polls? Do you have any thoughts on the
> resources (memory, cpu) used compared to a straight walk?
I'd be interested, too!
> Thanks
> Stuart
Szocske
-- !! This message is brought to you via the `tkined & scotty' mailing list. !! Please do not reply to this message to unsubscribe. To subscribe or !! unsubscribe, send a mail message to <tkined-request@ibr.cs.tu-bs.de>. !! See http://wwwsnmp.cs.utwente.nl/~schoenw/scotty/ for more information.
This archive was generated by hypermail 2b29 : Mon Jan 08 2001 - 15:27:37 MET