Your process takes 18 hours to run. You want it to be less.

You need to work out what the computer spends most of it's time doing in those 18 hours, so that you can try and reduce the most costly elements first (and get the biggest improvements).

Prime possibilities are:

• Waiting for a network response (latency). A lot of network protocols are request/response. If you need to do 50k requests and a single request takes 1s to process remotely, you'll end up waiting 50k secs == 14hours.
• Waiting to swap stuff in+out. If your app working set is bigger than the RAM available, performance will plummet as it waits for the disk to access virtual memory.
• CPU. If your app isn't waiting for anything else, it's trying to crunch through code. This is the case where profiling etc can help the most.
• Other disk I/O. If your process is reading and writing to storage, that can be slow. Particularly if you are on a slow device and/or are doing a lot of flushing after small writes (but sometimes that is that you need to do).

So you need to run some monitoring tools ('top' is a good first start, but solaris has many others - check out vmstat and iostat too). These will tell you which of the above issues is the problem. (Well, if it's not CPU, swapping or other disk I/O, then it's probably network latency.

And after all that about measuring first?

My best guess is that you're doing 50K LDAP operations and network latency is killing you (i.e. your box isn't busy when you're doing this, but perhaps your LDAP server (or network) is).

Look for a bulk import/export tool for your LDAP server and use that instead.

Total code took : 68258 secs

While LDAP Update/Add entry took:

the code took:67358 wallclock secs (29935.47 usr 60.38 sys + 0.00 cusr 0.00 csys = 29995.85 CPU) in considering each entry from PeopleFirst extract ...

1. Do older versions of PERL has bad algorithms for handling of HASHES ?
2. Will it help if we use Arrays instead of Hashes?

If upgrading the PERL version will help, I will try to convince the management if you can help me in locating some url which quotes this.

the code took:67358 wallclock secs (29935.47 usr 60.38 sys + 0.00 cusr 0.00 csys = 29995.85 CPU) in considering each entry from PeopleFirst extract ..

That's interesing. So 29935/67358=44% of your time was spent on user CPU. That is significant, and you might want to look into profiling the app's CPU usage (using Devel::Profile and Devel::DProf).

Of course, it also means that 56% of your time is spent doing other things. If that is net latency then you'd do well to look at bulk import/export instead.

Your timestamp logging appeared to show ~6secs for one request, is that right? That can't be representative, since as noted elsewhere in this thread, you'd never manage to do 50k updates in 18 hours if each takes 6s.

Lastly, if you do profile the app, then it will probably benefit you to produce a cut-down version which runs more quickly. This is useful because the profilers slow things down and generate large amounts of data - they'll probably break on such a big run.

Also, having a more quickly repeatable test case (e.g. ~10mins) will greatly accelerate your ability to test ideas on code and algorithm changes.

However, the hard part is knowing if your cut-down test case has the same performance profile as your main job run.

Another thought: if the 'missing' 56% of your time is overnight you might be sharing a network with a backup job, or something else which saturates the net and makes your network response times go very slowly.

You are asking the wrong questions. You already blame the hashes without really knowing what wastes all this time. Everyone above told you to do some profiling first and that is a really good idea. Often it is the algorithm used that kills the time.

For example: If your program often makes a copy of your hash then that will really thrash your memory and cost time. But you won't get it faster by changing to an array then.

Also you didn't show us what the code does with the hash you blame. Then how should we be able to tell you if an array ist better

So either post some code here or do some profiling or both.

%p1 = { 'dn' => 'st-eduid=ed988766,ou=People,dc=xyz,dc=com }

That's already wrong, use parenthesis instead:

%p1 = ( 'dn' => 'st-eduid=ed988766,ou=People,dc=xyz,dc=com )
[download]

I presume you have profiled the code so that you know where the major portion of the time is being spent. How about you show us a small working sample that demonstrates the actual issue? What you have shown so far doesn't really illustrate any problematic Perl code.

I'd have to agree with AM here ... sounds like you're blaming your hash structure when the LDAP server and/or network latency is your real issue. After upgrading your perl, take a look at Devel::Profile or Devel::NYTProf to try and pinpoint exactly where the code is spending all its time.

-derby

You could try printing timestamps immediately before and after the LDAP access - this would help to confirm the location of the (suspected) bottleneck...

I have already printed timestamps in the log files. We may see from here:

[09:20:29.260258 ]: 144248 (BATALLONES,Jayson) looks like a creation
[09:20:29.260564 ]: Creating entry 144248 (BATALLONES,Jayson) in ED...
[09:20:35.192125 ]: createEDentry: Successfully created ED entry 'st-e
+duid=ed953127,ou=people,dc=xyz,dc=com' 144248 (BATALLON
ES,Jayson) for location 000194
[download]

The above log shows that LDAP Server is not taking too much time in creating a entry. I have also used Benchmark for it and the result is as below:

the code took: 1 wallclock secs ( 0.01 usr  0.00 sys +  0.00 cusr  0.0
+0 csys =  0.01 CPU) in Creating entry 144248(BATALLONES,Jayson) in ED
+...
[download]

Mmmm, if I read correctly, a single lookup takes, on average, 6 secs - if it takes 6 secs/entry, it should take about 330,000 secs i.e. a lot longer than 18 hrs, for 55,000 entries, so I think you're getting off lightly!

It strikes me that the best answer to your problem is, as jbert suggests, to find a bulk LDAP download tool and use that - which will inevitably require knowledge of LDIF file processing.

A user level that continues to overstate my experience :-))

6s is quite slow. One question: Do you setup a new LDAP connection for each transaction? If yes, try to reuse an already opened LDAP connection (better: a pool of connections) since it is not necessary to bind and authenticate 50k times.