| View previous topic :: View next topic |
| Author |
Message |
rays
The Anomaly
(earlier version)
Joined: Sep 23, 2004
Posts: 475
Location: Geneva, Switzerland
|
 Posted: Fri Feb 26, 2010 2:50 am Post subject: Mail Merge and CPU consumption |
|
Hi Patrick,
We've been spending quite some time recently on using the Mail Merge functionality in NeoOffice and comparing performance with doing the same activity in OpenOffice for Mac 3.2.0.
I wonder if you could take a look at the following example activities on using a simple half-page Writer document when used in conjunction with a workbook containing 4 spreadsheets/tables each containing varying number of rows with names and an email address. Yes, we're merging and sending emails directly from within Neo/OpenOffice.
These results from MacBook Pro 2.53 GHz with 4GB RAM:
1. On opening the View>Data Sources dialogue (fn-F4), and switching between two of the available tables NeoOffice hits 100% CPU and takes around 30 seconds to render the result and return control to the user. OpenOffice doesn't even reach 10% CPU usage and switches in under 3 seconds.
2. Switching from a 3-row table to a 157-row table or back in the other direction both take over 30 seconds in NeoOffice with 100% CPU usage so it does not seem to be related to the number of rows in the tables we are switching between. As before, the same actions in OpenOffice is taking less than 3 seconds to perform.
3. Clicking on Mail Merge button; again NeoOffice hits 100% CPU and takes a couple of seconds to open the mail merge wizard window. OpenOffice is instantaneous with no detectable increase in CPU usage.
4. With 157 records to merge, OpenOffice is a bit quicker to compile the multi-page document created when clicking on step 8 (Save, print or send) in the mail merge wizard with NeoOffice again hitting 100% CPU while OpenOffice barely budges.
Bearing in mind these results above are obtained when testing on a MacBook Pro, consider the effect when using the same test activities on an iMac G5 1.83 GHz with 1 GB of RAM. Obviously I can't run the OpenOffice tests on it but, knowing that you stil have PowerPC computers there, you will be able to see for yourself how much longer these particular activities above take on a less powerful (but still pretty well-endowed with RAM) Leopard 10.5.8 system. It is virtually unusable, especially when we really need to be able to merge with 100+ rows in a table.
I'm just wondering whether there may be some way in which these basic mail merge processes could be better optimised in NeoOffice? Is there a particular reason that is preventing NeoOffice to perform on a par with OpenOffice in this area?
As ever, I will be happy to work with you towards a resolution of this issue if that is possible.
With kind regards,
_________________
Ray Saunders
World Scout Bureau |
|
| Back to top |
|
 |
pluby
The Architect
Joined: Jun 16, 2003
Posts: 11949
|
| Posted: Fri Feb 26, 2010 7:32 am Post subject: Re: Mail Merge and CPU consumption |
|
| rays wrote: | We've been spending quite some time recently on using the Mail Merge functionality in NeoOffice and comparing performance with doing the same activity in OpenOffice for Mac 3.2.0.
I wonder if you could take a look at the following example activities on using a simple half-page Writer document when used in conjunction with a workbook containing 4 spreadsheets/tables each containing varying number of rows with names and an email address. Yes, we're merging and sending emails directly from within Neo/OpenOffice. |
Does this same speed and CPU issue occur when using NeoOffice's built-in Bibliography data source or only when using a remote data source? If it does not occur with the Bibliography data source, then what kind of data source are you connecting to?
Since it will be extremely difficult to reproduce your case if you are using a remote data source, the real question is what speed and CPU numbers do you get when using OpenOffice.org 3.0.1 (the version that NeoOffice 3.0.2 is based on)? If you get the same numbers as NeoOffice, then we can conclude that Oracle's OpenOffice.org engineers have optimized the mail merge code after they release OpenOffice.org 3.0.1.
Patrick |
|
| Back to top |
|
|
rays
The Anomaly
(earlier version)
Joined: Sep 23, 2004
Posts: 475
Location: Geneva, Switzerland
|
| Posted: Fri Feb 26, 2010 1:15 pm Post subject: Re: Mail Merge and CPU consumption |
|
| pluby wrote: | Does this same speed and CPU issue occur when using NeoOffice's built-in Bibliography data source or only when using a remote data source? If it does not occur with the Bibliography data source, then what kind of data source are you connecting to?
Since it will be extremely difficult to reproduce your case if you are using a remote data source, the real question is what speed and CPU numbers do you get when using OpenOffice.org 3.0.1 (the version that NeoOffice 3.0.2 is based on)? If you get the same numbers as NeoOffice, then we can conclude that Oracle's OpenOffice.org engineers have optimized the mail merge code after they release OpenOffice.org 3.0.1. |
Fiirst observations from test on home Macbook 2GHz 2GB:
There is significant difference when using built-in Bibliography in NeoOffice - more responsive. I'm connecting to a .ods spreadsheet, using the same one as I've been using all week for these tests. Each application then automatically creates a .odb file which appears to act as a sort of soft/hard link through to the original spreadsheet.
In terms of switching between tables in the same test spreadsheet, OpenOffice 3.0.1 is acting with the same positive level of responsiveness as OpenOffice 3.2 with my external spreadsheet and no significant spike in CPU usage, so it doesn't appear to relate to any development work done between these versions of OOo.
The same huge performance difference exists on just comparing invoking fn-F4 on OOo 3.0.1 and NeoOffice. Could that be a place to start looking at how the two differ in their handling of that command? On this MacBook we're again looking at <3 seconds compared with NeoOffices nearly 30 seconds. It's as if this "secondary window" (View>Data sources) may at the heart of the problem as actions invoked in NeoOffice when this view is active are generally terribly slow compared with their OOo equivalents.
But I'll continue to with my investigations and see if I can pin-point something more specific but without reference to my current test .ods and .odt test files.
_________________
Ray Saunders
World Scout Bureau |
|
| Back to top |
|
|
rays
The Anomaly
(earlier version)
Joined: Sep 23, 2004
Posts: 475
Location: Geneva, Switzerland
|
| Posted: Fri Feb 26, 2010 1:59 pm Post subject: |
|
Early indication suggests that OOo handles .ods files but NeoOffice struggles with them.
With csv files, speeds are similar with no high CPU usage in Neo.
Too late tonight to look any further.
_________________
Ray Saunders
World Scout Bureau |
|
| Back to top |
|
|
pluby
The Architect
Joined: Jun 16, 2003
Posts: 11949
|
| Posted: Fri Feb 26, 2010 7:48 pm Post subject: Re: Mail Merge and CPU consumption |
|
| rays wrote: | | The same huge performance difference exists on just comparing invoking fn-F4 on OOo 3.0.1 and NeoOffice. Could that be a place to start looking at how the two differ in their handling of that command? On this MacBook we're again looking at <3 seconds compared with NeoOffices nearly 30 seconds. It's as if this "secondary window" (View>Data sources) may at the heart of the problem as actions invoked in NeoOffice when this view is active are generally terribly slow compared with their OOo equivalents. |
The speeds for NeoOffice and OpenOffice.org 3.0.1 are both only a few seconds when I press fn-F4. However, I only have the built-in Bibiliography database as my sole data source.
Can you obtain a sample of the NeoOffice process using the Activity Monitor application during that 30 second lag and attach it? That may give us a clue as to where in the code the slowdown is occurring.
Also, what types of data sources are listed after you press the fn-F4 keys? Specifically, are your data sources .odb files, the Mac address book, a Mozilla address book, or are they some other type of data? I suspect that the NeoOffice is connecting to each of your data sources when you press those keys and the high CPU and slowdown are occurring because NeoOffice is having trouble connecting to one of those data sources.
Patrick |
|
| Back to top |
|
|
rays
The Anomaly
(earlier version)
Joined: Sep 23, 2004
Posts: 475
Location: Geneva, Switzerland
|
| Posted: Sat Feb 27, 2010 12:49 am Post subject: Re: Mail Merge and CPU consumption |
|
| pluby wrote: | Can you obtain a sample of the NeoOffice process using the Activity Monitor application during that 30 second lag and attach it? That may give us a clue as to where in the code the slowdown is occurring.
Also, what types of data sources are listed after you press the fn-F4 keys? Specifically, are your data sources .odb files, the Mac address book, a Mozilla address book, or are they some other type of data? I suspect that the NeoOffice is connecting to each of your data sources when you press those keys and the high CPU and slowdown are occurring because NeoOffice is having trouble connecting to one of those data sources.
Patrick |
Samples taken while waiting for response to fn-F4 and while switching tables within the same .ods workbook.
I'm using a .odt workbook with four straightforward spreadsheets inside. When connecting it, Open/NeoOffice creates an .odb file of the same name which appears to act as a connector between the application and the original spreadsheets in the workbook.
In the tests, the number and types of data sources registered in NeoOffice and OpenOffice are the same. In addition to Bibliography, Open/NeoOffice create a new database for each of the .ods or .cvs files I connect to. But we're only talking about 2 or 3 being available in te applications at any time.
Hope these samples help.
_________________
Ray Saunders
World Scout Bureau |
|
| Back to top |
|
|
pluby
The Architect
Joined: Jun 16, 2003
Posts: 11949
|
| Posted: Sat Feb 27, 2010 9:02 am Post subject: |
|
Thank you for the samples. Nothing is jumping out at me in the samples and overall it looks like the NeoOffice's underlying code is busily reading the data from your data source.
I will look at it more closely but in the meantime can you test if the new native text highlighting code is somehow causing this problem? You can test this by checking the NeoOffice :: Mac OS X Options :: Disable Mac OS X Text Highlighting menu and closing and reopening your document.
Does the same problem occur when you disable native text highlighting? If the problem stops, then that means that my native text highlighting code is causing NeoOffice's underlying OpenOffice.org code to requery the data source repeatedly.
Patrick |
|
| Back to top |
|
|
rays
The Anomaly
(earlier version)
Joined: Sep 23, 2004
Posts: 475
Location: Geneva, Switzerland
|
| Posted: Sat Feb 27, 2010 10:46 am Post subject: |
|
Disabling native text highlighting made no difference but I have made progress at this end nonetheless.
The issue appears to be related to formatting used in the first row of the spreadsheets. OpenOffice is able to handle the current formatting but for some reason NeoOffice is choking on it. Selecting the first row and applying "Format>Default formating" un-chokes NeoOffice.
I'll now try to identify which or which combination of the format features used in our original spreadsheet triggers the choking in NeoOffice.
_________________
Ray Saunders
World Scout Bureau |
|
| Back to top |
|
|
pluby
The Architect
Joined: Jun 16, 2003
Posts: 11949
|
| Posted: Sat Feb 27, 2010 11:09 am Post subject: |
|
| rays wrote: | The issue appears to be related to formatting used in the first row of the spreadsheets. OpenOffice is able to handle the current formatting but for some reason NeoOffice is choking on it. Selecting the first row and applying "Format>Default formating" un-chokes NeoOffice.
I'll now try to identify which or which combination of the format features used in our original spreadsheet triggers the choking in NeoOffice. |
FYI. I have moved this forum topic to the NeoOffice Support forum since it is becoming clear that this problem is a bug in NeoOffice that we need to try to fix.
If you are able to, can you create a sample data file that I can use to create a data source with? If you can create such a data file, please attach it and then I can then try to reproduce this bug.
BTW, since the native text highlighting does not appear to be the cause of the problem and the problem does not occur in OpenOffice.org 3.0.1, my current theory is that this is a bug in Go-oo modifications to the OpenOffice.org code that we include in NeoOffice. If my theory is correct, then once I can reproduce this problem, I should be able to fix it by pulling out each of the Go-oo modifications one by one until the problem goes away.
Patrick |
|
| Back to top |
|
|
rays
The Anomaly
(earlier version)
Joined: Sep 23, 2004
Posts: 475
Location: Geneva, Switzerland
|
| Posted: Sun Feb 28, 2010 12:42 am Post subject: |
|
| pluby wrote: | | If you are able to, can you create a sample data file that I can use to create a data source with? If you can create such a data file, please attach it and then I can then try to reproduce this bug. |
Thanks for your concern, Patrick.
The attached .ods workbook comprises 2 worksheets. I have retained the header row formats exactly as my colleague set it up in NeoOffice.
Even with just 2 & 3 rows of 3 columns of data, NeoOffice struggles to View>Data sources and is also hitting 100% when switching between the two tables when viewed in the Data Sources window.
Hope it misbehaves for you too.
_________________
Ray Saunders
World Scout Bureau |
|
| Back to top |
|
|
rays
The Anomaly
(earlier version)
Joined: Sep 23, 2004
Posts: 475
Location: Geneva, Switzerland
|
| Posted: Sun Feb 28, 2010 1:45 am Post subject: Apparent cause is related to borders |
|
The attached .ods file is created from duplicating a copy of the previous problematicformat.ods.
By selecting all cells and removing borders from both worksheets, the problem disappears.
If I only remove the borders from one sheet but not the other, the problem remains.
I concluded this on the basis of the following sequence of changes I tried along the way, starting with a fresh copy of the file:
1. Changed cell background colour from cyan to grey - no change
2. Remove cell background colour to No Fill - no change
3. Changed font colour from blue to Automatic - no change
4. Select header row and re-applied Arial Bold 11pt (as viewed in OOo) - no change
5. Tried header row language setting to [None] - no change
6. Changed font from Arial Bold 11pt to Arial Regular 11pt - no change
7. Changed font from Arial Regular 11pt to Arial Regular 10pt to match remainder of worksheet cells - no change
8. Removed Auto-filter setting from "English" worksheet - no change
9. Select whole sheets and removed borders - aha!
I then went back a fresh copy of the original file and directly applied step 9 only, followed by re-testing with removing the borders in only one of the sheets in turn. Only by removing borders on both sheets was the problem resolved.
Trust this saves you some research time!
Kind regards,
_________________
Ray Saunders
World Scout Bureau |
|
| Back to top |
|
|
pluby
The Architect
Joined: Jun 16, 2003
Posts: 11949
|
| Posted: Sun Feb 28, 2010 8:39 am Post subject: |
|
I can definitely reproduce this bug with your first sample .ods document. I can also reproduce it in my Go-oo build so this bug is caused by one or more of the Go-oo code modifications that we include in NeoOffice.
I will start work on identifying which Go-oo code modification causes this bug. Once I am able to identify the modification and remove it, I will post a test patch for you to try.
Patrick |
|
| Back to top |
|
|
pluby
The Architect
Joined: Jun 16, 2003
Posts: 11949
|
| Posted: Sun Feb 28, 2010 11:37 am Post subject: |
|
FYI. I have created bug 3591 to track this bug.
Although I have not yet found the Go-oo code modification that causes this bug, I have found what behavior is causing this bug. What I found is that the Go-oo code is treating empty cells that have borders as cells with data. In your case, the number of cells that the NeoOffice Calc code returns to the Data Source table grid's rendering code is huge due to all of those empty cells with borders. The Data Source table grid rendering code does not process cells very fast so this huge number of cells causes the rendering code to get backlogged.
In contrast, in OpenOffice.org 3.0.1 all empty cells are ignored so only a handful of cells are returned to the Data Source table grid rendering code.
Now I have to find the Calc code that is treating empty cells with borders as non-empty cells.
Patrick |
|
| Back to top |
|
|
rays
The Anomaly
(earlier version)
Joined: Sep 23, 2004
Posts: 475
Location: Geneva, Switzerland
|
| Posted: Sun Feb 28, 2010 12:03 pm Post subject: |
|
Good luck! 
I'm looking forward to seeing this bug resolved - especially on the iMac G5 computers we still have in daily production use.
Thanks for your time on this.
_________________
Ray Saunders
World Scout Bureau |
|
| Back to top |
|
|
pluby
The Architect
Joined: Jun 16, 2003
Posts: 11949
|
|
| Back to top |
|
|
|
