Welcome to NeoOffice developer notes and announcements
NeoOffice
Developer notes and announcements
 
 

This website is an archive and is no longer active
NeoOffice announcements have moved to the NeoOffice News website


Support
· Forums
· NeoOffice Support
· NeoWiki


Announcements
· Twitter @NeoOffice


Downloads
· Download NeoOffice


  
NeoOffice :: View topic - Missing unicode characters
Missing unicode characters
 
   NeoOffice Forum Index -> NeoOffice Releases
View previous topic :: View next topic  
Author Message
DavidS
Agent


Joined: Jun 04, 2007
Posts: 14
Location: Reno, NV, USA

PostPosted: Fri Jan 06, 2012 3:08 pm    Post subject: Missing unicode characters

On another issue, I recently uploaded a file in Palatino done in V. 3.1.2 on a G5 running OS 10.4.11. I used a unicode character shown in the attached screen shot, highlighted in blue. When I view this file on a MacBook air with V. 3.2.1 installed under OS 10.6.7, not only this special character, but the whole row of characters and the one below it, are missing. I have the same issue with my G5 at home, also running OS 10.4.11 and same version of NeoOffice.
I found Palatino was in both installed in only the main library on hard disk, so I also installed it in user library. Even tried copying a folder off the desktop onto the MacBook air, which has an older Palatino font- 9/2000 vs. 3/2007. Even after restarting computer, these two rows of Unicode characters are missing.

David
Back to top
pluby
The Architect
The Architect


Joined: Jun 16, 2003
Posts: 11949

PostPosted: Fri Jan 06, 2012 4:16 pm    Post subject:

I suspect that Palatino does not have a glyph for that particular Unicode character (most fonts only cover a small subset of all possible Unicode characters) and so when NeoOffice searches for a "fallback" font to use as a substitute for that Unicode character, it does not find any other fonts on your machine that have a glyph for those Unicode characters.

To test if my theory is true or not, can you use the instructions below to open the Mac OS X Character Viewer window's Unicode tables and scroll down to the "0080" line? If my theory is correct, you will only see empty rectangles like as shown in the attached screen snapshot from my Mac OS X machine.

To open the Mac OS X Character Viewer window on Mac OS X 10.5 (I believe 10.4 steps are the same or very similar), launch the Mac OS X System Preferences application, click on the Language & Text icon, and then click on the Input Menu tab. In that tab, check the "Character Palette" checkbox.

After checking the "Character Palette" checkbox, a "flag" icon will appear in the Mac OS X menubar. Click on that icon and select the Show Character Palette menu. In the "Characters" window that appears, set the "View" option to "Code Tables". You can then see the fonts listed by Unicode character in the window like in my screen snapshot.

If you actually see glyphs in the "0080" or "0090" lines on a machine that NeoOffice does not display a character, can you click on the "0086" character and tell us which fonts are listed as having a glyph for that characters?

Patrick
Back to top
DavidS
Agent


Joined: Jun 04, 2007
Posts: 14
Location: Reno, NV, USA

PostPosted: Fri Jan 06, 2012 5:37 pm    Post subject:

Brilliant! Yes, you reach the character palette in OS 10.4 via the "International" option in System Preferences. What is curious is that when I do what you say on both machines, in the one that is displaying the character correctly, I find it is in a Microsoft true type font, PMingLiU.

Curiously, this font is also being used in the OS 10.6 machine for the same character, except it is NOT displaying. The font is in the fonts folder in main library, I also added it to my user library; still won't display the special character. Also curiously, this is NOT such an uncommon character- it is found in all kinds of fonts in my old Word Perfect 3.5, which still runs fine on 10.4 (see screen shot).
Back to top
pluby
The Architect
The Architect


Joined: Jun 16, 2003
Posts: 11949

PostPosted: Fri Jan 06, 2012 6:03 pm    Post subject:

DavidS wrote:
Curiously, this font is also being used in the OS 10.6 machine for the same character, except it is NOT displaying. The font is in the fonts folder in main library, I also added it to my user library; still won't display the special character. Also curiously, this is NOT such an uncommon character- it is found in all kinds of fonts in my old Word Perfect 3.5, which still runs fine on 10.4 (see screen shot).


On Mac OS X 10.5 and higher, adding fonts to specific folders will not work until the next reboot. Instead, try launching the /Applications/Font Book application, select the File :: Add Fonts menu, and select the font file that you want to add. After you add the font using the Font Book application, does the font appear in NeoOffice? You may need to restart NeoOffice after adding the font, but maybe not.

Patrick
Back to top
DavidS
Agent


Joined: Jun 04, 2007
Posts: 14
Location: Reno, NV, USA

PostPosted: Fri Jan 06, 2012 6:10 pm    Post subject:

Patrick-
I had already rebooted. Thing that is perplexing is font DOES show in NeoOffice, but cannot see the characters in the 10.6 machine, but can in 10.4. Will try my other 10.4 machine when I go home.

David
Back to top
sardisson
Town Crier
Town Crier


Joined: Feb 01, 2004
Posts: 4588

PostPosted: Fri Jan 06, 2012 6:14 pm    Post subject:

One other thing you should try is checking for duplicate or corrupt fonts using Font Book:
https://neowiki.neooffice.org/index.php/Troubleshooting_Tips#Check_for_duplicate_fonts
https://neowiki.neooffice.org/index.php/Troubleshooting_Tips#Disable_corrupt_fonts

Smokey

_________________
"[...] whether the duck drinks hot chocolate or coffee is irrelevant." -- ovvldc and sardisson in the NeoWiki
Back to top
pluby
The Architect
The Architect


Joined: Jun 16, 2003
Posts: 11949

PostPosted: Fri Jan 06, 2012 6:18 pm    Post subject:

DavidS wrote:
I had already rebooted. Thing that is perplexing is font DOES show in NeoOffice, but cannot see the characters in the 10.6 machine, but can in 10.4. Will try my other 10.4 machine when I go home.


If Smokey's suggestions for finding and eliminating duplicate instances of the font do not change anything, can you post a screen snapshot of the character palette window on your Mac OS X machine with the "0086" unicode character selected like in my previous screen snapshot?

I ask this because Apple replaced the entire text rendering functions in Mac OS X 10.5. Applications like NeoOffice 3.1.2 and Microsoft Office that work on Mac OS 10.4 use Apple's older "ATSUI" text rendering functions. In contrast, application like NeoOffice 3.2.1 use Apple's newer "CoreText" functions. So, while the font may be listed, it is possible that some very old fonts like your version of Palatino may no longer be supported by Apple's newer CoreText functions.

Patrick
Back to top
DavidS
Agent


Joined: Jun 04, 2007
Posts: 14
Location: Reno, NV, USA

PostPosted: Fri Jan 06, 2012 6:44 pm    Post subject:

I did try font book; could not find PMingLiu in my user library because it has already been moved to a Disabled Fonts folder on the 10.4 machine which IS showing the character.
I have taken a screen shot on the MacBook Air OS 10.6.7, which is NOT displying the character, although it shows in the NeoOffice font menu.

david
Back to top
pluby
The Architect
The Architect


Joined: Jun 16, 2003
Posts: 11949

PostPosted: Mon Jan 09, 2012 12:10 pm    Post subject:

DavidS wrote:
I did try font book; could not find PMingLiu in my user library because it has already been moved to a Disabled Fonts folder on the 10.4 machine which IS showing the character.
I have taken a screen shot on the MacBook Air OS 10.6.7, which is NOT displying the character, although it shows in the NeoOffice font menu.


So do you know where you obtained the PMingLiu font that is on your Mac OS X 10.6.7 machine? I ask because cannot find that font in my Microsoft Office 2008 installation or my Mac OS X 10.6.8 machine so, for me, there are still no fonts that have a glyph for the "0086" unicode character.

Patrick
Back to top
pluby
The Architect
The Architect


Joined: Jun 16, 2003
Posts: 11949

PostPosted: Mon Jan 09, 2012 12:53 pm    Post subject:

Ignore my last post. I found the PMingLiu font that Microsoft Office 2008 installs and I can now see its glyph for the 0086 unicode character in the character paletter and, like you, I cannot see it in NeoOffice's special characters dialog.

I will start investigating this to see why this range of PMingLiu glyphs are not displaying in NeoOffice. I will post again when I have some news to report.

Patrick
Back to top
pluby
The Architect
The Architect


Joined: Jun 16, 2003
Posts: 11949

PostPosted: Mon Jan 09, 2012 2:15 pm    Post subject:

I have found the reason why NeoOffice cannot display the PMingLiU font's glyphs for the 0080 through 009F range of unicode characters and the bad news is that this is not a NeoOffice bug. Instead, it is a Mac OS X bug.

What I found is that while the Mac OS X character palette can display these PMingLiU glyphs, Apple's text layout functions will always fail and will return the "no glyph available" glyph for these unicode characters even when the font is explicitly set to PMingLiU. Even worse, in Mac OS X 10.7, I could not even get the Font Book application to install the PMingLiU font so it is clear that newer versions of Mac OS X no longer fully support this particular font.

FWIW, I found that this was a Mac OS X bug by doing the following:

1. Launch TextEdit and then open the character palette and select the 0086 unicode character and press the character palette's Insert button. Nothing will get inserted in TextEdit for this unicode character, but for other unicode characters below 0080, inserting the PMingLiU font will work fine.

2. I put debug code in NeoOffice to print out what glyph number Apple's CoreText functions return for the 0086 unicode character and PMingLiU font and those functions return the 0xffff font which, if displayed, is an empty rectangle (that is, the "no glyph available" glyph).

I suspect that you will need to find a different font to use for the symbols in the 0080 through 009F unicode character range. Unfortunately, I cannot recommend any fonts as my quick Google search and my search of sil.org's sites did not find any free fonts that have these glyphs for that unicode character range.

Patrick
Back to top
sardisson
Town Crier
Town Crier


Joined: Feb 01, 2004
Posts: 4588

PostPosted: Mon Jan 09, 2012 3:36 pm    Post subject:

pluby wrote:
I have found the reason why NeoOffice cannot display the PMingLiU font's glyphs for the 0080 through 009F range of unicode characters and the bad news is that this is not a NeoOffice bug. Instead, it is a Mac OS X bug.

No, I think it's actually an "old pre-Unicode font" bug Wink Sorry, I didn't think to look at the Unicode glyph number before your last post. Sad

pluby wrote:
I suspect that you will need to find a different font to use for the symbols in the 0080 through 009F unicode character range. Unfortunately, I cannot recommend any fonts as my quick Google search and my search of sil.org's sites did not find any free fonts that have these glyphs for that unicode character range.

I don't think that will work; the only fonts that will have glyphs in that range are going to be old, pre-Unicode fonts, and as you've demonstrated, the OS is going to return a "no glyph found" result for any codepoints in that range, in any font (even, I suspect, LastResort). Why? Because the U+0080 to U+009F range is reserved for control characters (and has been since Unicode 1.1): http://unicode.org/charts/PDF/U0080.pdf (I'm a bit surprised it's showing up in the Character Palette screenshots on 10.6, actually.)

DavidS's best option, I think, is to find the Unicode codepoint where that glyph has moved (I think U+2020) and find a fully-Unicode-compliant font that displays the glyph as he likes it.

Smokey

_________________
"[...] whether the duck drinks hot chocolate or coffee is irrelevant." -- ovvldc and sardisson in the NeoWiki
Back to top
pluby
The Architect
The Architect


Joined: Jun 16, 2003
Posts: 11949

PostPosted: Mon Jan 09, 2012 4:43 pm    Post subject:

sardisson wrote:
DavidS's best option, I think, is to find the Unicode codepoint where that glyph has moved (I think U+2020) and find a fully-Unicode-compliant font that displays the glyph as he likes it.


Smokey's advice is definitely the best route for DavidS so please ignore my suggestion in my last post.

Also, thank you Smokey for catching that the U+0080 to U+009F range is reserved for control characters. I had forgotten about that special range and I didn't make the connection that "Latin-1 Supplement" was synonymous for "control characters".

Patrick
Back to top
sardisson
Town Crier
Town Crier


Joined: Feb 01, 2004
Posts: 4588

PostPosted: Mon Jan 09, 2012 5:59 pm    Post subject:

pluby wrote:
Also, thank you Smokey for catching that the U+0080 to U+009F range is reserved for control characters. I had forgotten about that special range and I didn't make the connection that "Latin-1 Supplement" was synonymous for "control characters".

FWIW, all of the Latin-1 Supplement isn't control characters, just those first 32 codepoints. (It's kind-of like the control characters were shoved in to some previously-empty spaces between in-use characters and code blocks.) The rest of the Latin-1 Supplement contains various symbol glyphs and multilingual variants of standard ASCII/Roman letters.

Smokey

_________________
"[...] whether the duck drinks hot chocolate or coffee is irrelevant." -- ovvldc and sardisson in the NeoWiki
Back to top
DavidS
Agent


Joined: Jun 04, 2007
Posts: 14
Location: Reno, NV, USA

PostPosted: Fri Jan 20, 2012 4:37 pm    Post subject:

Patrick-
Have been on vacation close to two weeks. I fear I do not buy the explanation that it is a Mac OS coding problem. On the plane last night, I went to insert a degree symbol that was repeatedly used in a document I was correcting. For various reasons, I decided to do this in text edit. By god, there in text edit, when I went to insert the degree, I found preselected the dagger from PMingLiu (see attached screenshot). To be doubly sure, I tried this also in Text Wrangler, an industrial strength text editor used for bioinformatics due to its ability to open monster (500mb) genome data files. It is also there in Text Wrangler. This was done with a MacBookAir, OS 10.6.7, running the latest version of NeoOffice with patch 3 installed.
I argue this points the finger back at NeoOffice, and again raises the nasty question of why this works on my office G5 DP running OS 10.4.11, and not on my home G5Dp. One clue I think is that the PMingLiu is an font folder labeled "Fonts disabled" on the office G5 (runing Neoffice 3.1.2.

David
Back to top
Display posts from previous:   
   NeoOffice Forum Index -> NeoOffice Releases All times are GMT - 7 Hours
Goto page 1, 2  Next
Page 1 of 2

 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

Powered by phpBB © 2001, 2005 phpBB Group

All logos and trademarks in this site are property of their respective owner. The comments are property of their posters, all the rest © Planamesa Inc.
NeoOffice is a registered trademark of Planamesa Inc. and may not be used without permission.
PHP-Nuke Copyright © 2005 by Francisco Burzi. This is free software, and you may redistribute it under the GPL. PHP-Nuke comes with absolutely no warranty, for details, see the license.