Search the Community

Welcome to the world of PIPE symbols, vertical lines and bars. Why are there two pipe symbols on a computer keyboard? Why are there two vertical lines on keyboards? Why does a solid line produce a broken line? ASCII? What does Ascii and character sets have to do with this? Why is the bar broken? Why is it no longer broken? What does ANY of this mean. Find out within (disclaimer: this video might actually confuse you more than you are right now).

It all started with 128 ASCII characters in the 1960s. In the 1980s a variety of (largely incompatible) 256 character codepages where used. Finally in the early 1990s a new system was invented that should overcome all the limitations and incompatibilities of the older codepages: Unicode—a system where all character of all writing systems are combined into one standard. It took some time, but today Unicode is the default encoding for basically all electronic communications. It doesn’t matter anymore if you use Windows or Mac OS or which font you use to display a text. I can put any of the 113,021 Unicode 7.0 characters on this website and you could safely copy and paste them to a local file (for example). Because every code point is just used once. There is no ambiguity anymore. A commercial Latin OpenType will probably have a rather complete character set for the first 256 characters, but there can be any number of unencoded characters as well. As an example: Arno Pro from Adobe includes the character sets for Latin, Greek and Cyrillic together using around 1000 glyph slots. But the fonts contain an additional range of around 1800 unencoded glyphs! The basic character set can be accessed directly with the keyboard using the appropriate keyboard layouts. To access Unicode characters that aren’t directly available this way, you can either copy them from certain websites or you can use character map apps for your operating system. Both of these methods are Unicode-based, which makes them a reliable way of accessing any character you want. But what about glyphs that don’t have a Unicode code point in the first place — like stylistic alternates, different figure sets, discretionary ligatures, small caps and certain pictograms? The glyphs in a font can be referenced in different ways. The glyph ID simply represents the position of a glyph in the list of all glyphs. But that isn’t a very reliable way to access a character. With the next update of the font the position of a certain glyph in the font might have changed. There is also no semantic meaning to a certain glyph ID. In one font, the ID 1 might be an A, in the next font it might be a space character. The glyph names on the other hand are used for OpenType functionalities. If you activate the ligature feature the combination of f + b can be replaced by an fb ligature which is accessed through its glyph name, e.g. “f_b”. But again, that is font specific. Another font might not have that ligature or use another glyph name. The only reliable way for users to access the glyph for a specific character regardless of the font or font version is a Unicode value and that is what most character map apps and character map websites offer. Glyphs without a Unicode value are usually simply omitted. To encode or not to encode Type designers need to choose one of two ways to deal with glyphs that don’t have an official Unicode code point: 1. Don’t encode them at all. This is the recommended way from a semantical and technical point of view. A small caps letter is just a visual/stylistic alternative to a lowercase letter. So it is accessed by typing the lowercase letter and then applying a styling, which activates the OpenType feature to switch out the lowercase letter with the small caps letter while retaining the character encoding of the lowercase letter. The replacement rule is set up as an OpenType feature within the font and the small caps character will be accessed only through an arbitrary glyph name, not a standardised Unicode value. The downside of this approach is: The app you are using needs to support the specific OpenType features or these glyphs will be inaccessible. Even though OpenType has been around for many years now, this is still a problem. People buy commercial fonts every day only to find out, that all the advertised ligatures and alternative characters cannot be accessed in their word processor or that their favourite browser cannot show the figure set they would like to use. 2. Use PUA codes Instead of relying on OpenType, glyphs that don’t match an existing Unicode character can also be assigned a Unicode value. For this purpose, Unicode has a so-called Private Use Area (PUA). A type designer can pick any Unicode value from that range and apply it to any glyph where no official Unicode value is available. When this is done, the glyph can appear in character map tools and can then be copied & pasted just like any other character. But when this technique is used, the PUA code point will only work for that specific font. Another font might show a completely different glyph (or nothing at all) and proper indexing or hyphenation will also fail, because these PUA characters have no standardised semantic meaning. So when you are looking for glyphs in a font, that don’t fall in the category of official Unicode characters, you need to check first which of the above methods is used. PUA characters might allow you to access any glyph in any app more easily, but you should keep in mind that their non-sematic encoding can cause problems. If you use PUA codes to access pictograms of a signage font for a vinyl cut you might be fine. But if you want to publish electronic documents which should be indexed, shared or copied, you might want to steer clear of the Private Use Area. To access official Unicode characters and PUA characters, you can try the Windows Character Map or alternative tools like PopChar or BabelMap. On Mac OS X the built in Character Viewer is quite handy, but it does not display font-specific PUA characters. For that, you might want to have a look at Ultra Character Map instead. Working with unencoded characters So how should one deal with unencoded glyphs in OpenType fonts? Unfortunately, there is no single app or no single trick, which makes them easily accessible across all apps on your operating system. But here are some recommendations which might be helpful to you. Know your type—study the “manual” When you buy a font, look out for the type specimen PDF and save it with the font. It might not always be easy to find, especially on reseller sites. So visit the designer’s or foundry’s site if necessary. The type specimen PDF will in most cases not only show the full glyph set, but also list the OpenType features to reach unencoded glyphs. You understand the possibilities of specific OpenType fonts much easier this way than by clicking through the OpenType menu of your app. This type specimen PDF from Typejockeys explains the OpenType features of Ingeborg Create your own glyph sheets if needed There is a nice free tool for Adobe InDesign called Font Table, which can generate an overview of all glyphs in a font. It just tries to access every possible glyph ID and so you get a full overview, which does not rely on Unicode code points. You can then save this glyph table as PDF or print it out. The glyph table of Arno Pro made in InDesign with Font Table Copy Glyph IDs across supported apps (Mac OS only) It isn’t heavily advertised because it doesn’t work across all apps, but Mac OS X actually allows you to copy & paste unencoded glyphs through their glyph ID. But it only works if both apps you are using support this. In FontBook, switch to the Repertoire view (⌘+2) and the just select and copy the character. You can then paste it to TextEdit for example or some other apps that use Apples text engine. If you use FontExplorer X as font manager you can also copy unencoded glyphs from the info panel of a specific font. And if you want to use this technique frequently you might want to take a look at Ultra Character Map. Accessing all those deliberately unencoded ligatures of Canapé? FontExplorer can help you. Using the Glyphs Panel (InDesign/Illustrator) If you use Adobe InDesign or Adobe Illustrator it’s very easy to work with unencoded glyphs. Not only do these apps provide a full glyph map, but you can also easily access glyph alternatives or filter the selection to only show the glyph replacements for certain OpenType features. Click on the triangles in the glyphs panel of InDesign or Illustrator to show alternative glyphs which will likely not be directly accessible through Unicode values. Find you special glyphs easily with the filter menu and double-click on the glyphs to add them to your text. The Photoshop trick While InDesign and Illustrator have a glyphs panel, Photoshop until the current version (CC 2014) does not and the options to activate OpenType features are also extremely limited. So even the unencoded glyphs of many of Adobe’s own Pro fonts might simply be unaccessible in Photoshop. But there is a trick! Photoshop allows you to copy unencoded glyphs from one app — and one app only: Illustrator. Create your text in Illustrator and apply OpenType features or select specific glyphs from glyphs panel. Select the text and copy it. Go to Photoshop and paste the text while the text tool is active. Photoshop will paste the text and maintain all glyphs. First line: Photoshop default. Second line: Pasted, but still editable text from Illustrator.

Tom Scott explains how and why we moved from ASCII to Unicode and why UTF8 has become the dominant standard to encode Unicode texts.