HTML Character Entities On Linux Or Unix - Follow Up

Hey there,

Today's post hearkens back to a post we ran back in December of 2007 regarding publishing Perl, shell and other code on Blogspot. The problem isn't really limited to blogspot.com, of course. Most sites that let you directly add content, and use a standard markup tagging system, have issues with many of the symbols found in Linux and Unix programming/shell code. I, personally, have gained a much deeper appreciation for bbcode, although it suffers from the same problems depending on what you want to "really" show on your page, just to a lesser degree.

The issue with publishing code in HTML (See, I'm going to stop kicking blogspot.com now ;) is that many of the symbols are the same as those used in the markup language itself. For instance, the < and > characters can't be written exactly as they appear when you're writing an HTML page to showcase your code, since all HTML tags (I believe ;) open with the < character and close with the > character. This can result in errors ranging from problems saving your posts to entire chunks of your code disappearing without any errors generated at all (the latter being the most disarming)

Obviously, I've been meaning to do this for a while (only 14 months from intent to action ;), but - for today - I've put together a list of common HTML character entities and how they should "really" be typed when you write your HTML. If I missed any, I'd love to hear from you, as I'd prefer for this to be a complete list, if that's at all possible :)

NOTE: Not all of these may display correctly in your web browser, since some may not be in your native character-set. Most of the fancy stuff (Greek, foreign characters, etc) was found on Elizabeth Castro's HTML XHTML Character Entity Page. Please visit that site to ensure that the information found here hasn't been expanded upon, and give her the credit she deserves for compiling so many more characters than I could ever imagine having to type :)

Enjoy, and welcome back to the work week (hopefully, you're on vacation :)

Cheers,

BASIC CHARACTERS (REGULAR):

&   is actually typed as: &
>    is actually typed as:  >
<    is actually typed as:  &lt;
"   is actually typed as:  &quot;
´  is actually typed as:  &acute;
¸  is actually typed as:  &cedil;
ˆ   is actually typed as:  &circ;
¯   is actually typed as:  &macr;
·  is actually typed as:  &middot;
˜  is actually typed as:  &tilde;
¨   is actually typed as:  &uml;
°   is actually typed as:  &deg;
÷  is actually typed as:  &divide;
½  is actually typed as:  &frac12;
¼  is actually typed as:  &frac14;
¾  is actually typed as:  &frac34;
≥    is actually typed as:  &ge;
≤    is actually typed as:  &le;
−  is actually typed as:  &minus;
²   is actually typed as:  &sup2;
³   is actually typed as:  &sup3;
×  is actually typed as:  &times;
¢   is actually typed as:  &cent;
¤ is actually typed as:  &curren;
€   is actually typed as:  &euro;
£  is actually typed as:  &pound;
¥   is actually typed as:  &yen;
¦  is actually typed as:  &brvbar;
•   is actually typed as:  &bull;
©   is actually typed as:  &copy;
†  is actually typed as:  &dagger;
‡  is actually typed as:  &Dagger;
⁄  is actually typed as:  &frasl;
…  is actually typed as:  &hellip;
¡  is actually typed as:  &iexcl;
ℑ  is actually typed as:  &image;
¿  is actually typed as:  &iquest;
‎   is actually typed as:  &lrm;
—  is actually typed as:  &mdash;
–  is actually typed as:  &ndash;
¬   is actually typed as:  &not;
‾  is actually typed as:  &oline;
ª   is actually typed as:  &ordf;
º   is actually typed as:  &ordm;
¶   is actually typed as:  &para;
‰  is actually typed as:  &permil;
′  is actually typed as:  &prime;
″  is actually typed as:  &Prime;
ℜ   is actually typed as:  &real;
®   is actually typed as:  &reg;
‏   is actually typed as:  &rlm;
§   is actually typed as:  &sect;
­   is actually typed as:  &shy;
¹   is actually typed as:  &sup1;
™  is actually typed as:  &trade;
℘  is actually typed as:  &weierp;
„  is actually typed as:  &bdquo;
«  is actually typed as:  &laquo;
“  is actually typed as:  &ldquo;
‹  is actually typed as:  &lsaquo;
‘  is actually typed as:  &lsquo;
»  is actually typed as:  &raquo;
”  is actually typed as:  &rdquo;
›  is actually typed as:  &rsaquo;
’  is actually typed as:  &rsquo;
‚  is actually typed as:  &sbquo;
    is actually typed as:  &emsp;
    is actually typed as:  &ensp;
    is actually typed as:  &nbsp;
   is actually typed as:  &thinsp;
‍   is actually typed as:  &zwj;
‌   is actually typed as:  &zwnj;

SOMEWHAT IRREGULAR CHARACTERS:

Á  is actually typed as:  &Aacute;
á  is actually typed as:  &aacute;
  is actually typed as:  &Acirc;
â  is actually typed as:  &acirc;
Æ  is actually typed as:  &AElig;
æ  is actually typed as:  &aelig;
À  is actually typed as:  &Agrave;
à  is actually typed as:  &agrave;
Å  is actually typed as:  &Aring;
å  is actually typed as:  &aring;
à is actually typed as:  &Atilde;
ã  is actually typed as:  &atilde;
Ä   is actually typed as:  &Auml;
ä   is actually typed as:  &auml;
Ç  is actually typed as:  &Ccedil;
ç  is actually typed as:  &ccedil;
É  is actually typed as:  &Eacute;
é  is actually typed as:  &eacute;
Ê  is actually typed as:  &Ecirc;
ê  is actually typed as:  &ecirc;
È  is actually typed as:  &Egrave;
è  is actually typed as:  &egrave;
Ð   is actually typed as:  &ETH;
ð   is actually typed as:  &eth;
Ë   is actually typed as:  &Euml;
ë   is actually typed as:  &euml;
Í  is actually typed as:  &Iacute;
í  is actually typed as:  &iacute;
Î  is actually typed as:  &Icirc;
î  is actually typed as:  &icirc;
Ì  is actually typed as:  &Igrave;
ì  is actually typed as:  &igrave;
Ï   is actually typed as:  &Iuml;
ï   is actually typed as:  &iuml;
Ñ  is actually typed as:  &Ntilde;
ñ  is actually typed as:  &ntilde;
Ó  is actually typed as:  &Oacute;
ó  is actually typed as:  &oacute;
Ô  is actually typed as:  &Ocirc;
ô  is actually typed as:  &ocirc;
Π is actually typed as:  &OElig;
œ  is actually typed as:  &oelig;
Ò  is actually typed as:  &Ograve;
ò  is actually typed as:  &ograve;
Ø  is actually typed as:  &Oslash;
ø  is actually typed as:  &oslash;
Õ  is actually typed as:  &Otilde;
õ  is actually typed as:  &otilde;
Ö   is actually typed as:  &Ouml;
ö   is actually typed as:  &ouml;
Š  is actually typed as:  &Scaron;
š  is actually typed as:  &scaron;
ß  is actually typed as:  &szlig;
Þ  is actually typed as:  &THORN;
þ  is actually typed as:  &thorn;
Ú  is actually typed as:  &Uacute;
ú  is actually typed as:  &uacute;
Û  is actually typed as:  &Ucirc;
û  is actually typed as:  &ucirc;
Ù  is actually typed as:  &Ugrave;
ù  is actually typed as:  &ugrave;
Ü   is actually typed as:  &Uuml;
ü   is actually typed as:  &uuml;
Ý  is actually typed as:  &Yacute;
ý  is actually typed as:  &yacute;
ÿ   is actually typed as:  &yuml;
Ÿ   is actually typed as:  &Yuml;
ℵ is actually typed as:  &alefsym;
∧   is actually typed as:  &and;
∠   is actually typed as:  &ang;
≈  is actually typed as:  &asymp;
∩   is actually typed as:  &cap;
≅   is actually typed as:  &cong;
∪   is actually typed as:  &cup;
∅  is actually typed as:  &empty;
≡  is actually typed as:  &equiv;
∃  is actually typed as:  &exist;
ƒ   is actually typed as:  &fnof;
∀  is actually typed as:  &forall;
∞  is actually typed as:  &infin;
∫   is actually typed as:  &int;
∈   is actually typed as:  &isin;
⟨   is actually typed as:  &lang;
⌈  is actually typed as:  &lceil;
⌊  is actually typed as:  &lfloor;
∗  is actually typed as:  &lowast;
µ  is actually typed as:  &micro;
∇  is actually typed as:  &nabla;
≠    is actually typed as:  &ne;
∋    is actually typed as:  &ni;
∉  is actually typed as:  &notin;
⊄   is actually typed as:  &nsub;
⊕  is actually typed as:  &oplus;
∨    is actually typed as:  &or;
⊗  is actually typed as:  &otimes;
∂   is actually typed as:  &part;
⊥   is actually typed as:  &perp;
±  is actually typed as:  &plusmn;
∏   is actually typed as:  &prod;
∝   is actually typed as:  &prop;
√  is actually typed as:  &radic;
⟩   is actually typed as:  &rang;
⌉  is actually typed as:  &rceil;
⌋  is actually typed as:  &rfloor;
⋅   is actually typed as:  &sdot;
∼   is actually typed as:  &sim;
⊂   is actually typed as:  &sub;
⊆   is actually typed as:  &sube;
∑   is actually typed as:  &sum;
⊃   is actually typed as:  &sup;
⊇   is actually typed as:  &supe;
∴  is actually typed as:  &there4;
Α  is actually typed as:  &Alpha;
α  is actually typed as:  &alpha;
Β   is actually typed as:  &Beta;
β   is actually typed as:  &beta;
Χ   is actually typed as:  &Chi;
χ   is actually typed as:  &chi;
Δ  is actually typed as:  &Delta;
δ  is actually typed as:  &delta;
Ε is actually typed as:  &Epsilon;
ε is actually typed as:  &epsilon;
Η   is actually typed as:  &Eta;
η   is actually typed as:  &eta;
Γ  is actually typed as:  &Gamma;
γ  is actually typed as:  &gamma;
Ι   is actually typed as:  &Iota;
ι   is actually typed as:  &iota;
Κ  is actually typed as:  &Kappa;
κ  is actually typed as:  &kappa;
Λ  is actually typed as:  &Lambda;
λ  is actually typed as:  &lambda;
Μ    is actually typed as:  &Mu;
μ    is actually typed as:  &mu;
Ν    is actually typed as:  &Nu;
ν    is actually typed as:  &nu;
Ω  is actually typed as:  &Omega;
ω  is actually typed as:  &omega;
Ο is actually typed as:  &Omicron;
ο is actually typed as:  &omicron;
Φ   is actually typed as:  &Phi;
φ   is actually typed as:  &phi;
Π    is actually typed as:  &Pi;
π    is actually typed as:  &pi;
ϖ   is actually typed as:  &piv;
Ψ   is actually typed as:  &Psi;
ψ   is actually typed as:  &psi;
Ρ   is actually typed as:  &Rho;
ρ   is actually typed as:  &rho;
Σ  is actually typed as:  &Sigma;
σ  is actually typed as:  &sigma;
ς  is actually typed as:  &sigmaf;
Τ   is actually typed as:  &Tau;
τ   is actually typed as:  &tau;
Θ  is actually typed as:  &Theta;
θ  is actually typed as:  &theta;
ϑ is actually typed as:  &thetasym;
ϒ  is actually typed as:  &upsih;
Υ is actually typed as:  &Upsilon;
υ is actually typed as:  &upsilon;
Ξ    is actually typed as:  &Xi;
ξ    is actually typed as:  &xi;
Ζ   is actually typed as:  &Zeta;
↵  is actually typed as:  &crarr;
↓   is actually typed as:  &darr;
⇓   is actually typed as:  &dArr;
↔   is actually typed as:  &harr;
⇔   is actually typed as:  &hArr;
←   is actually typed as:  &larr;
⇐   is actually typed as:  &lArr;
→   is actually typed as:  &rarr;
⇒   is actually typed as:  &rArr;
↑   is actually typed as:  &uarr;
⇑   is actually typed as:  &uArr;
♣  is actually typed as:  &clubs;
♦  is actually typed as:  &diams;
♥  is actually typed as:  &hearts;
♠  is actually typed as:  &spades;
◊   is actually typed as:  &loz;

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Using Iconv To Convert Character Sets On Linux And Unix

Hey There,

Today's topic was obliquely referenced in yesterday's post on using the online multilingual dictionary from the Bash CLI. Yes, after I woke up around 1 am and thought about it, the results did seem odd. It turns out, when I went to the page, that the multilingual dictionary on reference.com is actually house within a frameset under the online Thesaurus. My belated apologies for getting the name incorrect, although, if you think about it, it's not even a multilingual dictionary (not really). If you enter a word in English, get the definition in English and then get the word spelled for you in 30 different languages, that's more like an online dictionary with a keyword translator. If it were truly a multilingual dictionary, I would expect to get definitions in 30 different languages. Perhaps I want for too much at some times and not enough at others. Probably a good mix of the two ;)

Today, we're going to look at character conversion (Since, if you looked at the picture in yesterday's bash script post, you probably noticed that a lot of the foreign characters came up looking like garbage. Ok; maybe not garbage, but certainly not in the way they were intended to be shown. All manner of inflective symbols and more picturesque languages (Cyrillic) etc, just don't show up well on a standard U.S. computer.

There are, as I see it, two basic ways to get around this.

1. Install every font set and allow for every single type of encoding available. If your OS supports it, you should know be able to read Japanese Kanji and Russian, German, French, etc with the proper characters displayed. Still, I'm not sure there's any guarantee that this would always work, or that the benefits of keeping a huge cache of fonts and displays wouldn't end up under-weighing the burden.

2. Use a program called "iconv" (present on most Linux and Unix boxes nowadays), and convert when necessary.

I'll assume we want to proceed with option 2 and check out a simple example. But, before we do that, I'd like to point out that, from distro to distro, the implementations of the iconv program can vary greatly. For a striking difference, consider Sun's native version vs. Gnu's. In the Sun's version from 2.6 (Perhaps not a fair comparison, but their native client hasn't come up all that much either), you only have two flags to choose from (actually, you have to use them both): -f (The "convert from" character set) and -t (the "convert to" character set). Don't know what character sets you have installed? Your issue ;)

In the Gnu version, you've got those options (plus the long versions of both -- guess what those are ;) a few other options to suppress errors and gloss over characters that can't be decoded and, best of all, a "--list" option that will list all the character sets installed on your system. This list can be very useful if you want to convert to or from a character set but aren't sure if you can. With Gnu's iconv, you'll know in a few seconds without having to look anywhere else (For Sun 2.6, and later, you can find the definitions under /usr/lib/locale. If you do a "man" on "charmap" it's referring to the charmap source files in there - There may or may not be a standard directory like /usr/share/ii8n/charmaps on your Sun system). For useful information you can actually work with (on a less-than-abstract level) do a "man" on "localedef" and/or "locale." Didn't want to leave you with nothing on that side of the coin ;)

Here's a simple, but dramatically visual enough way, to show you what iconv can do. This example isn't really "practical," except insofar as it shows you proof from a few different angles that "iconv" actually can do what it says.

First, we'll check out our test file and make sure it's good to go:

host # ls
regular
host # file regular.ascii
regular: ASCII text
host # cat regular.ascii
This is a file created using our standard charset
host # cat -v regular.ascii
This is a file created using our standard charset

Then, we'll check the version of iconv (We want Gnu) to make sure we can work as efficiently as possible with as little charmap/locale/charset knowledge as possible:

host # iconv --version
iconv (GNU libc) 2.3.2... <-- Good :)

host # iconv --list
The following list contain all the coded character sets known.  This does
not necessarily mean that all combinations of these names can be used for
the FROM and TO command line parameters.  One coded character set can be
listed with several different names (aliases).

  437, 500, 500V1, 850, 851, 852, 855, 856, 857, 860, 861, 862, 863, 864, 865,
  866, 866NAV, 869, 874, 904, 1026, 1046, 1047, 8859_1, 8859_2, 8859_3, 8859_4,
  8859_5, 8859_6, 8859_7, 8859_8, 8859_9, 10646-1:1993, 10646-1:1993/UCS4,
  ANSI_X3.4-1968, ANSI_X3.4-1986, ANSI_X3.4, ANSI_X3.110-1983, ANSI_X3.110,
  ARABIC, ARABIC7, ARMSCII-8, ASCII, ASMO-708, ASMO_449, BALTIC, BIG-5...

There are literally 50, or so, more lines of available character sets we can convert from and to. Just run that command anytime if you need to check out the entire list. This is great because now we don't have to do any extra work to figure out that part.

Then we'll move on to converting the regular ASCII file to a UTF-16 file. We could convert it to UTF-8, but the "file" command doesn't consider the two charsets "different" enough (and neither do your eyes) and we want this to stand out. Also, UTF-16 has some fun features like not having complete EOL chars, etc, and is unsearchable by standard "grep." Notice the output from the "file," "grep," "cat" and "cat -v" commands once we've finished with the converse and do our comparisons:

host # iconv --from-code ASCII --to-code UTF-16 --output regular.utf16 regular.ascii
host # ls
regular.ascii  regular.utf16
host # file regular.ascii regular.utf16
regular.ascii: ASCII text
regular.utf16: Little-endian UTF-16 Unicode character data, with no line terminators
host # grep file regular.ascii regular.utf16
regular.ascii:This is a file created using our standard charset
host # cat regular.ascii
This is a file created using our standard charset
host # cat regular.utf16
ÿþThis is a file created using our standard charset
host # cat -v regular.ascii
This is a file created using our standard charset
host # cat -v regular.utf16
M-^?M-~T^@h^@i^@s^@ ^@i^@s^@ ^@a^@ ^@f^@i^@l^@e^@ ^@c^@r^@e^@a^@t^@e^@d^@ ^@u^@s^@i^@n^@g^@ ^@o^@u^@r^@ ^@s^@t^@a^@n^@d^@a^@r^@d^@ ^@c^@h^@a^@r^@s^@e^@t^@
^@

Man, did we make a mess of things ;) Now, we'll put everything back the way it was (hopefully) and run the same tests:

host # iconv --from-code UTF-16 --to-code ASCII --output regular.new regular.utf16
host # ls
regular.ascii  regular.new  regular.utf16
host # file regular.ascii regular.new
regular.ascii: ASCII text
regular.new:   ASCII text
host # grep file regular.ascii regular.new
regular.ascii:This is a file created using our standard charset
regular.new:This is a file created using our standard charset
host # cat regular.ascii
This is a file created using our standard charset
host # cat regular.new
This is a file created using our standard charset
host # cat -v regular.ascii
This is a file created using our standard charset
host # cat -v regular.new
This is a file created using our standard charset

And, voila, we've got everything back. "file" returns the same information, both files can be catted again and "grep" works on the file that we converted back from UTF-16 to ASCII.

That, in a nutshell is what "iconv" can do for you. The possibilities are endless, as (of course) are the amount of problems you can create for yourself ;)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Making Solaris 10 Zones Recognize Veritas Raw Devices

Hey there,

Here's something I learned today that I never knew before! ...Actually, when you think about it, everything you ever learn is something you never knew before. Otherwise, you'd already know it, so there'd be no learning involved... Where am I? ;)

Today's short-post is a little trick concerning Solaris 10 and the Veritas File System/Volume Manager. I've been using Veritas with Solaris 10 for quite a while, so this was pretty interesting to find out. In the usual case, I could always set up Solaris zones and hand them Veritas File System partitions with no incident. Usually, something like this:

host # zonecfg -z myNewZone
... <-- Here we skip past the create and "set zonepath" type stuff.  We have more information on dealing with Solaris zones in this old post
zonecfg:myNewZone> add fs
zonecfg:myNewZone:fs> set dir=/my/directory
zonecfg:myNewZone:fs> set special=/myNewZone/my/directory
zonecfg:myNewZone:fs> set type=lofs
zonecfg:myNewZone:fs> end
zonecfg:myNewZone> commit
zonecfg:myNewZone> exit

and life was beautiful ;)

Out of sheer luck (or horrible misfortune), I've always ended up at places that (while they may make heavy use of Veritas Volume Manager, File System, Cluster Server, etc) either went with some combination of SunCluster, Sun Solaris, HP and Oracle or Informix. Now, there's nothing wrong with any of the possible combinations you could end up with here. In fact, there's - more exactly - nothing wrong, since all of them (well, not HP-UX) can work with Solaris 10 zones using raw devices.

If you're not familiar with basic disk access concepts in Unix, at a very basic level, disk in Unix is generally accessible via two standard gateways: Block devices or Raw/Character devices. Most of the time, you probably deal with the block device interface, as this allows buffered sector reads of the device (generally a hard drive/disk) and is generally faster for most operations. To demonstrate what I mean, if you wanted to read a few bytes from HardDrive1, you could read those few bytes from your block device (almost to the byte) , but you'd have to read the entire device in order to get those few bytes when accessing the information via the raw device interface.

Raw devices do have their benefits, of course, or they probably wouldn't be around any more. Like their name suggests, they allow direct (raw) access to the disk. This can make it possible for you to recover information if your disk becomes corrupted (for instance) but is mostly used by software that likes to manage disk on its own and not have to rely on the Operating System. Most database software uses disk in this manner. Oracle, for instance, creates its own "file system" and (although it will work on UFS or any other filesystem) they say it works a lot more efficiently when it can be allocated raw disk and allowed to manage it on its own. It's probably true to some degree. Does anyone really know? ;)

Anyway (back from that two-paragraph aside ;), what I learned today is that Solaris 10 Zones can't (or won't) deal with raw Veritas devices (WT_?) I also found out that there's a way to get around that issue (hence, the post - I still owe you that one that goes on and on forever and leads nowhere ;).

You can get around this deficiency by doing the following (assuming you've already taken care of the basics of setting up your Veritas Volumes and Filesystem) and have created your zone with a base filesystem, like above, and want to add a raw device.

1. Check out your raw device file in the appropriate directory:

host # ls -l /dev/vx/rdsk/myRawDisk
crw------- 1 root root 289, 45000 Aug 24 11:13 myRawDisk

and make note of the Major and Minor numbers of the device (289 and 45000 respectively, above). You'll need these to feed the mknod command you're going to hammer out next (we'll assume you set your zonepath equal to /myNewZone - oh, yeah, and it's also important that you're not in your "Zone" when you do this):

host # cd /myNewZone/dev
host # mknod myRawDisk c 289 45000

2. And (it seems too simple!!! ;) then you just need to (well... you should ;) log into your zone and check it out, with a simple "dd" test or something:

host # zlogin -l root myNewZone
...
zone # ls -l /dev/myRawDisk
crw------- 1 root root 289, 45000 Aug 24 11:13 /dev/myRawDisk
zone # dd if=/dev/myRawDisk of=/dev/null
4096+0 records in
4096+0 records out

And, there you go. That problem you never knew you had is solved! ;)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Ignoring All Standard Characters Using Perl In Linux Or Unix

Hey there,

Every once in a while, you have to reverse the order of your thinking. For today's example of code to do cherry-picking of values, we'll demonstrate just that. In most scripts, you're either looking for a specific set of values (or a relatively specific set), or you're trying to ignore them. Unix and/or Linux will attempt to print any character it finds in a script if you ask it to(even if it's, technically, unrepresentable, like a control-character sequence), which can make for some interesting output. Here, we're going to look for everything that we don't want to find (??? ;)

Basically, our script attached to today's post is going to look for every character in a given file that counts as a "special" character. And, by special, I mean goofy :) Since we have no idea what kind of insane characters we might not want to see, we have to begin by defining everything that we know and excluding all of that so that we only match, and ignore, things we don't know about.

This starts out simple, of course. We know we want to ignore the alphabet (upper and lower case) and the regular set of numerals. The next step is relatively simple as well: We know we want to ignore all the other "normal" characters. If you recall from our post on generating all possible passwords using Perl, there are 94 regular characters (including the alphaBET and numbers, noted already) that we need to ignore, plus simple stuff like spaces, tabs, newlines, carriage return and bells. There may be more... we'll never know until we don't ignore it ;)

The trickiest part of script-work like this is the Hell of backslash-escaping that you'll inevitably get caught up in. Hopefully, the script we've attached today will help you out in that regard. If you feed this script any file, it should print out only the lines with"bizarre" characters in them.

For example, given a file with these contents:

0O
BBC
hey there
little ÿ¾åbÿ
756_RATT
LiP_$3rv1c3
ENTER

Running this Perl script will produce the following result (printing only lines with strange characters in them)

host # ./freaky.pl FILE
little ÿ¾åbÿ

Here's to finding out what you don't know :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# freaky.pl - only print out lines with unknown chars
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if ( $#ARGV != 0 ) {
 print "Usage: $0 FileName\n";
 exit(1);
}

$filename = $ARGV[0];

open(FILE, "<$filename");
while (<FILE>)  {
        if ( $_ =~ /[^A-Za-z0-9\s\t\\r\a`\-=\[\]\\;\',\.\/~!@#$%^&\*\(\)_+\{\}\|:\"<>\?)]/ )        {
                print $_
        }
}
close(FILE);

, Mike

Fun Using Banner To Send Subliminal Messages In Cygwin Linux

Greetings, once again,

Here's a good "Lazy Sunday Post" in the same vein as previous posts on using color in the shell and the like. Much like a psychic reading, this post is for entertainment purposes only ;)

Today, I put together a little no-variable script (from which a real script could easily grow) using the "banner" command in Cygwin Linux. It's very simple to use and takes no arguments at all. You can execute it simply, like this:

hot # ./subanner <--- If there were arguments they might be "what your banner says" and "what you really mean" ;)

I didn't bother putting in any error checking, either, since this is just for a goof and doesn't take any variables. This works well on any implementation of the "banner" command that accepts the -c flag to indicate what character the banner should be printed out with. If your distro of Unix or Linux doesn't have this capability, Perl has a Text::Banner module that can do the same thing.

Hope you have fun mucking with this (make it your own. We all have something unique to say ;) and, at the same time, let users that log on to your machines know what you really think :)

p.s. If anyone knows how to strip the ending \012 off of banner output (And I've tried octal dump, sed, awk, tr, hex and octal codes, Perl, etc), please email me - it's making me nuts ;) Therefore, this banner is vertical rather than horizontal.

Maybe if it wasn't the weekend my brain would be in better shape to tackle this issue ;)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# subanner - subliminal banner :)
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

message="Be Kind"
meaning="Go Away"

count=0

while [ $count -lt ${#message} ]
do
        banner -c "${meaning:$count:1}" "${message:$count:1}"
        let count=$count+1
done

, Mike

linux unix internet technology

Finding the Number of Characters In A Variable Regardless Of Your Linux Or Unix Shell

Hey there,

Here's a simple enough task: Finding (or determining) the amount of characters in a string variable. In most Unix or Linux shells, this has become a trivial exercise. For instance, in bash or ksh, you can easily tell how many characters are in a particular string variable by either writing one line of code, or skipping right to the end and printing out the answer. By way of example:

host # x=thisIsMyVariable
host # echo ${#x}
16

And it's that simple. The variable x, which contains the string value "thisIsMyVariable", does, indeed, contain 16 elements (or characters). As you can see, in the more advanced shells, taking care of this is no problem.

However, in certain circumstances, you may be required to use a less-advanced, but more highly-portable, shell, like the original Bourne shell. In that case, this same sequence of commands would result in the following:

host # x=thisIsMyVariable
host # echo ${#x}
host # <--- At best, you'll get zero here. Expect nothing. You might possibly get an error.

Our script attached today, uses the old-style expr (Well, it uses the new style expr with the old format and options, again, for maximum portability between systems and shells). It does the exact same thing that echo'ing ${#VARIABLE_NAME} does, but goes about it in a grueling and confusing manner. Remember back when you had to write scripts that took the shell's needs into consideration before your own? ;) Nowadays, things are much more intuitive.

Given a string, like "thisIsMyVariable," the following script will produce the exact same results. It will spit out how many characters are in that string, but take a longer, more confusing ,route to the answer. This is another good example of security through obfuscation as well. In general, having to use the expr command can be a hurdle for folks who've never used a shell that can't do arithmetic or string comparison on its own :)

Sample output (as boring as it is ;):

host # ./string.sh thisIsMyVariable
16

And now, you can get the string length no matter how limited your shell is!

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# string.sh - print out how many characters are in a string variable
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [ $# -ne 1 ]
then
 echo "Usage: $0 string"
 exit 1
fi

string=$1
echo $string|while read x
do
        chars=`expr "$x" : '.*'`
  echo $chars
done

, Mike

Using OD To Find Bad Characters In Files On Linux Or Unix

Hey there,

Today we're going to take a look at a command called "od" (short for "octal dump") and the sometimes elusive search for, and destruction of, those "mystery characters" that make your scripts not run correctly. Sure, there are programs out there like dos2unix that pretty much take care of this problem for you, but that's like taking candy from a baby ;) And there's always the possibility that they won't be able to find the bizarre character you're trying to get rid of. What do you do then?

We covered the other side of this issue in an earlier post on adding real control characters to your files. Now, it's time to finally undo that damage ;)

od is a great utility for finding out if there really are bad characters in your file in the first place. "cat -v" works, too, but you can only trust it as much as you trust your terminal settings. I generally assume my tty settings are correct, but they could be up to just about anything ;) For today's purposes, "cat -v" serves as an excellent corollary to od's output and, hopefully, makes it even easier to understand.

For instance, if you're looking for a control character in a file (we'll say ^M, since that one's the most popular), cat -v does a great job of bringing it out in the open, like this:

host # cat file
def
host # cat -v file
abc^Mdef

od can do the same thing, if only slightly differently:

host # od -c file
0000000 a b c \r d e f \n
0000010

Notice that, with the "-c" flag used, we can make the "^M" show up as an easily identifiable "\r". We also get to see what having that "actual" ^M in our file has done. Notice the second line in the "od" output (00000010). The first column of od's output is the byte-offset of the line. The first line, naturally starts at an offset of 0 and, since od only grabs the amount of bytes per line that you can actually see in the output, the second entry (the offset of the 6 letters and the new-line) should be 7 (This is because the numbering starts at zero - just in case you didn't know. No offense if you did ;). Another clue that something is amiss.

This is what the file would look like normally, using the same example as above, but without the ^M inserted:

host # cat file
abcdef
host # cat -v file
abcdef
host # od -c file
0000000 a b c d e f \n
0000007

Perfect :) The really nice thing about using od with the "-c" flag is that it prints out all regular characters in ASCII, which, studies have shown, most humans prefer reading ;) Only when od runs into a character it doesn't understand, does it do any substitution, as with the ^M. Below is the basic table of conversions the "-c" flag will do automatically:

null         \0
backspace    \b
form-feed    \f
new-line     \n
return       \r
tab          \t

After that, if od doesn't recognize a regular ASCII character, it will finally live up to its name and dump the octal value (of course, we could have forced it to do this all along, but we're trying to make this command seem accessible ;). Here's an example of what we see when od can't translate for us ASCII folks anymore:

host # cat file
bcdef
host # cat -v file
a^?bcdef
host # od -c file
0000000 a 177 b c d e f \n
0000010

Okay. So, in this instance, "cat -v" shows us what looks like a "backspace" or "erase" character (^?). od's output is slightly different. The "what would be the next line" offset is at 10. That's the sum of all 6 ASCII characters, the new-line and the 3-digit octal code, so everything's in order (even though it's not right).

The really sweet thing is that you, armed with the knowledge od has provided, can remove the character from the file with tr (or your fancy file/character manipulation command of choice) using the octal code, which makes it completely unnecessary for you to remember some bizarre symbol ;)

host # cat file
bcdef
host # cat -v file
a^?bcdef
host # od -c file
0000000 a 177 b c d e f \n
0000010
host # tr -d '\177' <file
abcdef
host # tr -d '\177' <file >file2 <--- Where the magic happens...
host # cat file2
abcdef
host # cat -v file2
abcdef
host # od -c file2
0000000 a b c d e f \n
0000007

Hope this makes the crusty old "od" command seem less daunting. You can do a whole lot more with it, of course, but there will be plenty of time for that later :)

Cheers,

, Mike

linux unix internet technology

Using Unpack To Convert Binary Values To Decimal

Hey There,

Today's post may seem like a step back for regular readers, but I like to keep this blog accessible, so I've decided to revisited the Perl unpack function to demonstrate how it can be used in one of its simplest forms on either Linux or Unix. A more advanced example can be found in our previous post on taking a look inside the wtmpx file.

The Perl script we're presenting today only consists of one simple unpack function surrounded by a whole bunch of processing code. If certain parts were modularized, this would make a decent simple function in a greater calculator/conversion script.

Notice that when we use the unpack value to convert a binary number, say:

11101110

to the decimal number:

238

all we're really doing is feeding the value (11101110) to the unpack function and telling it to convert the input using a standard template ("C" in this case - which stands for an unsigned character - octet - value). You'll not that we've actually put "C*" to account for an infinitely variable number of unsigned octet character values. If we didn't do this (Just put "C" instead), the unpack function would only process the first character the script fed it, so our original input:

11101110

would translate to this:

1

So, as you can see, in this simple function - unpack("C*", $binary) - the definition of the predefined template value C can make a huge amount of difference. It can also allow you to be as specific as you need to be. For instance - unpack("CC", $binary) - would produce:

3

from the same input. This can also be written as - unpack("C2", $binary) - which will save you a lot of typing if you want to match exactly 245 C's ;)

Hope you find this script useful and that it helps you to get more into Perl's pack and unpack functions. Once you get past the seeming incomprehensibility of the general statements, it all makes sense :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# bd - convert binary numbers to decimal
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

print "Binary Number:\t";
chomp ($binary=<stdin>);
@bitvalue = unpack("C*", $binary);
$bitvalue = @bitvalue;
$substring = $bitvalue - 1;
while ($substring ne -1) {
 $number = substr($binary, $substring, 1);
 $binarray[$substring] = $number;
 --$substring;
}
@decarray = reverse(@binarray);
$exponent = 0;
foreach $subscript (@decarray) {
 if ($subscript == 1) {
 $subscript = 2 ** $exponent;
 } elsif ($subscript == 0) {
  $subscript = 0;
 } else {
  print "Are you sure you know what a Binary Number is?\nMaybe you shouldn't be using this...\n";
  exit;
 }
 $exponent++;
}
$decimal = "0";
foreach $integer (@decarray) {
 $decimal += $integer;
}
print "Decimal Value:\t$decimal\n";
exit;

, Mike

linux unix internet technology

Simple Encryption and Decryption For Fun And No Profit

Here's another little fun thing for the weekend. It's based on Perl's pack and unpack functions and is a good introduction to using them for other purposes. It's also a good way to rediscover the fun of "passing notes in class," even though you're older and have a real job now ;)

You can use this script any way you want to; in fact it's written with a few things left out (how you want to deal with shell special characters -- another complete post on its own -- and if you want to expand on it to read and/or write to STDIN/STDOUT so you can pipe one instance to another, even though that defeats the purpose to a certain degree).

Hopefully, it will pique your curiosity about Perl and its ability to compress and convert different data types (I'm using hexadecimal and character here, but the available list is voluminous).

I've included usage points in the comments section of the script, but the basic usage would be:

tranz.pl encode your message here
tranz.pl decode 458616e6b6370264f62702659637964796e6760245865602c496e657870216e6460255e6968702d456e6167656279656 <-- The Hex output from a message encoded with this script.

Enjoy, and have a safe Sunday :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# 2007 - Mike Golvach - eggi@comcast.net
# Sanitize shell special characters in
# whatever manner you prefer - or just
# backslash them on the command line :)
#
# Sample usage:
# trans.pl encode hi there
# trans.pl encode hi there >FILE
# trans.pl encode `cat FILE`
# trans.pl decode 8696024786562756
# trans.pl decode `cat FILE`
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

$input = $ARGV[0];
shift @ARGV;
$message = join(" ",@ARGV);

if ( $input eq "encode")        {
        $output=unpack("h*",$message);
        @length=$output=~/.{0,256}/g;
        print("$output \n");
}       elsif ( $input eq "decode" )    {
        chomp($message);
        $output.=pack"h*",$message;
        print "\n$output\n";
}       else    {
        print "Usage: $0 [encode|decode] whatever you want to type\n";
        exit(1)
}

, Mike

linux unix internet technology