Why I was a Korn Shell Fan

A lot of Linux fanboys these days love everything Gnu, and like to talk shit about non-Gnu stuff. Perhaps it’s just a modern-day version of the empty Commodore vs. Apple vs. TRS-80 arguments we had back in the day: they each had strengths and weaknesses, so a lot depended on what you wanted to do. And a lot had to do with what you were familiar with.

Nevertheless, there are reasons I prefer Korn shell. If you have different reasons, you may prefer other shells. And that’s just fine too—but if you’re just sticking with something because it’s all the rage, that’s not a good reason.

So let’s start with the small stuff and work our way up.

Floating point math

You don’t need it often, but when you do, you don’t have to screw around using bc or dc to do it.

Korn shell’s built-in sleep accepts floating-point values too.

print command

echo has differing variants on different platforms. Ksh’s print works the same everywhere.

Compound variables

You can create a variable that has multiple fields in it. A simple struct of sorts. See typeset -C, which, by the way, you can do with…

Built-in documentation

Try it out: typeset --?? (or typeset --man) to learn about the myriad of options to typeset. No longer do you need to pore through a crazy long man page to find what you want, it’s right at your finger tips. Almost all the built-ins have them. And the man page isn’t all: -? gives a short synopsis, and --help shows the synopsis and option details.

And if you want to print that:

typeset --nroff 2>&1 | groff -man -T ps | ps2pdf - typeset.pdf

It also does --html, --about, --short, --long. See getopts --man for more, and getopts --??help for a full list of display options.

Self-documenting getopts

Those man pages just mentioned? They come from the getopts string using Ksh’s long format. (The commonplace short format is accepted too.) The long format is described in the getopts built-in man page, but you may also need to cookbook from an example; I’ve included some info after this essay. The format is a little weird, but easier than writing raw man pages, which you can create with the --nroff option. And it’s used to provide the same sort of short synopses, long synopses, and man pages on demand. Lastly, the getopts long options string is both directive and documentation, ensuring docs and code are in sync.

Good documentation

Bash has decent documentation, comparable to the quality of Korn shell. zsh documentation, however, can be sparse, especially regarding some of the command options. That and the frequently-corrupting shell history (especially if you do a lot of multi-statement commands on the command line, which seem to encourage the behavior, then are lost uninvoked when the corruption happens) drove me away.

FPATH

You can set a function path. If a function is requested and not known, Korn shell searches and loads it if found. You can have a library of functions, without having to copy-and-paste your code (or the bugs that go with that code).

True local variables

In modern shells, there are two ways to declare functions:

name () { ...

and

function name { ...

Usually, these behave the same. And because of backward combatability, the former must behave like the old Bourne Shell, which was insanely simple by today’s standards, to the point of being broken by modern expectations. But it’s a long-lived problem in software: we can’t fix anything because of all the things the fixes would break.

One of those simplicities is the lack of local variables. Recursion wasn’t possible in the old world. (Well, maybe with judicious use of eval. Is your hair gray yet?)

And looking back, Bourne shell only had the first form of function invocation. Korn shell introduced the second form with different behaviors to correct the defects.

So in Korn shell, if you create a local variable in a C-style function declaration (with the parenthesis), the shell ignores you and uses a global variable.

When you use function, you do get local variables. Real, true, lexical-scoped function scope locals: the functions you call can’t see them, nor fudge them up. And if they forget to declare their own locals, they pollute the global namespace instead of silently screwing up someone else’s locals.

In Bash, locals are created in the current context. But they are dynamically scoped, thus available to every function you call. Or any function they call in turn. And if one of those functions forgets to declare a local, it may very well unintentionally alter a variable belonging to another function. Pollution of the global namespace isn’t good, but it’s less hazardous than this nonsense.

Pipeline variable retention

When running a pipeline, Korn shell retains any values set at the end of the pipeline. So this construct works:

grep "^#" < source | sed -e ... | some other stuff | read variable

But in Bash, it doesn’t. The variable is read into a subshell which is disbanded when the pipeline completes. Now, the argument from Bash proponents is to just do this:

variable=$(grep "^#" < source | sed -e ... | some other stuff)

And that’s true enough when all you want is one line out of a file. But what if you want to stick a while loop on the end?

float total=0
grep "cleared" < checkbook | awk '{print $NF}' | 
while read value
do
        let "total += value"
done
print "The total is $total."

In Bash, this wouldn’t work because it’s using floating point math. But even if it was integer math, it wouldn’t work because it would total up the value, then the pipeline would close and the subshell would exit, taking total with it.

I realize there’s other ways to do this particular example. It wouldn’t be hard to run that through sed and make input you could feed to bc or dc to get a total. Or you could feed in a sentinel value to cause the loop to print its value, and capture that. But this is clearer and simpler. No hurdles.

And even if this is a hokey example, the concept stands. The pattern of read input, process it through a pipeline, and then amalgamate/ aggregate it at the end is not rare.

Note that since Bash 4.2, this is behavior is available via shopt -s lastpipe. However, this behavior only takes effect if job control is disabled, so it won’t work on the command line or in any functions you have loaded.

Interactive Use

For interactive use, the two shells seem very similar to me. Both have emacs and vi edit modes, quoting is identical (or nearly so) between the two, and many of the built-in variables are the same.

Bash has the bind command for keymapping; ksh has a KEYBD trap. I’m not sure which is better.

To the extent that Bash has emulated lots of Korn-shell pioneered behaviors, they’ve done well. Except the pipeline handling.

I have spotted at least one feature—case manipulation in parameter substitution—that’s been added to ksh based on its presence in Bash. So the copy-catting is going both ways.

And there are other unique Bash features, like the hook function for missing commands; ksh has no equivalent. (If I was implementing it, though, I’d make it a trap. It would make more sense than a magically named function.)

Imperfections

I do worry about the state of Korn shell. David Korn has moved on, and though AT&T has open sourced the code, will Korn shell be kept up? Only time will tell.

The code is old, and it looks it; like something written in the 80s. How much edit fatigue does it have? I’ve delved enough to know there code is ugly. And I’ve watched the current maintainers, at least one of whom seems to be a bull in a China shop. He’s done some awesome stuff—ripping out a legacy, AT&T-proprietary memory manager and moving to standard malloc—but he also aggressively removes anything he doesn’t care about, like Windows support. (I don’t care about it either, but I’ll bet there’s someone who does.)

Unfortunately

Unfortunately, it’s 2020, a year that in the future we’re going to wish we could just forget.

It’s also the year “ksh2020” started making the rounds. Several years ago, the code was open-sourced; after a few years of neglect, a few folks took notice and started maintaining the code base. I contributed a few minor fixes myself.

Unfortunately, one developer has proven to be a “bull in a pottery shop”, and while some of his efforts are things that definitely needed to be done— stripping out a proprietary memory manager and just using malloc, for one— he never wanted to hear different opinions.

He asked on ksh developers discussion if anyone used ksh on Windows, and when nobody responded, he ripped the Windows support out. I don’t think he asked anywhere else, where there might have been Windows users to object.

Someone had provided a patch to add a new format for shell parameter expansion, to allow capitalization, but it was poorly implemented, not entirely compatible and didn’t quite work how you’d expect. But I thought it would be handy, avoiding lots of upper=$(print -- "$thisvar" | tr 'a-z' 'A-Z') sequences, if it worked. I developed a patch, but there was some complaint (which I think was legitimate), so I reworked my patch again, and came up with a very nice implementation that was compatible with the bash version but better. Now krader faught it because it wasn’t traditional ksh, and why were we trying to be like bash anyway? He shot down the fix and persuaded them to rip out the feature entirely.

They did take my patch for fixing prompt column counting, adding some additional escape sequence support. Later on I had some reason to look at some older ksh code, and stumbled across the column-counting code—where I saw some code that wasn’t mine, but did something similar. I pulled up the later code and compared—parts had been diked out, and I unknowingly replaced them because their removal had broken things. I never tracked down who broke it, but I have my suspicions.

After a while, I realized my help wasn’t really wanted. In any discussion, Krader’s mind was set, his touted years of experience showing he was more right than anyone, and those who disagreed were wrong. Trying to help was more frustration than it was worth, so I stopped trying. I’m aware I’m not alone.

In July 2020, I upgraded to LinuxMint 20, which came with ksh2020 for ksh. I started having weirdness with file globbing, where ls would list files but echo * would return nothing; a single star matched nothing even though there were a bunch of files and subdirectories in the current directory.

If globbing cannot be trusted, the shell cannot be trusted.

And after ksh2020 flopped for being a buggy piece of trash, he bogged off and left his broken handiwork rotting on GitHub. Open-source developers should consider the lesson offered: how one self-appointed, over-confident, mindless-of-others, reckless developer who is blind to any damage he does can taint a codebase and destroy others' inspiration to contribute.

There seem to be some saner folks trying to start again from an earlier, untainted version, but only time will tell.

Replacement: zsh

Interactively, I have settled on zsh as a replacement. It isn’t a drop-in replacement because there are incompatibilities, but it’s somewhat close. There are various quirks moving to zsh:

• To void frequent “shell history corrupted” issues, don’t export HISTFILE; if you do and ksh gets invoked as a subshell, it appends to the history file and gives zsh a concussion.
• Disable ^D handling: bindkey -r '^D'
• Disable escape-slash handing: bindkey -r '^[/'
• Write a handler for ‘v’ in vi command mode to edit the command line in vi
• Update ^L handler, which should clear the screen and the scrollback buffer, thank you very much.

There were also scripts that required updates:

• I had to rewrite bits of my shell library (functions in my $FPATH) to accommodate variations, especially the lack of a -q option to whence.
• Script variables named $path need to be renamed, as zsh sets it to a space-separated version of $PATH, a crappy unexpected behavior imported from csh
• Similarly, $PROMPT is linked to $prompt.
• Array indexing starts at 1, not at 0.

zsh documentation is a hassle: there’s a lot of it, it’s split across several man pages. Losing the --help and --man options to the built-ins is inconvenient.

But so far, it’s proving to be a viable transition. Some things are improved, like the column-counting: there are escape sequences that direct zsh to stop and resume counting. It means a little more work to set the prompt, but avoids an ugly, unreliable counting algorithm hard-coded in the shell.

Conclusion

Admittedly, I’m more familiar with Korn shell, and that’s an influence. While I can use Bash just fine interactively, I find it frustrating to try to script with. After ksh, bash seems just broken—and in some cases, it’s probably because I’m not used to it. But in other cases, Bash requires jumping through hoops that aren’t necessary when working in Korn shell.

So, in programming features, ksh seems far ahead to me. Unfortunately, its maintenance and future are unclear.

zsh shows promise as a sanctuary for Korn shell users who are pushed away by a rogue developer and entropy.

I haven’t seen misbehavior from my existing ksh scripts, and unless I do, I’m letting that sleeping dog lie. But should it be necessary, zsh does seem like a viable candidate to move to without completely rewriting.

Despite name recognition, bash has a long way to catch up, although with constant attention it could catch up. I get the sense, though, bash isn’t even trying; they don’t mind, for example, that the shell can’t do floating point math.

Korn Shell’s self-documenting getopts

Insanely old Korn shells or other shells portending to be Korn Shell might not support the advanced getopts. If you are worried about this, you can detect availability with this construct:

if [[ $(getopts '[-][12:abc]' flag --abc; print -- 0$flag) == "012" ]]
then
    GETOPTS_STRING=$'
        ...new-style-getopts-document
        here...'
else
    GETOPTS_STRING="old-style-getopts-string here"
fi

The $'' quoting uses C-style quoting within, to keep you sane.

In your parsing loop, provide the selected string:

while getopts "$GETOPTS_STRING" flag
do
    case "$flag" in
        ... handle your options flags here
    esac
done

So what do you put in the new-style getopts?

A section title and a paragraph for it in the resulting man page:

[+TITLE?example - how to use the \bksh\b(1) getopts string]

If you need a new paragraph, leave out the title:

[+?This is another paragraph under the last title.]

To insert a definition list instead of a paragraph, wrap the entries in braces:

[+EXIT STATUS]
{
    [+0?The command succeeded.]
    [+1?The file was not mangled correctly.]
    [+2?The file could not be found.]
}

To define an option:

[s?Flag with short option only.]
[f:long-flag-name?Description of the flag.]
[p:option-with-argument-name?Description]:[argument-name]
[n:numeric-option-name?Explanation]#[argument-name]
[x:?This flag has no long-option name]
[12:flag-with-only-long-name?There is no short-form of this flag.]

The letter after the opening bracket is the short option. If a long option is used, it’s converted to the short form when provided to you. If you don’t want a short form, provide a 2 or more digit number to be used instead. Numeric argument specifications are enforced, but note 0xhex is allowed and decimal numbers (0.5) are an error.

For options with arguments, you can include a list of values. These are rendered in documentation, but they are not used for getopts processing. So don’t expect any restrictions they document to be applied by getopts.

[l:flag-with-list?Flag that takes only one of the defined values.]:[argument-name:type:attributes:=default-value]{
    [+one?This is the \afirst\a choice.]
    [+two?This is the \asecond\a choice.]
    [+three?This is the \athird\a choice.]
}

To include information about the script’s creators:

[-title?Name and contact information]

For example, [-author?Perette Barella] or [-license?This is free software released under the MIT license.]

Parameters go after a blank line following all your other text:

from-file to-file

If you have multiple forms, you can list them on separate lines:

from-file to-file
from-file ... to-directory

Lastly, there’s a handful of useful escape sequences: \b (backspace) toggles strong emphasis (bold). \a (bell) toggles emphasis (italic/underline). \v (vertical tab) toggles fixed-width display.

Putting it all together, here’s a demo program you can tinker with:

#!/bin/ksh
while getopts -a getopts_info $'
[+NAME?getopts info - document describing the ksh long-form getopts]
[+TITLE?This is a section and a paragraph in the resulting document.]
[+SOME OTHER SECTION?This is a spare section]
[+?Here is an example of a \adefinition list\a]
    {
        [+SOME] ?Text inside this \vdefinition list\v.]
        [+OTHER]?And text for another \vdefinition list\v entry.]
    }
[+?This is a continuation paragraph]
[s?Flag with short option only.]
[f:long-flag-name?Description of the flag.]
[p:option-with-argument-name?Description]:[argument-name]
[n:numeric-option-name?Explanation]#[argument-name]
[x:?This flag has no long-option name]
[12:flag-with-only-long-name?There is no short-form of this flag.]
[l:flag-with-list?Flag that takes only one of the defined values.]:[argument-name:type:attributes:=default-value]{
    [+one?This is the \afirst\a choice.]
    [+two?This is the \asecond\a choice.]
    [+three?This is the \athird\a choice.]
}
[-Author?Perette Barella]
[-Copyright?This document is public domain.]

argument argument2 ...
Other forms of this command
[Note that at this point, this is plain text]' option "$@"
do
        print -- "Argument: '$option' optarg:'$OPTARG'"
done