Bash/Sh is an objectively awful programming language. You could likely fill an entire book with all of the design fails that is Bash script syntax. Only in a shell script would you ever need eight backslash characters in a row in order to accomplish something...and "space as delimiter" is awful in so many ways.
But especially Sh runs everywhere, so it's good to know. This seems to be the article's point.
Honestly, though? The right answer is to transpile a better language to Sh.
Looking around, there have been several half-hearted, abandoned attempts. But apparently everyone is happy to just fall back on the "accidental syntax" of Bash that really doesn't make sense when compared to ... any other language.
I put up with Bash because there's no better alternative. That doesn't mean we should put up with Bash; just that we must put up with it at least to the point where it can run code written in a real language.
Bash is a user interface. The fact that you can string together words and symbols and make logic happen is basically just back-porting advanced user functionality into an existing interface.
Having said that, Bash is my favorite programming language, because it is so limiting, and simultaneously not limiting, because it wasn't designed to fulfill what A Real Programmer(TM) thinks a programming language should be.
A computer is supposed to serve a human. Not the other way around. But modern programmers, being wildly unimaginative creatures of habit, think the user is supposed to serve the computer. That the more you type, and the more hoops you jump through, the more "clever" you are with the "way" you tell the machine to do something, that this is superior. Never mind that the end result of all this incantation and tom-foolery doesn't actually result in a good outcome half the time. That the user is often unhappy with the result, if they even factor into the programmer's thinking at all. No, the true purpose of programming is to gratify the programmer and make the machine happy, not to make a human's life easier.
Programming is a mistake. It's a half-measure. A stutter in the evolution of digital computing. We weren't meant to program, we were meant to make a synthetic invention that relieved us of our labor. And what do we have to show for spending 1/3 of our life, and decades of work, for it? A metal and plastic box that can do pretty much exactly the same real work as 30 years ago, but with better graphics, because the hardware people gave us better screens.
There's almost nothing that computers do that isn't easier than the pre-computer way.
Look at it from an average user's perspective. They're thinking "Wow, this spreadsheet would have taken hours to do with a calculator and a pencil".
Most users are not programmers. Programming IS the act of making a synthetic invention that relieves us of our labor. It's no different from what a plumber does. If we had to connect pipes in a custom arrangement every time we wanted to do the dishes we would argue all day about the best pipes and threads and tapes.
But we don't. We use premade fixtures and we don't have to think about plumbing.
Those hoops you have to jump through are like the building codes. They make everything consistent. And more importantly, they make things inspectable. There are a million and one ways to plumb or frame a wall or something that would probably be fine. But that would be Real Engineering to do that and show that it's safe.
Wheras the codes are designed to not allow for "interesting" stuff that only an engineer would understand. The programming conventions developed over time based on programmers experience of what works and what doesn't when writing software that is so complicated not one person on earth could ever understand it.
The difference with programmers is they use programs to make programs. There's no user/designer distinction, and they all want to be creative and clever engineers, not just skilled tradespeople, so they don't see the benefits as much. If most of what you do is coding, you might thing nothing has advanced in the last 30 years, because you used text then and now you have more complicated text.
But from a user perspective, we went from drafting tables to CAD. We have Google Keep. We have voice assistants. We have 4k editing for consumers!
Bash pretty much just makes it explicit right in the language that there's no user/coder distinction and kind of keeps that mindset around.
Newer languages are designed to help people make user facing black boxes, and they do a great job of it it seems.
If only we had something like building codes, maybe we could stop fucking around trying to re-invent the building all the time, and just make a god damn building that didn't fall down after 3 months.
We make shit programs because our industry has no code, no apprentices, no journeymen, no masters. We have kids with no discipline who don't understand how anything works and just fake it 'til they make it. Everything from the shit designs to the inefficient basic programs to the overinflated titles to the lack of basic rigor shows that, generation after generation, we're actually getting worse at our profession, not better. We inherit good foundations and we shit on them. And we look at the people shitting stylistically and imitate them.
> we went from drafting tables to CAD. We have Google Keep. We have voice assistants. We have 4k editing for consumers!
We had CAD in the fucking 1960s!!! Google Keep is an overcomplicated frontend for RCS. I've literally programmed a better voice assistant in Perl in 1999. 4K isn't even usable on most devices, let alone necessary.
I literally knew more about software when I was 18 than any programmer I have met in the last 5 years. And I'm not that smart.
> Bash pretty much just makes it explicit right in the language that there's no user/coder distinction
Bash makes it explicit that there is no coder. There's only a user. Programmers don't seem to realize it, but the way they act, they don't consider that the user even exists. This is the defining characteristic of 21st century technology. Programmers are self-serving, and users are waiting for someone to make their lives easier. Like children waiting for the chefs to stop fucking around in the kitchen with sourdough starters and hydration ratios, and just bring out some god damn bread.
There was CAD in the 60s, but somewhere they have archives of paper plans for the moon landings. If it tried to use 60s cad I'd probably think it was just a drafting tool with some extras, not like modern cad with solvers, parametric modeling, and 3D views. I don't see how I could make a sketch with constraints on a face back then, modern CAD is slightly slow even on my laptop.
Deep learning voice recognition didn't exist in 1999, which is the critical tech that makes Assistant usable, along with the fact that there's now Keep to integrate with and IoT to control.
It's true that programmers hate users sometimes, but that's not an issue of tech or skill, that's an issue of programming culture is full of people who would like to go back to tech free simple living and mostly see computers as "bicycles for the mind" rather than "A UI layer for everything manmade we interact with" or "somewhat independent robots". They're in it for the exploration of thought, not because they want paying taxes to be fully automated.
But it's the managers and UI designers, and the small percent of programmers that don't hate tech, who actually direct a lot of the experience.
Bash is basically incapable of making anything I'd want as a user, because it's not good for managing insane amounts of complexity. It's a tool for directly controlling a computer and using it as a passive tool.
Modern languages are tools for essentially using the computer almost as a manager more than an employee, and for creating new models of interaction that the machine is better able to always have your back with.
If I want a block to be 35mm tall in total, but there's terraced layers and curves and stuff, I set a constraint.
I don't have to worry about the math. I'm not just virtually building stuff like a digital simulation of wood, I'm doing something completely synthetic, that was made up by someone who figured out a model for interacting that puts as much of the hard parts in the CPU instead of in the brain.
It can't just be one way "I tell the machine what to do" interaction or the whole thing is limited to my own ability just like paper.
It can't be text based or anything else that requires mental translation, or I'd just have an extra task in addition to doing the whole design myself.
Modern software's solution is to create a way to think about the task that is inherently something suited for both humans and machines, and then to teach the user that way. Bash is freeform, it says "think how you want then translate it to what machines know".
It's great if you want control, not great for computer-led experiences.
Sadly it doesn't seem like we are designing many new paradigms of interaction, but I blame lack of interest rather than lack of tech ability. Programmers think about the functionality first, rather than about the interaction model, because they like ideas and dislike software used in the real world when a pencil could have done instead.
We do have building codes. Unit tests, memory safe languages, design patterns, DRY, reuse... It's just that lots of devs don't like them.
Meh. That part is irrelevant in the extreme to the discussion.
Fact is that people write `.sh` and `.bash` scripts to get things done. They use conditional logic and loops and everything.
Functionally, it's a programming language with a (useful) REPL.
And this discussion is 100% about how bash is as a programming language.
> Having said that, Bash is my favorite programming language, because it is so limiting, and simultaneously not limiting, because it wasn't designed to fulfill what A Real Programmer(TM) thinks a programming language should be.
Umm... Then, yeah, I have to agree with your self-assessment of not being a "real programmer."
Sorry, but it's obviously true.
You're a scripter. I've known and worked with many scripters. They can do a lot of valuable work and contribute in a lot of ways to various projects. But they're not programmers.
Which is fine. Script away.
> But modern programmers, being wildly unimaginative creatures of habit, think the user is supposed to serve the computer.
After admitting you're not a programmer, you then go on to demonstrate that you don't really understand how programming works. Which I guess follows?
That's so far from the reality, that it's Not Even Wrong.
> That the more you type, and the more hoops you jump through, the more "clever" you are with the "way" you tell the machine to do something, that this is superior.
Programming languages are exactly the level of complexity needed to, with the least effort and most reliability, craft the code that, literally, civilization relies on.
It can't be made simpler. The complexity is inherent in the domain.
> Programming is a mistake. It's a half-measure. A stutter in the evolution of digital computing. We weren't meant to program, ...
YOU weren't meant to program. You admitted that above. Don't speak for the rest of us.
Everything significant in software that has been and will be accomplished moving forward will be created by programmers using programming languages. Your rant against programmers and programming is based on a lack of understanding of what programming really is and how programmers really think.
It's not even slightly about being clever. It's about the act of primal creation. We're effectively wizards of the modern world. We don't care at all about making a machine happy; we care about building things that work. That the things embody complexity isn't the goal, but it is a thing of beauty when it all works together elegantly.
Bash is a shell. A bad shell. Korn is a good shell.
It's not meant to be a real language. It's meant to be a shell. Here, let me back that up and reverse it for you.
<JAVA>
public Set<String> listFilesUsingJavaIO(String dir) {
return Stream.of(new File(dir).listFiles())
.filter(file -> !file.isDirectory())
.map(File::getName)
.collect(Collectors.toSet());
} //I'm not handling errors, and this doesn't actually print anything, it just lets me try to.
<SHELL>
ls
Shell is meant to be a common way to do common tasks on commandline, and make a little script if you want to save that and run it again. I can do little things in a shell one-liner quick, that would take a page of python or perl.
I put up with my sports car because there's no better alternative. It doesn't mean I should put up with a sports car for transporting my piano, just that I must put up with it at least to the point where the sports car can transport me to the car that transports a piano.
And it's an objectively awful programming language. Sh is even worse.
Korn isn't installed absolutely everywhere like Sh is. It's not even installed in 1/100 of the locations that Bash is installed. So it being better is irrelevant to the discussion of Bash as a programming language.
Because Bash is so awful, the only real justification for using it as a programming language is that it's already installed. Since Korn isn't already installed, that means you'd need to install it. If you're going to be installing something, there are far better choices.
So your comment doesn't at all contribute to the discussion of alternatives to using Bash as a programming language.
And the verbosity of using Java to list a folder contents is equally irrelevant. No one would argue that Java is better ... for listing directory contents.
If you do want to write a script that calls a bunch of shell commands, or even that operates on folders and files, there are far better languages and tools.
The main use case of a shell is interactive use. There is no reason to invoke a teanspiler when you type a command to be immediately executed. Often its syntax is a bit awkward because in needs to be compact and quick to type.
If you want a better language for complex scripts, use Python, or even Perl, they are now ubiquitous, or bring someting self-contained like Lua or Janet with you.
It would be great to have a different ubiquitous shell language on Unix machines, but it's unrealistic now.
There are two places a new shell language needs to be ubiquitous for success. One is installed on systems, which is pretty easy. With today's connectedness, new software is just an apt install, pacman, or curl | bash away. The other, much harder problem is that the people using the system need to know this new shell language. If I have to debug your shell script using Zsh, Csh, or Fish, I'm not going to be happy about it.
Shell is an interactive, concatenative string language with transparent file system access and binary execution capabilities. It doesn't need to be a general purpose programming language, because it's great at what it does. When I need something lower level, I can write that in low level and integrate into the shell language seamlessly.
>Bash is great, but when it comes to writing more complex scripts, many people prefer a more convenient programming language. JavaScript is a perfect choice, but the Node.js standard library requires additional hassle before using. The zx package provides useful wrappers around child_process, escapes arguments and gives sensible defaults.
> Honestly, though? The right answer is to transpile a better language to Sh.
I like this, but I think it's even better to just escape sh entirely. If you are on a system that supports sh, it almost certainly has a c99 compatible compiler which can be used to install lua(JIT)? or another better scripting language. I wrote a little script idempotently installs lua 5.4 (that I tried to include but HN formatting messed up) and then runs a luascript embedded in a heredoc. The first time I ran it, it took 5.4 seconds to print "Hello, world!" including downloading the lua source code from their website and compiling the whole thing. The second time it took 11ms. I really see no great reason to continue using sh for anything new except for the purpose of installing something else.
- Embedded devices that may or may not have an internet connection
- Government classified networks that aren't allowed to grab ANY random software off of the internet
- Simply not wanting to add another dependency to an operation.
I'm a former Lua fan. I've ... written it off at this point. [1]
Honestly if I were to take the approach you describe, I'd likely instead have the shell script only look at the architecture and then choose one of 2-3 precompiled Go binaries. Talk about simple and fast...and the binaries could be hosted behind a firewall somewhere.
Or maybe I was thinking of it wrong. Just embed the various Go binaries into a tar file embedded right in the shell script[2], and choose the right one based on architecture. Done. :)
Bash and most alternatives are full of footguns. However, they are really convenient to glue together different components, e.g. a data preprocessing system.
Personally, the way I avoid footguns is to use a functional style. Filter instead of conditions and map instead of loops. GNU Parallel makes this straightforward, including distributed commands across machines! Aside, it is easy to emulate relational algebra with grep, sed, and awk.
There are lots of better alternatives in almost all cases, just not super weird niche stuff.
A lot of things that one would use bash for involve servers, and there's Ansible for that, no need to use bash directly most of the time. If it were up to me, distros would include Ansible out of the box.
IMHO doing anything at all that's not programmatically repeatable isn't the best practice, so it's not like I want to be logging onto something and writing a quick script in Nano.
Outside of servers.... nobody uses Linux except embedded systems and FOSS enthusiasts. Nobody is paying me to do FOSS work, so I can just pretend that Pythonless systems don't exist.
Python does have some version compatibility issues, but for the most part it's stable, and at least it will crash with a traceback not a cryptic error code because you used some utility that had some nonstandard syntax for the flags.
On embedded, most of what you're doing is probably simple enough that if you do need to use sh you won't absolutely scream.
If running everywhere is actually a constraint (likely it’s not) and transpiling is a consideration, you could use Golang, that statically links and produces standalone binaries that run more or less anywhere. Though if you have a CD pipeline to transpile and deploy such, you likely have enough control over your environment already to have a less constrained choice to begin with.
Normally I’m not very fond of Gos verbosity, especially compared to bash, but i take anything over bash, it is just so filled with footguns, every single line has to be super carefully scrutinized. Note that I say this as someone actually quite good at it, not because I don’t understand it. In fact the more you learn, the more scared you get.
Yes, yet 'set -x' in Bash is more useful than in any other languages I know.
Because Bash is used to call utils so you see the partial trace of your script that you want to see, not the inside of the tools/utils that you don't want to see.
In contrast in gdb for C++ you 'step' inside the STL implementation that most of the time you don't care about..
I'm not very knowledgeable in this area as I don't do much shell scripting, but isn't zsh supposed to be some kind of enhancement over bash? Does it suffer from the same issues?
Regardless, the idea of a language that transpiles to bash is kinda interesting.
ZSH is basically the same thing. For all intents and purposes it is basically the same thing. The best attempt I’ve seen to truly make a better shell is Nushell.
Shell is an objectively _difficult_ programming language. But difficult != awful. Pointy and ready to bite you if you make a tiny mistake? Yes. Ubiquitous and not going anywhere? Also yes.
But given that *nix tooling output defaults to text, it's hard to argue against a universal language that talks to all of it.
This article is not specifically about Bash, but about the shell command language sh. The POSIX Shell is a standardized subset of Bash which has even better portability. I would recommend learning and using sh unless you really need Bash.
Your script will use sh if it starts with #!/bin/sh (instead of #!/bin/bash).
I can also recommend ShellCheck which is a shell script analysis tool (implemented in Haskell) which can find errors and potential problems in your scripts. On a Debian system, installing ShellScheck is as simple as `sudo apt install shellcheck'.
>I would recommend learning and using sh unless you really need Bash.
I don't buy this argument. Ok, it's a standard, but sometimes it's a pain-in-the-hole standard. Bash augmentations lessen some of the pains, and it's just nicer. The only situation is if you're using busybox in a very limited system or you reeeeeally need your script to run on many difference unices, which let's be honest, is not that common nowadays.
All of this was covered in great detail in the 2000s, when Debian and Ubuntu switched /bin/sh to the Debian Almquist shell, which was basically POSIX-only with some 3 things that Debian people simply couldn't live without, and encouraged system scripts to use it. If you weren't around then, go and read the discussions. They're mostly still available, and they cover some important stuff that straw man counterarguments regularly miss.
The Debian people were concerned, for starters, with how much time the Bourne Again shell spent, at process initialization, setting up things for extensions and interactive features that were never employed in non-interactive "sh" mode; a significant cause for concern given how much of the system was executable shell scripts.
But the aforementioned are what Debian people explicitly wanted and couldn't live without. Note that the StackExchange answer is a comparison of a 2022 dash to a 2017 standard, neither of which existed at the time. Debian Policy, the Debian Almquist shell, and the POSIX standard have all been revised since then.
I do small embedded systems with busybox, even that I can squeeze in bash when I need that, it's about 1MB in size and gives a lot more than busybox's sh. Another option is Lua which is also great.
Well, usually I have to decide between „do I look up the magic incantation to install bash in this container and risk fucking up the layer cache“ or „do I modify this script to not use bash-isms and feel like I’m a Unix purist in the process“, and the answer is the latter usually
> Bash augmentations lessen some of the pains, and it's just nicer
Both sh and bash are terrible, ugly hacks. Anything that requires more than 3 lines of them (including the #! line) should be written in a proper scripting language.
Though I do agree that cross-platformity to that level is rarely meaningful, and so things like python are just as likely available/can be made available.
My system Python tree is almost 1GB. That doesn't matter for desktop systems, but most software is published as container images these days and using a language like Python, Ruby or Perl for an image that doesn't already need it is pretty wasteful and pulls in a lot of extra dependencies.
I'm not sure there is a good alternative to sh/bash shell scripts as most dynamic languages have become pretty large dependencies these days.
That surely includes plenty of globally installed dependencies. I looked at the package sizes for debian and it is around 100 kB for the python3 package and on the order of another 100kb for python3-minimal. Couldn’t find a cumulative install size, but it is surely not larger than a few megabytes.
Until someone decides to write something that's only available in X version of Python. While there are certainly v4 and v5 bash-isms, they are far and few between compared to most languages.
...that said, having to support bash 3.2 for MacOS is a horrible thing.
AWK is part of the POSIX standard and is therefore on every UNIX-like. People really need to remember AWK exists and is a real, proper scripting language with a very sane syntax (it's not just for cryptic one-liners, really!)
Add a flake.nix file and you get completely reproducible program versions if you really want.
But I know that you meant it as a rhetoric, so my non-rhetorical answer would be that python3 is almost universally available on distros that are not minified deliberately (containers).
> python3 is almost universally available on distros that are not minified deliberately (containers)
Writing control scripts for use inside containers is currently my biggest application of BASH scripts. What makes BASH (or other shells) handy is the lack of supporting files that are needed - just copy in the script and you're ready to go.
I was going to say something similar. I try to avoid adding anything to my final images. Each extra dependency is something else to update, more surface areas for attacks, another thing to justify, etc. Much rather just use the slimmest base and the tools already available.
The various sub-versions of bash are a lot more compatible than the sub-versions of python 3. Just today stuff was breaking on me because of differences between python 3.6 and 3.8
It depends on what features of BASH that you want to use. If you target v3 (as I usually do), then that or a newer version will likely be already installed. There's very little issue with backwards compatibility.
If you wish require some users to install an interpreter, you might as well go for Korn or a real programming language. If you want to use something that's portable and installed everywhere, that's POSIX sh (or at most dash-like level). Bash is in the poor middle which isn't good enough for either side, unless you really know what's your target and don't care about elsewhere.
I usually target BASHv3 in my scripts, but then I know the environment that will be using those scripts. My biggest issue with POSIX is the use of the backtick which is difficult to read and parse.
ShellCheck is absolutely the most important thing to use when writing scripts. Each time that you get a warning that you don't recognise is a learning opportunity - just follow the link provided in the warning. Also, take the time to add the "# shellcheck disable=" comments so that scripts don't issue any warnings from shellcheck.
> My biggest issue with POSIX is the use of the backtick which is difficult to read and parse.
POSIX also has $(command) for command substitution. It can certainly be annoying when it's not used, though. I think shellcheck will recommend it over backticks.
> The POSIX Shell is a standardized subset of Bash which has even better portability.
It would be much more correct to say that bash's features are a superset of the shell features standardised by the POSIX standard, as is the case for several other shells (each adding their own parts on top of the standard) too.
Does the POSIX implementation vary for sh though? AFAIK some interpreters choose to add some extras (like `local`), but all of them at the very minimum must adhere to the POSIX spec from what I've understood.
Not only is it not more portable, POSIX doesn't even require that it exists at all.
Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH, ensuring that the returned pathname is an absolute pathname and not a shell built-in.
I don't know how this trend started but now it's been cargo-culted to death. Same with the /usr/bin/env thing in shebangs.
One use-case for /bin/sh over /bin/bash these days is that the alpine docker image (base image for a lot of docker images) has /bin/sh but not /bin/bash
"If bash is invoked with the name sh, it tries to mimic the
startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well"
BASH is like Obi Wan. It isn’t the most powerful or flashiest, but it survived a long time, where others didn’t, for very good reasons. Bash runs basically everywhere. It has many modern features you wouldn’t expect. Its syntax is literally what you would type on the command line if you were diagnosing or fixing systems so you don’t need to transpile to another language. Its reliance on other programs means it is glue and can easily incorporate highly cohesive functionality/tools others write and maintain. Also, it’s been around and is everywhere so you don’t worry about trying to incorporate the current latest and greatest declarative tool (which will blow over in 5 years) into your other workflows. Basically, don’t disparage a Jedi/tool that has survived where others didn’t. There is a reason.
Also, use shellcheck. Incorporate it into you editor. Fix all warning and don’t ignore them. This will push you deep into bash syntax rabbit holes but you come out better the other side.
A lot of bash errors are not understanding possible cases due to white space.. which shellcheck catches. After using it for a while, I don’t even really worry about white space because of the good habits I’ve learned/(been forced to use).
I didn't even know that Windows supported that, but I try to keep away from it as much as possible. Makes for an interesting test of scripts, I suppose. Sometimes, I script BASH to cope with any any filenames, but it depends on the usage as a lot of the time you can guarantee that files will be at least semi-sensible.
The style issue shellcheck reports here is probably what we both suspect it is.
The call to echo has a likely purpose is to trim some spaces from a string, but it's not at all obvious why this is done or what the benefit is. In the best case, it leaves the reader pondering this weird semi-no-op.
If you think shellcheck, and probably most readers, is wrong in this assertion, just speak your mind and let's learn from one another. But I certainly can't guess your intention here.
Using "echo" at all is a problem. It's recommended to switch to "printf" instead as echo has unfixable problems, especially with strings that start with a dash.
Not a problem until the random string starts with a dash and you get unexpected results from "echo" interpreting it as an option instead of an argument. Using "printf" bypasses that issue as it's clear what the argument is.
Just tested it and ShellCheck is happy with the printf version:
It's also got a stylistic improvement (in my opinion, anyhow) in that the line feed is explicit in the printf command rather than being almost a side effect of echo.
Edit: I see your point about the "tr" not enabling a dash, so my example falls a bit flat, but my point stands in other scenarios. It's good practise to avoid echo where feasible. I think the double quotes will also protect in this instance. The problem with relying on the tr string to not foul up echo is that you may later adapt the code and use a different translate string which could then introduce a tricky bug to track down.
I'd also add that in the cases where you disagree with shellcheck (it doesn't get everything correct all the time), you can include a disabling comment just before the line to tell shellcheck that you know best:
The only real advantage of bash (or other shells) is the ability to build up to it from the command line; to use it as a REPL then embed the stuff you've just done into a script.
The massive disadvantage of bash is that it uses a legal filename character " " as a delimiter ("$IFS"). That means it's very easy to write scripts which work fine on your test case but blow up in a different context. And the worst case of that is accidentally deleting the user's file system.
So many of the cute little bash examples you'll see, including I think the awk one on this page, will go wrong if you have a space in the wrong place in your filesystem. Or, god help you, a newline. Or a file named "*". Or "--".
Getting the 10% of the cases that have whitespace or metacharacters in variable values and filenames to work ... takes the other 90% of the time. (-:
I habitually quote variable expansions, even when I know that the variables are never going to contain whitespace. Because long experience has taught me that one day they will.
Just like I've learned never to assume that ${PATH} is always non-blank and so assume that PATH="${PATH}:/something" won't accidentally add the current directory to my search path. (-:
All the characters in my example are printable. Not all of the octets are, of course, but octets aren't characters. And spaces are extremely common in names, e.g. music track and album names contain spaces more often than not (and thus usually end up in file names). If you can't even support using the biggest key on every keyboard in file names you're doing something horribly wrong.
I guess the URL path in the Nginx log would be URL-encoded and have a %20 instead of a space. And otherwise, it would actually also a problem of that text based log format that uses spaces as field delimiters itself.
Anyway: When I can "build up from the command line", then apparently whatever I'm building up can be easily and intuitively done in the command line.
So, stuffing it into a bash script is fast and easy and should be easy to understand as well.
I just have real-world applications for small scripts that aren't much bigger than those "cute little bash examples". When you work with embedded systems, you do a lot of repeatable stuff via SSH. I once ported an Installer to Python and now I hate it because it has so much unhelpful noise between the actual relevant commands being executed.
Yes, Bash has it's issues and requires some discipline, but so do most other languages.
Shellcheck won't catch the missing dashes ('--') at the end of command options, so you could be in trouble if a variable starts with a dash and the command interprets it as an option rather than the filename. It's not particularly obvious, but if people can upload a file and specify its name, then they could compromise a script by choosing a suitably evil name. If you get into a habit of putting '--' at the end of the options and before the filename variable, then you protect against that.
Having to completely change the naming scheme you use for human-authored and human-read artifacts entirely to route around shitting, confusing escaping in a programming language is not a convincing selling point for that language.
If you have a file whose contents are Shakespeare's "As You Like It", you should be able to name that file "As You Like It" without having to worry that the computer will vomit and die on it. We deserve better tools.
I too often avoid spaces in filenames, but when writing scripts it's far better to ensure that you cope with them properly. The trick is to always quote your variables by default and then run shellcheck on your script so that it can flag up the corner cases where quoting acts to unquote the string. Of course, it'll also flag up a host of other possible issues which are well worth fixing or at least understanding why it's a possible issue.
Another common footgun is not including a double dash ('--') at the end of command options so that the following filename won't be interpreted as an option when it begins with a dash.
>Another common footgun is not including a double dash ('--') at the end of command options so that the following filename won't be interpreted as an option when it begins with a dash.
Yes. And if you have a file name that begins with a dash, and you want to delete it, you can use this command:
$ rm -- -filename
where -filename is that file. Here, the -- (double dash) means "end of the command-line options", so any argument after that is treated as a non-option argument, in this case, a filename for rm to delete.
This -- (double dash) works with many Unix commands. It was introduced relatively early on in Unix history, after the issue of filenames starting with a dash was found.
Although it is more likely that you would want to rename the file, to remove the dash from the name.
Another way of working around the issue of filenames beginning with a dash is to just put "./" in front of it. That's not always useful in scripts though if you don't know whether the variable will be a relative or absolute filename (though you could check for that easily enough).
If you ever need to use GNU make (still a popular tool) then you will be glad that you kept a policy of avoiding spaces in file names.
There are places in Makefiles where no amount of quoting will help you, and it simply cannot handle files with spaces.
Yeah, this is a problem with the tool that should be fixed, but my understanding is that it will never be fixed because of architectural reasons. And it's a very popular, useful, and most importantly, ubiquitous tool.
So this is another good reason to simply have a policy of banning filenames with spaces (and there are many more reasons).
Hm, I write bash scripts routinely and never do it that way. Always develop and run my scripts as files, starting with ‘set -x’ if I need debug info. I keep interactive mode solely for command line work, not development of scripts I plan to from files.
I have a counterpoint for many comments here. If you are writing some Python for example, and most of your code is executing commands such as tail, cat, wc, etc. and then doing something with their output, you are better off writing a shell script instead .
In my own experience people typically do that when they have some experience in bash, none in python, and try to do something in python the way they would do it in bash. So for those my advice, instead of getting back again to bash is to get a bit curious about the python standard lib, which has a nice doc unlinke bash, and not assume they know everything about scripting just because they have spent a long amount of time writing bash script.
I'm a computer scientist/software engineer that has used primarily Unix or Linux systems since the 80's. I write shell scripts occasionally, but I just find them error prone enough that I'd rather use python than sh for complex scripts. Python has some packages that make working with files, directories, and processes tolerable and at least I can understand the string quoting rules in python.
Generally my order of preference for shell scripting tasks is: python > bash > pearl > tcl > sh > anything > AppleTalk.
This argument isn't very compelling. Shell scripts are great at tricking you into believing you've got something 100% working when in fact you've only accounted for the happy path and maybe two or three failure cases. This is the primary reason shell scripts are so often broken. There are tons of assumptions baked in which are easily violated.
If your shell script keeps breaking, and you keep fixing it by writing more shell, then you're in an abusive relationship with your programming environment.
"Shell scripts are great at tricking you into believing you've got something 100% working"
Exactly. Shell scripting is basically a non-hygenic macro language, with very complex semantics around its primary data type, the string. It also lacks pretty much all of the affordances we use to write reliable programs. I have no doubt someone has written a typechecker or unit test framework for bash, but I've never seen one used. I'll take the ugly code with the straightforward semantics, thanks.
I feel like zsh is for pros maybe. I use Linux daily and write python and shellscripts. Zsh has been defaulted all over the place but all that meant for me is I have to figure out how to get it to do things the bash way once in a while.
I see no reason why distros like ubuntu or kali would default to zsh.I use the shell for sysadmin stuff, anything complex or app specific gets at least a python treatment.
The whole thing reminds me of python 2->3 or sysv init to systemd. There must be people somewhere deeply invested in these changes and their voice is certainly much more valuable than the average opensource joe user. Defaults matter and the time I and millions had to spend learning to transition for no direct benefit to us has value. The whole distro model of everyone getting a distro they like falls apart when not enough people can fork.
Overall, this devalues opensource as an investment. Because of unpredictable hidden costs like this.
I think this perspective is essential in our ever-evolving tech industry. Bash, despite its quirks and often steep learning curve, has its niche in our toolset.
Its power lies in its simplicity and direct access to system level functions, which makes it an efficient tool for system administrators and developers working in DevOps. It's excellent for automating small to medium-sized tasks on UNIX-like systems where the overhead of a full-fledged language would be overkill.
While it's true that bash has its limitations, the value of its ubiquity cannot be overstated. The bash shell is everywhere - from massive server clusters to tiny embedded systems. This means that a bash script written on one system is very likely to work unchanged on another.
The article is a bit confused about whether it's making a case for shell scripting in general, or for the Bourne Again shell in particular. By the looks of things, its target readership is people who don't use shell scripting, or interactive shells, at all; so the former case would be more appropriate, and the drawing of a distinction between the Bourne Again, Friendly Interactive, Z, and other shells just muddies the waters for that readership.
it seems like most people that have dislike bash never spent the time to learn it.
I have a love/hate relationship with is. I started with computers in 2000 and wrote a ton of bash for various things. Compiling software, running cicd, adhoc batch jobs, etc.
I've managed many cicd pipelines with bash in my time. While it's not great, once you know the basics, you can be very successful. That being said, rewriting the bash "program" in another language is much advised. If your bash is over a few hundred lines, chances are you've outgrown it
Some things in bash (mainly pipes and redirection) are just too easy. Job control is also great. The 'process' and 'shell script' are two primitives that make a lot possible.
Trying to do pipes or redirection in python is awful. There are a ton of subtle bugs you have to worry about
The thing that makes me loathe Bash more than anything else is it’s 3 types are string, int, and list and those are not the only types of data I work with day to day.
We should have better local shells that “drive” remote sh/bash shells through transpilation, so that we can have modern shells anywhere without concern for remote install, access to install or compile, compatibility, carrying customizations, etc.
IMHO, the best resource to learn about both Unix shell usage at the command-line prompt, and shell programming in .sh script files, is still the book "The Unix Programming Environment" by Kernighan and Pike, except for shell gotchas and subtleties which were discovered later.
Of course, they still would not have covered all possible issues known even at that time, because that was not the focus of the book, which was to be a Unix tutorial, not just on shell, but many other commands, general Unix usage, and the environment too (hence the title of the book).
But they did cover and warn about some issues, including giving solutions.
Sounds like a case for shells in general. No other script-execution tools make running other programs (and linking their inputs and outputs) so simple.
To overcome Bash’s problems, one can choose another shell. But I suppose you can’t just pick your personal favorite and script in its dialect, because then how would the human tasked with integrating the script into $wherever know which shells to have installed? You can stick with sh which is guaranteed to be installed, or bash (indeed an improvement over sh) which is nearly guaranteed to be installed.
Or one can learn the lessons that Debian learned over a decade ago, and Ubuntu almost two decades ago, about bashisms, maintainability, and performance; and script in at most the Debian Almquist shell, whatever fancy shell one may be using interactively.
Interesting that efficiency is cited as the main reason in several of those links. And here I see that dash starts up only .001 seconds faster then bash. Maybe things were different 15 years ago, I don't know.
The other concerns seem to have a very simple solution: use bash in the shebang instead of sh if your script uses bashisms.
That was one of the counterarguments to a common straw man of the time. Changing the interpreter line was a valid route. If you choose to use Bashisms, the counterargument went, you just make that choice explicit; just as people who used Korn/Z/whatever shell idiosyncrasies already had to.
As for efficiency, recall that there were hundreds of /bin/sh scripts in startup, administration, and everyday operation of the system, from all of the /etc/rc.d/ (sub)scripts through things in cron and build systems and package install/deinstall scripts to bunches of commands that were /bin/sh scripts under the covers. So multiply that difference by several orders of magnitude.
Amusingly, this is to a major extent still true. On these operating systems systemd is nowadays parsing .INI files over and over with its own custom .INI file parser, instead of a chosen flavour of /bin/sh parsing shell scripts over and over with whatever (Bison/YACC) parser they were using. They haven't actually gone down the SunOS SMF (and s6) route of compiling things into a machine-readable database. And there are still lots of /bin/sh scripts elsewhere, as service management is only one of several areas.
An interesting alternate history would have been if the Z shell had been /bin/sh on Debian and Ubuntu instead of the Bourne Again shell when this came up, and how much of a difference employing zcompile everywhere would have made.
> As for efficiency, recall that there were hundreds of /bin/sh scripts in startup, administration, and everyday operation of the system, from all of the /etc/rc.d/ (sub)scripts through things in cron and build systems and package install/deinstall scripts to bunches of commands that were /bin/sh scripts under the covers. So multiply that difference by several orders of magnitude.
And one of the links leads to a savings of 1 second on an eeepc. Not very convincing.
As I said, this was over a decade ago. The Debian Almquist shell has been a required part of the operating system for Ubuntu and Debian, and several (perhaps all) of their derivatives, for all of that time. They switched it over to a mandatory ("essential") package years ago.
The Almquist shell is the /bin/sh on NetBSD. A slightly variant Debian Almquist shell is the /bin/sh on FreeBSD. The Debian Almquist shell is in Core in Arch Linux (and Parabola Linux and Hyperbola Linux). The Almquist shell is the sh in busybox. The Almquist shell is the sh in MINIX 3.
One has to travel quite far to the likes of Illumos or OpenBSD or Toybox on Android to find an operating system where it's not in the box as a core part of the operating system. And on several of the ones where it is a core part, it's actually /bin/sh already.
More than a decade ago I needed a multi-platform way to manage Nagios client data collection (mostly) using whatever native tools were available, so I wrote one. bash, gawk and send_nsca were the only 3 binaries needed, and it ran well on 10 different *nixs (free and commercial) on Alpha, ARM, MIPS, SPARC, X86 and X86_64 platforms.
Of course, after a decade of incremental tinkering I ended up with exactly what you anecdon't want: ~2500 lines of shell script and ~1000 lines of awk...
LOC misleads of course, there were 15 modular plugins, so there was a lot of boilerplate, and this includes non-runtime ~1000 lines for install, self-test and sanity checks.
Maybe this already exists, but I imagine if someone wrote at library for interacting with gnu or posix commands from python (or ruby, or...) that built up the commands in a structured way and parsed the replies and put them in a dict, that would be very nice to use instead of bash-scripts. Every weird thing that you wanted to do would be a lot easier to do in python than in bash, and none of that looking out for spaces in strings and so on.
It uses a Haskell streaming library so you can do stuff like shell pipelines and file redirection (but in a more structured, safer and more powerful way)
Except a lot of the times you don’t need this. The standard library may have what you need. There is no reason to pipe ls through jc, when you could use os.listdir().
I’ve written a ridiculous number of shell scripts to automate many many things. It’s great for doing something now, and it’s great as a way to interact with a machine. It is not great in all cases. Particularly: array handling is bad, it’s slow if you start using actual Bashisms, very few younger devs seem to know it.
The shell is great cuz it’s there and it can cover about 50% of use cases for server side automation, but it’s bad for the reasons mentioned above.
As a serious programmer, I have printed out and read most of the bash v5 manual.
As a serious programmer, i do TDD even with bash, it is called bats-core
As a serious programer I use the right tool for the right job, so I write bash scripts when it makes sense.
Having done all these, I rearly write bash programs, but when I do, it is a joy! Plus i have gteater terminal understanding which is useful beyond bash programming.
sh = Shell.cd('/')
sh.transact do
system('tail -fq /var/log/nginx/access.log') | system("awk '{print $7}" | system("sort") | system("uniq -c") | system("sort -rn")
end
Pretty sure one could get without the system method with Shell having def method_missing(meth, * args) to call system([meth.to_s, * args]) or something, turning it into:
sh.transact do
tail '-fq' '/var/log/nginx/access.log' | awk '{print $7}' | sort | uniq '-c' | sort '-rn'
end
So it's both ubiquitous and a terse DSL for managing and coordinating programs. Yeah I agree. Also if you're on Windows it's nice that PowerShell is always preinstalled and serves the same purpose.
Indeed, in JS you wouldn't use awk and sort, you'd use JS array functions. It's fine to have examples that are contrived in the sense that the problem is contrived, but the solution should not be.
Good points. But I wish Bash had better syntax and built in support for basic concepts like boolean logic. Try to compose some complex boolean expression in Bash and you will quickly hit problems.
At the risk of being accused of bash bashing, I'll throw down that bash is a weak ineffective shell. A "shell" can and should be so much more than all bash has to offer.
The much earlier more powerful ITS (Incompatible Timesharing System) shell DDT (whose job name was HACTRN) had an integrated PDP-10 machine language debugger / assembler / disassembler, so you could interactively or batch "script" and patch DDT and even other jobs in full blown PDP-10 assembly language.
Why invent yet another half assed "scripting" language that no other jobs in the system are using, when you can use the same fully powerful machine language that every job in the system is using? And if you need to do anything complicated, there's always LISP!
DDT's syntax was even more obscure than bash (and only a bit less obscure than TECO), but it was much more powerful and elegant, unleashing the full undiluted power of the PDP-10 at your fingertips.
You could also write DDT commands in text files like your login file, i.e. assembling a few lines of code to print the prompt by making a system call to get the time and format it as text.
You could examine and deposit code and data, load and change symbols, set breakpoints, in your own and even other user's running jobs (processes)!
You could disconnect without logging out (accidentally or not) and all your jobs would stay around until you logged back in, at which time you could reattach your job tree and continue what you were doing, all without running something like "screen" -- that crucial feature was just built in and always worked.
You could also pass ownership of jobs (like a running ZORK game or LISP interpreter or FOOBAR feeper) back and forth between users to share, like passing a joint. ;)
>It helped that ITS had no security whatsoever! But it had some very obscure commands, like $$^R (literally: two escapes followed by a control-R).
>There was an obscure symbol that went with it called "DPSTOK" ("DePoSiT OK", presumably) that, if you set it to -1, allowed you to type $$^R to mess with other people's jobs, dynamically patch their code, etc. (The DDT top level shell had a built-in assembler/debugger, and anyone could read anybody else's job's memory, but you needed to use $$^R to enable writing).
Original song: The Raven
Original artist: Edgar Allan Poe
Filk author: Guy L. Steele Jr.
Intro: Notes for those not familar with the terms in this poem -- see below
[...]
DDT ("dee dee tee")
HACTRN ("hack-tran") = top level debugging and job controlling procedure, capable of controlling up to eight simultaneous jobs (which may themselves be DDTs!) and performing other miscellaneous functions. HACTRN specifically denotes a DDT at the top of a job tree, while DDT is the more general term. The two terms refer to the same job in the poem, and are thus treated as synonymous. Note that DDT requires its subjobs to have unique names for obvious reasons; hence the concern over seven jobs all named FOO.
Several of these sound like anti-features. Others are OS issues and have no bearing on the shell (i.e. ownership hot-swap). The ones that are useful have already been solved by other programs (screen, gdb, et al). What is the advantage of having it all packed inside the same executable?
ChatGPT is great for quickly getting the bones of a script but you still need to know bash reasonably well, ChatGPT will sometimes merrily write stuff that looks correct but is actually a foot-gun and that's not great when things like `rm` are involved.
I know enough bash to be able to spot those things. but I might forget how exactly `awk` should be used to split a string or such. So asking ChatGPT is a quick way of getting those things working...
You can't tell me how I function. I'm def learning new neat tricks asking things from ChatGTP. I'm 100% sure if you paste your bash scripts in there and ask for suggestion you'll learn too
But especially Sh runs everywhere, so it's good to know. This seems to be the article's point.
Honestly, though? The right answer is to transpile a better language to Sh.
Looking around, there have been several half-hearted, abandoned attempts. But apparently everyone is happy to just fall back on the "accidental syntax" of Bash that really doesn't make sense when compared to ... any other language.
I put up with Bash because there's no better alternative. That doesn't mean we should put up with Bash; just that we must put up with it at least to the point where it can run code written in a real language.