Commands tagged encoding (12)

  • Create executable, automountable filesystem in a file, with password!

    This is just a proof of concept: A FILE WHICH CAN AUTOMOUNT ITSELF through a SIMPLY ENCODED script. It takes advantage of the OFFSET option of mount, and uses it as a password (see that 9191? just change it to something similar, around 9k). It works fine, mounts, gets modified, updated, and can be moved by just copying it. USAGE: SEE SAMPLE OUTPUT The file is composed of three parts: a) The legible script (about 242 bytes) b) A random text fill to reach the OFFSET size (equals PASSWORD minus 242) c) The actual filesystem Logically, (a)+(b) = PASSWORD, that means OFFSET, and mount uses that option. PLEASE NOTE: THIS IS NOT AN ENCRYPTED FILESYSTEM. To improve it, it can be mounted with a better encryption script and used with encfs or cryptfs. The idea was just to test the concept... with one line :) It applies the original idea of http://www.commandlinefu.com/commands/view/7382/command-for-john-cons for encrypting the file. The embedded bash script can be grown, of course, and the offset recalculation goes fine. I have my own version with bash --init-file to startup a bashrc with a well-defined environment, aliases, variables. Show Sample Output


    6
    dd if=/dev/zero of=T bs=1024 count=10240;mkfs.ext3 -q T;E=$(echo 'read O;mount -o loop,offset=$O F /mnt;'|base64|tr -d '\n');echo "E=\$(echo $E|base64 -d);eval \$E;exit;">F;cat <(dd if=/dev/zero bs=$(echo 9191-$(stat -c%s F)|bc) count=1) <(cat T;rm T)>>F
    rodolfoap · 2013-01-31 01:38:30 13
  • Convert encoding of given files from one encoding to another


    4
    iconv -f utf8 -t utf16 /path/to/file
    imsaar · 2009-12-01 21:02:58 3
  • Uniformly correct filenames in a directory

    Useful if non-ascii characters in filenames have been improperly encoded. Replace "PROBLEM" with the incorrect characters (e.g. 'é'), and "FIX" with the correct ones (e.g. '?').


    3
    for i in *;do mv "$i" "$(echo $i | sed s/PROBLEM/FIX/g)";done
    AlecSchueler · 2009-06-28 01:50:25 18
  • Get gzip compressed web page using wget.

    Get gzip compressed web page using wget. Caution: The command will fail in case website doesn't return gzip encoded content, though most of thw websites have gzip support now a days.


    3
    wget -q -O- --header\="Accept-Encoding: gzip" <url> | gunzip > out.html
    ashish_0x90 · 2010-11-27 22:14:42 6
  • Convert encoding from cp1252 (MS Windows) to UTF-8 on source code files


    3
    find . -iname *.java -type f -exec bash -c "iconv -f WINDOWS-1252 -t UTF-8 {} > {}.tmp " \; -exec mv {}.tmp {} \;
    miguelbaldi · 2012-01-12 20:00:26 5
  • Detect encoding of a text file

    This command gives you the charset of a text file, which would be handy if you have no idea of the encoding. Show Sample Output


    2
    file -i <textfile>
    juvenn · 2009-09-08 01:33:19 4
  • Encoding from AVI to MPEG format

    http://www.mplayerhq.hu/DOCS/HTML/en/menc-feat-mpeg.html MEncoder can create MPEG (MPEG-PS) format output files. Usually, when you are using MPEG-1 or MPEG-2 video, it is because you are encoding for a constrained format such as SVCD, VCD, or DVD. To change MEncoder's output file format, use the -of mpeg option. Creating an MPEG-1 file suitable to be played on systems with minimal multimedia support, such as default Windows installs: mencoder input.avi -of mpeg -mpegopts format=mpeg1:tsaf:muxrate=2000 \ -o output.mpg -oac lavc -lavcopts acodec=mp2:abitrate=224 -ovc lavc \ -lavcopts vcodec=mpeg1video:vbitrate=1152:keyint=15:mbd=2:aspect=4/3


    1
    mencoder input.avi -of mpeg -ovc lavc -lavcopts vcodec=mpeg1video \ -oac copy other_options -o output.mpg
    slishan · 2010-01-18 13:12:03 3
  • Encode file path to URL

    Really helpfull when play with files having spaces an other bad name. Easy to store and access names and path in just a field while saving it in a file. This format (URL) is directly supported by nautilus and firefox (and other browsers) Show Sample Output


    1
    convert_path2uri () { echo -n 'file://'; echo -n "$1" | perl -pe 's/([^a-zA-Z0-9_\/.])/sprintf("%%%.2x", ord($1))/eg' ;} #convert2uri '/tmp/a b' ### convert file path to URI
    totti · 2013-07-01 08:54:45 7
  • Get gzip compressed web page using wget.

    Like the original command, but the -f allows this one to succeed even if the website returns uncompressed data. From gzip(1) on the -f flag: If the input data is not in a format recognized by gzip, and if the --stdout is also given, copy the input data without change to the standard output: let zcat behave as cat.


    1
    wget -q -O- --header="Accept-Encoding: gzip" <url> | gzip -cdf > out.html
    tempusername · 2014-11-29 20:42:21 8
  • Detect encoding of a text file


    0
    uchardet <filename>
    fltman · 2014-11-28 07:47:13 7
  • function to fix files starting with =?UTF-8 and ends with ?=

    I still don't know why mutt (or offlineimap or whatever in between), is borking the encoding of my files, but this fixes it. Show Sample Output


    0
    fix_saved_files() { for i in "$@" ; do mv $i $(php -r '$or = "'$i'"; mb_internal_encoding("UTF-8"); echo mb_decode_mimeheader($or) . "\n";') ; done; }
    guillaume · 2014-12-27 20:37:16 8
  • Encoding with base64

    Also works with files: cat file Hello world base64 file SGVsbG8gd29ybGQK To decrypt use the -d option: echo SGVsbG8gd29ybGQK | base64 -d Hello world Show Sample Output


    -3
    echo "Hello world" | base64
    P17 · 2009-04-27 15:00:08 16

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Stoppable sleep
A nice way to interrupt a sleep with a signal.

Copy an element from the previous command
'n' is a non-negative integer. Using 0 will expand to the name of the previous command.

Annoying PROMPT_COMMAND animation
unset PROMPT_COMMAND to disable.

Use /dev/full to test language I/O-failsafety
The Linux /dev/full file simulates a "disk full" condition, and can be used to verify how a program handles this situation. In particular, several programming language implementations do not print error diagnostics (nor exit with error status) when I/O errors like this occur, unless the programmer has taken additional steps. That is, simple code in these languages does not fail safely. In addition to Perl, C, C++, Tcl, and Lua (for some functions) also appear not to fail safely.

Remove color codes (special characters) with sed
Removes ANSI color and end of line codes to the [{attr1};...;{attrn}m format.

Terminal Escape Code Zen - Strace and Tput
Depending on the TERM, the terminfo version, ncurses version, etc.. you may be using a varied assortment of terminal escape codes. With this command you can easily find out exactly what is going on.. This is terminal escape zen! $ ( 2>&2 strace -f -F -e write -s 1000 sh -c 'echo -e "initc\nis2\ncnorm\nrmso\nsgr0" | tput -S' 2>&1 ) | grep -o '"\\[^"]*"' --color=always "\33]4;%p1%d;rgb:%p2%{255}%*%{1000}%/%2.2X/%p3%{255}%*%{1000}%/%2.2X/%p4%{255}%*%{1000}%/%2.2X\33\\\33[!p\33[?3;4l\33[4l\33>\33[?12l\33[?25h\33[27m\33(B\33[m" Lets say you want to find out what you need to echo in order to get the text to blink.. $ echo -e "`tput blink`This will blink`tput sgr0` This wont" Now you can use this function instead of calling tput (tput is much smarter for portable code because it works differently depending on the current TERM, and tput -T anyterm works too.) to turn that echo into a much faster executing code. tput queries files, opens files, etc.. but echo is very strait and narrow. So now you can do this: $ echo -e "\33[5mThis will blink\33(B\33[m This wont" More at http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html

Show a curses based menu selector
Not so much handy by itself, but very nice in shell scripts. This makes you a handy ncurses based checklist. Much like terminal installers, just use the arrow keys and hit 'Space' to adjust the selections. Returns all selected tags as strings, with no newline at the end. So, your output will be something like: "one" "two" "three" "four" "etc" For those who prefer bash expansion over gratuitious typing: $ whiptail --checklist "Simple checkbox menu" 12 35 3 $(echo {one,two,three,four}" '' 0"} ) Things to note: The height must includes the outer border and padding: add 7 to however many items you want to show up at the same time. If the status is 1, it will be selected by default. anything else, will be deselected.

Convert seconds to [DD:][HH:]MM:SS
Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

Which processes are listening on a specific port (e.g. port 80)
swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Convert seconds to [DD:][HH:]MM:SS
Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: