Questions tagged [awk]
AWK is a text-processing language. It is mainly used to manipulate and process structured data, and to generate reports.
631
questions
193
votes
18
answers
161k
views
Removing ANSI color codes from text stream
Examining the output from
perl -e 'use Term::ANSIColor; print color "white"; print "ABC\n"; print color "reset";'
in a text editor (e.g., vi) shows the following:
^[[37mABC
^[[0m
How would one ...
- 3,704
146
votes
12
answers
302k
views
How to get video duration in seconds?
How can I get video duration in seconds?
What I've tried:
ffmpeg -i file.flv 2>&1 | grep "Duration"
Duration: 00:39:43.08, start: 0.040000, bitrate: 386 kb/s
mediainfo file.flv | grep ...
- 1,847
62
votes
4
answers
26k
views
awk, mawk, nawk, gawk... WHAT?
I've just started learning awk and I'm a little confused about all those versions around. Is there any "version" which is found on all Unix-like systems? Like, you know, plain vi? Does the standard ...
Trollhorn
51
votes
11
answers
66k
views
sed: how to replace line if found or append to end of file if not found?
With a single input file that only contains comments (starting with #) and VARIABLE=value lines, is it possible to replace a value for a single variable if found and, otherwise, append the pair to the ...
- 1,258
23
votes
6
answers
74k
views
How to get the pid of a running process using a single command that parse the output of ps?
I am looking for a single line that does return the pid of a running process.
Currently I have:
ps -A -o pid,cmd|grep xxx|head -n 1
And this returns the fist pid, command. I need only the first ...
- 11.9k
19
votes
2
answers
50k
views
replace nth occurence of string in each line of a text file
I have large text files with space delimited strings (2-5). The strings can contain "'" or "-". I'd like to replace say the second space with a pipe.
What's the best way to go?
Using sed I was ...
- 387
18
votes
7
answers
39k
views
How can I replace a newline with its escape sequence?
Using sed to make text that contains certain characters suitable for use in string literals is straightforward:
sed "s/\\\\/\\\\\\\\/g"
sed "s/\\\"/\\\\\\\"/g"
But how can I do something similar ...
- 1,125
18
votes
2
answers
26k
views
How can I find my awk version?
If I want to know the version of awk I get the following:
$ awk --version
awk: not an option: --version
Checking in man awk I see that my awk is
mawk - pattern scanning and text processing ...
- 1,967
15
votes
3
answers
19k
views
How can I make a table of contents for a markdown document with Python/AWK/SED?
I have the following markdown document:
Heading-a
==========
---text---
Heading-b
------------
--- text ---
Heading-c
----------
--- text---
Heading-d
=======
--- text----
Heading-e
---
...
...
- 5,794
14
votes
5
answers
19k
views
Add to every end of line in Notepad++
I have a long text file
gallery-dl -g -i w4b027.txt >
gallery-dl -g -i a4b028.txt >
gallery-dl -g -i b4b029.txt >
gallery-dl -g -i c4b030.txt >
gallery-dl -g -i d4b031.txt >
gallery-dl -...
- 354
14
votes
7
answers
28k
views
Piping tail -f into awk
I am trying to pipe tail -f into awk to monitor a logfile in realtime, but according to examples, there should be not problem but I can't get it to work.
here is the command I'm running
tail -f ...
- 241
14
votes
4
answers
17k
views
How to pipe awk output (with periodic, continuous input) to output file?
I am trying to write a command that pipes the continuous output of a free command (run every second) to an awk command that parses a specific value (available free memory) and outputs this to a file ...
- 370
14
votes
2
answers
23k
views
How can I identify non-ASCII characters from the shell?
Is there a simple way to print all non-ASCII characters and the line numbers on which they occur in a file using a command line utility such as grep, awk, perl, etc?
I want to change the encoding of ...
- 3,704
13
votes
6
answers
22k
views
Randomly shuffle rows in a large text file
I have a text file of ~1GB with about 6k rows (each row is very long) and I need to randomly shuffle its rows. Is it possible? Possibly with awk?
- 367
13
votes
1
answer
17k
views
How to use awk separator of single quote?
I'd like to use awk to pull out information from SQL output like the following:
(count(distinct serverclass)='2')
And need to extract the number, which is 2 in this example.
I'd like to do ...
- 7,978
12
votes
3
answers
18k
views
How can I cat/print a file except the last 2 lines?
How can get I get the contents of a text file, except the last (for instance) 2 lines, with standard or GNU shell tools? (bash, tail, cat and so on.)
- 402
12
votes
1
answer
16k
views
Awk doesn't work when inside double quotes
This command doesn't work:
ssh $HOST "ls -l | awk '{print $1}'"`
Above ignores the command awk. I think it might be because of the double quotes?
Also, how would I add another set of double ...
- 505
12
votes
3
answers
25k
views
Filtering top command output
I'm running a Mac and want to filter the output of the top command for a particular process such that only the PID, COMMAND and %CPU columns are displayed.
After getting the PID of the process, I run:...
- 1,438
10
votes
3
answers
22k
views
How to find files within a size range?
Please suggest a way to find all the zip files which are more than 60 MB but less than 70 MB in size using the find command.
- 326
10
votes
1
answer
21k
views
Counting occurrences in first column of a file
We have this file:
1 2
1 3
1 2
3 3
52 1
52 300
and 1000 more.
I want to count the number of times each value occurs in the first column.
1 3
3 1
52 2
This means we saw 1 three times.
How can ...
- 726
9
votes
4
answers
58k
views
Print line X lines before/after found line
I want to search for a particular string in a certain file. If I found the string, I also want to print the line X lines before (or after) that line.
Can this with grep or awk, or do I need a ...
- 213
9
votes
3
answers
17k
views
Round off the decimal value in awk
i want to round off the value 262.5 to 263 using awk
i have tried the below
echo "262.5" | awk '{printf "%.f\n", $1}'
My expected answer is 263
- 109
8
votes
2
answers
9k
views
Watch with awk command
How do you watch a command like?
awk 'NR%2==0 {printf "%s %8.0f", $1, $5}' filename.txt
Preceding this with "watch" gives this error:
awk: cmd. line:1: fatal: cannot open file `{printf' for reading ...
- 227
8
votes
2
answers
3k
views
stdbuf not working while unbuffer does
I am trying to apply timestamps to stdout of a process. For the proper timestamps to be applied, I attempt to unbuffer stdout of the process. This works with unbuffer but not with stdbuf as I would ...
- 301
7
votes
6
answers
24k
views
Pipe output of awk to kill -9
I'm trying to kill any process that has to do with mysql. I'm piecing together a command, and so far, haven't come up with the right solution. One other consideration, is that the last process will ...
- 583
7
votes
4
answers
88k
views
AWK Equivalent Functionality on Windows
I need to troubleshoot some issues on a client PC that I am unable to install additional software on and need some basic functionality of awk on windows xp & 7.
Basically, I need to strip the ...
- 237
7
votes
6
answers
2k
views
Joining text files with 600M+ lines
I have two files, huge.txt and small.txt. huge.txt has around 600M rows and it's 14 GB. Each line has four space separated words (tokens) and finally another space separated column with a number. ...
- 387
7
votes
5
answers
8k
views
Using awk to split text file every 10,000 lines
I have a large gzip'd text file. I'd like to something like:
zcat BIGFILE.GZ | \
awk (snag 10,000 lines and redirect to...) | \
gzip -9 smallerPartFile.gz
the awk part up there, I basically ...
- 175
7
votes
2
answers
7k
views
Using ' (' (space followed by parenthesis) as field separator in awk
In an awk script I am trying to use ' (' as the field separator.
However unless I escape the bracket with double back-slash, like this:
BEGIN {FS=" \\("}
it does not work.
If I use FS=" \(" I get
...
- 3,785
7
votes
3
answers
7k
views
SQL like group by and sum for text files in command line?
I have huge text files with two fields, the first is a string the second is an integer. The files are sorted by the first field. What I'd like to get in the output is one line per unique string and ...
- 387
6
votes
3
answers
7k
views
Remove non-duplicate lines in Linux
how can I remove non-duplicate lines from text file using any linux program linke sed, awk or any other?
Example:
abc
bbc
abc
bbc
ccc
bbc
Result:
abc
bbc
abc
bbc
bbc
Second list have removed ccc ...
- 63
6
votes
4
answers
13k
views
Using grep to display second character in string?
I have a string that is always 3 characters... the first one and the last one are always the same.
example:
▁▅█
Is there a simple way to only display the one in the middle? (which is the only one ...
- 567
6
votes
4
answers
20k
views
awk match whole word
I want to use awk to match whole words from text file. Including words bounded by non alphanumeric characters.
For example -
string to search for - ABC
Source file -
HHHABCCCCH
HHH ABC
HH(ABC)...
- 8,448
6
votes
1
answer
32k
views
awk - awk concatenate string variable
I would like to awk concatenate string variable in awk. How can I do that?
I tried:
BEGIN{
t="."
r=";"
w=t+r
print w}
But I does't work.
Output:
0
Or I want to add variable and result of function.
...
- 907
6
votes
9
answers
18k
views
awk + print all line content except $1
I write the follwoing awk script:
% echo /var/sysconfig/network/my_functions alpha beta gama | \
awk -v word=alpha '$2 == word { print $0 }'
how to tell awk that I want to print all line ...
- 1,127
6
votes
3
answers
27k
views
using awk with find -exec
I have a directory structure with 14 directories containing a bunch of files containing data in a three-column format (separated with tabs). I intended to use find and awk to extract the second column ...
- 75
6
votes
1
answer
17k
views
"No such file or directory" for /bin/awk on Ubuntu
How can I install awk in this version of Ubuntu 11? I installed it from Ubuntu Software Center and I cannot use it.
I'm trying to run an awk script:
#!/bin/awk
Begin
{
print strftime("ora %H,...
- 259
6
votes
4
answers
16k
views
Rename CSV header using command line tools
Given a CSV file,
h1 h2 h3
a b c
z x b
how to rename the first header to One and the third header to Three using command line tools, namely
One h2 Three
a b c
z x b
- 165
6
votes
4
answers
7k
views
sudo to kill output of ps, awk, xargs
I want to kill root owned processes containing foo in the process name
sudo sh -c "ps aux | grep [f]oo | awk '{print $2}' | xargs kill -15"
this command fail as awk and xargs are ignored.
sudo sh -...
- 429
6
votes
4
answers
9k
views
Print back-reference in regular expression
I was hoping for a way to make sed replace the entire line with the replacement (rather than just the match) so I could do something like this:
sed -e "/$some_complex_regex_with_a_backref/\1/"
and ...
- 171
6
votes
1
answer
4k
views
How to split binary file based on pattern?
I would like to split a binary file into smaller files based on pattern 'xÚ' (it's 78 DA in hexadecimal), so when there's an 'xÚ' in the file the splitter script splits and pastes the content into a ...
5
votes
3
answers
6k
views
linux - replace all double newlines with a space
Is it possible to replace only double newlines in a text file using sed, awk, grep, tr, or whatever is needed?
this
is
data
this
is
more
data
What I need to do is to replace all double newlines ...
- 65
5
votes
4
answers
5k
views
Parsing the output of "uptime" with bash
I would like to save the output of the uptime command into a csv file in a Bash script. Since the uptime command has different output formats based on the time since the last reboot I came up with a ...
- 53
5
votes
3
answers
8k
views
output the items that showed up 2 times and more
How can I select the ips that showed up 4 times or more , for example here?
the input file is
192.168.1.28
192.168.1.100
192.168.1.31
192.168.101.2
192.168.1.31
192.168.11.02
192.168.1.31
192.168.1....
- 726
5
votes
3
answers
11k
views
Inserting string from xargs into another string
Say I have some command that outputs a bunch of strings. The output from this command could look something like:
...
foo
bar
blargh
...
Say I have some set string
myString$Blah
I want to insert ...
- 583
5
votes
1
answer
13k
views
Using grep to remove lines from a file which contain a string from another file
I have a file containing words (one per line) such as
Dog Fish Cat Shoes
I have a secondary file in CSV format such as
1,shoes,red 2,apple,black 3,fog,blue
I would like to use grep with the ...
- 210
5
votes
3
answers
11k
views
How can I zero pad a particular field in a text file?
I have a file like this:
123123213,456,be
124243233,4346,ytr
123123123,436535,uytr
324234324,322,yt
234324323,32,tyutr
I want to zero-pad the middle field to give the result:
123123213,00000456,be
...
- 123
5
votes
5
answers
3k
views
How to chain awk print statements?
Okay, here's a quick example; me isolating a host's IPV4 in three steps:
ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0 DOWN
wlan0 UP ...
- 2,741
5
votes
3
answers
1k
views
Search for files with more than one term (grep, awk?)
I am using a command like this to find files with the word 'term' in them:
grep -l term *
But I now want to be able to find files which have two different words in them (let's call them termA and ...
- 202
5
votes
2
answers
7k
views
Is it possible to change the default field separator in awk?
I'm always using awk to manipulate comma separated files, so the first line in my code is always changing the field separator to "," like so:
awk 'BEGIN {FS=","}
$2 < 20 {print $1}' myfile.csv
Is ...
- 183