All Questions
216
questions
0
votes
2
answers
104
views
bash + how to insert date before line
lets say I want to pipe the date output in the beggining of some text
for example
echo "this line is test line" | date
and expected output should be
Wed May 22 14:55:10 UTC 2024 this line ...
- 851
2
votes
1
answer
297
views
How to Print, when columns have spaces - awk?
Problem:
desired columns have text with spaces SAMSUNG MZNLN128HCGR-000H1
undesired columns have text with spaces, see VENDOR = M.2 SSD
awk only printing the first word.
awk not printing the column ...
- 367
0
votes
2
answers
423
views
awk string replacement - properly escape '&' character
I am working on one of our applications to update the docker image from using static configs for each environment to a single template using dynamic variable insertion with awk. Our Dockerfile calls a ...
2
votes
2
answers
311
views
I want to get every command that has 3 letters and only three letters
I don't get the output I expect using apropos -k . |awk '/^.../ {print $1}' am I completely overthinking this? In this sample output only zsh should have come back.
I also tried this
apropos -k . | ...
- 474
0
votes
1
answer
531
views
Moving files to a directory with spaces in its name
I'm trying to create a directory that has in its name the beginning of the names of the files it contains. I need this in order to archive it later. I have run into problems when moving files if the ...
alex2020
1
vote
1
answer
97
views
How to parse upload and download speed with ifstat and printf?
In bash I'm trying to display rounded (no decimals) download and upload speed in readable format i.e something like that:
Download 16KB Upload 200KB
I got this but its not quite working:
ifstat 1 1 |...
- 15
0
votes
2
answers
1k
views
Find and replace string in a file using 'sed' or 'awk'
Let's say I have the following input file:
yhara.runner.mng1.nna1.X9HCJG.1 yhara.runner.mng1.nna1.Z2HCJG.1 100.000 100
yhara.runner.mng1.nna1.X9HCJG.1 yhara.runner.mng1.nna1.AUEM0K.1 94.144 72
...
- 103
0
votes
1
answer
324
views
Loop with awk output field
the output of my script
grep 'Interface GigabitEthernet' * | \
sed 's/104-8-20-23.lightspeed.gnvlsc.sbcglobal.net/104.8.20.23/g' | \
sed 's/0.0.0.0: //g' | \
sed 's/.log:/ /g' | \
sed 's/Line protocol ...
1
vote
2
answers
297
views
Multiply numerical value after a string in a json file by 3?
I'm trying to replace a number in a set of text files. The number will be different for each file, but it will always occur after a specific string.
Here's the original file:
"structures&...
- 11
3
votes
4
answers
310
views
How to remove all newlines from a file?
I have the command cat file.txt | gawk '{print $2}' > test which displays:
1
1
0
0
1
1
Afterwards:
vi test
:%s/\n//
Which outputs: 110011
My question is I don't know how I could incorporate this ...
-1
votes
1
answer
165
views
write a efficient for loop in bash for multiple files (.PDB files)
I have problem with this script:
#!/bin/bash
for filename in /home/hb/pg/ensemble/pdbs/P24941/raw_pdb/*.pdb; do
grep COMPND $filename | grep "CHAIN:" -B 1 | sed 's/COMPND//g' | sed 's/...
- 9
0
votes
2
answers
511
views
Are linux commands (grep, awk etc) part of the system, or the shell?
I'm a developer, so I know my way around a terminal, but I never really dove into it. I can write bash scripts (albeit on a beginner level, not my area of expertise), but I now need to understand ...
- 3
1
vote
1
answer
398
views
How to find the "pattern" in FASTA file and record the cordinates along with header
I am looking for a solution to search for a string of 17 base pairs in the FASTA file that is the human reference genome.
To clarify, in simple words and put the resources, i have tried grep function ...
- 13
1
vote
2
answers
561
views
Delete lines between two patterns (with multiple occurrences) around a third pattern (single occurrence)
I'm trying to edit an uncompressed PDF on macOS Monterey, and I want to delete objects that contain a certain pattern in their description. Such objects start with "X 0 obj" (where X is the ...
- 11
2
votes
0
answers
898
views
Find and replace multi-line block in multiple files (bash/linux)
I'm trying to find a way to find a text block that meets the criteria:
block starts with "google = {"
block ends with next "}" character found - ignoring everything else.
Example
...
- 21
0
votes
0
answers
404
views
Awk regex: how to remove all punctuation except when in the middle of a word?
Following information gathered on this thread, I wanted to use this code to fill my needs, however the code provided wasn't working for more than one punctuation character (e.g. '...'). I thus wanted ...
0
votes
1
answer
423
views
Summing real numbers from a text file
We have the following file:
more /home/avrg.bc.txt
3.4
54.2
4.5
3.5
2.3
89.6
.
.
.
We want to calculate the sum of the numbers in the file as 3.4 + 54.2 + 4.5 + ....
- 851
0
votes
1
answer
57
views
keep each line containing x only if column y is the first containing x. remove the rest containing x
I'm trying to reduce a results file that contains multiple results for sometimes the same query and I only want the top result from each query.
Using bash and bash tools like awk, sed, etc. I would ...
- 11
0
votes
1
answer
280
views
Enclose the year in parenthesis in a filename
I have a group of files in a folder that I want to rename; each file has a 4-digit year as part of the filename, I would like to enclose the years in parenthesis.
"Ghostbusters 3 2016.avi" ...
0
votes
0
answers
518
views
using awk with single quote in a shell script will output error [duplicate]
we are on Debian 11.
I'd like to use grep and awk to retrieve information from the content of a single file, like the following:
file: /home/user/directory/otherdirectory/data.txt
file content:
$db['...
0
votes
2
answers
634
views
Remove the file name from a messy path in a text file awk or sed
I have an xml file with many lines containing something like this:
Save="C:\Users\Administrator\Desktop\my files\1020\A54f\Drawing965.DWG" Module="0"
Save="C:\Users\...
- 35
0
votes
2
answers
59
views
bash + gawk: set the number of the processed lines in CSV
As a part of my bash routine, I am using the following gawk code integrated to bash function which loop the CSVs and do their analysis:
test_ranking44 () {
for csv in "${rescore}"/*str*....
- 147
1
vote
1
answer
1k
views
Remove double quotes embracing variable strings from terminal
I have several files that contain this kind of line:
from="$variable_name" and other stuff
$variable name can change, so I could have
from="$myArray" and other stuff
from="$...
- 133
0
votes
1
answer
182
views
Using bash variables in the awk code
The following function written in bash incorporates AWK code carying out math operation on the multi-column data and eventually saves the results in the output file for all processed CSVs.
home="$...
- 147
1
vote
1
answer
1k
views
Use sed/awk to add specific text from one file to beginning of another file
I'm trying to use sed/awk to copy a range of text from the beginning of file1 to a specific line. Then add that output to the beginning of file 2 for a script I'm writing.
The following command does ...
- 11
0
votes
3
answers
2k
views
awk Removing Double Quotes from CSV File Quoted Strings separated by comma
I have a csv file I would like to remove any rouge " or , in it. The problem is all my fields look something like this "***********","*********", where * = can be any ...
- 1
0
votes
1
answer
317
views
Error filtering 2 files using AWK for a given condition
First of all, thank you for your help. I have a problem filtering 2 files using AWK conditionals. The two files I want to filter are these one: Fasta.fa
>SiiA lcl|NC_003197.2_prot_NP_463122....
- 1
0
votes
1
answer
569
views
Shuffle rows independently in a large file
I need to generate a random shuffle a very large csv (where I don't know in advance how many columns would be) in this way. So I have to go from this form
1,a,...
2,b,...
3,c,...
to something like ...
- 341
1
vote
2
answers
5k
views
Extract key from key-value pair
I have a file looking like that.
mke_version = "3.3.7"
msr_version = "2.8.5"
mcr_version = "19.03.14"
I want to print from file only key ...
- 33
1
vote
1
answer
436
views
Print sections in text file using awk and shell variables
I have a text file, materials.txt, with a large number of material definitions similar to this:
*FILESTART
line_a
line_b
line_c
*MAT_PLASTIC
1 property1 property2 property3
property4 ...
- 23
1
vote
0
answers
661
views
How to slice array of awk-output in bash?
Summary
Depending on what bash or awk version I have, I get different results when I try to slice an array that holds awk output. On older versions of bash and awk the "slice operation" ( e....
- 111
2
votes
1
answer
3k
views
Replacing words with specified words using AWK
My code generates several json file all of which look similar to
{
"instance": "sample_instance_05",
"steps": [
{
"129": "North&...
- 1,821
0
votes
1
answer
164
views
(awk) writing content between pattern 1 and second occurrence of pattern 2 into separate files
errors=$(mktemp -d) || exit 1
cat "$logFile" | awk -v d=$errors '/ ERROR /, /********************************************************************************/ {if ($0 ~ / ERROR /) {n++} ; ...
- 1
0
votes
1
answer
2k
views
Extract values following specific substrings in string containing spaces using awk
I have this single line string.
WordPress version: 5.5.3 Database revision: 48748 TinyMCE version: 4.9100 (49100-20200624) Package language: en_US
I would like to extract 5.5.3 and en_US
This is the ...
- 101
1
vote
3
answers
13k
views
find duplicate record in Text file
I have text file with this kind of structure
mv /XXX/20000/XXX-18245 /XXX/20000/XXX-28042
mv /XXX/10000/XXX-9942 /XXX/10000/XXX-18166
mv /XXX/10000/XXX-9962 /XXX/10000/XXX-18189
mv /XXX/20000/XXX-...
- 45
2
votes
2
answers
2k
views
Remove lines where either column is empty
I have a large colon-delimited text file containing two columns like this:
valueA:valueB
valueC:
valueD:valueE
:valueG
If there is no data to the left or right of the colon, I want to remove the ...
- 13k
3
votes
1
answer
1k
views
Remove spaces from first column of colon-delimited file
I have a very large colon-delimited file with two columns:
dhsj fdksjdk:dndkdlkfkd
shfdosod dkjsosjd:dkj dosjeod
dhswisoaidodisbw: dhsoi wjdos
fjfdododjfideefkjso sn:fhfdkd dkdjdf
If the first ...
- 13k
2
votes
2
answers
4k
views
Why am I getting this error in my script? awk: script.awk:19: “ syntax error
I have 2 text files
1.txt
AA;00000;
BB;11111;
GG;22222;
2.txt
KK;WW;55555;11111;
KK;FF;ZZ;11111;
KK;RR;YY;11111;
I try to generate this 3.txt output:
AA;00000;
BB;11111;KK;WW;55555;KK;FF;ZZ;KK;RR;YY;...
- 47
1
vote
0
answers
39
views
merging with format ;string;
I repost my question hoping to provide now a minimal reproducible example
Is basically doing a merge of the two files with format NAME;URL;?
Not exactly, string is not required to be a URL, it is ...
- 53
0
votes
0
answers
64
views
When script carries common strings from rows, lines are truncated
I have this script
BEGIN {
OFS=FS=";"
}
ARGIND == 1 {
map[$2] = ($2 in map ? map[$2] OFS : "") $1
next
}
{
r = $0;
for (i=1; i<=NF; ++i)
if ($i in map)
...
- 77
0
votes
0
answers
128
views
Cat awk command additional code implementation
I want to understand how to use awk cat command because I think to do not understand
In this page he suggest me to use his code
You should process g2.txt first like this:
cat join2.awk
BEGIN {
OFS=...
- 57
0
votes
0
answers
33
views
Not the same output of small bash-script on different Linux systems
I have a small script, that makes JOIN of LSBLK and DF commands.
Script:
lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT | egrep -v "fd0|ram|loop|sr0|hdc|cdrom|[SWAP]" |
while read kname ...
- 153
0
votes
1
answer
308
views
Helping with Join of output from DF and LSBLK Linux commands via bash
I already asked this question, but without update. I think this information should be related to this site.
I need to merge two outputs in Linux. This:
lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT |...
- 153
0
votes
2
answers
254
views
How to decrypt octal encoding from zfs diff output?
zfs diff replaces some of the characters in files names with \xxxx (in an octal encoding). Can I decrypt it back with awk or sed or any other common utility (without resorting to a perl or a python ...
- 109
0
votes
1
answer
1k
views
grep for same occurrence more than once
Is it possible to grep or awk for more than 1 occurrence of the same string from a file?
awk '/$STRING|$STRING/' file
grep -e $STRING -e $STRING file
I tried other options but still not able to get ...
1
vote
1
answer
641
views
AWK/SED/GREP - print M lines after the Nth occurrence of a match
I am looking to display M lines after the Nth occurrence of a match. Actually I need to do this in order to split a file containing Nth <regex> always followed by M lines into Nth different ...
- 21
2
votes
1
answer
2k
views
Remove all characters after first non-numeric character using sed or awk
TL;DR:
How can I create an expression using either sed or awk that will remove all characters after (and including) the first character that is not either numeric (i.e. [0-9]) or a period (.). I've ...
- 131
0
votes
2
answers
90
views
How to execute bash command on nth flat file column and print it out to output?
I need just basename cmd to be executed on 2'nd column
[ora@rac TEST]$ cat test.log | awk '{print $1 " " $2}'
9780c69e13e3dda64c3c6ddce2b37803 /test/test_202007140422_01.zip
...
- 3
1
vote
2
answers
77
views
pattern match special charecters and display contents between the pattern
My apache log format is as below
192.168.10.1 <"www.mytest.com"> <-"-"-> <--“192.168.10.90"--> [2020-07-10 03:35:46 PDT] >1008< 2470 3006 "https&...
- 417
1
vote
1
answer
229
views
recursively sort files with string "SENT_" somewhere in filename by the substring immediately after SENT_, and then display them. Linux
So I wish to run this from any directory, say ~/ and have it work for subdirectories. Example would be;
ls ~/:
/home/me/FILES/DIR_1/DIR_a/SENT_2222_....
/home/me/FILES/DIR2/SENT_3333....
/home/me/...
- 173