AWK

substituting awk variable in a bash script

2023-01-23T12:16:56+00:00

i am new to using awk.

i have written a simple one liner script to calculate memory consumed by a program as below.

```

!/usr/bin/env bash

read -p "enter process name: " item;

ps aux | awk -vi=$item '/i/ {sum += $6} END { printf "Memory Usage: %d MB\n", sum/1024 }'

```

in the above example, variable 'i' is not substituted when executing script.

where i am going wrong?

submitted by /u/mmkodali
[link] [comments]

Append to first matched line only, delete first matched line beginning from a matched line

2023-01-23T05:00:20+00:00

I have an xml file that I would like transform to this xml file in a shell script using awk. The resulting diff is:

2,3c2 < <name>debian-11-test</name> < <uuid>4ade684e-ce3e-4746-8292-528a84b98445</uuid> --- > <name>debian-11-test-1</name> 38c37 < <source file='/tmp/vm/debian-11-test.qcow2'/> --- > <source file='/tmp/vm/debian-11-test-1.qcow2'/> 88d86 < <mac address='52:54:00:14:fa:09'/> 89a88 > <ip address='192.168.122.11' prefix='24'/>

Looking for a mini awk script or command that can do this summary of the changes:

In the line containing <name>debian-11-test</name>, replace with <name>${host}</name> where $host is a shell variable with the resulting string to be placed in the xml file.
Delete the line with <uuid> and </uuid>, ideally only deleting the first matching beginning from the above <name></name> line or at least deleting the first match found in the file only.
Same as the first change: want find line containing <source file='/tmp/vm/debian-11-test.qcow2'/> and replace with <source file='/tmp/vm/${host}.qcow2'/>.
Same as second change: delete the line with <mac address='52:54:00:14:fa:09'/>, ideally only deleting the first match beginning with the line containing <interface type='network'> or at least deleting the first match found in the file only.
Finally, add a new line <ip address='192.168.122.$cnt' prefix='24'/> after the line matching <interface type='network'> and then exiting immediately.

Much appreciated. I should be able to learn from suggestions and tweak if they don't do exactly the above.

P.S. I'm aware of tools like virt-sysprep to prepare the VM image but they are for prepping a base image whereas I want to do bake these changes into the VM image so they are generated fresh every time without requiring a clean base image that needs to be maintained.

submitted by /u/exquisitesunshine
[link] [comments]

Can't figure this behavior out

2023-01-22T09:27:03+00:00

The relevant piece of awk code:

comm = n ? substr($0, m+1, n-m-1) : substr($0, m+1) jump = n ? substr($0, n+1) : 0 print comm printf("comm %s; jump %s;\n", comm, jump)

yields the output

A ; jump 0 D+A ; jump 0 D jump 0

with both gawk and mawk. Why is the value of comm disappearing in between the print and printf statement? Why isn't even the string literal "comm" within the printf argument being printed?

Entire code: https://pastebin.com/hD6PGFrP

Input file:

// This file is part of www.nand2tetris.org // and the book "The Elements of Computing Systems" // by Nisan and Schocken, MIT Press. // File name: projects/06/add/Add.asm // Computes R0 = 2 + 3 (R0 refers to RAM[0]) @2 D=A @3 D=D+A @0 M=D

submitted by /u/Pure_Needleworker536
[link] [comments]

Splitting a File and Extracting Text Between Two Strings

2023-01-21T00:26:30+00:00

Hi, y'all! I have a file where answers to questions were recorded and are preceded by a number and a right parenthesis, e.g. 1) and 9). What I'm trying to do is extract the number, the parenthesis, and the relevant information, i.e. any type of character that appears after the number and parenthesis BUT before the next number and parenthesis. For instance, if I have a file with the following content and then run the subsequent AWK script, it shows everything between 1) and 3). What I want to do is show everything between 1) and 2). Thank you in advance for your help!

test.txt 1) good 2) bad 3) ok

script.awk awk '/1\)/,/2\)/ { if ($0 ~ /1\)/) { p=1 } if (p) { print } if ($0 ~ /2\)/) { exit } }' test.txt

submitted by /u/tspier
[link] [comments]

Getting multiple near-identical matches on each line

2022-12-27T11:21:32+00:00

So the other day at work I was trying to extract data formatted like this:

{“5_1”; “3_1”; “2_1”;} (there was a lot more data than this spanning numerous lines, but this is all I cba typing out)

The output I wanted was: 532

I managed to get awk to match but it would only match the first instance in every line. I tried Googling solutions but couldn’t find anything anywhere.

Is this not what AWK was built for? Am I missing something fundamental and simple? Please help as it now keeps me up at night.

Thanks in advance :)

submitted by /u/rocket_186
[link] [comments]

Any way to reliably concatenate the elements of an array back into a string using gawk?

2022-12-21T19:07:18+00:00

Tapping the hive mind here. I need to sort letters in a string. I’ve put the letters into an array using Split, sorted the areay using Asort, now I need the elements of the array put back into a string. This seems to be unreasonably difficult in gawk due to it’s undefined concatenation. Is there a method or function that solves this problem? Thanks!

submitted by /u/AppleTreeBloom
[link] [comments]

How to split() a field variable (ex. $2) to an array?

2022-12-21T18:40:03+00:00

Looking to find options on how to split a ".tsv" file's field variable to an array.

I would like to keep it POSIX compliant if possible. (Using mawk 1.3.4 20200120)

Would prefer to use the built-in split() function since it gives incremental integer indexes.

These are the methods I tried. Only the last one works in an example script below.

array[$2];
split($2, array. "\n");
string = sprintf("%s ", $2); WITH split(string, array, " ");
printf(%s ", $2) > "field.tmp"; WITH split(file, array, " ");

#!/usr/bin/awk -f NR > 1 { #SAVE FIELD AS 1-LINE FILE printf("%s ", $2) > "/tmp/field.tmp"; } END { #ASSIGN PATH VARIABLE path = "/tmp/field.tmp"; fflush(path); #SPLIT 1-LINE FILE INTO ARRAY while ((getline file < path) > 0) { split(file, array, " "); }; close(path); #PRINT TO STDOUT TO CONFIRM for (i in array) { printf("Index %d is %s\n", i, array[i]); } }

Any help would be awesome :) I did notice that POSIX awk supports regex in the split function but no luck.

POSIX: (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html)

submitted by /u/JMP800
[link] [comments]

awk to convert a txt date to unix timestamp?

2022-12-18T21:50:01+00:00

Hi all. Can't get my brain around how to convert some netflow data into influxdb format via awk.

Have data, looks like

csv , columns , for , useful , data , 2022-12-15 12:24:15.410

I'm currently breaking this data up with a while loop and IFS delimiter *but* there are so many lines of data that this ends up being a very slow process.

I'm pretty sure an inline awk would do this much faster, but I need a little help in execution.

unixtimestamp=`date -d "2022-12-15 12:24:15.410" +"%s"` + 000 is what I need for influxdb.

And advice on how to take that data column in the csv and replace it with the computed unix timestamp plus 3 zeros? All other columns we go untouched.

Thanks.

submitted by /u/International_Pea500
[link] [comments]

ChatGPT playing awk. Giving the correct result in first attempt.

2022-12-18T01:46:43+00:00

data

the script

submitted by /u/Automatic-Ad-9530
[link] [comments]

Parentheses in short-circuit statements

2022-12-13T20:16:10+00:00

I'm learning about using logical operators in short-circuit statements, outside of logical tests. (Been using Awk for years, but these are new to me.) I noticed that parenthesizing sub-statements is vital if you want the whole statement to work as expected. But I'm not sure why, or how this works. For example:

BEGIN{ i = 0 ; i+=1 && i+=10 && i+=100 ; print "___ " i # 102 i = 0 ; i+=1 && i+=10 && (i+=100) ; print "__p " i # 102 i = 0 ; i+=1 && (i+=10) && i+=100 ; print "_p_ " i # 111 i = 0 ; i+=1 && (i+=10) && (i+=100) ; print "_pp " i # 111 i = 0 ; (i+=1) && i+=10 && i+=100 ; print "p__ " i # 102 i = 0 ; (i+=1) && i+=10 && (i+=100) ; print "p_p " i # 102 i = 0 ; (i+=1) && (i+=10) && i+=100 ; print "pp_ " i # 111 i = 0 ; (i+=1) && (i+=10) && (i+=100) ; print "ppp " i # 111 }

Only when the middle sub-statement is parenthesized do you get the expected result of 111. If it is not parenthesized you get 102, suggesting the first statement gets evaluated twice, and the last once. Anyone know why? Something to do with the Awk parser?

submitted by /u/ctenolophon
[link] [comments]

Newb here, is this messy?

2022-12-02T00:39:26+00:00

awk '/VM_pool|<name>/ { gsub(/<|>|\047/," "); print $(NF-1) }' $path

submitted by /u/Usually-Mistaken
[link] [comments]

Save the changes in the same file

2022-11-23T14:46:30+00:00

I am using awk to look for a pattern inside a line and change the way it's shown. It aims those lines which have at least three occurrences of nvl( or one occurrence of four trailing open parenthesis (

awk ' { if (tolower($0) ~ /(.*nvl\(){3}/ || $0 ~ /\({4}/) { print "/*" , $0 , "*/" } else { print $0 } }' teste.txt

If matched, this line is surrounded with /**/. It works fine. However, I'd like to make the changes in the file itself, just like the sed -i option makes.

Does awk have a mechanism to save the changes made in the same file?

submitted by /u/Mark_1802
[link] [comments]

Is there a way to do this with awk (using regex or with another alternative)

2022-11-20T19:05:58+00:00

submitted by /u/elnanoml
[link] [comments]

Capitalizing words in awk

2022-11-19T15:47:26+00:00

Hi everyone. Newly discovered awk and enjoying the learning process and getting stuck on an attempt to Capitalize Every First Letter. I have seen a variety of solutions using a for loop to step through each character in a string, but I can't help but feel gsub() should be able to do this. However, I'm struggling to find the appropriate escapes.

Below is a pattern that works in sed for my use case. I don't want to use sed for this task because it's in the middle of the awk script and would rather not pipe out then back in. And I also want to learn proper escaping from this example (for me, I'm usually randomly trying until I get the result I want).

echo "hi. [hello,world]who be ye" | sed 's/[^a-z][a-z]/\U&/g' Hi. [Hello,World]Who Be Ye

Pattern is to upper case any letter that is not preceded by a letter, and it works as I want. So how does one go about implementing this substitution s/[^a-z][a-z]/\U&/g in awk? Below is the current setup, but fighting the esxape slashes. Below correctly identifies the letters I want to capitalize, it's just working out the replacement pattern.

gsub(/[^a-z][a-z]/," X",string)

Any guidance would be appreciated :) Thanks.

submitted by /u/RyzenRaider
[link] [comments]

Newbie question: online awk simulator?

2022-11-18T15:59:28+00:00

Does anyone have an online awk simulator to recommend? Trying to teach myself awk for a project. Thank you!

submitted by /u/AppleTreeBloom
[link] [comments]

A brief interview with AWK creator Dr. Brian Kernighan

2022-11-16T01:22:21+00:00

submitted by /u/breck
[link] [comments]

Reshape existing data

2022-11-11T14:55:24+00:00

I have a text file, formatted

A [tab] 1,2,3... (Varying number of fields) [tab] 1,2,3... (Varying number of fields)

B [tab] 1,2,3... (Varying number of fields) [tab] 1,2,3... (Varying number of fields

C [tab] 1,2,3... (Varying number of fields) [tab] 1,2,3... (Varying number of fields ... [20k lines]

The first field is an IP address, the second column is a varying number of IPs, the third column is the same number of different IPs.

I want to separate everything out so I get

A 1 1

A 2 2

A 3 3

...

B 1 1

B 2 2

...

basically turning 20k lines into 200k+. The second and third columns have 1 - 20 comma-separated fields.

Thinking about constructing this, I'd go

while read p; do

fields=(Count number of fields in second column)

for i in 1..$fields; do

 IP=$(cat $p | awk '{print $1}') Srcaddr=$(cat $p | [awk to get $i'th value in second column]) Dstaddr=$(cat $p | [awk to get $i'th value in third column]) echo $IP $Srcaddr $Dstaddr >> outfile done

done

That actually doesn't look too bad for a first pass. The term in lines 5 and 6 will take a little work, figure I'll get the second and third fields respectively, then do another awk using $i and FS=, to get the appropriate fields from those columns.

Any tips for doing this better? I feel like what I wrote out above will get me there but it feels pretty graceless, and I'd love to learn some new things.

submitted by /u/tje210
[link] [comments]

Split a string with a delimiter and print first character

2022-11-02T19:45:52+00:00

I am trying to convert us-east-1 into ue1. I spent over some time but couldn’t figure out the right way to do it.

Can someone please help me out?

Edit: Thanks everyone for the input. I am going with the below one-liner. echo us-east-1 | awk -vRS=- '{printf substr($0,1,1)}'

submitted by /u/batman_9326
[link] [comments]

Newbie Question: Matching sub-string to field?

2022-10-31T03:39:11+00:00

I have a small Budgeting program going, and want to categorize items on a bank statement. I learned the absolute basics of AWK to combine multiple statement CSVs into one big CSV for the month or quarter. Since I am often getting groceries ect, I would like to knock off a good percentage of the categorizing with the use of matching against a lookup file.

Is there a straight forward way in AWK for every field on a record in a csv, run through an entire lookup table matching the keyword in the lookup table to the field in the CSV?

Dummy Tables

statement.csv:

Date	Description	Amount
10/20/2022	TRADER JOE'S CHICAGO IL	24.85
10/21/2022	SHELL GAS #1234	50.35
10/21/2022	Goldies Pub	10.15
10/22/2022	Dunkin Donuts	5.00

KeywordToCategory:

Keyword	Category
Shell	Automotive
Trader Joe	Grocery
Goldie	Entertainment

Thanks and I really appreciate the help!

submitted by /u/Hectic-Skeptic
[link] [comments]

Why is this code/command not working for negative numbers?

2022-10-28T09:38:14+00:00

awk '($6>max) && ($4!="AZ") {max = $6; line = $0}END{print line}' foo.txt

Whenever field #6 contains negative numbers, it doesn't correctly return the line that has the highest number in field 6.

The goal is to for example in the following file contents:

///////////////////////////////

Shellstrop Eleanor Phoenix AZ 85023 -2920765

Shellstrop Donna Tarantula_Springs NV 89047 -5920765

Mendoza Jason Jacksonville FL 32205 -4123794

Mendoza Douglas Jacksonville FL 32209 -3193274 (Donkey Doug)

Peleaz Steven Jacksonville FL 32203 -3123794 (Pillboi)

///////////////////////////////////

goal is to return the line containing Peleaz. (It wouldn't be Shellstrop Eleanor as she lives in AZ.)

This works as required for positive numbers but not negative. Or there could be some completely different bug Im missing. Im very new to awk.

submitted by /u/LeadershipComplex958
[link] [comments]

Does each statement inside the brackets always execute repeatedly? Even though the average only needs to be set once. How do you know that it does it repeatedly? Super noob question.

2022-10-23T01:40:51+00:00

https://preview.redd.it/l0cnu6v7mgv91.png?width=333&format=png&auto=webp&s=9db4aed991dd62f1eb0e7eefd37ccc34a2dc0f7f

submitted by /u/LeadershipComplex958
[link] [comments]

Processing a specific part of a text according to pattern from AWK script

2022-10-21T11:04:19+00:00

Im developing a script in awk to convert a tex document into html, according to my preferences.

```

!/bin/awk -f

BEGIN { FS="\n"; print "<html><body>" }

Function to print a row with one argument to handle either a 'th' tag or 'td' tag

function printRow(tag) { for(i=1; i<=NF; i++) print "<"tag">"$i"</"tag">"; }

NR>1 { [conditions] printRow("p") }

END { print "</body></html>" } ```

Its in a very young stage of development, as seen.

``` \documentclass[a4paper, 11pt, titlepage]{article} \usepackage{fancyhdr} \usepackage{graphicx} \usepackage{imakeidx} [...]

\begin{document}

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla placerat lectus sit amet augue facilisis, eget viverra sem pellentesque. Nulla vehicula metus risus, vel condimentum nunc dignissim eget. Vivamus quis sagittis tellus, eget ullamcorper libero. Nulla vitae fringilla nunc. Vivamus id suscipit mi. Phasellus porta lacinia dolor, at congue eros rhoncus vitae. Donec vel condimentum sapien. Curabitur est massa, finibus vel iaculis id, dignissim nec nisl. Sed non justo orci. Morbi quis orci efficitur sem porttitor pulvinar. Duis consectetur rhoncus posuere. Duis cursus neque semper lectus fermentum rhoncus.

\end{document} ```

What I want, is that the script only interprets the lines that are between \begin{document} and \end{document}, since before they are imports of libraries, variables, etc; which at the moment do not interest me.

How do I make it so that it only processes the text within that pattern?

[link] [comments]

help : newbie : How to use awk to specify from a field X to end of line

2022-10-11T17:36:41+00:00

I've seen some people say AWk don't really use ranges.

I have an input plain text file that I would like to convert to CSV using awk.

the problem is my last part of the record, where I want to preserve the input fields and not separate them with a delimiter, and This is pretty much a free format field(description) which can therefore contain say up to N random nr of words, which I would like to output as a single field.

given the input as an example

DATE TIME USERID NAME SURNAME SU-ID DESCRIPTION

10SEP22 17:26 UID01 John Wick root TEST 10SEP22 17:30 UID110 Bat Man DBusr Rerun Backup. 10SEP22 23:02 UID02 Peter Parker admin COPY FILE & EDIT DATE

As can be seen after the 6th field I would like to specify the rest as a single field and there can be N words present until the end of the line.

So currently I have this,

$awk '{print $1 "," $2 "," $3 "," $4 " " $5 "," $6 "," $7}'

and the output is this :

10SEP22,17:26,UID01,John Wick,root,TEST 10SEP22,17:30,UID110,Bat Man,DBusr,Rerun 10SEP22,23:02,UID02,Peter Parker,admin,COPY

It obviously cuts off after field 7 and only works if there is a single word in the description. Note I am also trying to keep the name and surname as a single field, hence separated by a space, not a comma.

I would like to get something like this to work in place of $7 above, while everything else($1 - 6) as per above still remains(on its own this works fine for my requirement) :

awk {'{i = 14} {while (i <= NF) {print $i ; i++}}'}

that way the output should be :

10SEP22,17:26,UID01,John Wick,root,TEST 10SEP22,17:30,UID110,Bat Man,DBusr,Rerun Backup. 10SEP22,23:02,UID02,Peter Parker,admin,COPY FILE & EDIT DATE

Any help is much appreciated.

submitted by /u/Gold-Ad-5257
[link] [comments]

What does $0=$2 in awk do? learn awk

2022-09-25T01:24:20+00:00

submitted by /u/morihacky
[link] [comments]

Match a pattern, start counter and replace the 5th field with the counter. Help Needed.

2022-09-04T18:55:29+00:00

I have a file which looks something like this:

ATOM 3667 CD1 ILE 237 12.306 -11.934 16.545 1.00 0.00 ATOM 3668 HD11 ILE 237 12.949 -12.488 16.075 1.00 0.00 ATOM 3669 HD12 ILE 237 11.408 -12.181 16.274 1.00 0.00 ATOM 3670 HD13 ILE 237 12.463 -11.002 16.328 1.00 0.00 ATOM 3671 C ILE 237 9.292 -11.489 20.242 1.00 0.00 ATOM 3672 O ILE 237 8.722 -10.388 20.078 1.00 0.00 ATOM 3673 OXT ILE 237 9.145 -12.132 21.279 1.00 0.00 TER ATOM 3674 N1 LIG 238 -1.541 3.935 2.126 1.00 0.00 ATOM 3675 C2 LIG 238 -0.418 6.199 2.597 1.00 0.00 ATOM 3676 N3 LIG 238 -3.604 3.076 2.842 1.00 0.00 ATOM 3677 C4 LIG 238 1.091 5.162 4.121 1.00 0.00 ATOM 3678 C5 LIG 238 0.498 4.906 5.503 1.00 0.00

After TER in $1 you can see that from next record the $4 field is LIG, and the $5 is 238, I want to change $5 to 1 for the first time LIG is matched then 2 for the next and so on.

This is how I want it to be:

ATOM 3667 CD1 ILE 237 12.306 -11.934 16.545 0.00 0.00 ATOM 3668 HD11 ILE 237 12.949 -12.488 16.075 0.00 0.00 ATOM 3669 HD12 ILE 237 11.408 -12.181 16.274 0.00 0.00 ATOM 3670 HD13 ILE 237 12.463 -11.002 16.328 0.00 0.00 ATOM 3671 C ILE 237 9.292 -11.489 20.242 1.00 0.00 ATOM 3672 O ILE 237 8.722 -10.388 20.078 1.00 0.00 ATOM 3673 OXT ILE 237 9.145 -12.132 21.279 0.00 0.00 TER ATOM 3674 N1 LIG 1 -1.541 3.935 2.126 0.00 0.00 ATOM 3675 C2 LIG 2 -2.491 3.845 3.151 0.00 0.00 ATOM 3676 N3 LIG 3 -3.604 3.076 2.842 0.00 0.00 ATOM 3677 C4 LIG 4 -3.852 2.404 1.633 0.00 0.00 ATOM 3678 C5 LIG 5 -2.826 2.559 0.663 0.00 0.00

I have banged my head around google, I need a quick fix. I could get till awk '{ print $0 "\t" ++count[$1] }' which adds the counter as an extra column. Thanks for the help!!!

submitted by /u/shrchem
[link] [comments]