Read and write few lines from the output of another program into a new file

nilkanthagholap · February 19, 2020, 9:35am

========================================================================
Wed 12-02-2020 19:06:20:: INFO : Processing 1925+211_gsb_band5_chan_field.ms.ap1.image.fits
Wed 12-02-2020 19:06:20:: INFO : PyBDSF version 1.9.2
Wed 12-02-2020 19:06:20:: INFO : Non-default input parameters:
adaptive_rms_box = True
advanced_opts = True
filename = ‘1925+211_gsb_band5_chan_field.ms.ap1.image.fits’
rms_box_bright = (33, 11)
Wed 12-02-2020 19:06:20:: USERINFO: → Opened ‘1925+211_gsb_band5_chan_field.ms.ap1.image.fits’
Wed 12-02-2020 19:06:20:: INFO : Original data shape of 1925+211_gsb_band5_chan_field.ms.ap1.image.fits: (1, 1, 6
144, 6144)
Wed 12-02-2020 19:06:20:: INFO : Final data shape (npol, nchan, x, y): (1, 1, 6144, 6144)
Wed 12-02-2020 19:06:20:: USERINFO: Image size … : (6144, 6144) pixels
Wed 12-02-2020 19:06:20:: USERINFO: Number of channels … : 1
Wed 12-02-2020 19:06:20:: USERINFO: Number of Stokes parameters … : 1

========================================================================
Thu 13-02-2020 10:05:41:: INFO : Processing 1925+211_gsb_band5_chan_field.ms.ap1.image.fits
Thu 13-02-2020 10:05:41:: INFO : PyBDSF version 1.9.2
Thu 13-02-2020 10:05:41:: INFO : Non-default input parameters:
adaptive_rms_box = True
advanced_opts = True
filename = ‘1925+211_gsb_band5_chan_field.ms.ap1.image.fits’
rms_box_bright = (33, 11)
Thu 13-02-2020 10:05:42:: USERINFO: → Opened ‘1925+211_gsb_band5_chan_field.ms.ap1.image.fits’
Thu 13-02-2020 10:05:42:: INFO : Original data shape of 1925+211_gsb_band5_chan_field.ms.ap1.image.fits: (1, 1, 6
144, 6144)
Thu 13-02-2020 10:05:42:: INFO : Final data shape (npol, nchan, x, y): (1, 1, 6144, 6144)
Thu 13-02-2020 10:05:42:: USERINFO: Image size … : (6144, 6144) pixels
Thu 13-02-2020 10:05:42:: USERINFO: Number of channels … : 1
Thu 13-02-2020 10:05:42:: USERINFO: Number of Stokes parameters … : 1
Thu 13-02-2020 10:05:42:: USERINFO: Beam shape (major, minor, pos angle) … : (8.94856e-04, 4.73865e-04, 33.4) deg
rees
Thu 13-02-2020 10:05:42:: INFO : Equinox of image is 2000.000000.
Thu 13-02-2020 10:05:42:: USERINFO: Frequency of image … : 1437.161 MHz
Thu 13-02-2020 10:05:42:: USERINFO: Number of blank pixels … : 0 (0.0%)
Thu 13-02-2020 10:06:05:: INFO : Raw mean (Stokes I) = 0.0011 mJy and raw rms = 0.7588 mJy
Thu 13-02-2020 10:06:05:: INFO : sigma clipped mean (Stokes I) = -0.0005 mJy and sigma clipped rms = 0.1164 mJ
y
Thu 13-02-2020 10:06:05:: USERINFO: Flux from sum of (non-blank) pixels … : 1.374 Jy
Thu 13-02-2020 10:06:05:: INFO : Estimated bmpersrc_th = 1465.60
Thu 13-02-2020 10:06:05:: INFO : Parameter confused is False
Thu 13-02-2020 10:06:05:: USERINFO: → Calculating background rms and mean images
Thu 13-02-2020 10:06:05:: INFO : Estimated size of brightest source (pixels) = 6
Thu 13-02-2020 10:06:05:: USERINFO: → Using adaptive scaling of rms_box
Thu 13-02-2020 10:06:12:: INFO : Maximum extent of largest 10-sigma island using clipped rms (pixels) = 25
Thu 13-02-2020 10:07:11:: USERINFO: Using user-specified rms_box … : (33, 11) pixels (small scale)
Thu 13-02-2020 10:07:11:: USERINFO: Derived rms_box (box size, step size) … : (614, 205) pixels (large scale)
Thu 13-02-2020 10:07:11:: USERINFO: Number of sources using small scale … : 6
Thu 13-02-2020 10:07:11:: INFO : Background rms and mean images computed
Thu 13-02-2020 10:07:11:: USERINFO: → Using 2D map for background rms
Thu 13-02-2020 10:07:12:: USERINFO: → Variation in mean image significant
Thu 13-02-2020 10:07:12:: USERINFO: → Using 2D map for background mean
Thu 13-02-2020 10:07:12:: USERINFO: Min/max values of background rms map … : (9.11e-05, 1.02e-03) Jy/beam
Thu 13-02-2020 10:07:12:: USERINFO: Min/max values of background mean map … : (-3.03e-04, 2.31e-04) Jy/beam
Thu 13-02-2020 10:07:12:: USERINFO: → Expected 5-sigma-clipped false detection rate < fdr_ratio
Thu 13-02-2020 10:07:12:: USERINFO: → Using sigma-clipping (‘hard’) thresholding
Thu 13-02-2020 10:07:12:: USERINFO: Minimum number of pixels per island … : 9
Thu 13-02-2020 10:07:13:: USERINFO: Number of islands found … : 105
Thu 13-02-2020 10:07:14:: USERINFO: Total number of Gaussians fit to image … : 73
Thu 13-02-2020 10:07:14:: USERINFO: Total flux density in model … : 1.402 Jy
Thu 13-02-2020 10:07:14:: USERINFO: → Grouping Gaussians into sources
Thu 13-02-2020 10:07:14:: USERINFO: Number of sources formed from Gaussians : 69
Thu 13-02-2020 10:07:14:: WARNING : All Gaussians were flagged for the following islands:
Island #3 (x=1815, y=2510)
Island #7 (x=2266, y=2893)
Island #8 (x=2772, y=3075)
Island #9 (x=2778, y=2981)
Island #10 (x=2777, y=2993)
Island #11 (x=2785, y=3034)
Island #12 (x=2792, y=3024)
Island #14 (x=2804, y=3156)
Island #17 (x=2836, y=3039)
Island #18 (x=2842, y=3033)
Island #19 (x=2852, y=3009)
Island #20 (x=2852, y=3103)
Island #21 (x=2862, y=3041)
Island #22 (x=2880, y=3209)
Island #23 (x=2892, y=2980)
Island #24 (x=2892, y=3292)
Island #25 (x=2892, y=3156)
Island #27 (x=2902, y=3051)
Island #30 (x=2914, y=3099)
Island #33 (x=2935, y=2828)
Island #35 (x=2943, y=3047)
Island #36 (x=2948, y=3058)
Island #38 (x=2976, y=3010)
Island #46 (x=3032, y=3090)
Island #47 (x=3037, y=2849)
Island #54 (x=3065, y=2817)
Island #57 (x=3071, y=3320)
Island #59 (x=3077, y=3301)
Island #66 (x=3166, y=3131)
Island #69 (x=3211, y=3318)
Island #71 (x=3231, y=3203)
Island #75 (x=3248, y=2830)
Island #76 (x=3249, y=2990)
Island #79 (x=3306, y=2856)
Island #80 (x=3326, y=3273)
Island #92 (x=3632, y=5509)
Island #93 (x=3634, y=1174)
Please check these islands. If they are valid islands and
should be fit, try adjusting the flagging options (use
show_fit with “ch0_flagged=True” to see the flagged Gaussians)
or enabling the wavelet module (with “atrous_do=True”).
To include empty islands in output source catalogs, set
incl_empty=True in the write_catalog task.
Thu 13-02-2020 10:07:14:: INFO : Calculating residual image after subtracting reconstructed gaussians
Thu 13-02-2020 10:07:25:: INFO : Statistics of the Gaussian residual image:
mean: -2.152e-08 (Jy/beam)
std. dev: 1.174e-04 (Jy/beam)
skew: 0.001
kurtosis: 2.788
Thu 13-02-2020 10:07:26:: INFO : Internally derived parameters:
beam : (0.0008948560555776, 0.0004738650057051, 33.408950805
66)
bmpersrc_th : 1465.600372721449
frequency : 1437161376.406
indir : ‘./’
minpix_isl : 9
rms_box : (614, 205)
thresh : ‘hard’

this is typical output of one of the programs i am using for image processing. i want to write few lines from this output into a new file. i have 1000 files of such output. how can i do that. i have recently started using linux and beginner in shell scripting. any help will be appreciated.

nixcraft · February 22, 2020, 11:19am

Have you tried the grep command? For example:

grep 'stuff you need' > output.txt

Say about output coming from a command named foo, try:

foo | grep 'stuff you need' > output.txt

nilkanthagholap · February 23, 2020, 3:38am

yes i have tried grep. but i want to automate the process as there are too many files.

nixcraft · February 24, 2020, 1:21pm

In that case process files using a bash for loop:

for f in /path/to/files/*
do
    grep "stuff you need" "$f" >> "${f}.output"
done

ivandb · February 26, 2020, 10:20am

Maybe you can use “grep -n”. In this way, the string found is written with the file name, something like:

xstrtod.c:3:#include <ctype.h>

where xstrtod.c is the file name and after the “:” you find the line found.

ivandb · February 26, 2020, 10:22am

Additional info:
If you exactly know the structure of what you are searching, consider to use “awk”.
awk is very powerful, but it requires some time to learn it. Nothing impossible, of course.
Anyway, awk works well when you exactly know how the field you are looking for is well structured.