sed '/^$/d' and grep -Ev '^$' failed to remove blank lines Announcing the arrival of Valued...
Statistical model of ligand substitution
I'm thinking of a number
Strange behaviour of Check
Is it possible to ask for a hotel room without minibar/extra services?
If A makes B more likely then B makes A more likely"
Estimate capacitor parameters
Can a monk deflect thrown melee weapons?
Area of a 2D convex hull
Estimated State payment too big --> money back; + 2018 Tax Reform
How to say that you spent the night with someone, you were only sleeping and nothing else?
Two different pronunciation of "понял"
Was credit for the black hole image misattributed?
Did the new image of black hole confirm the general theory of relativity?
How is simplicity better than precision and clarity in prose?
What is the electric potential inside a point charge?
What can I do if my MacBook isn’t charging but already ran out?
Simulating Exploding Dice
Blender game recording at the wrong time
Notation for two qubit composite product state
Problem when applying foreach loop
If I can make up priors, why can't I make up posteriors?
How to stop my camera from exagerrating differences in skin colour?
How do you clear the ApexPages.getMessages() collection in a test?
Can the prologue be the backstory of your main character?
sed '/^$/d' and grep -Ev '^$' failed to remove blank lines
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)Not able to execute a .sh file: /bin/bash^M: bad interpretersed, grep, and awk file sortingFilter lines based on length using sed and grepFilter consecutive identical characters using Sed and GrepHow to escape “.” with sed or grep?sed replace only first and last space in linesUse sed and grep to extract data for particular months in a file with timestampsremoving lines by sedgrep form bottom and display lines after matchSelect using grep and sedgrep and sed only the numbers from a text file's line
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
I have such a file with multiple blank lines in vscode
$ tail -n 20 draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Tried multiple methods to remove the blank lines
`grep -v -e '^$' failed
$ tail -n 20 draft3.py | grep -v -e '^$'
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
`grep -Ev "^$" failed
$ tail -n 20 draft3.py | grep -Ev "^$"
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
`sed '/^$/d' failed
$ tail -n 20 draft3.py | sed '/^$/d'
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
What's the problem? How could remove the blank lines?
command-line text-processing grep sed
edited Mar 23 at 9:06
Sergiy Kolodyazhnyy
75.5k9155329
asked Mar 23 at 8:32
AliceAlice
572111
add a comment |
I have such a file with multiple blank lines in vscode
$ tail -n 20 draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Tried multiple methods to remove the blank lines
`grep -v -e '^$' failed
$ tail -n 20 draft3.py | grep -v -e '^$'
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
`grep -Ev "^$" failed
$ tail -n 20 draft3.py | grep -Ev "^$"
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
`sed '/^$/d' failed
$ tail -n 20 draft3.py | sed '/^$/d'
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
What's the problem? How could remove the blank lines?
command-line text-processing grep sed
edited Mar 23 at 9:06
Sergiy Kolodyazhnyy
75.5k9155329
asked Mar 23 at 8:32
AliceAlice
572111
add a comment |
I have such a file with multiple blank lines in vscode
$ tail -n 20 draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Tried multiple methods to remove the blank lines
`grep -v -e '^$' failed
$ tail -n 20 draft3.py | grep -v -e '^$'
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
`grep -Ev "^$" failed
$ tail -n 20 draft3.py | grep -Ev "^$"
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
`sed '/^$/d' failed
$ tail -n 20 draft3.py | sed '/^$/d'
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
What's the problem? How could remove the blank lines?
command-line text-processing grep sed
edited Mar 23 at 9:06
Sergiy Kolodyazhnyy
75.5k9155329
asked Mar 23 at 8:32
AliceAlice
572111
I have such a file with multiple blank lines in vscode
$ tail -n 20 draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Tried multiple methods to remove the blank lines
`grep -v -e '^$' failed
$ tail -n 20 draft3.py | grep -v -e '^$'
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
`grep -Ev "^$" failed
$ tail -n 20 draft3.py | grep -Ev "^$"
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
`sed '/^$/d' failed
$ tail -n 20 draft3.py | sed '/^$/d'
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
What's the problem? How could remove the blank lines?
command-line text-processing grep sed
command-line text-processing grep sed
edited Mar 23 at 9:06
Sergiy Kolodyazhnyy
75.5k9155329
asked Mar 23 at 8:32
AliceAlice
572111
edited Mar 23 at 9:06
Sergiy Kolodyazhnyy
75.5k9155329
asked Mar 23 at 8:32
AliceAlice
572111
edited Mar 23 at 9:06
Sergiy Kolodyazhnyy
75.5k9155329
edited Mar 23 at 9:06
Sergiy Kolodyazhnyy
75.5k9155329
edited Mar 23 at 9:06
Sergiy Kolodyazhnyy
75.5k9155329
75.5k9155329
asked Mar 23 at 8:32
AliceAlice
572111
asked Mar 23 at 8:32
AliceAlice
572111
asked Mar 23 at 8:32
AliceAlice
572111
572111
add a comment |
add a comment |
3 Answers
3
active
oldest
votes
grep -v '^$' will remove empty lines. But what if we have spaces or tabs in some lines ? For example I added 3 spaces to parts of your text, and if we do cat -A we will see that it shows line terminator $, but it will be offset.
$
mid = (lo + hi) // 2$
$
if x == a[mid]:$
return x$
if x > a[mid]:$
The second line there has 3 spaces, first one doesn't. So we also want to use [[:blank:]] character class to account for those as well:
$ grep -v '^[[:blank:]]*$' text.txt
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Now you should see that the line with 3 added spaces is gone. The * signifies zero or more repetitions of the characters, so the pattern ^[[:blank:]]*$ also implies ^$ when there are zero whitespace or tab characters on the line. So this pattern handles both truly empty and seemingly empty lines. It also applies exactly the same to grep or sed, because we're using basic regex expressions and [[:blank:]] is one of the POSIX character classes, so it is portable.
We could also do something like this in python but without regex patterns:
$ python3 -c 'import sys; print("n".join([ l.rstrip() for l in sys.stdin if l.strip().split() ]))' < text.txt
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Why does this work ? Because .split() on a string will split at whitespaces to extract non-whitespace tokens. If a line contains only spaces, the resulting list from .split() will be empty.
As noted by ilkkachu in the comments, the issue can also occur if you use CRLF line endings ( used in DOS/Windows text files). It is easy to see if the file uses CRLF line endings via cat -A, they will be marked as ^M. For example,
$ printf 'hellonrnWorldn rntestnnnewtestn' | cat -A
hello$
^M$
World$
^M$
test$
$
newtest$
One thing that could be done to account for carriage return is this:
$ printf 'hellonrnWorldn ntestnnnewtestn' | sed '/^[[:blank:]]*r*$/d'
hello
World
test
newtest
It may be simpler to first use a dos2unix utility designed specifically for converting DOS files to Unix files, and then use sed and grep. See ByteCommander's answer that shows example of how to do that.
answered Mar 23 at 8:55
Sergiy KolodyazhnyySergiy Kolodyazhnyy
75.5k9155329
1
"regular" whitespace like spaces could do that, or they might have CRLF line endings
– ilkkachu
Mar 23 at 11:53
@ilkkachu Good point, thank you. I've edited to account for CRLF as well
– Sergiy Kolodyazhnyy
Mar 23 at 19:48
1
oh, I just noticed you used[:blank:]and not[:space:]. The latter would match the CR too.
– ilkkachu
Mar 23 at 20:11
add a comment |
Presumably you want to remove not only empty lines, but also lines with only whitespace characters. For that, use:
sed '/^s*$/d' # or respectively
grep -v '^s*$'
The sed expression deletes every line with any number (*) of whitespace characters (s) in it. grep -v outputs any line which does not match the expression.
Example usage
$ sed '/^s*$/d' <draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
answered Mar 23 at 8:37
dessertdessert
25.5k674108
add a comment |
use
perl -p -e 's/^[:blank:]*$//g' inputfile | sed '/^$/d'
and you will have
tail -n 20 draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Only with sed is
sed -r 's/^[[:space:]]*$//g' inputfile | sed '/^$/d'
answered Mar 26 at 15:52
aborrusoaborruso
21115
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "89"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f1128019%2fsed-d-and-grep-ev-failed-to-remove-blank-lines%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
grep -v '^$' will remove empty lines. But what if we have spaces or tabs in some lines ? For example I added 3 spaces to parts of your text, and if we do cat -A we will see that it shows line terminator $, but it will be offset.
$
mid = (lo + hi) // 2$
$
if x == a[mid]:$
return x$
if x > a[mid]:$
The second line there has 3 spaces, first one doesn't. So we also want to use [[:blank:]] character class to account for those as well:
$ grep -v '^[[:blank:]]*$' text.txt
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Now you should see that the line with 3 added spaces is gone. The * signifies zero or more repetitions of the characters, so the pattern ^[[:blank:]]*$ also implies ^$ when there are zero whitespace or tab characters on the line. So this pattern handles both truly empty and seemingly empty lines. It also applies exactly the same to grep or sed, because we're using basic regex expressions and [[:blank:]] is one of the POSIX character classes, so it is portable.
We could also do something like this in python but without regex patterns:
$ python3 -c 'import sys; print("n".join([ l.rstrip() for l in sys.stdin if l.strip().split() ]))' < text.txt
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Why does this work ? Because .split() on a string will split at whitespaces to extract non-whitespace tokens. If a line contains only spaces, the resulting list from .split() will be empty.
As noted by ilkkachu in the comments, the issue can also occur if you use CRLF line endings ( used in DOS/Windows text files). It is easy to see if the file uses CRLF line endings via cat -A, they will be marked as ^M. For example,
$ printf 'hellonrnWorldn rntestnnnewtestn' | cat -A
hello$
^M$
World$
^M$
test$
$
newtest$
One thing that could be done to account for carriage return is this:
$ printf 'hellonrnWorldn ntestnnnewtestn' | sed '/^[[:blank:]]*r*$/d'
hello
World
test
newtest
It may be simpler to first use a dos2unix utility designed specifically for converting DOS files to Unix files, and then use sed and grep. See ByteCommander's answer that shows example of how to do that.
answered Mar 23 at 8:55
Sergiy KolodyazhnyySergiy Kolodyazhnyy
75.5k9155329
1
"regular" whitespace like spaces could do that, or they might have CRLF line endings
– ilkkachu
Mar 23 at 11:53
@ilkkachu Good point, thank you. I've edited to account for CRLF as well
– Sergiy Kolodyazhnyy
Mar 23 at 19:48
1
oh, I just noticed you used[:blank:]and not[:space:]. The latter would match the CR too.
– ilkkachu
Mar 23 at 20:11
add a comment |
grep -v '^$' will remove empty lines. But what if we have spaces or tabs in some lines ? For example I added 3 spaces to parts of your text, and if we do cat -A we will see that it shows line terminator $, but it will be offset.
$
mid = (lo + hi) // 2$
$
if x == a[mid]:$
return x$
if x > a[mid]:$
The second line there has 3 spaces, first one doesn't. So we also want to use [[:blank:]] character class to account for those as well:
$ grep -v '^[[:blank:]]*$' text.txt
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Now you should see that the line with 3 added spaces is gone. The * signifies zero or more repetitions of the characters, so the pattern ^[[:blank:]]*$ also implies ^$ when there are zero whitespace or tab characters on the line. So this pattern handles both truly empty and seemingly empty lines. It also applies exactly the same to grep or sed, because we're using basic regex expressions and [[:blank:]] is one of the POSIX character classes, so it is portable.
We could also do something like this in python but without regex patterns:
$ python3 -c 'import sys; print("n".join([ l.rstrip() for l in sys.stdin if l.strip().split() ]))' < text.txt
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Why does this work ? Because .split() on a string will split at whitespaces to extract non-whitespace tokens. If a line contains only spaces, the resulting list from .split() will be empty.
As noted by ilkkachu in the comments, the issue can also occur if you use CRLF line endings ( used in DOS/Windows text files). It is easy to see if the file uses CRLF line endings via cat -A, they will be marked as ^M. For example,
$ printf 'hellonrnWorldn rntestnnnewtestn' | cat -A
hello$
^M$
World$
^M$
test$
$
newtest$
One thing that could be done to account for carriage return is this:
$ printf 'hellonrnWorldn ntestnnnewtestn' | sed '/^[[:blank:]]*r*$/d'
hello
World
test
newtest
It may be simpler to first use a dos2unix utility designed specifically for converting DOS files to Unix files, and then use sed and grep. See ByteCommander's answer that shows example of how to do that.
answered Mar 23 at 8:55
Sergiy KolodyazhnyySergiy Kolodyazhnyy
75.5k9155329
1
"regular" whitespace like spaces could do that, or they might have CRLF line endings
– ilkkachu
Mar 23 at 11:53
@ilkkachu Good point, thank you. I've edited to account for CRLF as well
– Sergiy Kolodyazhnyy
Mar 23 at 19:48
1
oh, I just noticed you used[:blank:]and not[:space:]. The latter would match the CR too.
– ilkkachu
Mar 23 at 20:11
add a comment |
grep -v '^$' will remove empty lines. But what if we have spaces or tabs in some lines ? For example I added 3 spaces to parts of your text, and if we do cat -A we will see that it shows line terminator $, but it will be offset.
$
mid = (lo + hi) // 2$
$
if x == a[mid]:$
return x$
if x > a[mid]:$
The second line there has 3 spaces, first one doesn't. So we also want to use [[:blank:]] character class to account for those as well:
$ grep -v '^[[:blank:]]*$' text.txt
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Now you should see that the line with 3 added spaces is gone. The * signifies zero or more repetitions of the characters, so the pattern ^[[:blank:]]*$ also implies ^$ when there are zero whitespace or tab characters on the line. So this pattern handles both truly empty and seemingly empty lines. It also applies exactly the same to grep or sed, because we're using basic regex expressions and [[:blank:]] is one of the POSIX character classes, so it is portable.
We could also do something like this in python but without regex patterns:
$ python3 -c 'import sys; print("n".join([ l.rstrip() for l in sys.stdin if l.strip().split() ]))' < text.txt
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Why does this work ? Because .split() on a string will split at whitespaces to extract non-whitespace tokens. If a line contains only spaces, the resulting list from .split() will be empty.
As noted by ilkkachu in the comments, the issue can also occur if you use CRLF line endings ( used in DOS/Windows text files). It is easy to see if the file uses CRLF line endings via cat -A, they will be marked as ^M. For example,
$ printf 'hellonrnWorldn rntestnnnewtestn' | cat -A
hello$
^M$
World$
^M$
test$
$
newtest$
One thing that could be done to account for carriage return is this:
$ printf 'hellonrnWorldn ntestnnnewtestn' | sed '/^[[:blank:]]*r*$/d'
hello
World
test
newtest
It may be simpler to first use a dos2unix utility designed specifically for converting DOS files to Unix files, and then use sed and grep. See ByteCommander's answer that shows example of how to do that.
answered Mar 23 at 8:55
Sergiy KolodyazhnyySergiy Kolodyazhnyy
75.5k9155329
grep -v '^$' will remove empty lines. But what if we have spaces or tabs in some lines ? For example I added 3 spaces to parts of your text, and if we do cat -A we will see that it shows line terminator $, but it will be offset.
$
mid = (lo + hi) // 2$
$
if x == a[mid]:$
return x$
if x > a[mid]:$
The second line there has 3 spaces, first one doesn't. So we also want to use [[:blank:]] character class to account for those as well:
$ grep -v '^[[:blank:]]*$' text.txt
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Now you should see that the line with 3 added spaces is gone. The * signifies zero or more repetitions of the characters, so the pattern ^[[:blank:]]*$ also implies ^$ when there are zero whitespace or tab characters on the line. So this pattern handles both truly empty and seemingly empty lines. It also applies exactly the same to grep or sed, because we're using basic regex expressions and [[:blank:]] is one of the POSIX character classes, so it is portable.
We could also do something like this in python but without regex patterns:
$ python3 -c 'import sys; print("n".join([ l.rstrip() for l in sys.stdin if l.strip().split() ]))' < text.txt
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Why does this work ? Because .split() on a string will split at whitespaces to extract non-whitespace tokens. If a line contains only spaces, the resulting list from .split() will be empty.
As noted by ilkkachu in the comments, the issue can also occur if you use CRLF line endings ( used in DOS/Windows text files). It is easy to see if the file uses CRLF line endings via cat -A, they will be marked as ^M. For example,
$ printf 'hellonrnWorldn rntestnnnewtestn' | cat -A
hello$
^M$
World$
^M$
test$
$
newtest$
One thing that could be done to account for carriage return is this:
$ printf 'hellonrnWorldn ntestnnnewtestn' | sed '/^[[:blank:]]*r*$/d'
hello
World
test
newtest
It may be simpler to first use a dos2unix utility designed specifically for converting DOS files to Unix files, and then use sed and grep. See ByteCommander's answer that shows example of how to do that.
answered Mar 23 at 8:55
Sergiy KolodyazhnyySergiy Kolodyazhnyy
75.5k9155329
edited Mar 23 at 19:48
answered Mar 23 at 8:55
Sergiy KolodyazhnyySergiy Kolodyazhnyy
75.5k9155329
answered Mar 23 at 8:55
Sergiy KolodyazhnyySergiy Kolodyazhnyy
75.5k9155329
answered Mar 23 at 8:55
Sergiy KolodyazhnyySergiy Kolodyazhnyy
75.5k9155329
75.5k9155329
1
"regular" whitespace like spaces could do that, or they might have CRLF line endings
– ilkkachu
Mar 23 at 11:53
@ilkkachu Good point, thank you. I've edited to account for CRLF as well
– Sergiy Kolodyazhnyy
Mar 23 at 19:48
1
oh, I just noticed you used[:blank:]and not[:space:]. The latter would match the CR too.
– ilkkachu
Mar 23 at 20:11
add a comment |
1
"regular" whitespace like spaces could do that, or they might have CRLF line endings
– ilkkachu
Mar 23 at 11:53
@ilkkachu Good point, thank you. I've edited to account for CRLF as well
– Sergiy Kolodyazhnyy
Mar 23 at 19:48
1
oh, I just noticed you used[:blank:]and not[:space:]. The latter would match the CR too.
– ilkkachu
Mar 23 at 20:11
1
1
"regular" whitespace like spaces could do that, or they might have CRLF line endings
– ilkkachu
Mar 23 at 11:53
"regular" whitespace like spaces could do that, or they might have CRLF line endings
– ilkkachu
Mar 23 at 11:53
@ilkkachu Good point, thank you. I've edited to account for CRLF as well
– Sergiy Kolodyazhnyy
Mar 23 at 19:48
@ilkkachu Good point, thank you. I've edited to account for CRLF as well
– Sergiy Kolodyazhnyy
Mar 23 at 19:48
1
1
oh, I just noticed you used
[:blank:] and not [:space:]. The latter would match the CR too.– ilkkachu
Mar 23 at 20:11
oh, I just noticed you used
[:blank:] and not [:space:]. The latter would match the CR too.– ilkkachu
Mar 23 at 20:11
add a comment |
Presumably you want to remove not only empty lines, but also lines with only whitespace characters. For that, use:
sed '/^s*$/d' # or respectively
grep -v '^s*$'
The sed expression deletes every line with any number (*) of whitespace characters (s) in it. grep -v outputs any line which does not match the expression.
Example usage
$ sed '/^s*$/d' <draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
answered Mar 23 at 8:37
dessertdessert
25.5k674108
add a comment |
Presumably you want to remove not only empty lines, but also lines with only whitespace characters. For that, use:
sed '/^s*$/d' # or respectively
grep -v '^s*$'
The sed expression deletes every line with any number (*) of whitespace characters (s) in it. grep -v outputs any line which does not match the expression.
Example usage
$ sed '/^s*$/d' <draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
answered Mar 23 at 8:37
dessertdessert
25.5k674108
add a comment |
Presumably you want to remove not only empty lines, but also lines with only whitespace characters. For that, use:
sed '/^s*$/d' # or respectively
grep -v '^s*$'
The sed expression deletes every line with any number (*) of whitespace characters (s) in it. grep -v outputs any line which does not match the expression.
Example usage
$ sed '/^s*$/d' <draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
answered Mar 23 at 8:37
dessertdessert
25.5k674108
Presumably you want to remove not only empty lines, but also lines with only whitespace characters. For that, use:
sed '/^s*$/d' # or respectively
grep -v '^s*$'
The sed expression deletes every line with any number (*) of whitespace characters (s) in it. grep -v outputs any line which does not match the expression.
Example usage
$ sed '/^s*$/d' <draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
answered Mar 23 at 8:37
dessertdessert
25.5k674108
edited Mar 30 at 7:04
answered Mar 23 at 8:37
dessertdessert
25.5k674108
answered Mar 23 at 8:37
dessertdessert
25.5k674108
answered Mar 23 at 8:37
dessertdessert
25.5k674108
25.5k674108
add a comment |
add a comment |
use
perl -p -e 's/^[:blank:]*$//g' inputfile | sed '/^$/d'
and you will have
tail -n 20 draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Only with sed is
sed -r 's/^[[:space:]]*$//g' inputfile | sed '/^$/d'
answered Mar 26 at 15:52
aborrusoaborruso
21115
add a comment |
use
perl -p -e 's/^[:blank:]*$//g' inputfile | sed '/^$/d'
and you will have
tail -n 20 draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Only with sed is
sed -r 's/^[[:space:]]*$//g' inputfile | sed '/^$/d'
answered Mar 26 at 15:52
aborrusoaborruso
21115
add a comment |
use
perl -p -e 's/^[:blank:]*$//g' inputfile | sed '/^$/d'
and you will have
tail -n 20 draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Only with sed is
sed -r 's/^[[:space:]]*$//g' inputfile | sed '/^$/d'
answered Mar 26 at 15:52
aborrusoaborruso
21115
use
perl -p -e 's/^[:blank:]*$//g' inputfile | sed '/^$/d'
and you will have
tail -n 20 draft3.py
hi = len(a)
if lo < 0:
raise ValueError('low must be non-negative')
if lo == hi:
return None
mid = (lo + hi) // 2
if x == a[mid]:
return x
if x > a[mid]:
lo = mid + 1
return self.bi_search(a, x, lo, hi)
if x < a[mid]:
hi = mid
return self.bi_search(a, x, lo, hi)
Only with sed is
sed -r 's/^[[:space:]]*$//g' inputfile | sed '/^$/d'
answered Mar 26 at 15:52
aborrusoaborruso
21115
answered Mar 26 at 15:52
aborrusoaborruso
21115
answered Mar 26 at 15:52
aborrusoaborruso
21115
answered Mar 26 at 15:52
aborrusoaborruso
21115
21115
add a comment |
add a comment |
Thanks for contributing an answer to Ask Ubuntu!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f1128019%2fsed-d-and-grep-ev-failed-to-remove-blank-lines%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
