Dtjytyk

I have such a file with multiple blank lines in vscode

$ tail -n 20 draft3.py

            hi = len(a)  



        if lo < 0:

            raise ValueError('low must be non-negative')



        if lo == hi: 

            return None 



        mid = (lo + hi) // 2



        if x == a[mid]:

            return x

        if x > a[mid]:



            lo = mid + 1

            return self.bi_search(a, x, lo, hi)

        if x < a[mid]:



            hi = mid

            return self.bi_search(a, x, lo, hi)

Tried multiple methods to remove the blank lines

`grep -v -e '^$' failed

$ tail -n 20 draft3.py | grep -v -e '^$'

            hi = len(a)  

        if lo < 0:

            raise ValueError('low must be non-negative')

        if lo == hi: 

            return None 

        mid = (lo + hi) // 2



        if x == a[mid]:

            return x

        if x > a[mid]:



            lo = mid + 1

            return self.bi_search(a, x, lo, hi)

        if x < a[mid]:



            hi = mid

            return self.bi_search(a, x, lo, hi)

`grep -Ev "^$" failed

$ tail -n 20 draft3.py | grep -Ev "^$" 

            hi = len(a)  

        if lo < 0:

            raise ValueError('low must be non-negative')

        if lo == hi: 

            return None 

        mid = (lo + hi) // 2



        if x == a[mid]:

            return x

        if x > a[mid]:



            lo = mid + 1

            return self.bi_search(a, x, lo, hi)

        if x < a[mid]:



            hi = mid

            return self.bi_search(a, x, lo, hi)

`sed '/^$/d' failed

$ tail -n 20 draft3.py | sed '/^$/d'

            hi = len(a)  

        if lo < 0:

            raise ValueError('low must be non-negative')

        if lo == hi: 

            return None 

        mid = (lo + hi) // 2



        if x == a[mid]:

            return x

        if x > a[mid]:



            lo = mid + 1

            return self.bi_search(a, x, lo, hi)

        if x < a[mid]:



            hi = mid

            return self.bi_search(a, x, lo, hi)

What's the problem? How could remove the blank lines?

grep -v '^$' will remove empty lines. But what if we have spaces or tabs in some lines ? For example I added 3 spaces to parts of your text, and if we do cat -A we will see that it shows line terminator $, but it will be offset.

$

        mid = (lo + hi) // 2$

   $

        if x == a[mid]:$

            return x$

        if x > a[mid]:$

The second line there has 3 spaces, first one doesn't. So we also want to use [[:blank:]] character class to account for those as well:

$ grep -v '^[[:blank:]]*$' text.txt

           hi = len(a)  

        if lo < 0:

            raise ValueError('low must be non-negative')

        if lo == hi: 

            return None 

        mid = (lo + hi) // 2

        if x == a[mid]:

            return x

        if x > a[mid]:

            lo = mid + 1

            return self.bi_search(a, x, lo, hi)

        if x < a[mid]:

            hi = mid

            return self.bi_search(a, x, lo, hi)

Now you should see that the line with 3 added spaces is gone. The * signifies zero or more repetitions of the characters, so the pattern ^[[:blank:]]*$ also implies ^$ when there are zero whitespace or tab characters on the line. So this pattern handles both truly empty and seemingly empty lines. It also applies exactly the same to grep or sed, because we're using basic regex expressions and [[:blank:]] is one of the POSIX character classes, so it is portable.

We could also do something like this in python but without regex patterns:

$ python3 -c 'import sys; print("n".join([ l.rstrip() for l in sys.stdin if l.strip().split() ]))'  < text.txt

           hi = len(a)

        if lo < 0:

            raise ValueError('low must be non-negative')

        if lo == hi:

            return None

        mid = (lo + hi) // 2

        if x == a[mid]:

            return x

        if x > a[mid]:

            lo = mid + 1

            return self.bi_search(a, x, lo, hi)

        if x < a[mid]:

            hi = mid

            return self.bi_search(a, x, lo, hi)

Why does this work ? Because .split() on a string will split at whitespaces to extract non-whitespace tokens. If a line contains only spaces, the resulting list from .split() will be empty.

As noted by ilkkachu in the comments, the issue can also occur if you use CRLF line endings ( used in DOS/Windows text files). It is easy to see if the file uses CRLF line endings via cat -A, they will be marked as ^M. For example,

$ printf 'hellonrnWorldn   rntestnnnewtestn' | cat -A

hello$

^M$

World$

   ^M$

test$

$

newtest$

One thing that could be done to account for carriage return is this:

$ printf 'hellonrnWorldn   ntestnnnewtestn' | sed '/^[[:blank:]]*r*$/d'

hello

World

test

newtest

It may be simpler to first use a dos2unix utility designed specifically for converting DOS files to Unix files, and then use sed and grep. See ByteCommander's answer that shows example of how to do that.

Presumably you want to remove not only empty lines, but also lines with only whitespace characters. For that, use:

sed '/^s*$/d'   # or respectively

grep -v '^s*$'

The sed expression deletes every line with any number (*) of whitespace characters (s) in it. grep -v outputs any line which does not match the expression.

Example usage

$ sed '/^s*$/d' <draft3.py 

            hi = len(a)  

        if lo < 0:

            raise ValueError('low must be non-negative')

        if lo == hi: 

            return None 

        mid = (lo + hi) // 2

        if x == a[mid]:

            return x

        if x > a[mid]:

            lo = mid + 1

            return self.bi_search(a, x, lo, hi)

        if x < a[mid]:

            hi = mid

            return self.bi_search(a, x, lo, hi)

use

perl -p  -e 's/^[:blank:]*$//g' inputfile | sed '/^$/d' 

and you will have

tail -n 20 draft3.py

            hi = len(a)

        if lo < 0:

            raise ValueError('low must be non-negative')

        if lo == hi:

            return None

        mid = (lo + hi) // 2

        if x == a[mid]:

            return x

        if x > a[mid]:

            lo = mid + 1

            return self.bi_search(a, x, lo, hi)

        if x < a[mid]:

            hi = mid

            return self.bi_search(a, x, lo, hi)

Only with sed is

sed -r 's/^[[:space:]]*$//g' inputfile | sed '/^$/d'