...making Linux just a little more fun!

### Talkback:164/lg_tips.html#1

[ In reference to "2-Cent Tips" in LG#164 ]

Paul Sephton [paul at inet.co.za]

Fri, 03 Jul 2009 23:14:47 +0200

@$# This was originally entitled: "about 2c ext2 fragmentation" - Kat $#@

On Fri, 2009-07-03 at 22:46 +0200, Carlos Baiget wrote:

    > another 0.5c: If any file in the given directory has a space in his
    > name, the defragmentation script will not work. To replace all spaces
    > inside the file name, i suggest to do a
    >
    >  rename 's/\ /_/g' *
    >
    > previously or modify IFS environment variable to not consider space as
    > field separator.
    >
    > Carlos

Ah, yes; or simply quote the file names as follows:

        fs=`echo "$line" | cut -f 1 -d':'`
        fn=`echo "$line" | cut -f 2 -d':'`
    # copy the file up to 10 times, preserving permissions
        j=0;
        while [ -f "$fn" -a $j -lt 10 ]; do
        ....

IFS also makes sense. Thanks for highlighting this rather serious error. I really should have taken the trouble to test this properly first!

Paul

Top Back

Paul Sephton [paul at inet.co.za]

Fri, 03 Jul 2009 23:17:55 +0200

Aaaahhhh! Top posted! Too late, she cried!

[[[ It's okay, Paul - for you, I manually corrected it. Thanks for noticing and caring. -- Kat ]]]

Top Back

Thomas Adam [thomas.adam22 at gmail.com]

Fri, 3 Jul 2009 22:24:27 +0100

2009/7/3 Paul Sephton <paul@inet.co.za>:

    > Ah, yes;  or simply quote the file names as follows:
    >
    >    fs=`echo "$line" | cut -f 1 -d':'`
    >    fn=`echo "$line" | cut -f 2 -d':'`
    > # copy the file up to 10 times, preserving permissions
    >    j=0;
    >    while [ -f "$fn" -a $j -lt 10 ]; do
    >    ....
    >
    > IFS also makes sense.  Thanks for highlighting this rather serious error.  I
    > really should have taken the trouble to test this properly first!
    >
    > Paul
    >
    > On Fri, 2009-07-03 at 22:46 +0200, Carlos Baiget wrote:
    >> another 0.5c: If any file in the given directory has a space in his
    >> name, the defragmentation script will not work. To replace all spaces
    >> inside the file name, i suggest to do a
    >>
    >>  rename 's/\ /_/g' *

Err, be careful here -- rename as a command is not the same across distributions. On Debian and I assume it's derivatives, it is indeed the classic perl script. On Redhat, it's something completely different. In which case "mmv" and friends can be used.

-- Thomas Adam

Top Back

Paul Sephton [paul at inet.co.za]

Fri, 03 Jul 2009 23:52:26 +0200

On Fri, 2009-07-03 at 22:24 +0100, Thomas Adam wrote:

    > 2009/7/3 Paul Sephton <paul@inet.co.za>:
    > > On Fri, 2009-07-03 at 22:46 +0200, Carlos Baiget wrote:
    > >> another 0.5c: If any file in the given directory has a space in his
    > >> name, the defragmentation script will not work. To replace all spaces
    > >> inside the file name, i suggest to do a
    > >>
    > >>  rename 's/\ /_/g' *
    >
    > Err, be careful here -- rename as a command is not the same across
    > distributions.  On Debian and I assume it's derivatives, it is indeed
    > the classic perl script.  On Redhat, it's something completely
    > different.  In which case "mmv" and friends can be used.
    >
    > -- Thomas Adam

Good point. Anyway, the other problem with the script is that it needs root privilege to run '/sbin/filefrag'. The script is not really safe either as pointed out by Carlos. I have tested the following against a directory of files that contain spaces. On the other hand, the fact that 'filefrag' can't be run as a normal user rather limits it's use. chmod +s /sbin/filefrag is an option.

Can you spot any more loopholes?

    #!/bin/sh
    # Retrieve a list for fragmented files, #fragments:filename
    FILEFRAG="/sbin/filefrag"
    if [ ! -f $FILEFRAG ]; then
      echo requires $FILEFRAG to defrag this directory
      exit 0
    fi

    flist() {
      for i in *; do
        if [ -f "$i" ]; then
          ff=`$FILEFRAG "$i"`
          fn=`echo "$ff" | cut -f1 -d':'`
          fs=`echo "$ff" | cut -f2 -d':' | cut -f2 -d' '`
          if [ -f "$fn" -a $fs -gt 1 ]; then echo -e "$fs:$fn"; fi
        fi
      done
    }
    IFS='
    '

    # Sort the list numeric, descending
    flist | sort -n -r |
    (
    # for each file
      while read line; do
        fs=`echo "$line" | cut -f 1 -d':'`
        fn=`echo "$line" | cut -f 2 -d':'`
    # copy the file up to 10 times, preserving permissions
        j=0;
        while [ -f "$fn" -a $j -lt 10 ]; do
          j=$[ $j + 1 ]

          TMP=$$.tmp.$j
          if ! cp -p "$fn" "$TMP"; then
            echo copy failed [$fn]
            j=10
          else
    # test the new temp file's fragmentation, and if less than the
    # original, move the temp file over the original
            ns=`$FILEFRAG $TMP | cut -f2 -d':' | cut -f2 -d' '`
            if [ $ns -lt $fs ]; then
              mv "$TMP" "$fn"
              fs=$ns
              if [ $ns -lt 2 ]; then j=10; fi
            fi
          fi
        done
        j=0;
    # clean up temporary files
        while [ $j -lt 10 ]; do
          j=$[ $j + 1 ]

          TMP=$$.tmp.$j
          if [ -f "$TMP" ]; then
            rm "$TMP"
          else
            j=10
          fi
        done
      done
    )
    # report fragmentation
    for i in *; do if [ -f $i ]; then $FILEFRAG $i; fi; done

Top Back

Thomas Adam [thomas.adam22 at gmail.com]

Fri, 3 Jul 2009 22:54:40 +0100

2009/7/3 Paul Sephton <paul@inet.co.za>:

    > Can you spot any more loopholes?

I scanned it very quickly, you, like everyone else, needs to "USE" ""MORE"" """""QUOTES""""".

I can't stress enough just how important that is.

-- Thomas Adam

Top Back

Paul Sephton [paul at inet.co.za]

Sat, 04 Jul 2009 00:02:26 +0200

On Fri, 2009-07-03 at 22:54 +0100, Thomas Adam wrote:

    > 2009/7/3 Paul Sephton <paul@inet.co.za>:
    > > Can you spot any more loopholes?
    >
    > I scanned it very quickly, you, like everyone else, needs to "USE"
    > ""MORE"" """""QUOTES""""".
    >
    > I can't stress enough just how important that is.
    >
    > -- Thomas Adam

... as in the last line,

       for i in *; do if [ -f $i ]; then $FILEFRAG $i; fi; done
    should be
       for i in *; do if [ -f "$i" ]; then $FILEFRAG "$i"; fi; done

oh well....

Top Back

Paul Sephton [paul at inet.co.za]

Sat, 04 Jul 2009 00:42:41 +0200

On Fri, 2009-07-03 at 22:54 +0100, Thomas Adam wrote:

    > 2009/7/3 Paul Sephton <paul@inet.co.za>:
    > > Can you spot any more loopholes?
    >
    > I scanned it very quickly, you, like everyone else, needs to "USE"
    > ""MORE"" """""QUOTES""""".
    > I can't stress enough just how important that is.
    > -- Thomas Adam

Ok, here's the latest. Gives a bit more feedback about what is happening in a more readable way. Also fixes a bug that left temp files lying around. I'm running out of still fragmented directories/files to test it on.

    #!/bin/sh
    # Retrieve a list for fragmented files, #fragments:filename
    FILEFRAG="/sbin/filefrag"
    if [ ! -f $FILEFRAG ]; then
      echo requires $FILEFRAG to defrag this directory
      exit 0
    fi

    flist() {
      for i in *; do
        if [ -f "$i" ]; then
          ff=`$FILEFRAG "$i"`
          fn=`echo "$ff" | cut -f1 -d':'`
          fs=`echo "$ff" | cut -f2 -d':' | cut -f2 -d' '`
          if [ -f "$fn" -a $fs -gt 1 ]; then echo -e "$fs:$fn"; fi
        fi
      done
    }

    # Internal Field Separator is now a newline
    IFS='
    '

    # Sort the list numeric, descending
    echo Reading files...
    flist | sort -n -r |
    (
    # for each file
      while read line; do
        fs=`echo "$line" | cut -f 1 -d':'`
        fn=`echo "$line" | cut -f 2 -d':'`
    # copy the file up to 10 times, preserving permissions
        j=0;
        echo -e -n "$fn\t$fs"
        while [ -f "$fn" -a $j -lt 10 ]; do
          j=$[ $j + 1 ]
          echo -n "."
          TMP=$$.tmp.$j
          if ! cp -p "$fn" "$TMP"; then
            echo copy failed [$fn]
            j=10
          else
    # test the new temp file's fragmentation, and if less than the
    # original, move the temp file over the original
            ns=`$FILEFRAG $TMP | cut -f2 -d':' | cut -f2 -d' '`
            if [ $ns -lt $fs ]; then
              mv "$TMP" "$fn"
              fs=$ns
        	  echo -n "$fs"
              if [ $ns -lt 2 ]; then j=10; fi
            fi
          fi
        done
        echo
        echo "    - file now has $fs fragments"
        j=0;
    # clean up temporary files
        while [ $j -lt 10 ]; do
          j=$[ $j + 1 ]

          TMP=$$.tmp.$j
          if [ -f "$TMP" ]; then
            rm "$TMP"
          fi
        done
      done
    )

Top Back

Thomas Adam [thomas.adam22 at gmail.com]

Fri, 3 Jul 2009 23:50:24 +0100

2009/7/3 Paul Sephton <paul@inet.co.za>:

    > if [ ! -f $FILEFRAG ]; then

    [ -x "$FILEFRAG" ]

Just on the off-chance the file isn't executable for some odd reason.

    >  echo requires $FILEFRAG to defrag this directory

    echo "requires $FILEFRAG to defrag this directory"

    >  exit 0
    > fi
    >
    > flist() {
    >  for i in *; do
    >    if [ -f "$i" ]; then
    >      ff=`$FILEFRAG "$i"`

To be honest, only the most ancient of shells, and I mean ancient, won't understand:

    $()

For command substitution, over backticks. Given nesting backticks is problematic, you should just use "$()".

    >      fn=`echo "$ff" | cut -f1 -d':'`
    >      fs=`echo "$ff" | cut -f2 -d':' | cut -f2 -d' '`
    >      if [ -f "$fn" -a $fs -gt 1 ]; then echo -e "$fs:$fn"; fi

Odd -- you claim "/bin/sh", yet "echo -e" isn't POSIX -- and why is it even invoked as such here, when you're not asking for backslash interpretation?

    >    echo -e -n "$fn\t$fs"

printf()

    >      j=$[ $j + 1 ]

((J++)) -- the square-brackets are an arcane syntax.

-- Thomas Adam

Top Back

Paul Sephton [paul at inet.co.za]

Sat, 04 Jul 2009 01:12:14 +0200

On Fri, 2009-07-03 at 23:50 +0100, Thomas Adam wrote:

    > 2009/7/3 Paul Sephton <paul@inet.co.za>:
    > > if [ ! -f $FILEFRAG ]; then
    >
    > ```
    > [ -x "$FILEFRAG" ]
    > '''
    >
    > Just on the off-chance the file isn't executable for some odd reason.

Cool. I had changed it to -x elsewhere, but the real problem is if the script is run as an ordinary user, the execution fails because of a kernel access restriction. I was thinking of checking

      -x $FILEFRAG -a $USER=root or
      -x $FILEFRAG -a -s $FILEFRAG.  Might be better.

    > >  echo requires $FILEFRAG to defrag this directory
    >
    > ```
    > echo "requires $FILEFRAG to defrag this directory"
    >
    > >  exit 0
    > > fi
    > >
    > > flist() {
    > >  for i in *; do
    > >    if [ -f "$i" ]; then
    > >      ff=`$FILEFRAG "$i"`
    >
    > To be honest, only the most ancient of shells, and I mean ancient,
    > won't understand:
    >
    > ```
    > $()
    > '''

... you can tell I'm rather ancient, can't you? I have never used $() before, other than in makefiles...

    > For command substitution, over backticks.  Given nesting backticks is
    > problematic, you should just use "$()".
    >
    > >      fn=`echo "$ff" | cut -f1 -d':'`
    > >      fs=`echo "$ff" | cut -f2 -d':' | cut -f2 -d' '`
    > >      if [ -f "$fn" -a $fs -gt 1 ]; then echo -e "$fs:$fn"; fi
    >
    > Odd -- you claim "/bin/sh", yet "echo -e" isn't POSIX -- and why is it
    > even invoked as such here, when you're not asking for backslash
    > interpretation?
    >
    > >    echo -e -n "$fn\t$fs"
    >
    > printf()

printf() is POSIX? I didn't know. Agreed, echo -e is unnecessary in the first instance. I'll just make it #bin/bash rather, or as you suggest, try printf() instead.

    >
    > >      j=$[ $j + 1 ]
    >
    > ((J++)) -- the square-brackets are an arcane syntax.

I'm not just ancient, but a bit arcane too

    > -- Thomas Adam

Thanks, Thomas. Your help is appreciated.

Top Back

Paul Sephton [paul at inet.co.za]

Sat, 04 Jul 2009 10:38:59 +0200

On Fri, 2009-07-03 at 23:50 +0100, Thomas Adam wrote:

    > 2009/7/3 Paul Sephton <paul@inet.co.za>:
    > > if [ ! -f $FILEFRAG ]; then
    >
    > ```
    > [ -x "$FILEFRAG" ]
    > '''
    >
    > Just on the off-chance the file isn't executable for some odd reason.

<snip> Ok, this actually reads and works a whole lot better. Thanks for the tips, Thomas:

    #!/bin/sh
    # Retrieve a list for fragmented files, #fragments:filename
    FILEFRAG="/sbin/filefrag"
    if [ ! -x $FILEFRAG ]; then
      echo requires $FILEFRAG to defrag this directory
      exit 0
    fi
    if [ ! "$USER" = "root" ]; then
      if [ ! -u $FILEFRAG ]; then
        echo $FILEFRAG can only be run as root, or should be SUID root
        exit 0
      fi
    fi

    flist() {
      for i in *; do
        if [ -f "$i" ]; then
          ff=$( $FILEFRAG "$i" )
          fn=$( echo "$ff" | cut -f1 -d':' )
          fs=$( echo "$ff" | cut -f2 -d':' | cut -f2 -d' ' )
          if [ -f "$fn" -a $fs -gt 1 ]; then printf "$fs:$fn\n"; fi
        fi
      done
    }

    # Internal Field Separator is now a newline
    IFS='
    '

    # Sort the list numeric, descending
    echo Reading files...
    flist | sort -n -r |
    (
    # for each file
      while read line; do
        fs=$( echo "$line" | cut -f 1 -d':' )
        fn=$( echo "$line" | cut -f 2 -d':' )
    # copy the file up to 10 times, preserving permissions
        j=0
        printf "$fn\t$fs"
        while [ -f "$fn" -a $j -lt 10 ]; do
          ((j++))
          printf "."
          TMP=$$.tmp.$j
          if [ -f "$TMP" ]; then
            echo "Error! target temp file already exists!"
          elif ! cp -p "$fn" "$TMP"; then
            echo copy failed [$fn]
            j=10
          else
    # test the new temp file's fragmentation, and if less than the
    # original, move the temp file over the original
            ns=$( $FILEFRAG "$TMP" | cut -f2 -d':' | cut -f2 -d' ' )
            if [ $ns -lt $fs ]; then
              mv "$TMP" "$fn"
              fs=$ns
        	  printf "$fs"
              if [ $ns -lt 2 ]; then j=10; fi
            fi
          fi
        done
        echo
        echo "    (file now has $fs fragments)"
        j=0
    # clean up temporary files
        while [ $j -lt 10 ]; do
          ((j++))

          TMP=$$.tmp.$j
          if [ -f "$TMP" ]; then
            rm "$TMP"
          fi
        done
      done
    )

Top Back