You are here

program gathering info from a large logfile?

17 posts / 0 new
Last post
Travis Carrico
Offline
Last seen: 15 years 4 months ago
Developer
Joined: 2006-10-22 00:30
program gathering info from a large logfile?

i want to be able to read fragmentation percentage out of a log file created by jkdefrag from my autoit script. is this possible and how could it be done. The lines i'm interested are burried in with tons of other lines and look like this:

09:32:55 - Total size of fragmented items: 680427520 bytes, 166120 clusters, 4.8814% of all items, 4.2249% of disk

and if it would have to read line by line would this cause the app to eat up a ton of cpu?

m2
Offline
Last seen: 13 years 2 months ago
Joined: 2006-12-12 12:00
Yes, it is possible. It

Yes, it is possible. It would be best if you chose a language that supports regular expressions (or can use library that does).
No, it won't eat ton of CPU, hdd use will be much more important.
Just a note - you're probably going to reread the file over and over, right? If it is so, always remember the last file position you read and start the next read from this place.

"Those people who think they know everything are a great annoyance to those of us who do." Asimov

wraithdu
Offline
Last seen: 10 years 10 months ago
Developer
Joined: 2007-06-27 20:22
Nope. You should be able to

Nope. You should be able to loop through the log file very quickly and find that line.

$logfile = FileOpen("logfilename", 0) ; 0 = open for read
While 1
     $line = FileReadLine($logfile)
     If StringInStr($line, "Total size of fragmented items:") Then
          ExitLoop
     EndIf
     If @error Then ExitLoop ; end of file
WEnd
FileClose($logfile)
MsgBox(0, "", $line)

That should get you started.

RogerL
Offline
Last seen: 1 month 3 weeks ago
Joined: 2007-03-14 09:17
Shouldn't the @error test

Shouldn't the @error test for end of file immediately follow the FileReadLine instruction (and be test for -1)?

As you have it the @error is as returned by StringInStr (0 = normal operation, 1 = occurence was 0) so the loop will exit on the first line that doesn't have the required string?

$logfile = FileOpen("logfilename", 0) ; 0 = open for read
While 1
     $line = FileReadLine($logfile)
     If @error = -1 Then ExitLoop ; end of file
     If StringInStr($line, "Total size of fragmented items:") Then
          ExitLoop
     EndIf
WEnd
FileClose($logfile)
MsgBox(0, "", $line)
ZachHudock
ZachHudock's picture
Offline
Last seen: 1 year 3 months ago
Developer
Joined: 2006-12-06 18:07
Not quite. I believe the

Not quite. I believe the @error checks for a failed (see: crashed) attempt to read in the file, or to compare the strings. By having it at the end, it will exit the loop if the user reached the end of the file, or if the string comparison crashed. The @error won't be touched if the string is not found, but the EOF hasn't been reached yet.

The developer formerly known as ZGitRDun8705

wraithdu
Offline
Last seen: 10 years 10 months ago
Developer
Joined: 2007-06-27 20:22
Nope nope, RogerL's correct.

Nope nope, RogerL's correct. @error is reset each time a new function is called. Thanks for catching that (I make that mistake too often cause I forget to look for it).

ZachHudock
ZachHudock's picture
Offline
Last seen: 1 year 3 months ago
Developer
Joined: 2006-12-06 18:07
His logic was still

My understanding of the true functionality was wrong, but his logic was still incorrect. He said that the script should die on the first line that the needed string isn't located in, which is not the case. It continues to read UNTIL it finds that string, or reaches the EOF

The developer formerly known as ZGitRDun8705

wraithdu
Offline
Last seen: 10 years 10 months ago
Developer
Joined: 2007-06-27 20:22
He was still correct I

He was still correct I believe. If StringInStr() doesn't find the substring, it sets the error flag. So my loop would exit after reading the first line and not finding the substring.

ZachHudock
ZachHudock's picture
Offline
Last seen: 1 year 3 months ago
Developer
Joined: 2006-12-06 18:07
I wasn't sure because Travis

I wasn't sure because Travis updated his JKDefrag package to include this, and it worked correctly. The substring is somewhere around line 15 in the file, so if Roger were correct, it would never make it past line 1....unless Travis came up with his own code for it.

The developer formerly known as ZGitRDun8705

RogerL
Offline
Last seen: 1 month 3 weeks ago
Joined: 2007-03-14 09:17
To wraithdu and Zach (and

To wraithdu and Zach (and Travis)

My first point was asking why wraithdu had a test for EOF after checking for the string. It is usual to test for EOF immediately after FileReadLine because it isn't logical to test the line to contain something if you haven't got a line of text (and of course to ensure you don't get stuck in a loop for ever if the string your searching for isn't there)

My next point was based on my interpretation of AutoIt's help for StringInStr. It says "@Error 0 - Normal operation, 1 - Occurrence was 0". On this basis as the first line doesn't contain the text, I assumed @error would contain 1 as there was no occurence and the loop would exit. However as I am continually surprised at the lack of detail and correctness in AutoIt's help on returned values I should have checked. I have now done so and @error contains 0 for both conditions and therefore Travis's code 'works'.

However the expression 'works' is too simplistic. One should not only code for normal conditions but should consider all the error conditions that might occur and handle them appropriately.

Because Travis opens a file he KNOWS is there, made by JkDefrag, which WILL have the string he is looking for there is a case for not bothering to cater for any EOF or any error condition. Just get lines until string is found then handle it parsing the line for the numbers he wants. I don't think that is good practice tho'. Best to getline, exit loop if any problem (telling user if appropriate),
if string found handle parsing the line BEFORE exiting the loop. That way he hasn't got any error condition with an indeterminate $line contents merging with the code parsing the 'good' line.

Hope I've made sense. I am a much better coder than English prose writer!

rab040ma
Offline
Last seen: 1 week 1 day ago
Joined: 2007-08-27 13:35
Are you just doing it for

Are you just doing it for your own use, or do you want it to be part of a Portable App you want to distribute?

Microsoft has a free utility that is designed for parsing log files.

But you could also use the Gnu grep utility. Depending on what you are doing, grep might be faster than looping in AutoIT. You could even use other Gnu tools to parse the resulting lines to display just the percentage.

MC

wraithdu
Offline
Last seen: 10 years 10 months ago
Developer
Joined: 2007-06-27 20:22
I figured out why it's not

I figured out why it's not exiting. When StringInStr() fails the error flag is set to 0 (which is its default state I guess)...which basically means no error, so the loop continues. I guess for an 'If @error' condition to be true, it must be a non-zero error value.

I think this is because @error returns 0 under normal operation. Like if your only line is

MsgBox(0, "", @error)

then you get '0'.

I'm not sure if I like StringInStr() returning a 0 error if it fails like that...

So both our scripts work, but I'd say RogerL's is *more* correct Wink

rab040ma
Offline
Last seen: 1 week 1 day ago
Joined: 2007-08-27 13:35
The docs say @error is only

The docs say @error is only set (to 1), after StringInStr, if there is a syntax error.

If you ask StringInStr where in one string another occurs, it returns 0 to say "nowhere". That's not an error condition, it's simply answering your question. It's like answering a numeric question with "0" meaning "none". So you need to test the return value (for something > 0), rather than @error.

If the only way to test for EOF is with @error, I suppose that's okay, but I prefer my programming languages to reserve error trapping for real errors and signal normal conditions (like reaching the end of the file) with something like a flag.

MC

wraithdu
Offline
Last seen: 10 years 10 months ago
Developer
Joined: 2007-06-27 20:22
Gotcha. And yes, EOF ==>

Gotcha.

And yes, EOF ==> @error = -1

Travis Carrico
Offline
Last seen: 15 years 4 months ago
Developer
Joined: 2006-10-22 00:30
Thanks

Thanks a lot guys. This is now implemented in JkDefrag Portable Dev Test 6!

RogerL
Offline
Last seen: 1 month 3 weeks ago
Joined: 2007-03-14 09:17
Hi Travis,See my post to

Hi Travis,

See my post to wraithdu, Zach and you above.

I have had a look at your code:

If FileExists("JkDefrag.log") = 1 Then
    $logfile = FileOpen("JkDefrag.log", 0)
    Do
        $line = FileReadLine($logfile)
        $i = $i + 1
        While 1
            $line = FileReadLine($logfile)
            If @error Then ExitLoop
            If StringInStr($line, "Total size of fragmented items:") Then
                 ExitLoop
            EndIf
        WEnd
        $line = StringTrimRight( $line, 8)
        ;code contines to end of loop parsing line
     Until.....

You might like to consider removing the first FileReadLine as I don't think it achieves anything, and move the code parsing the line with the fragemented items text to where the string is found as explained in my previous post

If FileExists("JkDefrag.log") = 1 Then
    $logfile = FileOpen("JkDefrag.log", 0)
    Do
        $i = $i + 1
        While 1
            $line = FileReadLine($logfile)
            If @error Then ExitLoop ; no line got: EOF (-1) or read error (1)
            If StringInStr($line, "Total size of fragmented items:") Then
                $line = StringTrimRight( $line, 8)
                ;code contines to end of loop parsing line
            EndIf
        WEnd
    Until..... 

You also might like to consider the neatness of a for loop over a do loop:

For $i = 0 To $max
    code
    body
Next
$i = 0
unrelated
code
Do
    $i = $i + 1
    code
    body
Until $i = $max

The initialisation of $i should at least be right above the Do statement.

I think do loops should only be used when you have to execute the body once to determine the end condition.

oh, and just in case you didn't know

$i += 1

is the same as

$i = $i + 1

Hope my comments are taken in the spirit they are meant. Don't want to appear overly critical or force my style of coding on anybody. Oh, and thanks for doing this JkDefrag GUI, I'm learnig AutoIt's GUI code too and your example is helping me - thanks.

Travis Carrico
Offline
Last seen: 15 years 4 months ago
Developer
Joined: 2006-10-22 00:30
thanks! i'll try to tidy up

thanks! i'll try to tidy up my code with those things for the next release.

Log in or register to post comments