QLCommander v2.2.x

[Andrew]

If it is a stack limit, you might try testing with a drive with only a few directories and see how that works. Test a minimal case to confirm that it works in that instance. Then expend with more directories until it fails. The issue could also be the depth of the directories. You could test with a drive with, say 4 directories that have only a single depth. Then expand a bit until failure to see how far you can go.

"data space can also run out if routines keep calling one another without RETuning." Maybe set data space to some large amount just to confirm if it still fails (again this is only for testing purposes).

The routine doesn't fail.
Using Turbo: I set the dataspace to 500k and stack to 9990 (the largest possible value). The routine correctly calculates the size of my entire drive (about 128 Mb of data) and it can be called twice in a row.
Then it crashes.
For win1_Games_ it can be executed 22 times in a row.
It looks like the compiled routine does not clear the used dataspace and/or stack when it ends, and each call eats away the available space.

Using Liberator: With heap size=2048 (default value) and stack=10k, the routine can be called any number of times for any directory, including full drive.
When it is QLiberated it doesn't leak heap, dataspace, stack. It behaves as expected.

From SuperBasic: the routine can be called any number of times for any directory, including full drive.

This looks to me to be a Turbo compiler issue. Maybe I am setting some compile parameters wrong, I have no clue ..

[dilwyn]

Compiled with Turbo the program iterates 5-7 times, then crashes with "Increase Dataspace". It makes no difference if I increase the dataspace to 200k
Compiled with QLiberator, it works!

Maybe Dilwyn can help, in Q-Trans he also has to read the directories recursively.

IIRC, I had to be very mindful of both stack and dataspace with my recursive routine - to allow for reasonable depths of recursion (my QXL.WIN called DIL.WIN is about 1GB in size on my QPC2, so a very large number of files, though it seems never more than 4 directory levels deep) Q-Trans had to have a stack of 32KB or so to deal with it. But that was in QLib, not Turbo.
On my hard drive, it called itself 784 times, recursing a max of 4 deep, running in SBASIC.

I only had a quick look at Andrew's code, couldn't figure out what was going on just from the short look, sorry. Recursive code can be awkward to debug - you may well get the impression from error messages that the problem is somewhere other than it really is.

And if any extensions have stack issues (i.e. leave unbalanced stack like VER$ can do in SuperBASIC) that can throw the result too. When I compiled it, the "adjust dataspace" message always seemed to come at 8700 b$=MDHEADR$(newname$), which may be just coincidence? I haven't tried looking in Per's code to see if there's any significant memory allocation going on within that routine.

I added a FREE_MEMORY line (note: Turbo's FREE_MEMORY, not FREE_MEM) in the procedure to watch the free memory of the task leak and reduce substantially when compiled, but stayed relatively constant in interpreted SBASIC. It was almost as though the MDHEADR$ wasn't releasing memory after each call when compiled or something like that. When another random string function was used (just for testing), the memory loss didn't occur in the same way, it was more akin to running interpreted. I must emphasise I haven't had time to look at Per's source assembler to look at what's going on in there, whether it allocates any memory to handle file headers etc which might not be getting released under Turbo for some reason).

I emphasise, before anyone jumps to the conclusion it's the MDHEADR$ extension is at fault, it probably only LOOKS that way because of the test code I put around it, which seemed to indicate a loss of between 600 bytes and 15K per call depending on the size of the directory. Hopefully, Andrew can put a similar test code around key points in the routine to try to isolate where the memory "leak" is happening - it's just some quick and dirty test I did just now.

That's as far as I've been able to get in the time I had this morning!

[pjw]

Andrew, here is part of a program that recursively descends a directory tree (counting files and adding sizes along the way). I compiled my program with Turbo and ran the same instance repeatedly across all my hard disks. The memory size never grew and it never crashed. It was just a rough test, so I set the data space to 10k without trying to optimise..
[

Code: Select all

184 rem + ------------------------------------------------------------------------ +
186 rem |<                            Count Directory                             >|
188 rem + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +
190 rem |                                Tree size                                 |
192 rem |                                                                          |
194 rem | Go through a directory sub tree recursively counting all files and       |
196 rem | directories encountered and add up their total size (including headers!) |
198 rem |                                                                          |
200 rem | GLOBal drc, fil, siz                                                     |
202 rem + ------------------------------------------------------------------------ +
204 rem | V0.01, pjw, 2014 May 18                                                  |
206 rem | V0.04, pjw, 2018 Nov 18, simplified and generalised version              |
208 rem | V0.04, pjw, 2025 Mar 09, GET#cd\pos+14 -> SET_POSITION pos+14:GET$(#cd)  |
210 rem + ------------------------------------------------------------------------ +
212 :
214 DEFine FuNction CountDir(fdv$, dir$)
216 LOCal cd, lp, er, fl, ps, l, ty%, nm$(36)
218 :
220 cd = FOP_DIR(fdv$ & dir$): IF cd < 0: RETurn cd
222 ps = 0: er = 0: fl = FLEN(#cd)
224 REPeat lp
226  IF ps >= fl: EXIT lp
228  SET_POSITION#cd; ps + 14: nm$ = GET$(#cd): rem Turbo doesnt get GET!
230  IF LEN(nm$) > 0 THEN
232   BGET#cd\ ps + 5, ty%
234   IF ty% = 255 THEN
236    drc = drc + 1
238    er = CountDir(fdv$, nm$): IF er < 0: EXIT lp
240   ELSE
242    lget#cd\ ps + 0, l: siz = siz + l
244    fil = fil + 1
246   END IF
248  END IF
250  ps = ps + 64
252 END REPeat lp
254 CLOSE#cd
256 RETurn er
258 END DEFine CountDir

My guess is there is some structural problem with your routine. Check your LOCals and parameters!

Interestingly: Turbo doesnt get TK2's GET on strings..

[pjw]

Heres the full utility:

DirSize.zip

(2.92 KiB) Downloaded 12 times

Sorry Andrew, I didnt see your original posting on the topic. Will take a look at your routine in my break.

[martyn_hill]

Hi Andrei and friends...

Like Dilwyn, I also tested the free memory - but with TK2's FREE_MEM - and I note that when interpreted, testsize_bas churns through available memory when b$=MDHEADR$ is used, but does not when replaced with a static assignment to b$.

Obviously, under the interpreter (and I guess, under QLib based on Andrei's earlier finding), that memory leak is normally unnoticed, but with Turbo's aim to maintain memory allocations within its own task-space, the task soon complains...

I've been scouring the source code for Per's MDHEADR$, and can't yet find anything un-toward - not unsurprisingly, knowing the typically brilliant quality of Per's coding - but something is definitely soaking-up memory in that routine, or else somewhere in the ut_stos SMSQe code or IO_OPEN/FSHDR/CLOSE calls that MDHEADR$ makes.

[pjw]

Uh-oh! I see there is a bug in MDHEADR$! that would explain the problem you were having! Awfully sorry for the time waste!
This toolkit was written over thirty years a go, and there was a changeover from level 1 to level 2 drivers at some point. However, I think its more likely the bug snuck in with my makeover in 2021. I usually stress test each command before I release a toolkit. Cant think how I missed this one.

I expect to release a fixed version later today on Knoware.no. Will let you know when its done.

[martyn_hill]

Hiya Per!

I'm sure you've made Andrei a very happy fellow

Cant think how I missed this one.

Good grief, man, give yourself a break! Given the sheer volume of truly useful stuff you've made available to us over the years, I think a little slip is acceptable, now and again!

Thanks again for your great library of stuff and for taking the time and care to maintain it for us!

[Andrew]

Uh-oh! I see there is a bug in MDHEADR$! that would explain the problem you were having! Awfully sorry for the time waste!
This toolkit was written over thirty years a go, and there was a changeover from level 1 to level 2 drivers at some point. However, I think its more likely the bug snuck in with my makeover in 2021. I usually stress test each command before I release a toolkit. Cant think how I missed this one.

I expect to release a fixed version later today on Knoware.no. Will let you know when its done.

Thank you very much, Per! You made my day!
I guess this is the first time someone uses this toolkit and stresses it to the max!
Thank you for building this toolkit!

[dilwyn]

Indeed, Martyn.

Glad I was scouting around the right point.

Thanks to Per for stepping up quickly to resolve this.

[tofro]

CRY FOR HELP
I narrowed down the problem to a single procedure, CalcDirSize, but I have no idea how to fix it. Everything works in SuperBasic, but crashes spectacularly when compiled with Turbo.
I wrote a small test program testsize_bas, so the problem is definitely the CalcDirSize procedure.
Compiled with Turbo the program iterates 5-7 times, then crashes with "Increase Dataspace". It makes no difference if I increase the dataspace to 200k
Compiled with QLiberator, it works!

I attached the TestSize_bas. Just change line 3200 text$="Win1_Games_" to a large directory from your system. One with with lots of subdirs - my Games dir has 41 subdirs, and some of them are 2-3 levels deep, with more tha 400 files in total.
The program uses Turbo Toolkit and also needs Per's MDInfo toolkit. It can be downloaded from: https://www.knoware.no/htm/toolkits.htm
Maybe someone can figure it out how to compile this with Turbo. I am stumped!

What I did first, was introduce a call counter in your code (i.e. count recursive calls of CalcDirSize). With your original code and your set data space of 96k, it managed to reach like 380 calls before running out of data space.
Then I made the following changes to your code:

Code: Select all

REMark Line 8700 removed (b$ = MDHEADR$(newname$)
REMark Line 8800 removed (x = LONGINTEGER (b$(1 TO 4)))
8810 x = FLEN (\fname$) : REMark added, should do basically the same as the above

(I think that should do roughly the same as your code)
And the code could happily do 900 calls and more with the same data space size set as above (and ran through even my longest directory trees with no problem whatsoever).

So.....
The problem is either with Per's extension (I didn't really check that) or with Turbo's ability to handle string arrays returned from machine code functions in a recursive setup (I'd suspect the latter EDIT: and turns out I was both wrong and late, apparently ).