Page 2 of 8
Re: MicroPicoDrive Popopo's version
Posted: Wed Aug 07, 2024 9:02 am
by Popopo
Hi all!
I must admit that it is a new field for me. Last night I was talking about it and getting more info from Johan, even he gave me his tool to calculate the data space of an image.
Till now the only thing I used to do was to copy MDV images into my SD and work with them, Dr Gusman tools I think solves the matter, but
there are things that I don't understand yet. Thus I decided
to start to investigate about it and to document it in PDF, Wiki and some videos for newbies like me.
Also mention that
thanks to Johan, I will make
a Java tool (with GUI) to share with the community to deal with the "magic fields" ("headers" and so on). If things goes right, I will ask to Johan and other advances users what other tools I could add to this APP in order to make much easier for everyone to deal with MDV images.
I work with Linux, that is why I didn't get any of those troubles with headers. Many users that use MDP not (most common Windows users), so it could be a great
tools multi-platform (Linux, Mac, Windows, OS/2... any OS that runs Java VM).
Sorry if I cannot answer some questions, but I have not enough level/information about all details for it.
About to put a folder inside others... hummm interesting,
I didn't find that problem in su-folders, only in the root directory, but yes... i
t makes sense and should happen too.
Thank you for all suggestions and sharing the information about that matter that is very new for me.
Re: MicroPicoDrive Popopo's version
Posted: Wed Aug 07, 2024 9:13 am
by Popopo
Pr0f wrote: Wed Aug 07, 2024 8:57 am
I will try that tool - see if it can generate a compatible mdv image for this microdrive emulator. Thanks for the heads up
Please, if you do, would you mind to document it?
I would like to reproduce the process in a video if it solves a wide known problem, and helps people like me.
XorA wrote: Wed Aug 07, 2024 8:50 am
I do wish people would stop calling it a header, as it isnt, the dataspace is stored in the directory file.
....
On Q-Emulator you can look in the documentation for how it stores the directory entry in a header on the front of a file in order to re-create this inside QDOS.
Oh!, interesting. Thanks for explanation in detail. What is then the "second header"?
XorA wrote: Wed Aug 07, 2024 8:50 am
mdvtool was written by the MiST core creator, I have forked it and added some features since, it produces QLAY format images which may or may not work with this form of uDrive emulator. It has the awesome feature of being able to convert zip->mdv directly.
https://github.com/xxoraa/mdvtool
Very interesting tool. By now that job also is done by Dr. Gusman tool, I didn't need to use it yet. But to have another tool to test is great.
I tried to install QEmu in my Linux but doesn't work fine. Then the fork uQm... horrible working on my system or too complex to use for me. So still looking for a emulator that allows me to start programming some "games or tools" and understand better the process that we are talking about now.
Also, it would be awesome me for, because in Sept I want to develop the Bluetooth feature for MDP and to send directly from my computer after testing in the emulator to my real QL the image generated and test it in real in just one click (without inserting mSD, copying files, etc) could be amazing for me. So I guess for other users too.
That is why, all that matter you are explaining to me is very very helpful.
Re: MicroPicoDrive Popopo's version
Posted: Wed Aug 07, 2024 10:24 am
by dilwyn
"Header". The term is certainly ambiguous, although used in some QDOS documentation for a 'file header'. When it comes to bytes added at the start of a file, a less ambiguous term is 'preamble', e.g. for the extra bytes QemuLator adds for the executables information, or for the extra bytes at the start of a PIC file indicating file identifier, image size and so on.
Executable file header is certainly a valid term in QDOSMS terminology, but when referring to data contained at the start of the file it is less ambiguous to refer to it as 'preamble' (i.e. something which is part of but comes before the main file content itself)
Re: MicroPicoDrive Popopo's version
Posted: Wed Aug 07, 2024 3:43 pm
by XorA
I've not found anything specific in the documentation about the file structure.
Its in Appendix II on the MacOS version!
Re: MicroPicoDrive Popopo's version
Posted: Thu Aug 08, 2024 9:01 am
by Derek_Stewart
XorA wrote: Wed Aug 07, 2024 3:43 pm
I've not found anything specific in the documentation about the file structure.
Its in Appendix II on the MacOS version!
In the Windows version as well.
Re: MicroPicoDrive Popopo's version
Posted: Thu Aug 08, 2024 10:30 am
by Pr0f
I should have gone to specsavers...
Re: MicroPicoDrive Popopo's version
Posted: Thu Aug 08, 2024 11:38 am
by Popopo
Please, I have some questions...
if that info is "only" in the doc of that emulator,
Does it mean that doesn't affect to others emulations or devices? (ie: vDrive, Oqtadrive, etc).
Is it a issue related with the OS? (ie. MS Windows, Mac OS, Linux, OS/2, BSD, etc).
I guess it is linked to the storage structure (FAT, NTFS, efs, HPFS...) more than the OS itself. Isn't it?
Johan, shared me his tool to calculate the data space. Dr. Gusman tool can set it but I didn't see anything to calculate it. How do you do it without an emulator?
Thanks
Re: MicroPicoDrive Popopo's version
Posted: Thu Aug 08, 2024 12:31 pm
by tofro
Popopo wrote: Thu Aug 08, 2024 11:38 am
Please, I have some questions...
if that info is "only" in the doc of that emulator,
Does it mean that doesn't affect to others emulations or devices? (ie: vDrive, Oqtadrive, etc).
Is it a issue related with the OS? (ie. MS Windows, Mac OS, Linux, OS/2, BSD, etc).
I guess it is linked to the storage structure (FAT, NTFS, efs, HPFS...) more than the OS itself. Isn't it?
Johan, shared me his tool to calculate the data space. Dr. Gusman tool can set it but I didn't see anything to calculate it. How do you do it without an emulator?
Thanks
QDOS sets aside 64 bytes in the file system (its native file systems) for this information. On microdrives and floppy disks, this information actually lives in the directory, and not in the actual file (but is still called a "file header"). As "foreign" (i.e. DOS, Windows, MacOS, Linux...) OS don't know anything about such a file header (and, thus, have no place to store it), if you simply zip and unzip QDOS files, that information is normally lost. QDOS zip/unzip knows about the header, of course, and packs and unpacks the header into the zip file when extracted to a native QDOS file system.
Emulators that don't use a native QDOS file system have the same problem - they don't have a place to store the additional information. Q-Emulator has found its own way to do that and simply prepends a specific header (here, it's really a "header") to the QDOS file and cuts it off when it presents the file to the emulated QDOS. That is why it can store QDOS files as "normal" DOS (or, whatever OS) files. Once the Q-Emulator header is attached, however, the file is no longer a valid QDOS executable - Which is somewhat of a bummer.. You need to use Q-Emulator to put the file onto a native QDOS file system, which will slice off the header during the copy (or zip, or whatever) process and put it to the proper place in the directory -
vDrive, OQtadrive and thelikes use native QDOS file systems (the "mdv_ file system") within their container files and can thus perfectly store the file header information where it belongs: in the directory of the container files.
Re your last paragraph: "re-calculating" the data space requirements of an executable is actually not possible - You can extract the information from the original file, but if the header information is lost, it's trial and error.
Re: MicroPicoDrive Popopo's version
Posted: Thu Aug 08, 2024 12:34 pm
by xelalex
Popopo wrote: Thu Aug 08, 2024 11:38 am
Please, I have some questions...
if that info is "only" in the doc of that emulator,
Does it mean that doesn't affect to others emulations or devices? (ie: vDrive, Oqtadrive, etc).
From my experience, the various QL (and also Spectrum) emulators are let's say somewhat liberal in their interpretation of the specification for the logical Microdrive structure. You will find all sorts of errors in images created by emulators, such as incorrectly numbered sectors, reverted sector numbers, missing sectors, or poorly placed file records, too name just a few. OqtaDrive fixes many of these problems on the fly, when loading an image. But I'm sure there's stuff out there that won't work
Regarding the particular issue with QDOS Zips, OqtaDrive can load these Zips directly and create a cartridge on the fly, provided the contents fits on one cartridge. The headers/preambles of the files in the Zip will be handled correctly. You can even use the oqtactl binary to convert a QDOS Zip into an mdv:
You don't need an actual OqtaDrive to do this, just need to get the binary from the release section of the project.
Re: MicroPicoDrive Popopo's version
Posted: Thu Aug 08, 2024 3:27 pm
by XorA
From my experience, the various QL (and also Spectrum) emulators are let's say somewhat liberal in their interpretation of the specification for the logical Microdrive structure. You will find all sorts of errors in images created by emulators, such as incorrectly numbered sectors, reverted sector numbers, missing sectors, or poorly placed file records, too name just a few. OqtaDrive fixes many of these problems on the fly, when loading an image. But I'm sure there's stuff out there that won't work
Im pretty sure most of the above exist on real MDVs in some form or other in the guise of "protection systems".