My understanding of the .mdv format and some q's.

[Chr$]

Just writing some stuff down on the structure of the .mdv dump files that mdump_task creates (to better understand it myself). Before I post it on my website I thought I would post here for comments/corrections. I also have a couple of questions at the end. Here we go...

The mdump mdv data format.

Basics:

Computers only understand highs and lows (1's and 0's), AKA binary. Mdv carts contain magnetic representations of highs (1) or lows (0) and these highs and lows (plus carefully positioned gaps and 'preambles' to aid syncing) whizz past the microdrive's stereo read/write head and are interpreted into data that can be understood by the QL.

Eight binary highs/lows make up an eight bit byte, e.g. 01110110 (which is 118 in decimal). To make things a bit easier for humans, bytes are usually converted to and written in hexadecimal, as that form of counting can represent every eight bit byte with just two characters in the hexadecimal range of 00 to FF (00000000 to 11111111 in binary, or 0 to 255 in normal decimal 'every day' counting). Written hex numbers are often prefixed by a dollar sign, a 0x or sometimes suffixed by a lower-case h, e.g. $0E, 0x0E or 0Eh.

The continuous loop of tape within a microdrive cartridge is divided into sectors when the cartridge is formatted - this lays out the template that any data saved to the cartridge should follow.

When you format a microdrive cartridge you'll see afterwards that the QL reports the number of good sectors that were achieved. This number of sectors is always a little different, as it depends on the precise length of tape within the cartridge and on the number of sectors that were found to be bad (i.e. not reliable). Formatting is deemed unacceptable and will fail if the cartridge formats with less than 200 good sectors, so you'll never see lower than 200. In my experience, with my equipment, cartridges with around 215 good sectors and less than 8 bad sectors seem to be the most reliable.

(I'll add screenshot of format complete message here)

There are always two special parts of each cartridge, one is the first sector, numbered $00, which contains the sector map (more on that later) and the other special area is the directory, which is always file number $00 and can take up multiple data blocks (if the directory contains a lot of file names). All other files saved onto the cartridge are numbered from $01 onwards and the files are numbered in the order in which they are saved onto the cartridge.

All of the normal sectors on the cartridge contain a 512 byte data block, plus a sector number, a random number (which is the same for every sector on the cartridge), a file number or flag, a block number, the cartridge name (which is also the same for every sector) and a checksum, which should always add up to the total of the bytes within the data block.

In total each sector represented in an mdump .mdv file is 530 bytes in length (18b in total for the name, flags, numbers and checksum etc, plus the 512b data block), so what I do when examining an mdump_task file is, using a hex editor like HxD, set the bytes per row to 530, that way everything is organised in easier to read columns.

(I'll add an image HxD showing an .mdv file here)

Note that there are other mdv cartridge file formats, all have an .mdv extension but are internally different. As I primarily deal with dumps of original cartridges, the info on this page relates to .mdv files created with the mdump_task program on a QL using the JS ROM. Note also that the actual data on the tape is not exactly the same as how it is represented in an mdump .mdv file.

The structure of a cartridge dump:

The mdump_task program adds its own 46 byte header (bytes $00 to $2D) to the raw .mdv dump data. After that follows sector $00 (which contains the sector map). Sectors then follow in descending order, from the highest that the tape length allows (e.g. $D9 for 217 sectors) all the way down to $01 at the bottom. After $01 mdump_task does sometimes save alternate copies of bad sectors.

So, after the 46 byte mdump header there are rows of similar 530 byte patterns. Here's an example of one such sector row, this one has no data in its data block:

Code: Select all

Byte no. 1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19 20 21 22 23 24                529 530 
     
         FF D9 62 6C 61 6E 6B 20 20 20 20 20 3E 70 FD 00 AA 55 AA 55 AA 55 AA 55 ............... 0E 0F

Byte 1 is always an $FF.
Byte 2 is the sector number, in this case $D9 or 217 in decimal.
Bytes 3 to 12 are the cartridge name assigned to it when it was formatted - in the example above the name $62 $6C $61 $6E $6B is the word 'blank', which is then padded out with 5 spaces (5x $20 in hex).
Bytes 13 and 14 ($3E $70) are randomly assigned to the cartridge when it is formatted and these 13th and 14th bytes are always identical for every sector of a given cartridge.
Byte 15 is the number of the file that is contained (or partly contained) within the data block of the sector, or it's a flag. When the sector contains no data, this is always the flag $FD.
Byte 16 is the number of the block of data of the file contained within the sectors' data block, or if there is no data contained within the block (as with the example above) then this byte is $00.
Bytes 17 all the way to byte 528 are the data block, which for empty sectors always alternate $AA $55.
Then right at the end, the last 2 bytes of a sector are always the checksum.

Here's another example of a sector which does contain data:

Code: Select all

Byte no. 1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19 20 21 22 23 24                529 530 

         FF BC 4E 45 4D 45 53 49 53 5F 32 20 1C 92 03 1B 6E 2E 22 3A 66 25 28 31 ................58 FF

It starts with $FF, as usual. It is sector number $BC, which is 188 in decimal. There is a 9 character name, followed by a padding space ($20). All of the sectors on the cartridge have the random number $1C $92. Byte 15 shows that its data block contains (part of) file $03 - in fact in byte 16 you can see that it contains part $1B, or part 27 in decimal(actually the 28th part, because 0 counts!) of file $03. Then follows the actual data and right at the end is the 2 byte checksum.

How are files actually saved?:

Files are not saved within data blocks of consecutive sectors, instead, they are spread out evenly across the tape. The sector map contains the info that tells the QL where the various parts of a file can be found. Small files that are less than 512b in size will only take up one data block of one sector and the data is always padded out at the end to use the full 512b data block space.

Larger files that take up multiple data blocks obviously have to be spread over 2 or more (non consecutive) sectors, so for example, a file that is the first file saved onto a cartridge (and is therefore file $01 - remember file $00 is always the directory) that is 1,400 bytes in size will live in a total of 3 sectors. These sectors will have the following pairs of 15th and 16th bytes: $01 $00, $01 $01 and $01 $02 - that's $01 for file number 1 and then its three blocks $00, $01 and $02. As the file is smaller than 3x 512b, the last part of the data block that contains part $02 of the file will be padded out.

The sector map:

The sector map tells the QL where to find the parts of each file. As mentioned above, the sector map is always sector $00. Here is an example of part of a sector map:

Code: Select all

Byte no. 1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

         FF 00 71 75 65 73 74 69 6F 6E 73 20 08 31 80 00 F8 00 05 07 04 04 03 01 FD 00 FD 00 FD 00 12 08 0B 03

Byte 1, as usual, is $FF.
Byte 2 is the sector number, $00 for the sector map.
Bytes 3 to 12 are the cartridge name with, in this example, one padding space ($20) at the end.
Bytes 13 and 14 are the random number.
Byte 15 is $80, which I believe is a flag to indicate the sector map.
Byte 16 is $00, the block number - the sector map is always only one block long.
Sector mapping starts at byte 17. First, the sector map gives its own position, thus
Byte 17 is the designated file number for the sector map. It is always $F8.
Byte 18 is the block number for the sector map, always $00.

From byte 20 all the way to the end of the sector (where there are 2 checksum bytes as usual) there are pairs of file and block numbers, in total 255 pairs. Their position in the sector map represents their position on the tape, so the first pair represents sector $01, and being $05 $07 it tells the QL that part $07 of file $05 is held within the data block of sector $01. Remember that sector $01 is at bottom of the file, as mdump_task starts with sector $00 (the sector map) and then descends from the highest sector number to the lowest.

The next (2nd relevant) pair of numbers in the example above tells the QL that its sector - $02 - contains part $04 of file $04. The next pair represents sector $03 and within that sector is the $01 part of file $03.

A file number with the flag $FD and block number $00 indicates that the block is empty, and you can see three empty sectors on the example above. These empty sectors correspond to sectors $04, $05 and $06. Another flag used in the sector map is $FF (and block $00) - where this appears indicates that the corresponding sector has been marked as bad by the formatting process and it will not be used to store data.

A note on file headers:

At the beginning of each file, i.e. at the start of its $00 block, is always a 64 byte header before the actual data begins. This header contains information such as the file length, type and its name. It also contains space for the file save date, but this extra info was not implemented within the original QL ROMs.

Questions:

The QL Advanced user guide mentions a 2 byte checksum at the end of the sector header, but these don't seem to show within mdump .mdv files - the headers appear 14 bytes long and then immediately follows the file number etc. Is it simply not shown in the mdv file, just as the preambles are also omitted? Or is it an error in the advanced user guide?

Checksums - How do they work exactly, I can't seem to get my head round them. The advanced QL user guide states that the checksum at the end of the data block is preset to $0F0F (when the cart is formatted?) but whenever I add that to each byte of the data block I always get something completely different to the checksum shown in an mdump dump.

[xelalex]

The QL Advanced user guide mentions a 2 byte checksum at the end of the sector header, but these don't seem to show within mdump .mdv files - the headers appear 14 bytes long and then immediately follows the file number etc. Is it simply not shown in the mdv file, just as the preambles are also omitted? Or is it an error in the advanced user guide?

It's an omission in the mdv format as used by mdump.

Checksums - How do they work exactly, I can't seem to get my head round them. The advanced QL user guide states that the checksum at the end of the data block is preset to $0F0F (when the cart is formatted?) but whenever I add that to each byte of the data block I always get something completely different to the checksum shown in an mdump dump.

When calculating a checksum, you start with 0xf0f0 (the preset), then keep adding the bytes.

[Chr$]

When calculating a checksum, you start with 0xf0f0 (the preset), then keep adding the bytes.

Thanks xelalex. Can you show me an example and walk me through that? Perhaps I'm not adding things up correctly. I use the windows calculator in 'programmer mode'.

So e.g. for a blank sector it's just pairs of $AA and $55 and the check sum is always $0E0F.

[xelalex]

So e.g. for a blank sector it's just pairs of $AA and $55 and the check sum is always $0E0F.

That's the correct data block checksum after formatting. So your calculation is correct.

[Chr$]

So e.g. for a blank sector it's just pairs of $AA and $55 and the check sum is always $0E0F.

That's the correct data block checksum after formatting. So your calculation is correct.

Ok, but that wasn't my calculation, that's what is always shown as the checksum at the end of a blank data block. I can never seem to get that result, I think I must be adding hex incorrectly.

[xelalex]

Ok, but that wasn't my calculation, that's what is always shown as the checksum at the end of a blank data block. I can never seem to get that result, I think I must be adding hex incorrectly.

Ah, ok. Then let's calculate for a fresh data block:

Code: Select all

0x0f0f + 0x100 * 0xaa + 0x100 * 0x55 = 0x10e0f

That's the preset plus 256 bytes with 0xaa and 256 bytes with 0x55. Since the accumulation of the check sum is done in a 16 bit register, we finally get 0x0e0f.

[Chr$]

Ok, but that wasn't my calculation, that's what is always shown as the checksum at the end of a blank data block. I can never seem to get that result, I think I must be adding hex incorrectly.

Ah, ok. Then let's calculate for a fresh data block:

Code: Select all

0x0f0f + 0x100 * 0xaa + 0x100 * 0x55 = 0x10e0f

That's the preset plus 256 bytes with 0xaa and 256 bytes with 0x55. Since the accumulation of the check sum is done in a 16 bit register, we finally get 0x0e0f.

Ah, got it, thanks. The 'chopping' off of the 1 due to the 16-bit register thing must have thrown me.

[xelalex]

Ah, got it, thanks. The 'chopping' off of the 1 due to the 16-bit register thing must have thrown me.

Yes, when calculating this separately, you need to modulo the total sum (preset + bytes) with 0x1000, to get the actual checksum. Here's the function I have in OqtaDrive for this. Input is the sum of all bytes for which to get the check sum:

Code: Select all

func toQLCheckSum(sum int) int {
	return (0x0f0f + sum) % 0x10000
}

[Martin_Head]

Also on a real microdrive the checksum is stored least significant byte first. I.e. The wrong way round to the way normally stored in a QL.

And the checksum at the end of an mdump data block is the 'wrong' way round, to the way it would normally be stored.

Note that you are describing the format of a mdump2 image file. The mdump1 image has sectors of 528 bytes. With no checksum at the end of the data block.

I have documented the image file formats of both of the mdump versions and the Qlay version. And of course the _mdi image file format.
This is all hand written at the moment. So I would need to type it up, unless you would just like a scan of it.

I also have some SBASIC programs I have written for examining image files. Displaying the map, sectors, etc. If your interested.

[Chr$]

Also on a real microdrive the checksum is stored least significant byte first. I.e. The wrong way round to the way normally stored in a QL.

And the checksum at the end of an mdump data block is the 'wrong' way round, to the way it would normally be stored.

Note that you are describing the format of a mdump2 image file. The mdump1 image has sectors of 528 bytes. With no checksum at the end of the data block.

I have documented the image file formats of both of the mdump versions and the Qlay version. And of course the _mdi image file format.
This is all hand written at the moment. So I would need to type it up, unless you would just like a scan of it.

I also have some SBASIC programs I have written for examining image files. Displaying the map, sectors, etc. If your interested.

Type it up as and when you have the time/inclination, no hurry with that. I would certainly like to try out the progs to display the map, sectors etc.