in reply to Losing bytes with Device::SerialPort ?

I think it unlikely that the problem you have is related to the UART buffer being off or not working properly. You are holding the serial port open so the serial port driver will certainly receive all the data that comes in for you, whether or not your process is scheduled.

The problem might be an unwanted character translation as others have pointed out. Another possibility is that there is something else also listening to the serial port and it is absorbing a character once in a while. This could be a getty or some other process watching for something to happen on a serial port. Some program might also be running that expects to talk to some kind of transparent serial port dongle like an X10 Firecracker that really isn't so transparent after all.

Since it sounds like you have a magic sequence at the start of every received message (0x55 0xaa), you can use this to synchronise to the start of a message and recover from the situation where you have fallen out of sync due to a lost character. The details of doing this depend on whether or not the protocol is designed so that the magic sequence cannot possible occur in the body of a message (maybe it would be quoted if it does appear). To do this, you should read characters one at a time until you find a 0x55 followed by a 0xaa, use this to detect that you have come to the start of a new message, and then read the rest of the header in bulk (and then the body in bulk).

In order to solve your problem, you will probably want to dump everything you read from the serial port into a debugging log. When you get a corrupted message, analyze the PREVIOUS message in the log to see how it ended. You might find that the end of the previous message contained the start of the current one, which would be wrong and would indicate the size of the previous message was one byte off.

Replies are listed 'Best First'.
Re^2: Losing bytes with Device::SerialPort ?
by BrowserUk (Patriarch) on Dec 22, 2005 at 16:05 UTC
    You are holding the serial port open so the serial port driver will certainly receive all the data that comes in for you, whether or not your process is scheduled.

    You are wrong. There is absolutely no certainty that the device driver will be able to offload each byte from the UART in a timely manner, and with the FIFO buffering disabled, it is 16 or 64 times more likely not to be able to do so.

    It has nothing to do with whether the process is scheduled, or whether the serial port is open or not. The problem is one of timely response by the device driver to each IRQ request. With the hardware buffering turn off, data loss is endemic at anything above very low transmission speeds. Any higher priority IRQ activity (like DMA requests) can prevent the DD from storing the inbound byte before the next one replaces it. This can be mitigated to some extent by enabling rts/cts or XON/XOFF handshaking, but the net result is that you reduce throughput to a crawl.

    This may have nothing or everything to do with the OPs problem, but it is certainly one way that his symptoms could occur.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      You are right that there are some ways that data can be dropped on the floor, however I stand by assessment that it is very unlikely here, assuming the computer in question is halfway recent.

      Sure, in theory IRQs from the serial port can be ignored in critical sections long enough for it to matter, but asynchronous serial ports are so slow compared to modern CPUs and UARTs without buffers are so rare nowadays (even in embedded equipment (except for TTL serial ports)) that it hardly ever matters. At 115200 bps, it takes more than a millisecond to completely fill up a 16550 buffer, and that's a long time.

      My empirical basis for this is that we could run upwards of 32 dumb serial ports on commodity PC hardware all transferring at 14400 bps (or more, thanks to modem compression), in or around 1993. And we know how much faster PCs are now than they were then...

        Agreed. Circa 1992, we had a PS/2 model 80 (20MHz 386) running OS/2 happily handling 64 ports at 9600; but it was important that the FIFO buffering was enabled. It's hard to see why it wouldn't be by default, but if it wasn't that could explain the symptoms described.

        It's also possible to set the IRQ threshold below the buffer maximum. This is useful when conducting bidirection conversations involving short sequences; for example when handshaking.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re^2: Losing bytes with Device::SerialPort ?
by Anonymous Monk on Dec 23, 2005 at 08:42 UTC
    Hi,

    thanks for all info. I'll try to log communication when I get home. But I have following further questions:

    - is there any easy way to determin if something else is listening on serial port ?

    - possibility that end character might be swallowed is mentioned. I'd like to clear this in my head: I have start sequence of bytes ,and then info about length -so I receive only declared number of bytes - is there any way that end byte of previous message would be swallowed if it is a part of starting ?

    - do I have any way to declare two byte starting sequence as start event - so I can receive bytes only after it - or should I receive byte per byte and see if it's start sequence ?

    - do timings ($PortObj->read_const_time(100);$PortObj>read_char_time(5);) anyhow influence my situation ?

    Thanks in advance,

    Regards,

    Rob.

      - is there any easy way to determin if something else is listening on serial port ?

      There's no portable way but on some operating systems you might try fuser.

      - possibility that end character might be swallowed is mentioned. I'd like to clear this in my head: I have start sequence of bytes ,and then info about length -so I receive only declared number of bytes - is there any way that end byte of previous message would be swallowed if it is a part of starting ?

      So you get a declared length and then read that many bytes. If any one of the bytes that is part of the body of the message got lost in transmission, you will end up reading one byte past the end of the message (therefore, probably the start of the next message).

      A really robust protocol like the asynchronous version of HDLC that is used in PPP defined a sequence of bytes (sequence of bits in the true synchronous version of HDLC) that can never appear anywhere inside a frame. It always marks the start of a new frame. So what you do is that you read until you find a frame header, then start reading the contents of a frame. If, while you are doing this, you find a copy of the FLAG sequence prematurely, you know that something has gone wrong and the frame you are presently reading is corrupted. You will have to discard it and start a new frame header from the present position (i.e. the position where you just found the FLAG sequence). You lost a frame but at least you are synchronized to the start of the next frame (which will hopefully not be corrupted).

      This robustness requires using a protocol that provides such a guarantee of a sequence that can never occur except at the very start of a frame. 0x55 0xaa might serve that purpose in your protocol, but I suspect that the protocol you are dealing with is a little dumber than that and that 0x55 0xaa can probably appear inside the data if you are unlucky. My guess is that this is some kind of embedded device and it doesn't use a very wonderful protocol. In that case your ability to resynchronize to the start of a frame in the face of a corrupted frame is quite limited and you should complain to the protocol designer. By the way, the ability to synchronize is also important when your application starts, if the first thing it reads is some gibberish message fragment from something that was sent before your application started, so that you know when the gibberish is over and the real fun starts.

      Synchronization over data transmission lines is not a concept that is reserved for the telecommunications field. An example of a data encoding quite different from HDLC that also provides excellent resynchronization capability is UTF-8.

      In answer to your last question, you could probably use timing heuristics to work around a bad protocol. That will be some trial and error.

      Hi

      I've learned my lesson. Another process was running and under linux obviously more apps can access serial port in parallel (there is no way to do the same under Windows)..

      Thanks all for great reponses, I certainly learned a lot about Serial port under Linux ....

      Thanks,

      regards,

      Rob.