Mailing List Archive

High Speed Serial Problem
G'day,

I'm porting a DOS application to Linux and have encountered problems
with the high speed serial devices from which the app receives data.

The hardware consists of a PC-104 board with a 486 compatible SOC
(system on a chip) providing 4 serial ports. Ports 2 and 4 connect to
external devices that communicate at 115,200 baud. The device sends a
42 byte record (containing a CRC), asserts a break to indicate end of
record, then does it again. Every hour or so the app sends a message
to the device. The message is sent immediately after the break is
recognized.

As an additional complication, the device connection is RS-485. Being
half duplex, the code enables receive mode and transmit mode as needed.

The DOS app has an ISR to handle the communication. Measured
throughput is approx 4,230 characters per second. There is an error
rate of approx 2.5 errors per 1,000 records.

The Linux app has a separate thread for each of the 2 serial ports.
Data is received using select() and read(). Read is called with a 32
byte buffer. As neither BRKINT nor IGNBRK is set, the input thread
recognizes 0xFF, 0x00, 0x00 sequences as breaks. When the encoded
break is recognized, any pending message is sent.

The Linux app receives approx 5,169 bytes per second and has an error
rate of 100 errors per 1,000 records.

Errors generally show up as dropped bytes. Usually only 1 or 2 bytes.

The first (and most obvious) problem is that the Linux error rate is
40X higher rate than the DOS error rate.

The second problem is that output from Linux isn't working. After
writing to the device, reading back doesn't show any change. It's
suspected that there's a timing issue, i.e. a timing window for
receiving messages after the device sends a break.

I'm using a 2.6.29 kernel and the 486 SOC is running at 300mhz.
Processor speed is not believed to be the root because a newer 500mhz
version of the PC-104 board exhibits the same symptoms.

Any suggestions of what to do or where to look to solve the problems?

Thank you.

David
Re: High Speed Serial Problem [ In reply to ]
David,

I suggest you post this question to linux-kernel@vger.kernel.org and
linux-serial@vger.kernel.org
instead.

One question though: does your target have a real (physically real) uart chip?
There are some x86 systems out there which emulate a 8250 through SMM code,
which exhibit these kinds of failures at high baud rates. Do lower
baudrates work?

Best regards,
Manuel Lauss
Re: High Speed Serial Problem [ In reply to ]
On Sun, 14 Mar 2010 17:34:50 +0100
Manuel Lauss wrote:

> David,
>
> I suggest you post this question to linux-kernel@vger.kernel.org and
> linux-serial@vger.kernel.org
> instead.
>
> One question though: does your target have a real (physically real)
> uart chip? There are some x86 systems out there which emulate a 8250
> through SMM code, which exhibit these kinds of failures at high baud
> rates. Do lower baudrates work?
>
> Best regards,
> Manuel Lauss

Hello Manuel,

The UARTs are real and work very very well with DOS (remember the 2.5
errors per 1000 records).

The high speed serial device has a fixed 115KB rate and is
transferring 4000+ characters per second. With the fixed baud rate,
lower baudrates aren't usable, hence can't be tested.

David
Re: High Speed Serial Problem [ In reply to ]
David Relson wrote:
> Any suggestions of what to do or where to look to solve the
> problems?

Look to the hardware. Linux is not introducing these problems.

Start with tapping the RS-485 signal with two other receivers, and
look at what bytes are being sent when.

Linux can do plenty of buffering in it's RX interrupt so you're not
going to lose any incoming data.


//Peter
RE: High Speed Serial Problem [ In reply to ]
G'day,

I'm porting a DOS application to Linux and have encountered problems
with the high speed serial devices from which the app receives data.

Any suggestions of what to do or where to look to solve the problems?


OK, this sounds like a timing problem to me - I assume you are controlling
the bus transceiver from your userland code? The Linux serial handler has a
lot of buffering in it and it enables the FIFOs on the UART chip, so you may
only get an interrupt some time after the data has been received.

My first approach would be to get a scope and look at the signal lines (TX,
RX and transmit enable) and see if the timing looks plausible. If you have
a spare I/O line you can play with, then set that up so that it pulses or
toggles when you detect a break - you can then see how long it's taking for
the code to recognize the condition.

Pete
Re: High Speed Serial Problem [ In reply to ]
Hi,

> The Linux app has a separate thread for each of the 2 serial ports.
> Data is received using select() and read(). Read is called with a 32
> byte buffer. As neither BRKINT nor IGNBRK is set, the input thread
> recognizes 0xFF, 0x00, 0x00 sequences as breaks. When the encoded
> break is recognized, any pending message is sent.

There should be no problem handling 115K2 data at full speed on a 300MHz
processor although some SOC UARTs are realy gnarly. Using line break as
a protocol signalling element is so 1970s but again shouldn't be a problem.

What are your VMIN, VTIME settings ? They can have a big impact on read
performance/timing. VMIN=VTIME=0 makes a mockery of using select for
example.

What size of return are you actually seeing from the read ?

If you are seeing reasonably full blocks, does you code handle the three char
break sequence spanning the read boundy ? Sorry if this is teaching grandma
to suck eggs.

--
Bob Dunlop
Re: High Speed Serial Problem [ In reply to ]
David Relson wrote:

> I'm porting a DOS application to Linux and have encountered problems
> with the high speed serial devices from which the app receives data.



Well here is link to a home brewed Serial Chip buffering
circuit, that may better allow you to look at the signals
and protocol interactions on your serial ports. I have found
the hardware ckt to help (stabalize) bit sniffing on serial
communications. The keyword is 'eavesdropping' as used
by embedded designers. What good about this project, is it
is an easy, practical work that allows you to learn the art
of eavesdropping on a com link, rs232C being the vintage
form. RS485 will come much easier into enlightenment for you....


You can download the tar file from here:

http://sourceforge.net/projects/serialsniffer/



If you look at the file: LinuxSPA.png (after unpacking the
tar file), you'll see the circuits; I used gimp, then you
can see the chips used to buffer the signals are the old
reliable 1488 and 1489.

I'm not sure about the max speed, I believe its around
500kbps, so you might have to find newer, faster
replacements chips for the 1488 and the 1489.....


When you go to test your hardware/software between the new
embedded linux system and a tried and proven dos based
system, this sort of buffering circuit is very valuable to
keep from "loading/distorting" the signals between the 2
different devices, particularly as the speed of
communications increases.


You can look at both ends of the serial communications
with 2 different windows on each side of the comm link, to
discerning transmission issues, such as timing sync, skew
etc etc.


hth,
James
RE: High Speed Serial Problem [ In reply to ]
Hello Bob,

VMIN and VTIME are the default values of 1 and 0, respectively. If I
understand correctly, these settings will provide data as rapidly as
possible.

At present, I am using a 256 byte buffer for reading data. I've added
some statistics gathering. Every 1000 reads, the program logs the total
bytes read and the maximum read. The program was run for 30 seconds to
provide 6 statistics records

With _no_ processing of data, 1000 reads got from 27,545 to 28,520
chars. The max read was 128 chars. 3 of the 1000 read samples had max
reads of 56 or 57 and 3 had max reads of 110, 126, and 128.

With copying of data to a buffer (which involves several function calls,
incrementing counters, etc), 1000 reads got from 27,187 to 28,241 chars
and each of the 6 sample sets had a max read of 110 to 128 bytes.

One would expect processing to increase the total bytes received, but
this was not the case.

The code _is_ handling the 3 char break sequence and the (undocumented)
doubling (escaping) of 0xFF chars (to distinguish them from break
sequences).

Regards,

David
-----Original Message-----
From: Bob Dunlop [mailto:bob.dunlop@xyzzy.org.uk]
Sent: Monday, March 15, 2010 9:35 AM
To: gentoo-embedded@lists.gentoo.org
Subject: Re: [gentoo-embedded] High Speed Serial Problem

Hi,

> The Linux app has a separate thread for each of the 2 serial ports.
> Data is received using select() and read(). Read is called with a 32
> byte buffer. As neither BRKINT nor IGNBRK is set, the input thread
> recognizes 0xFF, 0x00, 0x00 sequences as breaks. When the encoded
> break is recognized, any pending message is sent.

There should be no problem handling 115K2 data at full speed on a 300MHz
processor although some SOC UARTs are realy gnarly. Using line break as
a protocol signalling element is so 1970s but again shouldn't be a
problem.

What are your VMIN, VTIME settings ? They can have a big impact on read
performance/timing. VMIN=VTIME=0 makes a mockery of using select for
example.

What size of return are you actually seeing from the read ?

If you are seeing reasonably full blocks, does you code handle the three
char break sequence spanning the read boundy ? Sorry if this is
teaching grandma to suck eggs.

--
Bob Dunlop