293 lines
21 KiB
HTML
293 lines
21 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en-us">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<title>How to talk to your microcontroller over serial | jaseg.de</title>
|
|
<link rel="stylesheet" href="/css/style.css" />
|
|
<link rel="stylesheet" href="/css/fonts.css" />
|
|
|
|
<header>
|
|
|
|
|
|
<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/atom-one-light.min.css">
|
|
<script src="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script>
|
|
<script>hljs.initHighlightingOnLoad();</script>
|
|
<nav>
|
|
<ul>
|
|
|
|
|
|
<li class="pull-left ">
|
|
<a href="https://blog.jaseg.de/">/home/jaseg.de</a>
|
|
</li>
|
|
|
|
|
|
|
|
|
|
</ul>
|
|
</nav>
|
|
</header>
|
|
|
|
</head>
|
|
|
|
<body>
|
|
<br/>
|
|
|
|
<div class="article-meta">
|
|
<h1><span class="title">How to talk to your microcontroller over serial</span></h1>
|
|
|
|
<h2 class="date">2018/05/19</h2>
|
|
<p class="terms">
|
|
|
|
|
|
|
|
|
|
|
|
</p>
|
|
</div>
|
|
|
|
|
|
|
|
<main>
|
|
<div class="document">
|
|
|
|
|
|
<p>Scroll to the end for the <a class="reference internal" href="#conclusion">TL;DR</a>.</p>
|
|
<p>In this article I will give an overview on the protocols spoken on serial ports, highlighting common pitfalls. I will
|
|
summarize some points on how to design a serial protocol that is simple to implement and works reliably even under error
|
|
conditions.</p>
|
|
<p>If you have done low-level microcontroller firmware you will regularly have had to stuff some data up a serial port to
|
|
another microcontroller or to a computer. In the age of USB, a serial port is still the simplest and quickest way to get
|
|
communication to a control computer up and running. Integrating a ten thousand-line USB stack into your firmware and
|
|
writing the necessary low-level drivers on the host side might take days. Poking a few registers to set up your UART to
|
|
talk to an external hardware USB to serial converter is a matter of minutes.</p>
|
|
<p>This simplicity is treacherous, though. Oftentimes, you start writing your serial protocol as needs arise. Things might
|
|
start harmless with something like <tt class="docutils literal">SET_LED ON\n</tt>, but unless you proceed it is easy to end up in a hot mess of command
|
|
modes, protocol states that breaks under stress. The ways in which serial protocols break are manifold. The simplest one
|
|
is that at some point a character is mangled, leading to both ends of the conversation ending up in misaligned protocol
|
|
states. With a fragile protocol, you might end up in a state that is hard to recover from. In extreme cases, this leads
|
|
to code such as <a class="reference external" href="https://github.com/juhasch/pyBusPirateLite/blob/master/pyBusPirateLite/BBIO_base.py#L68">this gem</a> performing some sort of arcane ritual to get back to some known state, and all just because
|
|
someone did not do their homework. Below we'll embark on a journey through the lands of protocol design, exploring the
|
|
facets of this deceptively simple problem.</p>
|
|
<div class="section" id="text-based-serial-protocols">
|
|
<h2>Text-based serial protocols</h2>
|
|
<p>The first serial protocol you've likely written is a human-readable, text-based one. Text-based protocols have the big
|
|
advantage that you can just print them on a terminal and you can immediately see what's happening. In most cases you can
|
|
even type out the protocol with your bare hands, meaning that you don't really need a debugging tool beyond a serial
|
|
console.</p>
|
|
<p>However, text-based protocols also have a number of disadvantages. Depending on your application, these might not matter
|
|
and in many cases a text-based protocol is the most sensible solution. But then, in some cases they might and it's good
|
|
to know when you hit one of them.</p>
|
|
<div class="section" id="problems">
|
|
<h3>Problems</h3>
|
|
<div class="section" id="low-information-density">
|
|
<h4>Low information density</h4>
|
|
<p>Generally, you won't be able to stuff much more than four or five bit of information down a serial port using a
|
|
human-readable protocol. In many cases you will get much less. If you have 10 commands that are only issued a couple
|
|
times a second nobody cares that you spend maybe ten bytes per command on nice, verbose strings such as <tt class="docutils literal">SET LED</tt>. But
|
|
if you're trying to squeeze a half-kilobyte framebuffer down the line you might start to notice the difference between
|
|
hex and base-64 encoding, and a binary protocol might really be more up to the job.</p>
|
|
</div>
|
|
<div class="section" id="complex-parsing-code">
|
|
<h4>Complex parsing code</h4>
|
|
<p>On the computer side of thing, with the whole phalanx of an operating system, the standard library of your programming
|
|
language of choice and for all intents and purposes unlimted CPU and memory resources to spare you can easily parse
|
|
anything spoken on a serial port in real time, even at a blazing fast full Megabaud. The microcontroller side however is
|
|
an entirely different beast. On a small microcontroller, <a class="reference external" href="http://git.musl-libc.org/cgit/musl/tree/src/stdio/vfprintf.c">printf</a> alone will eat about half your flash. On most small
|
|
microcontrollers, you just won't get a regex library even though it would make parsing textual commands <em>so much
|
|
simpler</em>. Lacking these resources, you might end up hand-knitting a lot of low-level C code to do something seemingly
|
|
simple such as parsing <tt class="docutils literal">set_channel (13, <span class="pre">1.1333)\n</span></tt>. These issues have to be taken into account in the protocol design
|
|
from the beginning. If you don't really need matching parentheses, don't use them.</p>
|
|
</div>
|
|
<div class="section" id="fragile-protocol-state">
|
|
<h4>Fragile protocol state</h4>
|
|
<p>Say you have a <tt class="docutils literal">SET_DISPLAY</tt> command. Now say your display can display four lines of text. The obvious approach to this
|
|
is probably the <a class="reference external" href="https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol">SMTP</a> or <a class="reference external" href="https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol">HTTP</a> way of sending <tt class="docutils literal">SET_DISPLAY\nThis is line 1\nThis is line 2\n\n</tt>. This would certainly
|
|
work, but it is very fragile. With this protocol, you're in trouble if at any point the terminating second newline
|
|
character gets mangled (say, someone unplugs the cable, or the control computer reboots, or a cosmic ray hits something
|
|
and <tt class="docutils literal">0x10 '\n'</tt> turns into <tt class="docutils literal">0x50 'P'</tt>).</p>
|
|
</div>
|
|
<div class="section" id="timeouts-don-t-work">
|
|
<h4>Timeouts don't work</h4>
|
|
<p>You might try to solve the problem of your protocol state machine tangling up with a timeout. "If I don't get a valid
|
|
command for more than 200ms I go back to default state." But consider the above example. Say, your control computer
|
|
sends a <tt class="docutils literal">SET_DISPLAY</tt> command every 100ms. If in one of them the state machine tangles up, the parser hangs since the
|
|
timeout is never hit, a new line of text arriving every 100ms.</p>
|
|
</div>
|
|
<div class="section" id="framing-is-hard">
|
|
<h4>Framing is hard</h4>
|
|
<p>You might also try to drop the second newline and using a convention such as <tt class="docutils literal">SET_DISPLAY</tt> is followed by two lines of
|
|
text, then commands resume.". This works as long as your display contents never look like commands. If you are only ever
|
|
displaying the same three messages on a character LCD that might work, but if you're displaying binary framebuffer
|
|
data you've lost.</p>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="solutions">
|
|
<h3>Solutions</h3>
|
|
<div class="section" id="keep-the-state-machine-simple">
|
|
<h4>Keep the state machine simple</h4>
|
|
<p>Always use a single line of text to represent a single command. Don't do protocol states or modes where you can toggle
|
|
between different interpretations for a line. If you have to send human-readable text as part of a command (such as
|
|
<tt class="docutils literal">SET_DISPLAY</tt>) escape it so it doesn't contain any newlines.</p>
|
|
<p>This way, you keep your protocol state machine simple. If at any time your serial trips and flips a bit or looses a byte
|
|
your protocol will recover on the next newline character, returning to its base state.</p>
|
|
</div>
|
|
<div class="section" id="encode-numbers-in-hex-when-possible">
|
|
<h4>Encode numbers in hex when possible</h4>
|
|
<p>Printing a number in hexadecimal is a very tidy operation, even on the smalest 8-bit microcontrollers. In contrast,
|
|
printing decimal requires both division and remainder in a loop which might get annoyingly code- and time-intensive on
|
|
large numbers (say a 32-bit int) and small microcontrollers.</p>
|
|
<p>If you have to send fractional values, consider their precision. Instead of sending a 12 bit ADC result as a 32-bit
|
|
float formatted like <tt class="docutils literal">0.176513671875</tt> sending <tt class="docutils literal">0x2d3</tt> and dividing by 4096 on the host might be more sensible. If you
|
|
really have to communicate big floats and you can't take the overhead of including both <a class="reference external" href="http://git.musl-libc.org/cgit/musl/tree/src/stdio/vfprintf.c">printf</a> and <a class="reference external" href="http://git.musl-libc.org/cgit/musl/tree/src/stdio/vfscanf.c">scanf</a> you can
|
|
use hexadecimal floating point, which is basically <tt class="docutils literal"><span class="pre">hex((int)foo)</span> + "." + <span class="pre">hex((int)(65536*(foo</span> - <span class="pre">(int)foo)))</span></tt> for four
|
|
digits. You can also just hex-encode the binary <a class="reference external" href="https://en.wikipedia.org/wiki/IEEE_754">IEEE-754</a> representation of the float, sending <tt class="docutils literal"><span class="pre">hex(*(int</span> <span class="pre">*)&float)</span></tt>.
|
|
Most programming languages will have a <a class="reference external" href="https://docs.python.org/3.5/library/struct.html">simple, built-in means to parse this sort of thing</a>.</p>
|
|
</div>
|
|
<div class="section" id="escape-multiline-strings">
|
|
<h4>Escape multiline strings</h4>
|
|
<p>If you have to send arbitrary strings, escape special characters. This not only has the advantage of yielding a robust
|
|
protocol: It also ensures you can actually see everything that's going on when debugging. The string <tt class="docutils literal">"\r\n"</tt> is easy to
|
|
distinguish from <tt class="docutils literal">"\n"</tt> while your terminal emulator might not care.</p>
|
|
<p>The simplest encoding to use is the C-style backslash encoding. Host-side, most languages will have a <a class="reference external" href="https://docs.python.org/3.5/library/codecs.html#text-encodings">built-in means of
|
|
escaping a string like that</a>.</p>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="encoding-binary-data">
|
|
<h3>Encoding binary data</h3>
|
|
<p>For binary data, hex and base-64 are the most common encodings. Since hex is simpler to implement I'd go with it unless
|
|
I really need the 30% bandwidth improvement base-64 brings.</p>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="binary-serial-protocols">
|
|
<h2>Binary serial protocols</h2>
|
|
<p>In contrast to anything human-readable, binary protocols are generally more bandwidth-efficient and are easier to format
|
|
and parse. However, binary protocols come with their own version of the caveats we discussed for text-based protocols.</p>
|
|
<div class="section" id="the-framing-problem-in-binary-protocols">
|
|
<h3>The framing problem in binary protocols</h3>
|
|
<p>The most basic problems with binary protocols as with text-based ones is framing, i.e. splitting up the continuous
|
|
serial data stream into discrete packets. The issue is that it is that you have to somehow mark boundaries between
|
|
frames. The simplest way would be to use some special character to delimit frames, but then any 8-bit character you
|
|
could choose could also occur within a frame.</p>
|
|
<div class="section" id="slip-ppp-like-special-character-framing">
|
|
<h4>SLIP/PPP-like special character framing</h4>
|
|
<p>Some protocols solve this problem much like we have solved it above for strings in line-based protocols, by escaping any
|
|
occurence of the special delimiter character within frames. That is, if you want to use <tt class="docutils literal">0x00</tt> as a delimiter, you would
|
|
encode a packet containing <tt class="docutils literal">0xde 0xad 0x00 0xbe 0xef</tt> as something like <tt class="docutils literal">0xde 0xad 0x01 0x02 0xbe 0xef</tt>, replacing the
|
|
null byte with a magic sequence. This framing works, but is has one critical disadvantage: The length of the resulting
|
|
escaped data is dependent on the raw data, and in the worst case twice as long. In a raw packet consisting entirely of
|
|
null bytes, every byte must be escaped with two escape bytes. This means that in this case the packet length doubles,
|
|
and in this particular case we're even less efficient than base-64.</p>
|
|
<p>Highly variable packet length is also bad since it makes it very hard to make any timing guarantees for our protocol.</p>
|
|
</div>
|
|
<div class="section" id="bit-framing">
|
|
<h4>9-bit framing</h4>
|
|
<p>A framing mode sometimes used is to configure the UARTs to transmit 9-bit characters and to use the 9th bit to designate
|
|
control characters. This works really well, and gives plenty of control characters to work with. The main problem with
|
|
this is that a 9-bit serial interface is highly nonstandard and you need UARTs on both ends that actually support this
|
|
mode. Another issue is that though more efficient than both delmitier-based and purely text-based protocols, it still
|
|
incurs an extra about 10% of bandwidth overhead. This is not a lot if all you're sending is a little command every now
|
|
and then, but if you're trying to push large amounts of data through your serial it's still bad.</p>
|
|
</div>
|
|
<div class="section" id="cobs">
|
|
<h4>COBS</h4>
|
|
<p>Given the limitations of the two above-mentioned framing formats, we really want something better. The <a class="reference external" href="https://en.wikipedia.org/wiki/Serial_Line_Internet_Protocol">Serial Line
|
|
Internet Protocol (SLIP)</a> as well as the <a class="reference external" href="https://en.wikipedia.org/wiki/Point-to-Point_Protocol">Point to Point Protocol (PPP)</a>, standardized in 1988 and 1994 respectively,
|
|
both use escape sequences. This might come as a surprise, but humanity has actually still made significant technological
|
|
progress on protocols for 8-bit serial interfaces until the turn of the millennium. In 1999, <a class="reference external" href="http://www.stuartcheshire.org/papers/COBSforToN.pdf">Consistent Overhead Byte
|
|
Stuffing (COBS)</a> (<a class="reference external" href="https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing">wiki</a>) was published by a few
|
|
researchers from Apple Computer and Stanford University. As a reaction on the bandwidth doubling problem present in
|
|
<a class="reference external" href="https://en.wikipedia.org/wiki/Point-to-Point_Protocol">PPP</a>, COBS <em>always</em> has an overhead of a single byte, no matter what or how long a packet's content is.</p>
|
|
<p>COBS uses the null byte as a delimiter interleaves all the raw packet data and a <a class="reference external" href="https://en.wikipedia.org/wiki/Run-length_encoding">run-length encoding</a> of the non-zero
|
|
portions of the raw packet. That is, it prepends the number of bytes until the first zero byte to the packet, plus one.
|
|
Then it takes all the leading non-zero bytes of the packet, unmodified. Then, it again encodes the distance from the
|
|
first zero to the second zero, plus one. And then it takes the second non-zero run of bytes unmodified. And so on. At
|
|
the end, the packet is terminated with a zero byte.</p>
|
|
<p>The result of this scheme is that the encoded packet does not contain any zero bytes, as every zero byte has been
|
|
replaced with the number of bytes until the next zero byte, plus one, and that can't be zero. Both formatter and parser
|
|
each have to keep a counter running to keep track of the distances between zero bytes. The first byte of the packet
|
|
initializes that counter and is dropped by the parser. After that, every encoded byte received results in one raw byte
|
|
parsed.</p>
|
|
<p>While this might sound more complicated than the escaping explained above, the gains in predictability and efficiency
|
|
are worth it. An implementation of encoder and decoder should each be about ten lines of C or two lines of Python. A
|
|
minor asymmetry of the protocol is that while decoding can be done in-place, encoding either needs two passes or you
|
|
need to scan forward for the next null byte.</p>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="state-machines-and-error-recovery">
|
|
<h3>State machines and error recovery</h3>
|
|
<p>In binary protocols even more than in textual ones it is tempting to build complex state machines triggering actions on
|
|
a sequence of protocol packets. Please resist that temptation. As with textual protocols keeping the protocol state to
|
|
the minimum possible allows for a self-synchronizing protocol. A serial protocol should be designed such that if due to
|
|
a dropped packet or two both ends will naturally re-synchronize within another packet or two. A simple way of doing that
|
|
is to always transmit one semantic command per packet and to design these commands in the most <a class="reference external" href="https://en.wikipedia.org/wiki/Idempotence#Computer_science_meaning">idempotent</a> way possible.
|
|
For example, when filling a framebuffer piece by piece, include the offset in each piece instead of keeping track of it
|
|
on the receiving side.</p>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="conclusion">
|
|
<h2>Conclusion</h2>
|
|
<p>Here's your five-step guide to serial bliss:</p>
|
|
<ol class="arabic simple">
|
|
<li>Unless you have super-special requirements, always use the slowest you can get away with from 9600Bd, 115200Bd or
|
|
1MBd. 8N1 framing if you're talking to anything but another microcontroller on the same board. These settings are
|
|
the most common and cover any use case. You'll inevitably have to guess these at some point in the future.</li>
|
|
<li>If you're doing something simple and speed is not a particular concern, use a human-readable text-based protocol. Use
|
|
one command/reply per line, begin each line with some sort of command word and format numbers in hexadecimal. You get
|
|
bonus points if the device replies to unknown commands with a human-readable status message and prints a brief
|
|
protocol overview on boot.</li>
|
|
<li>If you're doing something even slightly nontrivial or need moderate throughput (>1k commands per second or >20 byte of
|
|
data per command) use a COBS-based protocol. If you don't have a better idea, go for an <tt class="docutils literal">[target <span class="pre">MAC][command</span>
|
|
<span class="pre">ID][command</span> arguments]</tt> packet format for multidrop busses. For single-drop you may decide to drop the MAC address.</li>
|
|
<li>Always include some sort of "status" command that prints life stats such as VCC, temperature, serial framing errors
|
|
and uptime. You'll need some sort of ping command anyway and that one might as well do something useful.</li>
|
|
<li>If at all possible, keep your protocol context-free across packets/lines. That is, a certain command should always be
|
|
self-contained, and no command should change the meaning of the next packet or line that is sent. This is really
|
|
important to allow for self-synchronization. If you really need to break up something into multiple commands, say you
|
|
want to set a large framebuffer in pieces, do it in a <a class="reference external" href="https://en.wikipedia.org/wiki/Idempotence#Computer_science_meaning">idempotent</a> way: Instead of sending something like <tt class="docutils literal">FRAMEBUFFER
|
|
<span class="pre">INCOMING:\n[byte</span> <span class="pre">0-16]\n[byte</span> <span class="pre">17-32]\n[...]\nEND</span> OF FRAME</tt> rather send <tt class="docutils literal">FRAMEBUFFER DATA FOR OFFSET 0: [byte
|
|
<span class="pre">0-16]\nFRAMEBUFFER</span> DATA FOR OFFSET 17: [byte <span class="pre">17-32]\n[...]\nSWAP</span> BUFFERS\n</tt>.</li>
|
|
</ol>
|
|
</div>
|
|
</div>
|
|
</main>
|
|
|
|
<footer>
|
|
|
|
<script>
|
|
(function() {
|
|
function center_el(tagName) {
|
|
var tags = document.getElementsByTagName(tagName), i, tag;
|
|
for (i = 0; i < tags.length; i++) {
|
|
tag = tags[i];
|
|
var parent = tag.parentElement;
|
|
|
|
if (parent.childNodes.length === 1) {
|
|
|
|
if (parent.nodeName === 'A') {
|
|
parent = parent.parentElement;
|
|
if (parent.childNodes.length != 1) continue;
|
|
}
|
|
if (parent.nodeName === 'P') parent.style.textAlign = 'center';
|
|
}
|
|
}
|
|
}
|
|
var tagNames = ['img', 'embed', 'object'];
|
|
for (var i = 0; i < tagNames.length; i++) {
|
|
center_el(tagNames[i]);
|
|
}
|
|
})();
|
|
</script>
|
|
|
|
|
|
<div id="license-info">
|
|
©2020 by Jan Götte. This work is licensed under
|
|
<a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">CC-BY-NC-SA 4.0</a>.
|
|
</div>
|
|
<div id="imprint-info">
|
|
<a href="/imprint">Impressum und Haftungsausschluss und Datenschutzerklärung</a>.
|
|
</div>
|
|
</footer>
|
|
</body>
|
|
</html>
|
|
|