Let's step back for a moment and take a look at the big picture. In a networked application, you rarely employ just one protocol, but lots of them, possibly without even being aware of them. In order to send a bunch of data over an Ethernet cable, there needs to be a protocol that defines how bits are encoded onto carrier signals, how chunks of data are delimited, how multiple stations share the media, and so on. This is basically the realm of the guys and gals that used to have soldering irons and oscilloscopes on their desks in the good old days. At the highest level, there's the concept in the developer's head about how the application should work, and how parts of it running on different machines should interoperate with one another. At this level, it's mostly flipcharts and boxes and arrows, but it's a protocol all the same.
Inbetween these two, you have several more layers of protocols, each offering a different level of abstraction. If you ever attended a course on networking, or read a book about it, you've probably heard about the OSI model,5.2 which defines seven categories, or layers, at which networking takes place. For instance, the lowest one is called the physical layer; this is the soldering iron level. The boxes and arrows level focused on how all the parts are supposed to work together, is called the application layer.
The OSI model is quite useful in looking at different aspects of networked applications, and the security issues involved with them. Unfortunately, as with every classification system, you can't just put this bug into that category, and that one into the other one; there's always some overlap. A taxonomy of security bugs that is strictly aligned with the seven OSI layers is something that probably not even academics will find interesting.
Still, perspectives differ a lot between applications based on a middleware framework such as CORBA, where you can usually ignore all issues surrounding data representation (or at least blame your vendor for any security bugs), and applications that implement their own data representation.
Having said that, let's look at the three OSI layers that I believe are the ones most useful to our discussion.
The main concern at this level is, what's the security impact of an attacker watching your network traffic, or injecting packets into it. These activities are colloquially referred to as snooping and spoofing.
For instance, if your application processes information that's confidential (think of home banking), then of course you'd be concerned about attackers learning your credit card number, or changing the recipient account number of a bank transaction you're making remotely.
On the other hand, if you're programming a distributed shoot'em-up game, you're probably not too worried about these things.
The presentation layers is concerned with how to package your data when sending it across the network. In particular, this level tries to hide peculiarities of the host system such as integer byte ordering, native character encodings, etc.
There are several protocols such as SMTP, the Simple Mail Transfer Protocol, which transmit everything as one or more lines of ASCII encoded text. Others, such as NFS, rely on Sun's Remote Procedure Call (RPC) protocol, which has a fairly complex binary data presentatation called XDR, the External Data Presentation.
Security consideration at this layer usually revolve around questions of correctly decoding data received from the network, and converting it to the application level format.
For instance, a network protocol may specify that a certain string contained in a packet should be delimited by the NUL character and not exceed a certain length. A common mistake is to just assume that the client conforms to this specification, and blindly copy this string to a local buffer using strcpy. If the client doesn't conform, e.g. because it's a mean, nasty and ugly cracker, the packet might well contain a sequence of ASCII characters considerably longer than what the protocol allows, causing your application to overflow the buffer (more on this below, including a real life example).
For instance, in the example of the finger.cgi script mentioned priviously, the application protocol could be expressed as ``client sends login name as argument, and server responds with the output of the Unix finger command for this user.''
Note that there is no reference to the underlying mechanisms here, such as the HTTP protocol, and the CGI convention. And in fact, the bug discussed above (passing the unchecked user name argument to the shell for evaluation) is equally deadly in an application built around CORBA or SOAP as it is in a CGI script.
To sum it up, the main security concern at the application layer is to make sure that the data received conforms to implicit protocol constraints, and does not have unintended side effects when used.
Throughout the following, I will try to follow this classification to a certain degree, in order to help you decide which things to worry about when writing networked applications, and which not.