We have been running a wireless network throughout the Karekare area (or more correctly the Waikarekare area), starting with an experimental network in 2004. The terrain is hilly and bush clad (Aerial Photo with network sites overlayed).
Step one was finding the antenna sites that maximise coverage and are handy to power sources. That involved lots of climbing on roofs with a laptop with an external antenna. The rule, "If you can see the remote site, everything will work", turned out to be mostly true. The reverse, "If you can't see it, it will not work", has always been true (except some trees are relatively transparent to 2.4GHz, while other are not. Damp ones are always a problem, unless the two routers are very close). The exceptions were sometimes baffling. We have had examples of sites with a perfectly clear path between the two wireless routers, and reasonably close at that, but the links have been marginal at best. Other, seemingly less optimal sites, have worked far better. We have put some of this down to reflections from the terrain, causing signal cancellation, and/or added noise. Some of it, we have put down to signal splatter off of trees off of the signal path. Large pine trees near the signal path, with needles about the length of an antenna, look to be adding to the received noise level. We assume that this is a similar effect to shining a light on to a tree full of tinsel. Raising the transmitter power, just increased the noise level.
We experimented with many antenna types, and still are (see Antenna ). We settled on single and double sided slotted wave guides as the antenna for our core network sites as they are high gain, have a low wind profile, good signal pattern, and are relatively easy to make. We have used bicircle antenna at the client end. These are small enough to put into the router enclosure, easy to make, and also have a good gain. A variant, with a small parabolic reflector, rather than a flat one, has worked well for long distance links. We are looking at circularly polarised antenna to see if these will help in some of the marginal signal areas, where reflections seem to be a problem.
We started by using LinkSys WRT54G and WRT54GS basestations (see router build), both in the core network and at client sites, with OpenWRT firmware using routed WDS links between the base stations. We have switched to Ubiquity Nanostations, as they are ready made for external use and have built in high gain antenna. These have proved very reliable. The core network backbone is now using Nanostation M5's, the newer clients use Nanostation 2's at the client end and we still have WRT54-GL's as distribution nodes for the clients to connect to.
The core was built with nodes positioned to cover the three main housing areas. There is a fourth central node, used as both a relay point, and a distribution site. These three are essentially isolated from one another by the terrain. We get about 90% coverage, but we need further hops to reach all houses. The central node, at Te Ahuahu, is the connection point to the Internet. No other site is close enough to the exchange, to fill this roll. A satellite link was looked into, but the costs were excessive compared to ADSL or Frame Relay offerings.
We have found that using a single router at a relay point, to receive, then send on the signal, introduces quite a large latency into the transmission. The resulting data throughput reducing to about a third. This is due to the half duplex nature of 802.11. The sending router needs to negotiate access to the relay, then send the packet. The relay needs to buffer the entire packet, negotiate with the destination router, then send on the packet. This buffering step, at least halves the throughput rate. The same is true for the acknowledgements being sent back. Hence the overall throughput is much lower than a half. Some of this additional delay is probably due to establishing the links. As a result of this, we are changing all the key relays to be two routers back to back. Where we have tried this, it has worked well. It has the added advantage of being able to use multiple channels, to reduce contention.
The current network is using OSPF routing to isolate each of the home networks from the wireless network. This will allow multiple paths to be constructed through the core. Each wireless link is on its own 255.255.255.252 subnet (i.e. only two IP addresses). Thus confining local site traffic to the site, and only sends wireless traffic across the links it needs to traverse (rather than broadcasting all traffic everywhere). We also split the wireless Access Point network from the local lan (wired network ports), also to confine to local traffic to the local wire (and thus not having to have the wireless side using bandwidth retransmitting it). OSPF has proved to be a little unstable in high traffic conditions, where the routing time outs come into affect because the link is two busy to send the keep alive packets.
The network has recently been divided into 4 areas, with client routers in each area, being allocated IP addresses in a fixed range for that area. This allows us to put in consolidated static routes in the core routers, that mean that we don't have to worry so much about routes bouncing in and out. There will always be a route to the local distribution site. At the client nodes, we now put in a default route pointing to the local backbone site. This gives the client nodes a starting point, that doesn't rely on OSPF. In most cases, it is the only route that they need to know. The backbone's distribution routers still relies on the client OSPF packets to know that the client is still alive, though traffic from a client will also get it put into the routing tables.
We also ping all nodes, every few seconds, from a central server. This is for our own network diagnostics, but has the benefit of ensuring that client routes are kept alive in the backbone.
We have recently found a way to further boost performance and reliability. We now run each routers local wifi , and the 100M switch ports, on different networks. This is in addition to running each WDS link on its own subnet. Before we did this, all local client traffic, was also being sent out on the local wifi interface, as the hardwired and wifi nets were bridged. By routing, we separate these two interfaces, and this leaves the wifi transmitter free for link traffic.
The external links are both behind a openBSD pf firewall. We also run an email relay, a softflowd (modified to record upstream adsl routers AS (fake) numbers) with netflow flow-tools monitoring suite (graphed with gnuplot), an ntp time service, this web site, and the domain name server.
