pyngrok – a Python wrapper for ngrok

pyngrok is a Python wrapper for ngrok that manages its own binary and puts it on our path, making ngrok readily available from anywhere on the command line and via a convenient Python API.

install

pyngrok is available on PyPI and can be installed using pip.

pip install pyngrok

That’s it! pyngrok is now available as a package to our Python projects, and ngrok is now available from the command line.

open a tunnel

To open a tunnel, use the connect() method, which returns the public URL generated by ngrok.

from pyngrok import ngrok

# Open a tunnel on the default port 80
public_url = ngrok.connect()

The connect() method can also take an options parameter, which allows us to pass additional options that are supported by ngrok.

get active tunnels

It can be useful to ask the ngrok client what tunnels are currently open. This can be accomplished with the get_tunnels() method, which returns a list of NgrokTunnel objects.

from pyngrok import ngrok

tunnels = ngrok.get_tunnels()
# A public ngrok URL that tunnels to port 80 (ex. http://<public_sub>.ngrok.io)
public_url = tunnels[0].public_url

closing a tunnel

All open tunnels will automatically be closed when the Python process terminates, but we can also close them manually.

from pyngrok import ngrok

public_url = "http://<public_sub>.ngrok.io"

ngrok.disconnect(public_url)

the ngrok process

Opening a tunnel will start the ngrok process. This process will remain alive, and the tunnels open, until ngrok.kill() is invoked, or until the Python process terminates.

If we are building a short-lived app, for instance a CLI, we may want to block on the ngrok process so tunnels stay open until the user intervenes. We can do that by accessing the NgrokProcess.

from pyngrok import ngrok

ngrok_process = ngrok.get_ngrok_process()
# Block until CTRL-C or some other terminating event
ngrok_process.process.wait()

The NgrokProcess also contains an api_url variable, usually initialized to http://127.0.0.1:4040, from which we can access the ngrok client API.

If some feature we need is not available in this package, the client API is accessible to us via the api_request() method. Additionally, the NgrokTunnel objects expose a uri variable, which contains the relative path used to manipulate that resource against the client API. This package also gives us access to ngrok from the command line, as shown below.

other useful configuration

authtoken

Running ngrok with an auth token enables additional features available on our account (for instance, the ability to open more tunnels concurrently). We can obtain our auth token from the ngrok dashboard and install it like this:

from pyngrok import ngrok

ngrok.set_auth_token("<NGROK_AUTH_TOKEN>")
# Once an auth token is set, we are able to open multiple tunnels at the same time
ngrok.connect()
ngrok.connect(8000)

This will set the auth token in the config file. We can also set it in a one-off fashion by setting it for the “auth” key of the options parameter passed to connect().

config file

The default ngrok config file lives in the home directory’s .ngrok2 folder. We can change this in one of two ways. Either pass the config_path parameter to methods:

from pyngrok import ngrok

CONFIG_PATH = "/opt/ngrok/config.yml"

ngrok.connect(config_path=CONFIG_PATH)

or override the DEFAULT_CONFIG_PATH variable:

from pyngrok import ngrok

ngrok.DEFAULT_CONFIG_PATH = "/opt/ngrok/config.yml"

ngrok.set_auth_token("<NGROK_AUTH_TOKEN>")

binary path

The pyngrok package manages its own ngrok binary. However, we can use our ngrok binary if we want in one of two ways. Either pass the ngrok_path parameter to methods:

from pyngrok import ngrok

NGROK_PATH = "/usr/local/bin/ngrok"

ngrok.get_tunnels(ngrok_path=NGROK_PATH)

or override the DEFAULT_NGROK_PATH variable:

from pyngrok import ngrok

ngrok.DEFAULT_NGROK_PATH = "/usr/local/bin/ngrok"

ngrok.connect()

command line usage

This package puts the default ngrok binary on our path, so all features of ngrok are also available on the command line.

ngrok http 80

For details on how to fully leverage command line usage, see ngrok’s official documentation.

contributing

Clone the pyngrok repository and contribute on GitHub. If you find issues, report them on GitHub. Pull requests for fixes and features are also warmly welcomed.

Twilio-Powered Air Quality Texting Service

With wildfire season upon us, use this handy texting tool to find out what the air quality is in your area. Simply text your zip code to (415) 212-4229 for air quality updates. You can also add “map” to the text to be sent an image of your region.

This service isn’t just useful for individuals with limited access to smartphones or the Internet. It also alleviates the load put on air quality sites like AirNow, which are often overloaded and unavailable during wildfire season due to the spike in traffic. Texting this number instead is a great way to get the same information without bogging down those sites, helping them to stay up for others who need to access them.

Spread the word and stay safe! If you’re interested in nerding out over the code, feel free to check it out on GitHub, and if you have questions or comments, tweet @alexdlaird to let him know.

Alex and Jess Are Raising Our Future

“Talk is cheap.” That’s what we say. And, to a degree, it’s true. But bear this in mind: all action is precipitated by talk. People will often try to silence your voice expressly for that reason — because they know it will lead to action.

In the age where hating on millenials is trendy, dismissing the value of social media is equally in vogue — there’s a correlation there, but that’s another post for another time. But like any form of communication, it has its pros and cons, and you get out of it what you put into it.

This is what we put into it. A group of like minded and motivated parents from all across the country banded together using social media to encourage, educate, and challenge each other (and our peers) ideologically and politically while raising the level of discourse. You’ve seen us posting since well before the election, and you’ve seen us continue to join our collective voices as concerned citizens and parents since.

Today, we launched our next initiative: a PAC (Political Action Committee). Raising Our Future is focused on funding federal candidates who are fighting to make a better world for our children on issues of educational equity, social justice, and family planning.

We just launched yesterday. In one day, we raised over $15,000 and took our new Facebook page to 1,300 members. We shared the stories of dozens of founding members throughout the day (we’ll continue this in the days and weeks to come), each post reaching, on average, 2,500 people.

“Stop talking and do something about it.”

We did. We have. We are. We are a force to be reckoned with. Join us.

#RaisingOurFuture #ROFPAC #WeStartedAPAC

Like Us on Facebook: https://www.facebook.com/ROFPAC
Follow Us on Twitter: https://twitter.com/rof_pac
Our Story: https://www.raisingourfuture.org/2017/07/25/alex-and-jess-are-raising-our-future

DONATE: https://www.raisingourfuture.org/donate

Django Dropzone Uploader

Ever been on a trip and, upon return, needed a quick and easy way for all your friends to send you their pictures and videos without burning CDs, sending massive emails, or using third-party services? Or, maybe a better question, ever wondered how to construct a basic Django application with Amazon’s web services, for instance S3?

Look no further. Below is the basic code for a drag-and-drop Django web application that allows users to upload files directly to an Amazon S3 bucket.

Deployment Setup

The code for this project can be found on GitHub.

You’ll need the following installed before cloning or forking the source code:

This project will write to an Amazon Web Services (AWS) S3 storage bucket, so it’s assumed you have an AWS account. If not, create one. S3 is a storage platform from Amazon, and EC2 allows you to spin up virtual servers, which you can use to host this project. If you’re new to AWS, Amazon will likely give you the first year of their smallest EC2 instance free.

This project also includes a deployment script, which allows you to easily deploy the project from your local computer to your server.

Here’s what you need to setup in AWS to ensure your account is ready to receive a deployment of this project:

  • Launch an EC2 instance running Ubuntu Server (or some other Debian-based operating system)
  • Save the .pem key pair file for the EC2 instance as ~/.ssh/myserver.pem
  • Create an EC2 Security Group that has port 80 opened
  • Create an S3 bucket
  • Generate an AWS Access Key and Secret Access Key
  • (Optional) Create an elastic IP and associate it with the EC2 instace you created
  • (Optional) Create a DNS entry of your choosing to point to the elastic IP (AWS will generate their own DNS entry that you can also use, if you don’t have your own domain name)

Fork the Code

Now you’re ready to clone, configure, and deploy the code to your EC2 server.

  • Fork the repository on GitHub
  • Clone your forked repository
  • Modify the variables at the bottom of djangodropzonetos3/settings.py to customize the application
  • You must specify valid values for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_STORAGE_BUCKET_NAME in settings.py
  • Modify the HOSTNAME variables at the top of fabfile.py to point to your EC2 instance’s DNS entry
  • Modify the REPO_URL variable at the top of fabfile.py to point to your fork of the repository

Deploy

The fabfile.py in the repository will take care of setting up the environment for you, including installing and configuring a web server. Isn’t that handy? So you’re ready to deploy by doing the following:

  • From the Command Line at the root of the cloned source, execute “pip install -r reqs.txt”
  • From the Command Line at the root of the cloned source, execute “fab deploy”

That’s it. If this deployment is successful, you should be able to navigate to the hostname for your server in a web browser, drop and save the files, and see them stored in your S3 bucket.

Now, start poking around in the code to learn the ease and awesomeness of Django and how this was accomplished! Leave your thoughts in the comments section below!

DD-WRT NAT Loopback Issue

NAT loopback is what your router performs when you try to access your external IP address from within your LAN. For instance, say your router forwards port 80 to a web server on your LAN. From an outside network, you could simply visit your external IP address from a browser to access the web server. Internally, if NAT loopback is disabled or blocked, you would not be able to access this the same way.

There are any number of valid reasons why you’d want to allow NAT loopback on your network. If you’re like me, you simply want internal and external access to operate in the same way. NAT loopback is needed to accomplish this, and it is simple and safe. Don’t be fooled by the plethora of forum posts crying that NAT loopback is disabled on routers purposefully, that it opens up dangerous security holes, or that it will destroy your network and ultimately your livelihood as you know it. Like the vast majority of scare tactic-based content on the internet, it’s false. Your router will not stab you in your sleep if you allow NAT loopback … although it may emit higher levels of radiation, lace your lipstick and food with carcinogens (compliments of the government, of course), and kill Brad Pitt. Again. Coincidentally, the posts never specify why the claims might be true, lack credible sources, and are rarely found outside of back alley forums. We’re still talking about NAT loopback, right? The internet has made us so gullible …

The primary reason for the security concern is that some consumer routers appear to intentionally disable NAT loopback by default, and there is no way around this with stock firmware. However, this is not an intentional barrier, it’s just a constraint of limited stock firmware. Nothing new there. The simplest solution to this is, as usual, to flash DD-WRT to your router. Then, follow this tutorial to allow NAT loopback.

Implementation

Before proceeding, ensure NAT loopback actually doesn’t work with your version of DD-WRT. Different versions of DD-WRT implement NAT with slight variances, so it’s possible your version of DD-WRT may not actually need the special rules below.

To check if NAT loopback is working on your router, you’ll need your external IP address. If you don’t know your external IP address, just Google “what is my ip”. Now, open a Command Prompt and ping your external IP address. If the command times out, NAT loopback is not working.

In the DD-WRT Control Panel, navigate to the “Administration” tab and click on “Commands”. Add the following rules, then click “Save Firewall” to ensure the rules execute even after the router is rebooted.

insmod ipt_mark
insmod xt_mark
iptables -t mangle -A PREROUTING -i ! `get_wanface` -d `nvram get wan_ipaddr` \\
-j MARK --set-mark 0xd001
iptables -t nat -A POSTROUTING -m mark --mark 0xd001 -j MASQUERADE

Conclusion

That’s it! Now, try pinging your external IP again from the Command Line. This time you should receive packets.

DD-WRT is a always evolving. The developers have stated that they aren’t planning on fixing this issue, but if this procedure doesn’t work for you, leave a comment below and I’ll check to see if something has changed in the latest version of DD-WRT. I’ll try to always keep the tutorial updated with instructions for the latest DD-WRT build.

Also, if you previously followed my DD-WRT Guest Wireless tutorial, this fix should work for both interfaces.

DD-WRT Guest Wireless

If you’ve done any amount of work with routers, you know that it doesn’t take long to start craving consistency. And more advanced functionality that the cheap home interfaces simply don’t grant you. This is the point where you usually break down and start research things like Tomato, OpenWrt, and DD-WRT, just to name a few of the more popular alternatives.

These alternate firmwares don’t just provide a consistent administrative experience across all compatible models and brands, they also turn a cheap home router into a flexible and competitive enterprise router.

My Setup

DD-WRT is my personal firmware of choice. Powerful, flexible, and stable. One thing that I demand in a router is the ability to broadcast a secondary SSID for my guest’s to be able to access wireless internet in my home without also having access to my entire network of computers and devices.

Gladly, because my router’s firmware was extremely slow and buggy, I flashed my Cisco E2500 router with “mini” DD-WRT firmware (the E2500 also supports the “big” firmware). But after reviewing getting the two wireless networks setup on my router, it was brought to my attention that there are no good tutorials for how exactly you are to do this using DD-WRT. The tutorial provided on their own website, in fact, does not work. So, I find that it falls upon me to put out my particular configuration for two mutually exclusive wireless networks from a single router, both networks having access to the WAN port (that is, internet access). There are, of course, multiple ways to do this. Feel free to leave alternative suggestions in the comments.

Create Two Wireless Networks

First, create your wireless networks by clicking clicking on “Wireless” and then “Basic Settings”. We’ll setup security in a moment. After you’ve configured your private wireless network setup, click “Add” under “Virtual Interfaces” to add the “wl0.1 SSID”. Give your guest network a separate SSID, and select “Enable” for “AP Isolation”.

Now click “Save” and “Apply Settings”.

ssid

Setup Wireless Security

Navigate over to the “Wireless Security” tab. After you’ve setup the wireless security for your private network, setup similar security for your guest SSID. I would advise against leaving your guest wireless completely open, but since you’re going to be giving out this password to your guests, it should probably be a little simpler than your private network’s key.

Now click “Save” and “Apply Settings”.

security

Create Bridge

At this point, you have two wireless networks broadcasting on two separate SSIDs. Both networks should have internet access, but you’ll also notice both networks dish out IPs in the same subnet, and both networks are clearly able to see each other. While you may like and trust your guests, that doesn’t mean you necessarily want them to have access to all your network devices. To separate the network routing, we need to create a bridge and place the guest network into a different subnet.

Click on “Setup” and then on the “Networking” tab. Under “Create Bridge” click “Add” to add a new bridge. Give the bridge a name, and modify the IP address of the bridge to be in a different subnet than your private network. For example, my private network grants IPs in the subnet 192.168.1.0/24, so my guest network in the image below is setup to grant IPs in the subnet 192.168.2.0/24.

Now click “Save” and “Apply Settings”. Though the page may refresh right away, you may need to wait about a minute before the bridge is available to use in the next few steps.

create-bridge

Assign Guest Network to Bridge

Under “Assign to Bridge” click “Add”. Select the new bridge you’ve created from the first drop-down, and pair it with the “wl0.1” interface.

Now click “Save” and “Apply Settings”.

assign-bridge

Create DHCP Server for Guest Network

We’re almost there! We’ve created a bridge in an alternate subnet, but the alternate subnet doesn’t have a DHCP server, so our guests currently cannot access the guest SSID (unless they assign themselves a static IP). Scroll to the bottom of the “Networking” page and under “Multiple DHCP Server” click “Add”. Ensure your newly created bridge name is selected from the first drop-down menu.

Now click “Save” and “Apply Settings”. Congratulations, we now have a working, separate guest network! Unfortunately, while users can connect to the network and DHCP is running, guest users aren’t able to access the internet quite yet.

bridge-dhcp

Create Firewall Rules for Guest Network

Navigate to the “Administration” tab and click on “Commands”. We need to add three rules to our firewall settings before our private network is completely secure and our guest network has internet access. Add these three rules (one per line) to the “Commands” text field, then click “Save Firewall” to ensure the rules execute even after the router is rebooted.

iptables -t nat -I POSTROUTING -o `get_wanface` -j SNAT --to `nvram get wan_ipaddr`
iptables -I FORWARD -i br1 -m state --state NEW -j ACCEPT
iptables -I FORWARD -i br1 -o br0 -m state --state NEW -j DROP

firewall

Improve Guest Security

Pete Runyan commented with a few more ways to nail down the security of the guest network. For one, your guests likely assume that their device on the guest network is not accessible from other devices on the same network, so you’ll want to add the firewall rules below to make that true. It’s also probably unnecessary (depending on your needs) to allow users on the guest network SSH, Telnet, or GUI access to the router. Append these firewall rules to harden the security of all of your networks!

iptables -I FORWARD -i br0 -o br1 -m state --state NEW -j DROP
iptables -I INPUT -i br1 -p tcp --dport telnet -j REJECT --reject-with tcp-reset
iptables -I INPUT -i br1 -p tcp --dport ssh -j REJECT --reject-with tcp-reset
iptables -I INPUT -i br1 -p tcp --dport www -j REJECT --reject-with tcp-reset
iptables -I INPUT -i br1 -p tcp --dport https -j REJECT --reject-with tcp-reset

Conclusion

You should now have two working SSIDs: a private one for your home network, and a guest network for your visitors. Both networks should have internet access. The private network will function the same as a LAN and single wireless network did before, with the wireless network having full access to the LAN connections. The guest network, on the other hand, is separated from the private network. Additionally, each individual device on the guest network is separate from another, so guests cannot see each other.

If you’ve gotten to this point and something is not working, or your guest network does not have internet access, don’t be alarmed. DD-WRT is a always evolving, and it’s entirely possible bridge settings or firewall rules for the latest build have changed. If this tutorial does not produce the desired result, please leave a comment below. I’ll try to always keep the tutorial updated with instructions for the latest DD-WRT build.

Geocentral Location; Addresses to Coordinates

Recently, I needed to plot numerous addresses on a map and, ultimately, find the geocentral location of all addresses. The geocentral location is the weighted center of all the addresses, which can be useful in helping determine numerous things, including the average distance between all addresses and some other location.

The geocentral location is attained through relatively simple vector math. Let’s say, for instance, you have a set of points on a graph. Adding each point together would give you the weighted center of all the points, which can help you determine quite a bit about how that population of points interacts with you or each other.

I’ve put together a simple script below that interacts with Google Maps to do just that. Input a list of addresses in the text box below, attain the coordinates for each address, and plot each address, and the address’ geocentral location, on the map below.

 

A few things to keep in mind:

  • One address per line
  • Addresses must be properly formatted
  • Ensure no address lines are blank
  • The geocentral location is marked with a blue flag
  • In order to keep strain on my server low, the tool below only allows 150 or less addresses to be processed. The source is available on GitHub here, so you’re welcome to modify the tool for use on your own server.

 

 

 

If you try unable to use this tool, either an address is malformed or Google has changed a part of their Maps API. If you’re certain that all of your addresses are properly formatted and the tool still does not appear to work for you, send me an email so I can update check if Google has updated their Maps API.

A Correction for the WSJ: So, Who Did Invent the Internet?

Recently, Gordon Crovitz wrote an opinion piece for the Wall Street Journal titled Who Really Invented the Internet? Fortunately, it’s only an opinion piece, because there was little more than opinion, littered with plenty of misinformation, in the writing. You can read the article here.

Now, it’s not like I look to the WSJ for the latest technology information (or, in this case, technology history). Far from it. And generally when a here’s-the-truth-you-never-knew article starts with political propaganda, it’s pretty safe to assume that whatever comes next is going to be absurd. The article’s introduction could essentially be summarized as, “Obama said something that was true, but I’ll be darned if I can’t find a way to make it sound false!”

Even still, to those of us in the technology field, the “first computer” and “who invented the internet” discussions are highly revered and hotly debated, so when someone not in the industry starts boasting that they have a complete and final answer to these discussions, we usually just scoff. In Crovitz’s defense, he seems to be confusing “internet” with “World Wide Web” and many other terms that merely relate to networking and computers. But that’s about the extent I’d go to to defend him; he’s a conservative author trying to make something out of nothing just because a liberal said it.

 

Due to the fact that I’m more than a little OCD, I wound up relating the history of internet technology through the ages to my Grandpa, who originally sent me the Crovitz article. Much of the details below are in response to specific parts of Crovitz’s article, so, as painful as it may be, I recommend you read that article first. Alright, ready? Begin.

 

Personal Computer: The term “personal computer” was not coined until 1975 for the Altair 8800. However, it is highly disputed that Xerox created the first “personal computer”, by whatever modern definition you use. IBM created the first electronic computer in 1953 (the IBM 701), Digital Equipment Corporation created the first digital computer in 1960, and Hewlett-Packard release the first mass-produced digital computer in 1968, the HP 9100A.

Personal Workstation: This is the term the WSJ author is looking for in their article. The first personal workstation, a “workstation” being a computer that can be connected to another computer (in this case, through the Ethernet technology he referenced), was created by Xerox in 1974. However, the computers used by ARPANet were technically also workstations, just not mass produced.

Intranet (take special note of the “a”): A connection between two or more computers within the same network. The network in your house is an “intranet”.

Internet (take special note of the “e”): A connection between two or more networks. The wires that connect your house’s network to mine are the “internet”.

ARPANet: The first computer network (or “intranet”), created by the Department of Defense, which was fully implemented in 1969. I’ve never heard it associated with nuclear strikes or anything of the sort. It was created merely to replace slow and overused satellite communication between government agencies. When originally created, it did not use TCP/IP, it used NCP.

DNS: DNS stands for “Domain Name System”. It’s interesting that, for an article claiming Ethernet was more defining to the internet than TCP/IP, the article makes no mention of DNS, the third essential component to the modern internet. Though you type in “google.com” to get to Google, Google’s website actually lives at an Internet Protocl (IP) address of 173.194.34.165. This IP address is similar to your human’s street address. People cannot be expected to remember an IP addresses for their favorite websites, so DNS was invented to resolve a host name (google.com) to an IP address. This is similar to me saying “Ben and Jerry’s on Navy Pier” instead of “Ben & Jerry’s – NAVY PIER, 700 East Grand Ave., Chicago, IL 60611-3436”.

RFC: RFC stands for “Request for Comment”. The article does not mention these, but they are crucial to understand when things were adopted. They’re sort of like the Congressional bills of the technology world, but more well-defined. RFC documents are official definitions of technological protocols or interfaces. When something is adopted as a standard, an RFC fully defining it is written, and, if other people want to interface with it, they use that “law” to know how things work. The very first RFC, RFC 1, was called “Host Software” and dictated the infrastructure of ARPANet. RFC 791 was for TCP/IP 1981. RFC 894 was for Ethernet in 1984. RFC 1035 was for DNS in 1987. These dates do not necessarily correspond to when the interfaces were created, but they do indicate when the interfaces were standardized and/or adopted.

World Wide Web: The World Wide Web was formally introduced in 1989. The World Wide Web is, in very loose terms, the combination of HTTP, HTML, and database communication that transfers web content by a standardized means to a web browser.

 

Difference Between Intranet and Internet

So, what is the difference between the an “intranet” and the “internet”. First of all, the foundational structures of the “internet” are identical to the “intranet” (that being TCP/IP referenced in the article). Once there was the possibility for the intranet, the possibility for the internet also existed, but it was not realized until a bit later, which is why Xerox is trying to claim credit for that. It’s a chicken-or-the-egg argument. Naturally, each company (and the Pentagon) claim different loose definitions of all these terms so that they can claim credit for actually inventing the end result. The fact is, none and all of them invented it … which coincides with Obama’s remarks pretty well, if you ask me.

 

TCP/IP and Ethernet

First of all, it’s sad that the article references Vinton Cerf but makes no mention of Bob Kahn. They collaborated together to define TCP/IP, but Kahn rarely gets the credit he deserves. Kahn was actually the one with the idea of TCP/IP, and Cerf was in charge of the implementation and later the RFC definition.

Secondly, it should be highly suspect that much of the WSJ author’s claims come from a book written about Xerox. More significantly, after the WSJ article was published, the author of the cited book released a statement refuting the article and saying the article misrepresented the content of his book.

Naturally, Xerox will claim “full credit” for a discovery, as many other companies have done as well, but they cannot claim full credit as they utilized standards that had already been put in place (namely TCP/IP). However, their contribution to the internet’s development was equally strong. Ethernet was merely a communication standard that allowed passing data (at very high speeds) between two computers using TCP/IP. Neither technology would ever have been adopted by the private sector (and ultimately the world) without something like …

 

DNS

The Domain Name System was invented in 1983, and the internet would not exist without it, just like TCP/IP and Ethernet. I won’t go into details of why it was necessary, but it was created when issues were seen in how hosts were resolved with ARPANet. It was obvious that as ARPANet got larger, the way hosts were resolved (me asking, “Hey, what’s Mom and Dad’s address?) would become weaker and weaker (and certainly slower and slower). So they decentralized their host resolution to several Domain Name Systems rather than a centralized location at the Pentagon. This was essentially the birth of the privatized internet, as we know it, but that is not to discredit its foundations.

 

So Did Xerox Invent the Internet or Not?

Short answer? No. Xerox has never been one of the discussion points in the “who invented the internet” within knowledgeable circles.

Long answer? It’s a bit arrogant for Xerox (or any one company or government organization) to accept or take full or even majority credit for the invention of the modern day internet. It was a combined effort of multiple unrelated parties, companies, and government entities. People usually credit the Department of Defense with the creation of the internet because, well, they created the first internet. And without the funding and research for TCP/IP, the advancement toward what we have today would have been much slower (assuming it ever happened at all). Additionally, though Xerox coupled TCP/IP with their own technology to make Ethernet, they did not use Ethernet on the internet. They used it on their own intranet, or internal network, because at the time only government organizations had access to the internet. More importantly, TCP/IP and other internet protocols could exist outside of an internal network, which is where Ethernet is used. Ethernet is used to join computers to an intranet, not to join networks to the intranet. Xerox’s contribution certainly increased the speed and reliability of internal network communication, but that is an indirect contribution to the internet. It is not an essential part of the components that makeup the internet.

 

What About the Privatization of the Internet?

The reason the internet became privatized had little to do with little government/big government politics, as the WSJ implies, and everything to do with decentralization. The fundamental structure and combination of TCP/IP and network-to-network communications led to DNS, and once DNS was introduced it became obvious that the internet was going to become a worldwide tool that could not be contained or centralized by any one government or entity. Ironically, however, the U.S. government did still control the all DNS servers, and government organizations were the only ones with access to the internet.

Though Xerox enabled reliable intranet communications with Ethernet (which, by the way, was given back to the government for their use primarily), ARPANet expanded to become the internet, and DNS offered the potential to use the service around the globe, it was not commercialized. It was not until 1992 when Congress passed a bill (spearheaded by Al Gore, which is usually why people misquote him to make the joke in which he claims to invent the internet) that allowed commercial access to the internet. This began the privatization of the internet, but the government still controlled all DNS servers.

For six more years the internet was essentially still controlled by the U.S. government, but commercial entities were allowed to use it. In 1998 (not sure what event the article is referring to when it says 1995), the Clinton administration issued a mandate to form a non-profit organization called the International Corporation of Assigned Names and Numbers (ICANN). The U.S. government gave control of all DNS servers, maintenance, and documentation of internet infrastructure to ICANN. And you thought Google owned the internet. At that point, the internet became officially and completely privatized.

 

Doesn’t Britain Claim They Invented the Internet?

Actually, no. If you watched the Olympics 2012 Opening Ceremonies, Tim Berners-Lee was paraded through the stadium and loudly proclaimed as the “inventor of the World Wide Web”. And there’s the distinction. London never claimed he invented the “internet”. There is a difference. The “internet” and the “World Wide Web” are two distinct things, though they obviously operate together and are essentially synonymous to the average internet user today.

In 1989, Tim Berners-Lee had an idea for a database of hypertext links. Berners-Lee implemented what he called the World Wide Web with the collaborative help of Robert Cailliau. It didn’t take long for the two of them to realize the potential the World Wide Web could offer to the internet, so in late 1990 Berners-Lee developed the protocol necessary to transmit World Wide Web data across the internet: HyperText Transfer Protocol (HTTP) and HyperText Markup Language (HTML). Along with this, he developed the first web browser, which he called simply the WorldWideWeb. Joining HTTP, HTML, and a browser with the internet gave Berners-Lee the ability to pass much more valuable data from point to point, displaying that data in a specifically intended way to the end-user.

In regards to the WSJ article, it’s also possible that the author of the WSJ was confusing the term “internet” with “World Wide Web”. By 1994, better graphical browsers had been created, and the World Wide Web standard had pretty well been adopted, but primarily only by universities and research labs. In late 1994, Berners-Lee founded the World Wide Web Consortium (W3C), which maintains many of the standards for the World Wide Web still today. After W3C was founded, and in early 1995, the potential the World Wide Web coupled with the internet had to offer the commercial world became apparent, and the internet really started taking off.

 

Conclusion

Even still, the Department of Defense, Vinton Cerf, and Bob Kahn do deserve full credit for the creation of the first intranet/network and the initial ideas for networking protocols. The natural successor to that was Ethernet, DNS, and ultimately a privatized and distributed internet as we know it today.

Here’s a more simple example to help with the comparison. Assume for a moment that, prior to Henry Ford, nobody had ever done anything with a vehicle that moved (without assistance from an outside force) from point A to point B. Ford created the Quadricycle as his first vehicle. He then adapted that into the Model T. Is the Model T any more or less of a vehicle? It has more of the parts that we’re used to today, and it was certainly much more luxurious. But to say then that, because the Model T is more like what we have today, the Quadricycle was not a vehicle is silly. The Quadricycle was still a vehicle that moved you from point A to point B. The Model T was the natural successor to that, and cars have progressively become more and more advanced (with newly invented technology added to them) as society has advanced.

In the same way, ARPANet moved network information from point A to point B. The internet was the natural successor to an intranet, but the same ideas and fundamental technology were used for it, so it is safe to say that the government formed what has become the internet. Which, I believe, was President Obama’s point. No argument here that the internet boomed came in 1998 when it was fully privatized, but the internet also would not have been established in the first place without government research and funding.

The Napster Revolution

I’ve recently been reading through Steve Jobs’ biography, a phenomenal work by Walter Isaacson. A point that Isaacson keeps coming back to throughout the book is that Steve Jobs revolutionized six different industries: animated movies (through Pixar), personal computing, tablet computing, phones, digital publishing, and music.

I don’t disagree with Isaacson. Jobs did revolutionize the way that digital media (including music, movies, books, and more) is marketed and sold today. But before you can have the corner on the market, there needs to be demand. And the revolution that realized the screaming demand for easily accessible digital media around the globe started in a college dorm room during the summer of 1999.

 

The Beginning

It started with two adolescents, Shawn Fanning and Sean Parker, who shared mutual interest in hacking and programming. Though the teenagers never met at this point, they continued to chat over IRC in the years to come, bouncing various software ideas off each other.

During his Freshman year of college at Northeastern University in Boston, Fanning had an idea to simplify online music acquisition for him and his roommate. It was 1998, and the easiest way to download MP3s was through various websites. Each website had a different interface. Each a different library of music. Many broken links. All were very slow.

Fanning wrote a piece of software that fixed this. It provided a single, clean interface that searched all the major MP3 websites, providing results only for working links. It was effective. But it still wasn’t a comprehensive library.

By the end of his Freshman year, Fanning had dropped out of college and was mulling over ideas for a music sharing program that didn’t rely on limited libraries and websites that were taken down and relaunched on a weekly basis. He worked out the good and bad ideas for such a program with his internet buddy, Parker, over IRC, slowly growing more confident in his idea and its architecture.

By midsummer, 1999, Fanning sat down at his uncle’s for a sixty-hour programming spree, and it was during those sleepless hours that Napster was officially born.

 

The Architecture

His idea was simple enough. All he needed to do was combine three existing protocols into one client: computer-to-computer connectivity (which was accomplished in instant messaging clients like IRC), file sharing (which was implemented in many instant messaging clients and exhibited in operating systems like Windows), and advanced search (which was illustrated by MP3 and internet search engines).

Fanning had already implemented two of the three features in the MP3 search program that he wrote during his Freshman year of college. The third feature, computer-to-computer connectivity, was the innovation that led his first program to become Napster.

The issue with Fanning’s first program was the same issue independent MP3 websites had: it relied on the servers of third-party websites that were frequently taken down for a number of reasons. Using computer-to-computer connectivity, Napster utilized each user’s computer as a server on the Napster network. Rather than searching the server of a website, Napster searched the computer’s of user’s that were currently logged onto the network.

There still was a centralized server for Napster—which is what eventually led to the service’s downfall—that indexed MP3 files and their locations. This allowed the Napster to still provide a very rapid search functionality.

Amazingly, in its two years of operation, the centralized server for Napster never went down. Not once.

 

Sharing and Searching

Napster did not blindly search a user’s entire computer for MP3 files—Fanning was originally a hacker, but he still understood privacy. Nor was Napster able to search a client’s computer if the Napster client was not running. So how did user’s music library become part of the Napster network?

  1. The user would need to install the Napster client on their computer
  2. The user would need to share a specific folder on their computer
  3. The user would need to have the Napster client running

Assuming these three criteria were met, any MP3 files within the user’s shared folder would be indexed on the centralized Napster server and available for download by other logged on users.

Any other user using the Napster client could then browse for songs by artist, song, album, etc. The search would be indexed through the centralized Napster server, and results returned from the index would be shown to the user. When a user selected a song for download, the Napster server would return the IP address of the user’s computer that contained the desired song, connect the two user’s computers, and transfer the file.

 

The Rise …

After Fanning’s sixty-hour programming marathon, Napster was born. It was June of 1999, and Fanning and Parker released the beta of Napster to thirty of their friends. It was meant to be a small group for testing. But obviously, given the described architecture above, the more computers that user Napster, the larger the Napster library would be. Fanning and Parker’s friends saw this potential, and less than seven days later, the purposely small test group had spread the download from thirty to 15,000 users.

Its users unaware of the legal implications, Napster went viral. Less than a year from its release, Napster was the fastest growing website in history and had acquired over 25 million users. This growth rate was unprecedented and was a surprise to everyone except Fanning, Parker … and anyone that used the service. Napster was wildly popular on the internet for two blissful years. Before Napster’s user base started to decline (due to the hot legal attention it was receiving), the service peaked at 80 million registered users.

Fanning believed his idea would be popular. But he had no idea of the demand that it would generate. Prior to the release of Napster, digital media was not easily accessible to the general public. Napster opened our eyes to the convenience we could be affording. Unfortunately, the convenience Napster offered was relatively short lived. The Recording Industry Association of America (RIAA) had taken its focus off nearly every other legal dispute it had to focus its crosshairs squarely on Napster.

 

… And Fall

How much damage (if any) Napster did to the music industry will be a topic of debate that will never find a good answer. While the RIAA may point out that, at its peak, Napster shared roughly 2.79 billion MP3s per month among its users, others would tell you that a song downloaded for free does not always correlate to revenue lost. A statistician on the other side of the argument might point out that, during the year Napster was most popular, revenue for the music industry increased by $500 million. Neither of these facts provide hard evidence for either side of the case, but they make for good argumentation.

Regardless, the service Napster provided was solely free MP3 distribution, and there’s no doubt that the means by which Napster did this violated copyright law. The RIAA, along with major record labels, artists, producers, and other corporate giants, banded together to file litigation against Napster. The litigation itself wasn’t overly complicated, and the Napster company effectively dissolved in July 2001, two years after launching the Napster service, one year after injunction.

But the Napster rise, fall, and lawsuit were extremely interesting. No, the litigation itself was nothing to write home about. It was the companies sponsoring the litigation, as well as Napster, that illustrated both the irony of the situation and the need for something like Napster with a legal face. Because many of the same companies that sponsored the litigation against Napster, and even sued Napster itself, were the same companies that had (and continued to, even after injunction) funded Napster.

While the litigation departments of media companies around the world were building cases against Napster, the software departments were integrating components of Napster into their own applications. AOL, Yahoo, and Microsoft, for instance, each introduced instant messaging clients that had a Napster button on every chat window, which allowed you to quickly share a song with a friend. You may recall that AOL merged with Time Warner in late 2000, which caused Warner Music to be renamed to Warner Music Group. Point being, Warner Music Group was one of the many groups involved in litigation against Napster, but their parent company AOL Time Warner was funding the very company they were suing.

AOL was not the only house divided. German media giant Bertelsmann saw the potential in Napster, but they also saw how susceptible it was to legal disputes. So they invested $85 million into the company, asking them to develop a better, more secure distribution system. All this, even while Bertelsmann’s media division was also funding the RIAA and its lawsuit  against Napster. And finally, when the dust was still settling in early 2002, Bertelsmann offered to purchase Napster for $20 million. The offer was rejected, and Napster quickly disintegrated as its employees (and executives … and board) took their severance pay and fled at the sight of bankruptcy. For as spectacular as the formation of Napster was, the day the company finally closed its doors was downtrodden and quiet.

 

The Gnutella Network

The end of the Napster service did not end the Napster idea. And even though Bertelsmann offered Napster $85 million to develop a secure distribution system that the company never had time to develop, someone else did: two people named Justin Frankel and Tom Pepper, co-founders of Nullsoft, a small software company recently purchased by none other than AOL. Justin and Tom developed a more robust and secure peer-to-peer file sharing network, and they called it Gnutella. Mind you, this wasn’t years after the RIAA smashed Napster into the ground. They began working on their alternative to Napster in 2000, and the Gnutella network began to catch the attention of the public’s eye in early 2001, when Napster’s legal battles were ramping up.

To AOL’s credit, they did try to stop Gnutella from growing up and living a long and healthy life. The day after the Gnutella source code was publicly released on Nullsoft’s website, AOL demanded the project be shut down. But, of course, it had already been downloaded thousands of times, and it was already being redistributed on countless sites. So AOL’s move to pull it off their servers was said to be similar to closing the barn door after you let the horse out.

The Gnutella network, unlike Napster, was not a client. It was both a protocol and an idea. The downfall of the old MP3 sites was that both their index server and their libraries were centralized servers owned by the sites. The downfall of Napster was that, though their libraries were on their user’s computers, they still had a centralized server that indexed all MP3 files and the computers on which they were stored. The Gnutella network removed all centralized server and instead used each user’s computer as a server and also a relay. The relay was what acted in place of a centralized index server. And since the Gnutella protocol was open source, anyone could make a client that connected to it. And there are … many.

When you logged on to the Gnutella protocol using a Gnutella client, for instance LimeWire or Morpheus, the protocol on your computer would ping several other computers that it thought might be logged on. Each of those computers also had a list of computers they knew were logged on, so they would return that list to your Gnutella client. Once Gnutella found other logged on users, it would remember those addresses the next time you started the service—this way if one of the servers did go down for any reason (even copyright violation), it still had other alternatives. When you searched for a file on Gnutella, it would send the request down the chain of clients you were attached to, and clients attached to those clients, until it found a match.

In this way, the Gnutella network was completely distributed. There were no centralized servers, so there was nothing for copyright holders to seize when an infraction was suspected. Sure, they would seize your computer, since it was one of Gnutella’s servers. But there were millions of other servers out there just like you. And for this reason, the Gnutella network has never been (and likely never will be, as its effectively impossible) shutdown; it has only grown since its inception. Certain clients have legally been shutdown before, but since they are open source, they would simply reemerge a few days later.

 

Modern Distribution

The Gnutella network today is the most widely used peer-to-peer distribution network (aside from torrenting). Though illegality popularized these distribution systems, they are primarily used for legitimate transfers today, though obviously they do still house illegal content.

More importantly, however, the digital media revolution that Napster started, however controversial it was and is, finally forced the media sources to reevaluate demand for their products. Piracy initially caused CD sales to plummet and thus the music industry to lose money. But once key distributors like Apple, Amazon, and even eventually Napster again (purchased by Rhapsody and reintroduced legally for a fee) finally saw the demand that Napster enlightened them to, the music industry recovered (though they’d like you to believe they’re still limping along). Sure, CD sales have all but died, and some stores like Best Buy don’t even carry CDs, but digital sales have surpassed what CD sales used to be. The digital revolution also opened new possibilities. For instance, Pandora, offering you a digital streaming alternative to your radio.

But the demand for digital content didn’t stop at music. Companies like Netflix, Hulu, and Amazon Instant emerged. Products like the iPad and Kindle are hugely popular. And TV stations started streaming their content online. Even non-internet-based companies like Redbox were formed based on consumer’s desire for on demand content.

 

Now, I’m not condoning illegal activity. And downloading music or movies from LimeWire or The Pirate Bay is very bad, kids. But there is a lesson to be learned here. When the culture begins to change, and the culture realizes a new possibility that never existed to them before is now a reality, don’t resist it. The amount of money the record labels and RIAA put into legislation and litigation before they even consider changing with the culture to provide legal alternatives to services such as Napster far surpassed the money they were losing in records sales. When the culture advances, advance with it. That’s what technology is all about.

 

Unless the culture advances into a murdering machine. That should still be frowned upon.

Reagan.com Email is a Misguided Effort

I heard a commercial with the booming and illustrious voice of Rush Limbaugh. After I recovered from banging my head against my desk, I reflected on what was said in the commercial.

Rush pointed to the popular free email providers (Yahoo, Google, and others) to remind you that they scan your email. To remind you that they sell your email address, and other information about you, to the highest bidder. To remind you that the use of these free email addresses may increase your risk of spam mail. In contrast, purchasing an email address from Reagan.com provides you with private and secure email, and your information will never be sold.

I was intrigued.

I found that Rush was not the only conservative advertising this servic. Fox, CBS, and many others also endorsed it, though for slightly different political reasons; they primarily portrayed it as an email alternative “for conservatives”. They said that, unlike these free services, Reagan.com email would not have you unknowingly contributing to “the liberals”. These are hard-and-fast definitions, people.

Michael Reagan, founder of Reagan.com and son of, you guessed it, Ronald Reagan, has this to say about his service:

[…] every time you use your email from companies like Google, AOL, Yahoo, Hotmail, Apple and others, you are helping the liberals. These companies are, and will continue to be, huge supporters financially and with technology of those that are hurting our country.

Because apparently liberals are the only ones that are interested in using technology to advance our country. And apparently “the liberals” are the only people benefiting from these huge corporations. Obviously, they would never help “the conservatives”. Regardless, this is a relatively empty claim as its never actually substantiated.

 

Politics aside, allow me to explain to you from a technical perspective why the commercials endorsing Reagan.com and even the information on Reagan.com is largely misleading.

First, let’s address the script Rush was fed in his advertisement. It is well known and accepted that free email providers, along with many paying internet providers as well, will harvest and sell your information to advertising companies. It’s well known because these companies clearly state this in their Privacy Policies. The claim is that the Reagan email service, which costs you $40 per year, does not do this. However, if you read through the Privacy Policy for Reagan.com, it is true that Reagan.com says they will not collect your information, but they do allow their affiliates to collect your information.

We may also use one or more advertising network providers to help present advertisements or other content on this website. These advertising network providers use cookies, web beacons, or other technologies to serve you advertisements or content tailored to interests you have shown by browsing on this and other websites you have visited. Advertising network providers collect non-personally identifiable information such as your browser type, your operating system, web pages visited, time of visits, content viewed, ads viewed, and other click stream data.

The key phrases here are that their “advertising network providers” have the right to collect information about “content viewed”. I don’t know about you, but the content I primarily view while logged onto my email is … email.

The use of cookies, web beacons, or similar technologies by these advertising network providers is subject to their own privacy policies, not our privacy policy for this website or its Service.

Reagan.com uses the affiliate networkadvertising.org for their ads (why they show ads on a service they charge for is beyond me). Ironically, if you look through the list of partners of Network Advertising, four companies may quickly jump out at you: Microsoft (Hotmail), AOL, Yahoo, and Google. Just to name a few. Which means much of the same ad revenue that these companies may generate from your use of their free email services may still be generated for them through your use of Reagan.com.

This last point is key to highlighting the disconnect between the claim of the Reagan.com email service and the reality of the internet’s interconnectivity. This disconnect has also recently been highlighted with the controversial SOPA and PIPA bills passing through Congress. You have politicians proposing bills, or in this case making a buck using the influence of politics, on technical subjects in which they have little to no understanding.

If privacy is what you seek, you cannot use the internet, and you certainly cannot use email (unless it is isolated to an internal network). Even if a given email was secure and private while on the Reagan.com servers, any incoming and outgoing messages will go through a server at some point somewhere in the world that is likely owned, operated, or affiliated with one of the internet or server giants, including Google. Coincidentally, even if you had a Reagan.com email address and sent an email to yourself, the email would still go through one of these external servers before returning to you.

 

Next claim. Reagan.com is email for conservatives, right? So supposedly using Reagan.com will support a conservative agenda rather than a liberal agenda. Perhaps directly, and on the very surface, but indirectly (and about half an inch below the surface down to bedrock) no. As I said before, you can’t take something as intertwined and complex as the internet and expect to take the biggest internet giants out of it. Ironically, on the same site that Michael Reagan is falsely boasting that his service will get you away from those Big Brother liberal companies, he provides instructions for how to configure his email service to work on your mobile device. You know, the one made by Blackberry, Apple, or Motorola (owned by Google) running the Android OS (also owned by Google).

Let’s give Reagan the benefit of the doubt. Let’s assume he’s not trying to insinuate it’s Big Business we should distrust. Maybe he’s suggesting Google, Yahoo, and the like sell your information to the government, and that’s where the privacy risk comes in. This is half true … although they don’t sell it. And, again, Reagan.com won’t get you away from this. Even when using Reagan.com, as soon as the email leaves the Reagan.com servers, the United States government will have the opportunity to seize and view the email. They probably won’t, unless you’re a terrorist suspect, but they always have the right, no matter your provider, thanks to the Patriot Act. Heck, even on the Reagan.com servers the government has the right to seize it under this act.

 

There’s a phrase that somebody said once goes something like:

Is it really free if it costs you your privacy?

That’s up to you to decide, really. But if you believe internet companies are the only ones tracking personal information about your daily habits … well, let’s just say you should stop shopping at Target. Or Wal-Mart. Or Best Buy. Or really any major chain in America. Personally, I don’t think a corporation tracking your habits to better serve you with ads related to your interests is an invasion of your privacy.

The cost of Reagan’s supposedly private and secure email service is $40 per year. This service is rented from a man who has no technical expertise and is not a server administrator. His Terms of Service clearly and painfully guarantee you nothing in terms of support, up-time, warranty, or back-up. And if you’re expecting new features in the future … well, don’t hold your breath.

On the other hand, companies like Google and Yahoo have incentive to provide you with new features. They have incentive to guarantee you up-time, because every second their servers are down is ad revenue lost for them. They have dedicated support teams to ensure their servers are always running at peak health, and they have redundantly connected servers and farms, just in case.

Reagan’s servers go down? I’m sure they’ll get it back up eventually. But, you know, you’ve already paid them your $40, so they don’t lose money by the second when the service is down. And it is owned by a politician … so don’t expect a quick turnaround.