Introduction



Modifying Tinyproxy to Cache User Requests

Joe Larsen

Abstract

In order to provide internet connectivity to users in areas of the world were power is a limited resource solutions must be implemented that allows for a user request to be cached while networking equipment is turned on. To achieve this while appearing to have an “always on” internet connection this document will describe how Tinyproxy can be modified user Python 2.5 to check for a connection before it allows internet requests to be sent to a web server.

Keywords

Tinyproxy, proxy server, Python, Ubuntu, Linux, HTTP, vim, lynx, Internet-on-demand, always on internet

Introduction

Today there is a large focus on energy efficiency to reduce one’s carbon footprint or as a result of limited resources. In the developed world, most people enjoy “always on” internet connectivity and while seen as a convenience this is an inefficient use of resources. This is especially true when someone’s network is comprised of computers that act as clients and not servers. As a result a solution is sought in which networking equipment can be turned off when not in use but still provide the same seamless user experience that one would expect with a “always on” connection.

Objective

In order to achieve the desired user experience their must be software that operates in the background which will:

• Detect the presence of an internet connection

• Catch outgoing user requests until the internet connection is detected

• Include an interface to turn on networking devices

There are a number of ways that one can achieve these goals. Because this particular application is part of an existing project that runs off of Linux the solution that has been chosen is to modify an application currently being used called Tinyproxy.

Tinyproxy is a proxy server. A proxy server can be used to create a local cache of web sites, to implement a content filter, or to provide a layer of anonymity to someone’s internet browsing activity. All of these activities involve Tinyproxy being used as an intermediary between the user and the web server in all outbound connections. In being an intermediary Tinyproxy is processing the data of the user request before sending it out. In effect this is caching the user requests before they are sent out.

In order to cache the user requests until the internet connection is detected, either a function that involves processing the request or sending a request out to a network interface will have to modify. This application note will describe how to achieve this by modifying the function used to send requests onto a network interface. While Tinyproxy is written in C, this application note will describe how to do this with Python 2.5. This is done for no other reason than the author of this note is most familiar with python. The logic used should be easily ported to C by someone familiar with the language.

Implementation

This section will provide a step by step description of how to achieve the objectives outlined above. First the code for Tinyproxy must be analyzed to determine functionality that requires modification. Then the modifications have to be made. Finally the system has to be tested.

In order to achieve the desired functionality the function handling all outgoing network communications will be edited. This function is called handle_connection() and is located in the file {tinyproxy_root}/src/reqs.c. This function was chosen because request can be cached immediately before being sent to a network interface. As a result any processing such as blacklist content filtering can be preformed first.

Modifying TinyProxy

The first thing that needs to be done is acquiring the source code of Tinyproxy. Then uncompress the file that was received. To do this open terminal and type:

$ wget

$ tar xsvf ./tinyproxy-1.8.1.tar.bz2

This application note will use vim to edit all C files. Open the file tinyproxy-1.8.1/scr/reqs.c. Find the function handle_connections(). An easy way to do this would be to type:

:/handle_connection

Once the handle_connections() function is found, a system call needs be added to run an external script. As was alluded to in the introduction Python will be used to achieve the objectives. In order to do this the function system() will be used. This allows a command to be passed to terminal while an application is running. In this case the location of the executable python script that will be created is passed to terminal. This will effectively run that script while suspending the other processes of Tinyproxy (not allowing the request to be sent out over a network).

In order to use the system() function you have to place the following code in the handle_connection() function:

1. int res;

2.

3. log_message (LOG_CONN, "Beginning IP test");

4.

5. res = system ("/home/ece480/test/test.py");

6.

7. if (res == -1)

8. log_message (LOG_CONN, "Error Running Python Script");

9.

10. else

11. log_message (LOG_CONN, "Connection Confirmed");

This would be placed below the block of code where already existing variable declarations for the function exist.

Line 1 is simply declaring a variable that will be used to determine if the system() function was run successfully. Lines 3, 8, and 11 are log entries that will be written to the tinyproxy.log file. Line 5 is where the system() function is used. The input to the function is a terminal command; in this case it is the location of an executable file. If the command ran successfully, the system() function will return a positive number or 0. If it fails it will return a negative number.

Creating a python script

When creating a Python application it is best to use an IDE that is either built for or has plug-ins for Python, due to the importance of white space and tabs. Two suggestions would be IDLE, the GUI command shell that comes with Python or NetBeans from Oracle with the Python 2.5 plug-in.

The fist thing that must be done is declaring the application this script is to be run with (#! /usr/bin/python) and import any libraries that we will be using. The command library is used to make calls to terminal and return the output from terminal to Python. The sys library is used to determine the condition under which the python script is exiting and redirect stdout to a file for log entries. The exit condition is the value passed to the res variable on line 5 in the Tinyproxy modification code block. The time library will be used to generate time stamps for the log entries. This can be done with the following code:

1. #! /usr/bin/python

2. import commands

3. import sys

4. import time

5.

6. saveout=sys.stdout

7. fsock=open("/usr/local/var/log/python.log","a")

Lines 6 and 7 are used to direct print statements to stdout and open a log file where information will be appended to. Any strings handled by the print function will be written to that log file.

The remainder of this script will be broken up into a series of functions called getIP(), sendSig(), exiting(), and main(). The names of these functions should be self explanatory, but they will be covered in detail. The first function getIP() makes a system call using the command ifconfig to get the computers IP address. The information is then piped into grep, cut, and awk to return a usable string. The function then checks the length of the string, if there is no IP address, the value of the IP Address is set to “none”. This is done for logging purposes. Finally the function returns the IP address.

1. def getIP():    

2.     ip=commands.getoutput("ifconfig eth0 |grep 'inet addr'|cut -d':' -f2|awk '{print $1}'")

3.     if len(ip)==0:

4.         ip="none"

5.     return ip

Next a function has to be written to send a turn on signal. This function will take care of turning on the networking equipment. This functionality is outside the scope of this application note, but it is still important to log. As of March 31, 2010 the sendSig() function looks like:

1. def sendSig():

2. print "%s STATUS    Sending turn on single" \

3. % time.strftime("%Y-%m-%d %H:%M:%S")

4.    

5. # TODO - Write code for this function

The next function handles how the python script terminates. This function handles logging the close of the script, stdout control being returned to the system, closing the log file, and sending an exit code to the res variable in the code that was added to Tinyproxy.

1. def exiting():  

2.     print "%s CLOSING" % time.strftime("%Y-%m-%d %H:%M:%S")

3.     sys.stdout=saveout

4.     fsock.close

5.     sys.exit(0)

The final function is the main function. This starts an infinite while loop. After making a call to the getIP() the script checks the IP Address against a value defined in the application. In this particular instance it is 35.9.132.239. If the actual IP address and the value supplied are the same the script will break send a turn on signal, log the event, and continue the proxy server functionality in Tinyproxy. The reason for sending a turn on signal is to ensure the networking equipment stays on, as other components on in this project run off of a timer and that turn on signal will reset the timer. All of the IP Addresses in this implementation are assigned as static IP addresses. This application is assuming that if an IP address is assigned by a router that router is connected to the internet. If this were to be implemented on a DHCP network regular expressions could be used to ensure the IP address falls within an excepted range.

1. def main():

2. sys.stdout=fsock   

3. ip=getIP()

4. print "%s STARTING   Entering main loop. IP Address is: %s" \

5.           % (time.strftime("%Y-%m-%d %H:%M:%S"), ip)

6.     count=0

7.     while True:

8.         n=getIP()

9.         if n=="35.9.132.239":

10.         print "%s STATUS    IP Address Confirmed" \

11.               % time.strftime("%Y-%m-%d %H:%M:%S")

12.         sendSig()

13.         exiting()

14.         break   

If the condition on line 9 is not met, then the application runs the sendSig() function ever 100 cycles through the while loop. This information is then logged. The code used to implement this is:

15. else:

16.        sendSig()

17.        counter=count%100

18.        trueCount=count/100

19.        if counter==0:

20.            print "%s STATUS    IP Address invalid: %s" \

21.                  % (time.strftime("%Y-%m-%d %H:%M:%S"), n)

22.            print "%s STATUS    IP Address has been checked %d" \

23.                  % (time.strftime("%Y-%m-%d %H:%M:%S"), trueCount)

24.         count=count+1

Finally you will want to include a main()function at the end of the script. Then save the script in the same location defined by line 5 in the C code used to modify Tinyproxy. Once this has been completed you will need to provide permissions for the script to be executed. The easiest way to do this is by typing the following command into terminal:

chmod a+x /home/ece480/test/test.py

Compiling Tinyproxy

Compiling Tinyproxy is incredibly easy. Before compiling if an already existing binary is installed, it needs to be removed by issuing the command:

sudo apt-get remove tinyproxy

After doing this docbook-xsl needs to be installed

sudo apt-get install docbook-xsl

At this point issue the following commands:

cd ~/tinyproxy-1.8.1

./configure

make

make install

After running each command, ensure that there are no errors. If you have encountered errors ensure that they are corrected, and reissue the command before moving on. Otherwise the Tinyproxy will not install properly.

Two errors that I encountered when attempting to run Tinyproxy were:

• The log file in /usr/local/var/log/tinyproxy.log could not be found or created

• Tinyproxy could not write the PID to the file /user/local/var/run/tinyproxy.pid

These two problems are very easy to fix, simply issue the following commands:

sudo touch /usr/local/log/tinyproxy.log

sudo touch /usr/local/run/tinyproxy.pid

sudo chmod 666 /usr/local/log/tinyproxy.log

sudo chmod 666 /usr/local/run/tinyproxy.log

The above commands are creating the files, if they do not already exist, and then changing the permissions associated with the file so information can be written to them. Once this is complete the application can be tested.

Testing - Setting up Lynx

To test Tinyproxy an HTTP browser is needed. This section will explain how to configure a web browser to pass all requests though a proxy server. This document will be using lynx, a terminal based web browser. First one needs to ensure that lynx is on their system. This can be done by typing the command:

sudo apt-get install lynx

Once lynx is installed the configuration file needs to be edited to forward all requests to a proxy server. Because tiny proxy is operating on the same computer that lynx will be running on the domain will be localhost and the forward port will be 8888. To edit the lynx configuration file type:

sudo vim /etc/lynx.cfg

Inside of vim type :/http_proxy: This will search for the string “http_proxy:”. A section of the configuration file will be presented, and should appear similar to:

#http_proxy:

#https_proxy:

#ftp_proxy:

#gopher_proxy:

#news_proxy:

#newspost_proxy:

#newsreply_proxy:

#snews_proxy:

#snewspost_proxy:]:port/

#snewsreply_proxy:

#nntp_proxy:

#wais_proxy:

#finger_proxy:

#cso_proxy:

#no_proxy:host.domain.dom

Edit the first line of this section to read:

http_proxy:

Testing Tinyproxy

Once this is completed the application can be tested to ensure that it meets the objectives. In order to do this open two additional terminal windows and run the following commands:

tail –f /usr/local/log/tinyproxy.log

tail –f /usr/local/log/python.log

These two commands will dynamically output the log files into terminal, so that Tinyproxy’s activity can be followed in real time. In a third terminal window type:

lynx msu.edu

The web page should load, additionally the log files should look like:

1. CONNECT Mar 31 22:54:14 [13853]: Beginning IP test

2. CONNECT Mar 31 22:54:14 [13853]: Connection Confirmed

3. CONNECT Mar 31 22:54:14 [13853]: Connect (file descriptor 6): localhost [127.0.0.1]

4. CONNECT Mar 31 22:54:14 [13853]: Request (file descriptor 6): GET HTTP/1.0

5. INFO Mar 31 22:54:14 [13853]: No proxy for msu.edu

6. CONNECT Mar 31 22:54:14 [13853]: Established connection to host "msu.edu" using file descriptor 7.

7. INFO Mar 31 22:54:14 [13853]: Closed connection between local client (fd:6) and remote client (fd:7)

8. NOTICE Mar 31 22:54:14 [13853]: Waiting servers (1766203423) exceeds MaxSpareServers (20). Killing child.

Looking at lines 5 and 6 provides evidence that Tinyproxy did break to run the python script. In order to ensure the script actually ran as expected the python.log file needs to be examined:

1. 2010-03-31 23:07:33 STARTING Entering main loop. IP Address is: 35.9.132.239

2. 2010-03-31 23:07:33 STATUS IP Address Confirmed

3. 2010-03-31 23:07:33 STATUS Sending turn on single

4. 2010-03-31 23:07:33 CLOSING

This is how a log should look if there was an active internet connection. If the internet connection was to be disconnected by unplugging the Ethernet cable and a request was made to a website the log file would look similar to this:

1. 2010-03-31 23:07:33 STARTING Entering main loop. IP Address is: 35.9.132.239

2. 2010-03-31 23:07:33 STATUS IP Address invalid: none

3. 2010-03-31 23:07:33 STATUS IP Address has been checked 0

4. 2010-03-31 23:07:33 STATUS Sending turn on single

5. 2010-03-31 23:07:33 STATUS Sending turn on single

6. 2010-03-31 23:07:33 STATUS Sending turn on single

7. 2010-03-31 23:07:33 STATUS Sending turn on single

8. 2010-03-31 23:07:33 STATUS Sending turn on single

9. 2010-03-31 23:07:33 STATUS Sending turn on single

10. 2010-03-31 23:07:33 STATUS IP Address Confirmed

11. 2010-03-31 23:07:33 STATUS Sending turn on single

12. 2010-03-31 23:07:33 CLOSING

Lines 10-11 are a result of plugging the Ethernet cable back into the physical interface, allowing the computer to negotiate with a router for its IP Address. The browser should also load the webpage.

Conclusion

This application note describes how to modify Tinyproxy to work in an “Internet-on-demand” architecture to detect an internet connection, catch outgoing user requests, and create the interface to send a turn on signal. Hopefully the methodology from this application note can be used to include additional functionality into Tinyproxy or other applications.

Some areas are still in need for improvement. An evaluation of the security of the overall system because in more than one instance execute, read, and write privileges was given to all users because it was easy. Also depending on the application it would be worthwhile to check internet connectivity instead of just checking for a connection on the LAN.

References





-









Tinyproxy man page

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download