websphinx
Class RobotExclusion

java.lang.Object
  |
  +--websphinx.RobotExclusion

public class RobotExclusion
extends java.lang.Object


Constructor Summary
RobotExclusion(java.lang.String userAgent)
          Make a RobotExclusion object.
 
Method Summary
 void clear()
          Clear the cache of robots.txt entries.
 boolean disallowed(java.net.URL url)
          Check whether a URL is disallowed by robots.txt.
static void main(java.lang.String[] argv)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

RobotExclusion

public RobotExclusion(java.lang.String userAgent)
Make a RobotExclusion object.

Parameters:
userAgent - name of the robot using this object, as shown in the User-Agent header fields of its HTTP requests. Use null for anonymous robots.
Method Detail

disallowed

public boolean disallowed(java.net.URL url)
Check whether a URL is disallowed by robots.txt.

Parameters:
url - URL to test
Returns:
true if url's Web site denies robot access to the url

clear

public void clear()
Clear the cache of robots.txt entries.


main

public static void main(java.lang.String[] argv)
                 throws java.lang.Exception