Using Norbert

Utilising Norbert is very simple.

1. Import the class import org.osjava.norbert.NoRobotClient;
2. Create an instance for your user-agent NoRobotClient nrc = NoRobotClient("googlebot");
3. Parse a robots.txt at a site nrc.parse( new URL( "http://www.osjava.org/" ) );
4. Ask if a url is allowed boolean test = nrc.isUrlAllowed( new URL( "http://www.osjava.org/index.html" ) );